Understand approaches to monitoring both individual AWS services and your applications hosted on AWS.
- [Instructor] Before we start actually performing monitoring on AWS services, let's consider in this section what we should monitor and why we should monitor it. Let's start at the highest level. So the key question is how much monitoring is right for your application, and of course, the answer is it depends, but there are some guidelines that you can use. So the rule of thumb is that monitoring should be driven off of business needs, so what's an example of that? One of the key ones is application availability.
If your, for example, website is not functioning, then that's going to be most normally the highest level business priority, so you would be monitoring for, they call them the bilities, for example, the availability. Now another bility would be securability, or security. So if your website is hacked and you have the wrong information, that again is a significant business disruption, so all the monitoring should be drawn off of business needs or risks, you could say it another way, because monitoring costs money.
Even if the AWS service that you're using to monitor is actually free, so in the case of, for example, storing files in S3, you can have a bucket inventory as part of S3, it's not really free because a person or some other service has to read, interpret and act on the reported results. One of the things that I hear over and over in production teams is, yeah, we have logs, but no one reads them, so that's really useless monitoring and actually, it's negative monitoring because the act of producing the logs could also be putting unnecessary drain on the system, so just like anything else in the world of application development and software, there should be thought put into monitoring strategies and processes and procedures and we're going to dive into that.
So drilling down to the next level, we want to look at the idea of monitoring approaches, so you want to think about with your team and with your applications, why and how to monitor. Which activities do you want to monitor for? Generally, it's almost always availability 'cause you want it to be up, but then is your priority performance or is it security or is it something else? Or in which order are these priorities? What is the level of risk that you perceive around your application? And just to give a real world example here, I was working with an enterprise IOT company, or internet of things, and as you may or may not know, IOT in the enterprise world has become recently more of an attack surface because of some new protocols and some new devices and new methods of security, and it's really easy for, you know, developers or teams to miss aspects, for example, of security, so there have been several unfortunate incidents in the news where IOT companies have been maliciously hacked for ransom, basically, so in this particular case, we decided to prioritize auditing and monitoring of security because of the level of risk, so that's an example of tying a business driver to a monitoring strategy.
So we're going to look at which activities and which tools or services, so your file system tier, your network tier, your database tier, or look at the services by themselves when we first start this course, but as we move through the course, we'll start combining the services into components and then eventually into applications. The next aspect here is how to select metrics for monitoring. I see this done incorrectly really frequently too, where developers or DevOps people who really aren't familiar with some of these cloud services, they will just equate them with what they did with on premise services, so for example, DBA, who's worked with on premise databases, will look at some of the metrics that she or he has been monitoring for years around their Oracle database or the single server database and when they move to a cloud service, the metrics that they should be looking at might be different because this service is deployed in the cloud.
Now, really, the best tip to get started is to look at what Amazon suggests, and we're going to do that pretty extensively in this course. We're going to make sure that we understand for the core services like RDS and Kinesis, what does Amazon suggest that we look at, how do we look at that, and at what level do they expect that metric to be running at? I'm really surprised how often people don't do this. It's really, really a tip. Look at what the service provider is suggesting that you monitor and start with that.
You may add additional monitoring capabilities based on, like I said, your particular business needs, but start with what they recommend and make sure that you understand and utilize those metrics first. Going along with that, you have to figure out which data, and really, less is more because as I said in the previous movie, I've constantly heard from teams, yes, we have logs but no one looks at them or we make the new guy look at them and it's like a punishment kind of thing, and again, this is really not a useful approach.
You want to pull only the data that you need, and oftentimes, you want to aggregate and visualize it, and a big tip I have here, and I'm going to be showing some of this throughout this course, is yes, you can do this manually, but there is a whole partner ecosystem out there that provides different levels of log aggregating, reading, alert creation capabilities and visualizing, and these services can be very, very good value because if you think, well, gosh, they cost a lot of money, what's your time worth? And again, this is a mistake that I've seen a lot of teams making, where they say, well, partner product X costs too much money.
We can just do it on our own, and then what happens is, of course, they run out of time and then the application goes down and then we're kind of in a bad situation, so you know, you don't want to be focusing on the wrong thing here, and taking a high level look at monitoring and what your monitoring strategies are, what your monitoring tools are, what your monitoring targets are, is an important part of successful application deployment and we're going to look at that in this course.
- Understanding approaches to monitoring
- Matching business application priorities to monitoring approaches
- Tools for monitoring and logging
- Monitoring core AWS Services
- Using CloudFormation templates
- Monitoring S3, EC2, Lambda, RDS, Kinesis, and EMR
- Using core AWS tools for metrics and logging
- Using AWS services for advanced monitoring