Query logs for insights - Amazon Web Services (AWS) Tutorial

- [Instructor] In order to query our logs from AWS CloudWatch console, under Logs, you need to navigate to the Insights menu. Here, we have a query editor we can use to write our queries in order to make a query, first, we need to select one or more log groups. For our demo, we are going to select the API gateway execution logs. Next, we need to choose a timeframe. We can either use one of the common values between five minutes to 12 hours or if we want to retrieve all the records we can use the Custom value. If you are not getting results from the queries make sure that you have the correct timeframe set. And finally we have our query editor. Here is where we write our queries. First, we need to specify the fields that we want to select on the right, we have a helper UI, clicking on the fields will show us all available fields that this log group has. This makes it very convenient to find out what type of fields we have and what type of values those fields have. Now let's look at querying some data from our logs. To make a quick select, we use the fields command, where we specify the fields. Here, I specify the timestamp and the message that I want to display. Then if we want to sort the data, we can use a sort command separated by a pipe, where we need to specify the field name and the sort order which can be ascending or descending. And finally, we can limit the number of records that we want to show using the limit command. It's always a good practice to limit the number of records to show so we can incur the least amount of charges possible. Now that's execute our query to see what we get. And as we can see, we can see the total summary of showing 20 records out of 30,272 records. If we scroll down, we can see the actual logs. One of the most common command in querying is filtering. To filter log entries, We can use the filter command. After fields, we need to specify the filter command. We use a pipe and then we specify filter. For example, let's filter for all entries that contain the word exception. Now let's run our query. This should return all the log entries that contain the word exception. And as we can see, we have 360 records that match this filter. The API data tool will intentionally create bad records, and we can see here the exception of those bad records. And if you look at the message it says that there was a validation exception. One or more parameter values were invalid, missing the key ID in the item. The API tester tool does not send the ID key which is required resulting in a validation error. The query editor also supports aggregate operations like sum, average, maximum, and minimum. To use the aggregate operations, we need to use a stats command, then specify the aggregation that we want. For example, if we want to get the total number of requests that ended in error, we can do the following. First, we need to remove the field section as we don't need it anymore. We are going to leave the filter as it is, as we want to filter all the requests that ended in exception. Next, you specify the stats. We are specifying the stats command, and then specifying to count them all and output that variable as error count. Now let's run the query and this should return one record with the total number of requests that match the word exception. And as we can see, we have the number 360. The application inside query editor also supports grouping. For example, we can group the error count into different periods. Let's say, we want to group the error count in a period of five minutes. Then we can do the following. We need to specify the by key word and then use the bin function, inside which we specify the period, to specify five minutes, We specify 5m, indicating five minutes. Now let's run the command and see the output. And now we can see that the error count is grouped in a period of five minutes. The first five minutes we have 146, the next 168, and then two. Using the CloudWatch Insights to query our logs is pretty straightforward and we can easily identify different errors and reasons behind the errors to resolve them as quickly as possible.