In this video, see third-party cost control tool Looker block for AWS, and report and suggest optimizations for cost control and reduction.
- [Instructor] In addition to considering third party dedicated products for AWS cost control, there are some new interesting blended type solutions. And one I wanted to call out is something called the AWS block for a product called Looker. So this is referenced in this blog post. And it's really pretty new. It's only a couple weeks before the time of this recording. And what the folks at Looker have done is they have basically extended their platform so that you can get customizable cost reports. So what does this mean? This is a little bit complicated.
You're note really buying a product like you are with BotMetric for ingesting your product service logs and then munging the logs and doing machine learning against them and getting recommendations. This is simpler. But the price of this is different as well. So many customers will purchase a third party set of visualization tools. Looker is one of those. So similar types of tools will be Tableau, ClickView, so on and so forth. So, Looker's main sort of product offering is to offer licenses for visualization of data that's stored in some kind of database.
But what Looker does is they provide these dashboards based on an underlying text language called LookML. And what they've done is they've leveraged the fact that Amazon introduced a new service that Amazon allows you to do SQL-like queries or SQL queries against text files. And that service is Athena. So really interesting. I'm actually going to kind of go through this so you can see how it works. So the idea here is that they talk about the AWS Cost Explorer, which we saw in an earlier movie, which is a great tool and works for some of my customers.
But you might want some more custom queryability of the underlying service usage data, which is in the CSV file. So the idea of Athena is that you can define a table-like structure, tabular structure using a sequel-like statement. It's based on Apache Hive DDL. And there's an example so you can see what it looks like here against your CSV file. And then you can run SQL queries, basically any SQL queries against it and you're paying Amazon for the computation there.
So what Looker does here is they take the results of that and then they visualize that into a dashboard. So really you're kind of allocating the cost differently here. You're paying Amazon for the query cost whereas in some of the other third party products like BotMetric that I showed, you're paying BotMetric. One's not better than the other. It's just differences in terms of who you're paying and what the capabilities are and what you need. Just I like to talk about different products so you guys can make the informed choices that work best for you. So the way that this works is you have to have your account set up and then you have to have a bucket like we showed for some of the other Amazon internal tools that basically gets the CSV file of all the service usage.
And then you have to have appropriate permissions on the bucket. We saw on earlier videos we needed permissions for some of the internal services. What we would have to do for this one is we have to permission it for Athena because we define a table structure on it. And then we do SQL queries. So we've seen this in earlier movies. Enable the cost in usage report. We did that. And you may remember, you can associate it with Amazon internal visualizers like QuickSight. So there's all kinds of choices for us now which is fantastic. Then we want to configure the S3 bucket and files for querying and to just concisely summarize what it's saying here.
Basically, there's a manifest file that gets dumped into the bucket that you need to remove 'cuz Athena needs files of the exact same structure in order to work properly. So you can manually remove that, but if you're going to do this in production you're going to programmatically remove it. Then, you're going to go into Athena and you're going to define this tabular structure. And it really looks like SQL DDL. It's a customized kind of DDL. It's based on Hive. But what that does is that allows the Athena query engine to query the underlying CSV files as if they were a relational table basically.
And if you've not seen Athena, this is what it looks like. And basically the concept is select data sets. So, where's your file? And then you create a table using the wizard or Hive. And then you basically just can run ANSi SQL queries. And there's a charge by the amount of data that's scanned. So it's a servula service actually. It's a charge by the query. So you set that up. And then you would have the table set up properly. And that's where Looker comes in. So you would have to already have a Looker subscription probably for your business usage visualizing your production data.
And assuming you had that, then one of the big advantages of going this route is the LookerML block for presenting this data in a dashboard is actually free. So it can be a real cost effective, but flexible solution. So you can see that you will work with the Looker implementation and get the block and then you basically just connect to the Athena table and then you can get a visualization. And let me just actually show you what this looks like.
So here we are in the Looker visualization. And you can see that we have total cost summaries at the top and we have them broken up by week, month, and year to date. Beneath that, we have a time period over time periods. So you can see it, for example, on the left. 44% of 1,278 per cost per prior week. So it's for your priors. Below that, we have the most common service cost which we've been talking about all throughout this course, which is a summary of EC2 cost.
Now one of the great things about Looker, because it's retrieving information based on a SQL query that you write in Athena and then visualizing it. Although this block has been set up by default with EC2, it's more flexible than some of the other tools that you would buy for cost management where the vendor would have to create the filters and features. For example, if you were using a lambda based architecture, you could change the query to say return to me the lambda cost rather than the EC2 cost in your SQL query and in your LookerML block.
And you could have a more dynamic cost dashboard. Now that being said, we're just going to look a little bit more at this starting block. So we've got these rolled up costs. And then as we scroll down, we see some visualizations of EC2 reserved instance costs. We see not only total costs, but we see trend lines. Again, one of the great things about using a dedicated visualization client is the sophistication of the data coming in can be more quickly understood through these visualizations.
Here I'm pointing out the reserved instance usage over time. And the red line you want to be near 100 because of course, you may remember when we talked about EC2, if you've purchased reserved instances, you're going to be charged for them whether you use them or not. And this type of visualization to me is more intuitive. It's one of the reasons I'm excited about using custom visualization clients to work with these complex cost metrics. Now as we scroll down, another aspect of working with a sophisticated visualization tool is it allows you to go beyond the restrictions of the Amazon tools.
And what I mean by that is here we're looking at most costly services, which is EC2 again. And we can for the data that we've set up with the drill down, drill down to the individual service spend levels. And you can't get this out of the Amazon tools. So again, it allows you to find your areas of interest and to get to the level of detail that's actually actionable. So you can see in this table we're looking at cost by usage. And this is by the type of EC2 instance. So here you could figure out I'm spending on these m3 mediums and this might not be the appropriate size for my particular need and I can just reduce my cost by just changing to a smaller machine size.
Now in addition to this, I've talked in this course about costs that could be surprising. This is data transfer. And this is another area where these custom visualization tools really shine. So here what we've got is an overlay of data transfer cost. And yes, most of the lines are green. But what you can do is you can apply metric values so you could quickly see which data transfer usually transfer out costs, are impacting your storage cost, your s3 cost. And you can see we've got one yellow line there. And in this case that means that this was transferred not really geographically so much different, but over a different method.
Over the public internet. And these are the kinds of costs that can really come back to bite you if you can't understand them. And again, one of the things I really really like about using these customizable clients that are designed for data visualization is you can more effectively understand where your costs are coming from, drill in, and control them appropriately.
- Approaches to cloud service cost control
- Why control AWS service costs?
- Controlling costs by service
- Starting services with CloudFormation
- CloudWatch billing alarms
- AWS Trusted Advisor cost control
- Using third-party products
- Cost control scenarios