Learn how to upload S3 objects via the AWS Console and set up logging of all S3 access events.
- [Instructor] Now that we have both a primary bucket and a logging bucket, let's go ahead and generate some activity that will create some logs for us. The logs that appear in the logging bucket will reflect all the activity that any user does in the primary bucket, from uploading to downloading to deleting and more. Let's go into the primary bucket and we will generate some activity and we'll look at how to upload files. Here we have the upload button. And you can see that you actually have the opportunity to drag and drop files directly in, or you can do what I'll do which is click add files.
Here I have some CSVs of sample data. Let's start with this states.csv which is a comma separated value file that has some basic information about the US 50 states. I'll click next and you can see that we have the opportunity to set individual permissions on this object just like we could at the bucket level. We can also set access across accounts and we can make the object public if we so desire. I want to leave these at default and click next. Now here you see the storage tiers that I described in the S3 overview.
They're not set at the bucket level, but at the individual object level. We have standard, which is the default, standard-IA, which stands for infrequent access and sacrifices a little bit of the availability nines for a slightly cheaper price, and finally we have the opportunity to set reduced redundancy which sacrifices slightly on both availability and durability in order to achieve a lower price point. Just keep in mind that no matter what storage tier you choose, they're all going to be more reliable and more durable than most storage setups that you would do on your own.
So I'll keep this at default and then we'll go into encryption. Now the encryption options include the Amazon S3 master key, which as I said, is generated by AWS, and is managed by the S3 service so you don't ever see it, it's just behind the scenes, it encrypts the objects, and you don't have to worry about it. The third option there is the AWS KMS option, which if we select it will prompt us to choose a KMS key that's already in existence. You see the first option says AWS slash S3, that's actually the same as the S3 master key in the second option.
The second one here is a KMS key that I've already created in this account. I don't want to use that right now, so I'll use the Amazon S3 master key and allow this object to be encrypted. Down below we have both metadata and tags, and this is where I want to clear some things up. As you recall, object storage means you have data with metadata attached. That is what the capital M metadata is for. The metadata that we set here in this section will be permanently attached to this object and it will be visible from get request to S3 whether using rest or an STK, things like that, you can review the metadata key value pairs on this object.
You can see that there's some prebuilt metadata keys. And at the very bottom there's the X-AMZ meta option. Now it's not completely obvious, but I can actually click into this section here and give this a name. So what I can do is create as many X Amazon metadata key value pairs as I like. So similarly to what I did with the bucket, I'm going to give this a creator metadata tag, and give it my name and click save. So I could create as many of those as I want. The tag section here refers to the kind of tags that you would see throughout AWS, like the ones that you might put on an EC2 instance.
They're similar to the metadata, but they're not going to show up as attached to the object. Rather, it would be a separate call into the S3 rest API for instance to see what the tags are that you've attached to the object. They're similar to metadata, but they're slightly different and one key difference is that with actual capital T tags you can use them for IM policies to control access to this individual object, or they can also be used in bucket life cycle policies. Right now I don't need to add any tags to this object, so I'll click next and go on.
Here's my review screen and I'll click upload. Okay, let's generate a little bit more activity by uploading some more files. I'll take these other CSVs which I have pulled from the open data website of the US government. We've got precipitation data and demographic data. And click next. We'll keep all the defaults, in fact let's keep all the defaults and we'll just click upload. Now if something doesn't show up like it just did here, you can always click the refresh option over here and it will refresh the file list without going through a full page refresh.
So now that we have a few objects in a bucket, let's head outside to the logging bucket and see what we've got. Now don't be concerned if your logging bucket doesn't have anything in it right away. It can often take S3 a little while to start logging your activities from the source bucket. Let's click in here and see if we have anything. All right, there we go, demo primary logs, that's the prefix that we asked the source bucket to place on its logs. If we click in, we can see that there are numerous logs from just the activities that we've generated so far in this lesson.
If we click on one we can download it by clicking the download as button, then right-clicking the resulting pop up link and choosing save link as. Here in the terminal I can see that file has downloaded and I can cat out the contents so you can see what's inside it. You see that we have detailed information on activity that took place on the bucket. Here's my bucket name. Here's the activity, it was a put action. And you can see the timestamp and the source IP. So you can see how these logs give you a really detailed audit trail on everything that happens in your bucket.
Join AWS architect Brandon Rich and learn how to configure object storage solutions and lifecycle management in Simple Storage Service (S3), a web service offered by AWS, and migrate, back up, and replicate relational data in RDS. Find out how to leverage flexible network storage with Elastic File System (EFS), and use the new AWS Glue service to move and transform data. Plus, learn how Snowball can help you transfer truckloads of data in and out of the cloud.
- What is data management?
- AWS S3 basics
- S3 bucket creation
- S3 upload and logging
- S3 event notifications
- S3 data lifecycle configuration
- Working with Amazon Elastic Block Store volumes
- Creating and mounting an EFS
- Creating an AWS RDS instance
- RDS backup and recovery
- Moving data with AWS Database Migration Service
- Moving data with Data Pipeline and Glue