Learn how to configure S3 lifecycle transition and object deletion rules, and be able to map concepts of the data lifecycle to the S3 lifecycle rule setup.
- [Instructor] In this video, I'll demonstrate how to add object lifecycle rules to an S3 bucket. Let's click into the S3 service. I'll choose my primary bucket and go to the furthest right tab, Management. We're on the Lifecycle section, and we currently don't have any lifecycle rules. Click Add lifecycle rule. The first step is to create a name for the rule. Before we do that, let's think for a moment about what we actually want to do here. Let's assume that we want to transition our data out of standard storage and into infrequent access after a certain amount of time.
From there, we want to go to Glacier, and from there, after a long enough time, we want to permanently delete the objects. So we'll start out with a rule that says GoToinfrequentAccess. If we'd like, we can enter a prefix here to make sure that this rule is only applied to a subset of objects in the bucket. Instead, we'll leave the prefix blank, and we'll click Next. This rule will apply to every object in this bucket. On the Configure transition tab, we can choose Current version or Previous versions. In a version bucket, we could select the latter and apply this lifecycle rule only to noncurrent versions of an object, but we want to apply the rule to current version.
So check that. This new section that appears has the button Add transition. Let's click that. This is where we'll select infrequent access. Transition to Standard-IA after, and we can stick with the default, and by the way, the minimum, of 30 days. We can't transition to infrequent access any sooner than that. The next tab allows us to configure expiration. That is, to fully delete an object. We can do this in one lifecycle rule, but my preference is to space them out over multiple rules with clear names so we can manage them separately.
Click Next to go on. Review your choices and click Save. So that's one part of our lifecycle transition figured out. The next step is to make sure that after a long enough time, these objects go from infrequent access to Glacier storage tier. We'll add a new lifecycle rule, GoToGlacier. Again, we want to apply to the whole bucket and not add a prefix or tag filter. We'll apply this transition to the current version and click Add transition. Now we choose Transition to Amazon Glacier after, let's say, 100 days.
Click Next, skip Configure expiration, and Save. Before we save, you'll notice the warning. It says to check the first rule that we made because it has prefixes that are overlapping. Now, of course, we didn't specify a prefix, but what this means is that the scope of the rule is overlapping. We've applied both these rules to the entire bucket. That's okay because we know that the rules themselves are not overlapping. S3 will transition all objects in the bucket to infrequent access after 30 days and to Glacier after 100.
Click Save to commit your changes. Finally, we do want to add a lifecycle rule that will delete objects after, let's say, one year. Click Add lifecycle rule again, and we'll call it DeleteAfter1Year. Again, no prefix will apply to the whole bucket. Click Next. Skip the configure transition tab, and go straight to Configure expiration. Like with our transitions, we'll apply to the current version. You can see that the default here is actually already 365 days to expire the current version of the object.
That's exactly what we want, so we'll click Next, review our changes, ignoring the warning, and click Save. This bucket is now configured with a complete set of lifecycle rules that will transition objects from standard storage to infrequent access to Glacier, and finally, to permanent deletion. In order to demonstrate the Glacier tier of S3 storage, I've moved to another bucket that already has an object in the Glacier storage class. States.csv, shown here, is in the storage class Glacier, as you can tell from the right-hand side.
If I click the check mark next to it and go to the More drop-down, you can see that I no longer have an active Download as option. Rather, I must initiate restore. This is how cold storage works. I don't get to access the file immediately. I must ask for it to be given to me. This brings up a new UI. The first thing I need to specify is the number of days that the archive data will be accessible. I want to make sure that I don't miss this, so I'm going to set it for three days. Remember that the Glacier storage tier of S3 has no easy way to notify you when the retrieval is complete, so we'll have to come back and look.
We have a few retrieval options in the drop-down as I mentioned earlier, bulk retrieval, expedited, and standard. Each comes with different time constraints and cost associated. I'd like to show you the expedited retrieval, but the expedited retrieval is quite expensive, $100 per retrieval, so I'll just select Standard retrieval, and we'll come back later. From here, click Restore. By the way, take a look at the note. Restored objects are going to have the Standard storage class for the period of time that they're retrieved from Glacier.
Now all we have to do is come back in a few hours, and we'll see that the states.csv file is ready for retrieval. Okay, it's been a few hours, and now we want to see if our states.csv object is available for retrieval from Glacier. You can see that nothing major has changed on this display. The storage class still says Glacier. However, if we check on the checkbox next to states.csv and let the side menu come up, we can see a few new things. Notice that the More drop-down still does not allow us to choose Download as.
However, in the side bar, we have a big Download button, and the restoration expiry date, which is in a few days as I specified. Simply click Download, and now we have our file that has been retrieved from cold storage. Though this is a very small file in this example, you can imagine how this could be used to retrieve larger archives from cold storage. If you've set your lifecycle rules in a way that maps to your data usage, this should be a relatively rare event, but Glacier gives you the assurance that your data is always available to you at a cheap storage price.
Join AWS architect Brandon Rich and learn how to configure object storage solutions and lifecycle management in Simple Storage Service (S3), a web service offered by AWS, and migrate, back up, and replicate relational data in RDS. Find out how to leverage flexible network storage with Elastic File System (EFS), and use the new AWS Glue service to move and transform data. Plus, learn how Snowball can help you transfer truckloads of data in and out of the cloud.
- What is data management?
- AWS S3 basics
- S3 bucket creation
- S3 upload and logging
- S3 event notifications
- S3 data lifecycle configuration
- Working with Amazon Elastic Block Store volumes
- Creating and mounting an EFS
- Creating an AWS RDS instance
- RDS backup and recovery
- Moving data with AWS Database Migration Service
- Moving data with Data Pipeline and Glue