Learn how to set up a synchronization job between a local file system and S3 using the AWS command-line interface.
In this video, I'm in a terminal to demonstrate how the aws command line interface can be used to interact with s3 and sync large amounts of files to it on a continuing basis. If you don't have the cli installed and configured, simply search the video library for aws cli. If I type aws s3 ls, I can list the buckets that we have available in our account. And if I type aws s3 mb for make bucket, I can give a bucket name to create using the s3:// protocol format.
We'll call it s3-cli-test. Now it happens that that bucket name is available, but remember that your bucket names must be globally unique so you may want to append your initials. I can do a listing of what's in the bucket, but since there's nothing there yet, I want to actually upload some files. The folder that I'm currently in has one reasonably sized file and a directory full of many small files. We want to sync all of these to s3. We could copy one file at a time using the CP command, but there is an actual sync function that we can use.
So we type aws s3 sync a dot to represent the current directory that we're in and again s3://s3-cli-test. This will recursively sync everything in this folder to the bucket. Now that it's done, we can go to the console and see what that looks like in the aws GUI. Sure enough, our new bucket has been created. We can click inside and see all the files that have been moved. Let's head back to the terminal.
You mmight be wondering is this a true sync? Let's test things out by creating a new file. Now we have one new file. Let's execute the sync again and see what aws does. As you can see, the sync function knows that the previous files have been synced and only uploads the newly created file. We can confirm by looking at GUI. Sure enough, there's hello.txt. So you can see how the sync function is very powerful. If you want to make sure a certain folder or files is always synced to aws s3, a good way to achieve this would be to set up a recurring crime job to run the sync command on the same folder.
It will only incur bandwidth for the newly created files. This way you can use the cli to stay in sync with s3.
Join AWS architect Brandon Rich and learn how to configure object storage solutions and lifecycle management in Simple Storage Service (S3), a web service offered by AWS, and migrate, back up, and replicate relational data in RDS. Find out how to leverage flexible network storage with Elastic File System (EFS), and use the new AWS Glue service to move and transform data. Plus, learn how Snowball can help you transfer truckloads of data in and out of the cloud.
- What is data management?
- AWS S3 basics
- S3 bucket creation
- S3 upload and logging
- S3 event notifications
- S3 data lifecycle configuration
- Working with Amazon Elastic Block Store volumes
- Creating and mounting an EFS
- Creating an AWS RDS instance
- RDS backup and recovery
- Moving data with AWS Database Migration Service
- Moving data with Data Pipeline and Glue
Skill Level Intermediate
Amazon Web Services: Monitoring and Metricswith Sharif Nijim2h 4m Intermediate
Amazon Web Services: Data Serviceswith Lynn Langit4h 30m Intermediate
Amazon Web Services: High Availabilitywith Sharif Nijim2h 17m Intermediate
Amazon Web Services for Data Sciencewith Lynn Langit3h 56m Intermediate
2. Object Storage
3. File Systems
4. Database Services
5. Getting Data to AWS
6. Moving Data in AWS
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.