Understand the multitude of control planes for tuning distributed compute jobs. These include cluster, job, and more.
- [Instructor] As I've previously mentioned,…when sizing on Databricks…there are a number of common Control Planes.…And those include optimizations around the data,…optimizations around the job,…and the activities associated to the job,…and optimizations around the cluster.…So let's take a look at these in the context…of our example.…So in the world of data,…the most common optimizations that I see…in working with customers on Spark…are compressing the input data.…
A number of compression formats are supported.…We'll be working with a compression format called EZ-2.…Partitioning the data, splitting input data files into…files that are of smaller size so they can be distributed…more quickly and easily.…This strategy I've been using in combination…with the next one which is converting…to a format that is more optimized for the type of compute…that you're performing.…So in my particular use case where we evolved to was…we started with basically CSV or text files.…
And we then moved to compressing them,…we then moved to converting them to .parquet which is…
- Business scenarios for Apache Spark
- Setting up a cluster
- Using Python, R, and Scala notebooks
- Scaling Azure Databricks workflows
- Data pipelines with Azure Databricks
- Machine learning architectures
- Using Azure Databricks for data warehousing
Skill Level Intermediate
1. Big Data on Azure Databricks
2. Core Azure Databricks Workloads
Use a notebook with scikit-learn11m 29s
3. Scaling Azure Databricks Workloads
4. Data Pipelines with Azure Databricks
5. Machine Learning Architectures
Next steps1m 1s
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.