Lynn discusses open source Hadoop version numbers and features. She also talks about public cloud-based Hadoop distributions, from AWS and Google.
- [Instructor] As we're thinking about Hadoop clusters, of course, open source is where this all started. Most of my customers though, other than startups, really don't use open source because they need tooling, they need management around it, and frankly, although the underlying structure changes, not a lot has been happening since I recorded my Hadoop fundamentals in comparison to the amount of change in the commercial and public cloud distributions. Now, for commercial distributions, of course, Doug Cutting, who created Hadoop, and his team are at Cloudera, and they still are a market leader in many ways.
I find they're used more often on premise, although obviously could be deployed into the cloud. The new player, kind of the shining star, is Databricks, which has created the Apache Spark libraries and has a commercial version. We'll be using that in this course 'cause that's what I'm using more and more with my customers. Now that actually sits on top of the Amazon cloud currently, so it spins up virtual machines on Amazon. But we'll get into that as we get into the demos. A new aspect of Hadoop distributions is that in addition to the managed distribution that's been on the Amazon cloud for several years, EMR or Elastic MapReduce, we now have some competition from Azure with HDInsight and most notably from Google with the Google Cloud Dataproc.
We're going to take a look actually at Google Cloud Dataproc because I think it's an exciting new player and it's been out less than, I think, six months in general availability at the time of this recording. But it is a really fast, really performant and has a great value. Of course, EMR is the market leader because it's been out for the longest, but I wanted to include a look at GCP, Google Cloud Dataproc, because it's really interesting in terms of speed and value.
Author
Released
7/5/2017- Relate which file system is typically used with Hadoop.
- Explain the differences between Apache and commercial Hadoop distributions.
- Cite how to set up IDE - VS Code + Python extension.
- Relate the value of Databricks community edition.
- Compare YARN vs. Standalone.
- Review various streaming options.
- Recall how to select your programming language.
- Describe the Databricks environment.
Skill Level Intermediate
Duration
Views
Related Courses
-
Apache Spark Essential Training
with Ben Sullins1h 27m Intermediate
-
Introduction
-
Welcome53s
-
-
1. Hadoop Core Fundamentals
-
Modern Hadoop1m 53s
-
Hadoop libraries1m 23s
-
Run Hadoop job on GCP1m 52s
-
Databricks on AWS2m 32s
-
-
2. Setting Up a Hadoop Dev Environment
-
Load data into tables1m 51s
-
3. Hadoop Batch Processing
-
Processing options1m 2s
-
Resource coordinators1m 30s
-
Compare YARN vs. Standalone1m 30s
-
-
4. Fast Hadoop Options
-
Big data streaming1m 57s
-
Streaming options1m 10s
-
Apache Spark basics1m 46s
-
Spark use cases1m 2s
-
5. Spark Basics
-
Apache Spark libraries3m 24s
-
Spark shell1m 53s
-
-
6. Using Spark
-
Tour the notebook5m 29s
-
Import and export notebooks2m 56s
-
Calculate pi on Spark8m 19s
-
Import data2m 50s
-
Transformations and actions4m 43s
-
Caching and the DAG6m 49s
-
7. Spark Libraries
-
Spark SQL8m 34s
-
SparkR6m 11s
-
Spark ML: Preparing data4m 21s
-
Spark ML: Building the model3m 50s
-
MXNet or TensorFlow2m 30s
-
Spark with GraphX2m 12s
-
-
8. Spark Streaming
-
Spark streaming4m 21s
-
9. Hadoop Streaming
-
Pub/Sub on GCP3m 59s
-
Apache Kafka1m 26s
-
Kafka architecture1m 6s
-
Apache Storm1m 30s
-
Storm architecture1m 36s
-
-
10. Modern Hadoop Architectures
-
Conclusion
-
Next steps26s
-
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.
CancelTake notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.
Share this video
Embed this video
Video: Apache and commerical Hadoop distributions