From the course: Cloud Hadoop: Scaling Apache Spark

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Streaming ingest services

Streaming ingest services - Apache Spark Tutorial

From the course: Cloud Hadoop: Scaling Apache Spark

Start my 1-month free trial

Streaming ingest services

“

- [Instructor] I've been mentioning streaming ingest services for modern Hadoop pipelines, and you might be curious as to what these are. So, this is an introduction to this topic. It's a huge topic, but I just wanted to get some of the names out there because we're going to be seeing how they work or drilling into them in subsequent sections. So, probably the most popular one is open source. It's Apache Kafka and there are many different flavors and configurations. It's sort of the defacto standard. Now, that being said, it's interesting to see that the three major cloud providers have come up with streaming ingest services on their own and they've been extremely popular. So, Amazon's is Kinesis, Google Cloud is Pub/Sub, and Azure has the Service Bus.

Contents