Spark architecture for interactive analytics

- [Instructor] Now, here we've got Hadoop Architecture…running on the Google Cloud.…And what I think is of note here, is that you have,…on the left side, both batch and streaming.…And notice, that for the stream data,…the recommended architecture is to use Cloud Pub/Sub.…Of course, if you had requirements around guarantees…of messages in order, do you know what you'd use?…Probably Kafka.…So, you have different options that you can use…for the stream ingest layer.…The batch ingest layer, on the upper left-hand side,…is just using Google Cloud storage.…

Now, the data's coming in, and it's being…aggregated by Google's Cloud data flow service.…What this is, is a productized implementation of…yet another open source set of libraries called Apache Beam.…This is a relatively new service,…and it allows for sophisticated data pipelines…that are scalable to be created.…This is in beta at the time of this recording,…so if you're building a pipeline on the Google Cloud,…you're going to want to look at data flow, as well.…