Spark Streaming architecture for dynamic prediction

- [Instructor] As we're continuing our tour…of Hadoop architectures,…I wanted to end on an aspirational note.…A couple of things, first of all,…I'm obviously a big fan of Spark,…you can tell from the focus of this course.…And I see lots of customer demand,…and I see lots of development,…and exciting stuff happening in the Spark ecosystem,…and Spark is a key part of the majority…of the Modern Hadoop Pipelines that I see.…So the business situation is the company needs…to do dynamic price prediction of airline ticket prices…which in and of itself doesn't sound super exciting…but how they solved it really is quite intriguing,…using a Modern Hadoop architecture.…

What they did is, it's a variant of…what we actually looked at earlier in this course…where we ran a notebook…and we looked at using Machine Learning on Spark.…The results were kind of underwhelming.…Do you remember back there,…where it was like 29% prediction correct?…And then we used, basically, an implementation of a matrix…so that we could examine all different possibilities…