From the course: Cloud Hadoop: Scaling Apache Spark

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Build a quick start with Databricks AWS

Build a quick start with Databricks AWS - Apache Spark Tutorial

From the course: Cloud Hadoop: Scaling Apache Spark

Start my 1-month free trial

Build a quick start with Databricks AWS

“

- [Instructor] Again, I think something that is unexpected in our story here is that we started not with infrastructure, rather the opposite side. We decided to start with Databricks, with SaaS, and the reason for that was before we got into scaling, we really wanted to test out viability, we wanted to get feedback from the users, the researchers, and we wanted to have a way they could get up and running really quick and give us feedback. So to do this, there were several steps. First we selected the implementation, so SaaS on AWS, and the reason we selected AWS is because it was simply easier to sign up using their community edition than the Azure implementation. It didn't require a work email, for example. In terms of pairing the algorithm or in this case, the workload, to run on Spark, that actually took quite a lot of time. And that's like a step zero, basically, in scaling Spark workloads. In our case, we went into…

Contents