From the course: Data Science Tools of the Trade: First Steps

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Spark: Application

Spark: Application

From the course: Data Science Tools of the Trade: First Steps

Start my 1-month free trial

Spark: Application

“

- [Instructor] We can do some simple interactions with Spark through its native Spark shell or PySpark. But using the interactive shell environment is quite cumbersome as your program gets more complicated. This is why Spark has a mechanism that allows you to create and submit a self-contained application. Let's create a new Python script called Simpleapp.py. Change your directory by typing cd, space, slash, usr, slash, local, slash, spark, slash, examples, slash, src, slash, main, slash, python. Press Enter. Type gedit, space, Simpleapp.py. Press Enter. You will be presented with a blank document because the contents have not been imported yet. Using your favorite web browser, navigate to this website and copy and paste the code as such. Then change the line that contains YOUR_SPARK_HOME to slash, usr, slash, local, slash, spark. Save the file and exit. This program simply counts the number of lines containing letter a and the number of lines containing letter b. To run this…

Contents