Learn how to submit a self-contained Spark application.
- [Instructor] We can do some simple interactions with Spark…through its native Spark shell or PySpark.…But using the interactive shell environment…is quite cumbersome as your program gets more complicated.…This is why Spark has a mechanism…that allows you to create and submit…a self-contained application.…Let's create a new Python script called Simpleapp.py.…Change your directory by typing cd, space,…slash, usr, slash, local, slash, spark,…slash, examples, slash, src, slash, main, slash, python.…
Press Enter.…Type gedit, space, Simpleapp.py.…Press Enter.…You will be presented with a blank document…because the contents have not been imported yet.…Using your favorite web browser,…navigate to this website…and copy and paste the code as such.…
Then change the line that contains YOUR_SPARK_HOME…to slash, usr, slash, local,…slash, spark.…Save the file and exit.…This program simply counts the number of lines…containing letter a…and the number of lines containing letter b.…To run this self-contained application,…type spark-submit,…
- Enabling technologies in data science
- Cloud computing and virtualization
- Installing and working with Proxmox, Hadoop, Spark, and Weka
- Managing virtual machines on Proxmox
- Distributed processing with Spark
- Fundamental applications of machine learning
- Distributed systems and distributed processing
- How Hadoop, Spark, and Weka can work together
Skill Level Beginner
Course organization1m 17s
1. Introduction to Data Science
2. Cloud Computing
3. Distributed File Systems
4. Distributed Processing
5. Machine Learning
6. Case Study
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.