Learn about how to set up a Hadoop cluster.
- [Instructor] Setting up a Hadoop cluster…is much more involved than installing Hadoop…in a standalone mode.…It requires a tedious process…of installing underlying software,…adding environment variables,…and changing configuration files.…Much of this can, of course, be automated…by writing shell scripts or by cloning virtual machines.…In this demo, I have two Proxmox virtual machines,…an Ubuntu desktop VM, shown here.…
It's configured to be my master node,…while an Ubuntu server VM, shown here, serves as a slave.…One additional software requirement for a Hadoop cluster,…is secure shell server,…which needs to conduct public key authentication.…For example, I created a new user called hduser…for both my Hadoop master and slave.…
I also edited their /etc/host and /etc/hostname files…so that I can do this.…Let's first log in as hduser on my Hadoop master VM.…Type su - hduser as your user name,…and then type your password.…Here, the dash allows me to log in to the home directory…of the hduser.…
Now, type ssh master.…As you can see here, I don't need a password to log in,…
- Enabling technologies in data science
- Cloud computing and virtualization
- Installing and working with Proxmox, Hadoop, Spark, and Weka
- Managing virtual machines on Proxmox
- Distributed processing with Spark
- Fundamental applications of machine learning
- Distributed systems and distributed processing
- How Hadoop, Spark, and Weka can work together
Skill Level Beginner
Course organization1m 17s
1. Introduction to Data Science
2. Cloud Computing
3. Distributed File Systems
4. Distributed Processing
5. Machine Learning
6. Case Study
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.