Learn how to download, unzip, and understand the exercise files and include files.
- [Instructor] In this video, I'm going to show you how to download and use your exercise files. So the exercise files are available to you as a part of the course, so download them and put them in your desktop. Once it is here, unzip them to the same folder, preferably. Okay, now that we have extracted the files, let us go and explore what is inside this folder. So you have one folder here called code. Let's go into this one.
So what you see here, you have a lot of .sh files. Now, these are script files that we would be using as a part of different exercises later for Kafka Connect. There is also a MySQL database setup that has got instruction to set up MySQL with certain tables and databases that are needed for our exercise. Under the include folder is a bunch of jar files that are required for Kafka Connect for using JDBC connectors. We will show them where to deploy them when we get to that specific exercise.
And then you have a Spark data engineering project. Now, this is a Java project, so I recommend first you extract it to the same location, 'cuz this is another zip file, so now it is extracted, and end of that you'll see a normal Java Maven project setup. So we will be importing this into Eclipse in the next video.
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators