From the course: Apache Kafka Essential Training: Getting Started (2021)
Join today to access over 22,700 courses taught by industry experts or purchase this course individually.
Logs in Kafka
From the course: Apache Kafka Essential Training: Getting Started (2021)
Logs in Kafka
- [Instructor] Where does the data that is sent by producers stored physically? It is in the Kafka logs. Kafka logs are the physical files in which data is stored before they are consumed by the consumers. Logs are managed by Kafka brokers. Each broker has an assigned log directory where it stores the log files. There are multiple log files created in Kafka. Each broker will have its own log territory. And in these directory, there are separate files for each topic and partition. These are rolling files so when a file gets filled up it's rolled over and a new file is created to continue with the logging process. So each partition will have multiple log files in the log directory. Data in Kafka is only kept for a configured interval of time. The default is seven days. A separate thread in Kafka keeps pruning files that are over this period. Log files are an important consideration for managing a Kafka instance…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.