This video is a brief introduction about all the product offerings of Elastic and how each product fits into the stack.
- [Karthik] Section one, the Elastic Stack, In this section we are going to take a look at all the products of the Elastic family, followed by the importance of analyzing your data with Elastic Stack. Finally, we'll be installing and starting Elasticsearch, Logstash, and Kibana in a system running with Windows operating system. Chapter one, the Elastic family. In this video we are going to take a look at a brief introduction about all the product offerings of Elastic and how each product fit into the stack.
The first and most important product of the Elastic Stack family is Elasticsearch. A distributed, full text search server and analytic engine. That is build on top of Apache Lucene. It serves as the heart of the Elastic Stack. Elasticsearch was first developed by Shay Banon, and was open-sourced as a distributed search engine in 2010. Right from its first release, it gained popularity for its ease of use and was implemented by many companies.
Later in 2012 the Elasticsearch company was founded by the people behind Elasticsearch and Apache Lucene. In March 2015 the company was renamed to Elastic, to better align the company with the broad solution it provided. The next product in the Elastic Stack family is Logstash. An open source data collection engine with real-time pipelining capabilities. It was started by Jordon Sissel in 2009, later in 2013 Logstash was added to the Elastic family.
A few months later, a new product by the name Logstash Forwarder was introduced. The main difference between Logstash and Logstash Forwarder was that Logstash Forwarder was used only to ship logs from server. Where as Logstash had capabilities of converting the files to JSON document and had the ability to send to various output nodes like Kafka file, etc. The other reason being for the existence of Logstash Forwarder was that Logstash is a Java-based program and it was heavy to install on all the host servers.
In contrary Logstash Forwarder was a light-weight product and was used to collect log files from all these servers. The next member in the product line is Kibana. An open-source browser based analytic and search dashboard for Elasticsearch. It was created by Rashid Khan in 2011. Later in Jan 2013, Rashid joined the Elasticsearch team and it became the visual front end of the Elastic Stack. Together these three products became a very powerful tool called the E L K Stack or the ELK Stack.
Where E stands for Elasticsearch, L stands for Logstash and K stands for Kibana. The ELK Stack become an instant hit and was widely adopted by many companies. Wikipedia, GitHub, Ebay, Facebook, Cisco, and NASA are a few examples, among a large number of companies who have benefited by the ELK Stack. In May 2015 Elastic introduced the Beats family.
A platform for building lightweight open sourced data shippers that can send data to Elasticsearch, which can be analyzed later. You'll be seeing in detail about the Beats platform in the upcoming slides. With Beats added to the product line, Elastic unveiled the Elastic Stack in Feb 2016. Which consisted of Elasticsearch, Logstash, Kibana and Beats. A new product by the name Xpack was introduced in Feb 2016.
That extends the Elastic Stack with features such as security, alerting, monitoring, reporting and graph. We will look into each product of Xpack later in this video. The Beats platform. Packetbeat, a real-timie network and packet analytics solution was developed by Tudor and Monica. And was acquired by Elastic in May 2015. Along with this, they also unveiled the Beats platform. In the same release another Beats was also introduced by the name File beats, which was the replacement of Logstash Forwarder.
Metricbeats. Metricbeats is a lightweight shipper that can be installed on servers to periodically collect metrics from the operating system and the services running on it. Previously the product was knows by the name Topbeats. In Feb 2016 Winlogbeats, a lightweight shipper for Windows Event Logs was introduced. It acts like a Windows Service and can ship even log data to Elasticsearch or Logstash.
Later in July 2016, the libbeat framework was unveiled through which the developers can develop their own Beats. Some of the famous community beats are BingBeats and DockerBeats. Xpack. Monitoring. In Jan 2014 Elastic introduced their first commercial product by the name Marvel. A product which will monitor Elasticsearch deployments in real-time giving system administrators complete transparency into the state of their deployments.
As part of the Xpack package it was later renamed to Monitoring. Shield. In Jan 2015, Elastic introduced a commercial plug-in by the name Shield to enhance the security and administrative functionalities in Elasticsearch. A few features of Shield are providing role-bases access control, LDAP-based authentication system support, and audit logging. Alerting. In May 2015, a new product providing alerting and notification capabilities for Elasticsearch was introduced which was called by the name Watcher.
Watcher allowed companies that used Elasticsearch for real-time search and analytics to set up alerts and notifications around the changes and trends in their data. A new feature called Reporting was introduced along with the Xpack package. Reporting was used to generate scheduled and email dashboard as PDF reports to any user on or group to collaborate among the team. Graph. In March 2016, Graph a new extension of Elasticsearch and Kibana was introduced which allows users to uncover, understand and explore the relationship between the data.
In the next video, we will be seeing how to make sense of your data using the Elastic Stack.
This course was created and produced by Packt Publishing. We are honored to host this training in our library.
- Elasticsearch concepts
- Working with Logstach and Kibana
- Elasticsearch Query DSL
- Aggregation and analyzers
- Scripting in Elasticsearch
- Using plugins and APIs
- Building an interface with dashboards
- Filtering and processing input
- Loading data to Elasticsearch