ELK- Elastic Logstash Kibana

1. Elastic Search is a robust search and analytics tool that stores data in a document oriented data store. It is open source, meaning you can download, use and modify the program free of charge.
2. The most popular usage of ElasticSearch today is Log Management.
3. Similar products in the market today are Splunk and Solr. Splunk is the enterprise software, and expansive. 
4. Elastic Search is built on top of high performance open source search engine Apache Lucene. The documented oriented storage differs sharply from traditional table oriented RDBMS (Such as Oracle, MS SQL Server). With document oriented data storage, data is stored as structured JSON (JavaScript Object Notation) documents. Every field is indexed by default. This is why the search speed is incredible. 
5. The architecture of Elastic Search favors distribution, meaning you can scale your Elastic Search infrastructure massively and seamlessly. The infrastructure is resilient for failures – it automatically relocates and balances resources. 
6. The user interface to perform search and analytics is Kibana, an open source data visualization platform. 
7. With its intuitive, clean and responsive interface Kibana makes searching for data a joy. 
8. Logstash, an open source tool does the heavy lifting of consuming the logs from various systems and sends them to Elastic Search. 
9. It is the workhorse that collects the log files from Application Servers, parses them, formats them and sends them to Elastic Search. 
10. Together, Elastic Search, LogStash and Kibana form the commonly known System ELK. It provides powerful RESTFul API’s  to do every action imaginable using the data set.

 

Logstash collects the log files from Application Servers, parses them, formats them and sends them to Elastic Search. Elastic Search stores and indexes the data, which is presented by Kibana. The end users access Kibana Web Interface to view the data.

1  Application Servers to LogStash Typically, a Logstash Forwarder component is installed on the Application Servers. This piece of software is responsible for collecting the logs based on the configuration that you setup via a configuration file. Logstash Forwarder is an instance of Logstash whose input is Application Log files and output is Logstash Indexer (The 2nd block from the right in the above diagram). Optionally you can introduce a Message queueing component between the Application Servers and the LogStash Indexer. Rabbit MQ is a popular choice in ELK implementations.

2  Logstash to Elastic Search Cluster Logstash (indexer) parses and formats the log (based on the log file content and the configuration of LogStash) and feeds Elastic Search Cluster. The protocol used is a Native Elastic Search Transport. Elastic Search consumes data from Logstash and creates Indexes (for log files, typically a date-based index is created by Elastic Search).

3    

   Kibana is a data visualization platform that is the presentation layer in the ELK stack. It consumes data from Elastic Search Indexes. A user accesses Kibana interface via a web browser.