Kibana Pie Chart Elastic Search, Logstash and Kibana – the ELK Stack – is emerging as the best technology stack to collect, manage and visualize big data. If you came here looking for help installing the ELK stack, you don’t need an introduction, so let’s get right down to the dirty work. The following guide shows how to install Java 8, Elasticsearch 2.3, Logstash 2.3 and Kibana 4 on Ubuntu with init.d (system v) or alternatively with systemd. You can do one or the other depending on your system and/or preferences. In two previous posts Integrate Bro IDS with ELK Stack and How to Install Bro Network Security Monitor on Ubuntu
, we showed how to install Bro and parse the generated Bro logs with Logstash. With the entire stack installed, running, and parsing logs generated by Bro, Kibana allows for a wonderful data analysis and discovery process. Of course, almost any data source can be used, and not just Bro.

Oracle Java 8

Elasticsearch

Note: Check for the latest Elastic Search release version here: downloads/elasticsearch

System V

Systemd

Configure

Note: If you want to access your Elasticsearch instance from clients on a different IP address via Javascript, add the following inside elasticsearch.yml:

Also note that if you want to access Elasticsearch of any of the plugins like kopf from a host besides local host, you’ll need to add the following to elasticsearch.yml:

FYI, the Elasticsearch stores your actual data in /var/lib/elasticsearch/elasticsearch/nodes/....

Test

In browser: http://localhost:9200/

Hello World Data

Debugging

Debug startup errors by running elasticsearch in the console

Elasticsearch Kopf Plugin (an aside)

The kopf plugin provides an admin GUI for Elasticsearch. It helps in debugging and managing clusters and shards. It’s really easy to install (check here for latest verion):

View in browser at: http://localhost:9200/_plugin/kopf/#!/cluster. You should see something like this:

Elasticsearch Kopf

Elasticsearch Kopf

Logstash

Note: Check for the latest Logstash release version here: downloads/logstash

System V

Systemd

Configure

By default Logstash filters will only work on a single thread, and thus also one CPU core. To increase the number of cores available to LogStash, edit the file /etc/default/logstash and set the -w parameter to the number of cores: LS_OPTS="-w 8".

You can increase the Java heap size here as well. Make sure to uncomment the line you are updating. Don’t forget to restart logstash afterwards.

Test

Directly change java options in script if you are starting logstash from the command line and not as a linux service.

Hello World (warning Logstash at the command line is slow to start, so be patient)

Hello World with Elastic Search

Plugins

For non-standard parsing features, we access plugins. The following terminal commands show how to install the logstash-filter-translate plugin. For a more in-depth explanation of installing logstash plugins see How to Install Logstash Plugins for Version 1.5.

Kibana

Note: Check for the latest Kibana release version here: downloads/kibana

System V

Systemd

Configure

Test

In browser: http://localhost:5601

Final Words

If all went well, the next step is to tap into a datasource with Logstash and view it with Kibana. In two previous posts Integrate Bro IDS with ELK Stack and How to Install Bro Network Security Monitor on Ubuntu
, we showed how to install Bro and parse the generated Bro logs with Logstash. The following is a screen shot from a Kibana dashboard we made for one of our websites bitcoinium.com showing some nice bar and pie charts. Once everything is set up and running it immediately becomes clear how useful the ELK stack is.

ELK Stack for Bitcoinium

ELK Stack for Bitcoinium

Related Resources

Integrating Bro with the ELK Stack: http://knowm.org/integrate-bro-ids-with-elk-stack/
How to Created a Bonded Network Interface: http://knowm.org/how-to-create-a-bonded-network-interface/

Related Posts

Subscribe To Our Newsletter

Join our low volume mailing list to receive the latest news and updates from our team.

Leave a Comment

Knowm 32X32 Crossbar

Knowm Newsletter

Are you ready for memristor AI processors? With our newsletter, you will be.