Configuring a Single Node ELK Instance

2020-01-15 about 1536 words 8 min

Contents

Introduction

This post will aim to guide you in setting up a ELK stack instance, other posts will go into detail about other aspacts such as monitoring logs on Windows. A brief description will be given on all the components that will be used.

If you are interested in reading more about the technical details of how the different components work please consult the References section, I have linked some interesting articles and videos that can be of use.

Elasticsearch

Elasticsearch is a database that can be classified as NoSQL, it provides searching and analytics capabilities at a distributed level. There are options to cluster multiple different nodes which can help with search speed and scalability, however, this post will only be concerned with setting up a single node. The core component of Elasticsearch is Apache Lucene which is what provides the actual searching capabilities. Elasticsearch itself is a layer built on top of Apache Lucene that allows for different features, such as clustering and sending queries as RESTful API requests.

The main thing to understand is that Elasticsearch is the component on which all the logs will be stored, this means that logs will be sent to Elasticsearch and quires regarding logs will be sent to Elasticsearch .

Kibana

Kibana is a visualization layer of the ELK stack, it allows an analyst to go to a webpage and view all the logs that are stored on Elasticsearch. Kibana allows an analyst to create queries for specific logs and create visualizations.

Logstash

Logstash makes it easy to send data from endpoints to Elasticsearch and simplifies the process of ingesting data into Elasticsearch. It acts as a point to which logs are initially sent, they can be sent in a few different formats such as Syslog or Beats. This is refereed to as the input stage. After they can be filtered which can provide structure to logs and attach other information such as geographical location of an IP address. Lastly, the output stage sends the data to Elasticsearch.

Beats

Beats are data shippers that can send information to either Logstash or Elasticsearch directly. There are a few different types, for example Metricbeat can send information regarding metrics of the operating system and Packetbeat can send network packet analytics information.

For this post Winlogbeat will be used on a Windows endpoint system in order to ship Windows logs to Elasticsearch.

Configuring ELK

Configuration of ELK will be done on CentOS 7 (these steps have also been tested on CentOS 8), all three components (Elasticsearch, Logstash, and Kibana) will be setup on the single server.

Creating a Partition for Logs

This is more of a personal preference, but I like to create a dedicated disk and folder for Elasticsearch to put all the logs in. There is a default location, which you are free to use, but this section will show how to format a disk for the logs.

We assume that the new disk will be /dev/sdb, and that it has been partitioned with 1 partition of /dev/sdb1. Begin by formatting the partition with your favourite filesystem, here I will use XFS.

1

[user@ELKNode]$ mkfs.xfs /dev/sdb1

Now we will create the folder in which all the logs will be stored.

1
2


[user@ELKNode]$ mkdir /logs
[user@ELKNode]$ mount /dev/sdb1 /logs

After we will mount /dev/sdb1 and figure out the UUID of the mount point using the blkid command. As can be seen below the UUID is 08a16a80-45b3-4aee-87ff-48d43a55a63e.

1
2
3
4
5
6
7
8


[user@ELKNode]$ blkid
/dev/sda1: UUID="3D03-8706" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="ad07c298-d43e-4d99-8bd3-54e60788fb33"
/dev/sda2: UUID="7bd4dd91-d7ca-4af1-9c7c-aec527b6e43b" TYPE="ext4" PARTUUID="8e3e5911-eaf5-4bf5-8df0-4ba4a7b73195"
/dev/sda3: UUID="2k6w9c-tYU2-HCAZ-g2Sf-e4ke-jsyt-UwWMQk" TYPE="LVM2_member" PARTUUID="267d8f89-5e8a-4c2e-918e-3cb6d2622eaa"
/dev/sdb1: UUID="08a16a80-45b3-4aee-87ff-48d43a55a63e" TYPE="xfs" PARTUUID="7fb72529-01"
/dev/mapper/cl_centos8template-root: UUID="6b1b2d75-934e-4451-99c4-be8c4d9ec1ae" TYPE="xfs"
/dev/mapper/cl_centos8template-swap: UUID="da38cf22-263b-4d5b-8bc4-007d85f26597" TYPE="swap"
[user@ELKNode]$

Next, add the following line into the /etc/fstab file so this partaitoin can be mounted on boot.

1

UUID=08a16a80-45b3-4aee-87ff-48d43a55a63e       /logs   xfs     defaults        0 0

Installing Elasticsearch

Elasticsearch is written in Java and so it will need to be installed, run the following command to install Java.

1

[user@ELKNode]$ sudo yum install java-1.8.0-openjdk

In order to be able to install any components of the ELK stack the Elasticsearch repository must be added. In order to do this create a file located in /etc/yum.repos.d/elasticsearch.repo and add the following contents to it:

1
2
3
4
5
6
7
8


[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

After the Elasticsearch repository is added Elasticsearch can be installed. We also need to change the owner of the /logs directory, if you are using a different disk to store the Elasticsearch data.

1
2


[user@ELKNode]$ sudo yum install elasticsearch
[user@ELKNode]$ chown elasticsearch:elasticsearch /logs/

Configuring Elasticsearch

Now that Elasticsearch is installed it must be configured. The configuration file is located at /etc/elasticsearch/elasticsearch.yml. You can either copy paste the configuration from this website, or uncomment and edit the needed fields that are there by default.

1
2
3
4
5
6


cluster.name: TestELK
path.data: /logs # Default is /var/lib/elasticsearch
path.logs: /logs # Default is /var/lib/elasticsearch
network.host: 10.0.0.10
http.port: 9200
discovery.type: single-node

Elasticsearch has the ability to cluster multiple nodes, however, this functionality is out of scope for this post as only a single node will be setup. The cluster.name field describes the name of this cluster, it will not change much on a single node aside from the name in the logs. It is not mandatory to have this value since by default it is set to the name elasticsearch.

The path.data field will specifies the location that Elasticsearch will store the data. More over, path.logs specifies the location that the log fields for Elasticsearch will be stored.

By default Elasticsearch will be run on 127.0.0.1:9200, by running it on localhost it will also be in development mode. This is completely valid way to run, however, in this example I am binding it to a interface with the IP address 10.0.0.10 using the network.host field.

Lastly, discovery.type field will be set to single-node. This setting will cause Elasticsearch to elect itself as the master and will not join any clusters. We need this since it will be running as a stand alone node.

Elasticsearch License

Elasticsearch does have a license , most people looking to play around with it will probably use the Basic license which will have most of the important features. If the license is not set it may cause issues with querying Elasticsearch.

Begin by starting Elasticsearch, this may take a moment or two. You will know for sure that Elasticsearch has started when it has binded to the specified port in the configuration, which is 9200 in this case.

1
2


[user@ELKNode]$ sudo systemctl start elasticsearch
[user@ELKNode]$ sudo systemctl enable elasticsearch

When it has started run the following command in order to view the current license.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


[user@ELKNode]$ curl -X GET "10.0.0.10:9200/_license?pretty"
{
  "license" : {
    "status" : "active",
    "uid" : "729469ff-c9dc-4687-8c25-4136aed5a230",
    "type" : "basic",
    "issue_date" : "2019-09-19T04:15:59.184Z",
    "issue_date_in_millis" : 1568866559184,
    "max_nodes" : 1000,
    "issued_to" : "TestELK",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}

This can also be done using PowerShell as seen below:

1

PS C:\> Invoke-RestMethod -Method GET -Uri '10.0.0.10:9200/_license?pretty'  | Format-List

Take note of the status and type field, which are set to active and basic. This means that the basic license is active, in this case it was automatically generated when it was started.

In the case where it was not generated you may have to start the license manually , this can be done using the following command:

1

curl -X POST "10.0.0.10:9200/_license/start_basic?pretty"

Installing Kibana

Now Kibana must be installed and configured. The Elasticsearch repository has already been added from the last section, so it is enough to just run the installation command.

1

[user@ELKNode]$ sudo yum install kibana

The configuration file is located at /etc/kibana/kibana.yml make sure to alter it in order to make Kibana accessible. The default port is set to 5601, once again it is binded to the interface with the IP address of 10.0.0.10. Lastly, the Elasticsearch node is specified.

1
2
3


server.port: 5601
server.host: "10.0.0.10"
elasticsearch.hosts: ["http://10.0.0.10:9200"]

Kibana will run on the port specified using HTTP, you will be able to connect to it through the browser. If HTTPS is desired a reverse proxy can be setup to point to Kibana, for this you may wish to alter some settings such as which IP address Kibana is binded.

Start the Kibana service using the following command.

1
2


[user@ELKNode]$ sudo systemctl start kibana
[user@ELKNode]$ sudo systemctl enable kibana

Installing Logstash

After Kibana is setup Logstash will need to be installed. Run the following command to install Logstash.

1

[user@ELKNode]$ sudo yum install logstash

Start the Logstash service using the following command.

1
2


[user@ELKNode]$ sudo systemctl start logstash
[user@ELKNode]$ sudo systemctl enable logstash

After following these steps an instance of Elasticsearch, Kibana, and Logstash should be running and ready to take in logs. Following posts will discuss how to ingest logs into ELK.