Updated March 14th to reflect that I’m now installing the second Elasticsearch instance on TheHive VM and not in Docker.
In the first installment we’ll begin by creating our Elasticsearch cluster. For the VMs I’ll be using Ubuntu 16.04.
-Note on Security-
There is none. I’ll follow-up with a later blog post on setting up certificates and SearchGuard.
Update: I’m now looking at utilizing Open Distro instead with its built-in alerting and security.
-Note on Networking-
On my network I have MAC address reservations on my DHCP server to ensure the VMs consistently receive the same IP without having to configure the host. You may have to configure static addresses on the host.
I also have a Vagrantfile with Ansible Playbook that can configure the Elasticsearch VM for you.
ELK Installation
Caveats:
TheHive Project is undergoing some back-end changes currently which leads to a complication in the setup. The team behind TheHive has decided that Elasticsearch no longer meets their needs and release 4.0 will use GraphDB in the back-end. The current stable 3.2.1 release, as well as the current beta of 3.3.0 which we’ll be using in this deployment, require a back-end of Elasticsearch 5.6. As a result I’ve decided to use two Elasticsearch instances for this deployment. I’m deploying Elasticsearch, Logstash, and Kibana 6.6.1 on a VM as the log repository, and deploying Elasticsearch 5.6.15 on the Hive VM for its back-end. The installation of ES 5.6 will be covered in a later post.
Although I’ll summarize the steps below, I’m following the Elasticsearch Debian installation guide should you need further explanation.
I’m assuming you’re beginning with a freshly patched Ubuntu 16.04 server.
Start by installing Java - I’ve opted for the OpenJDK installation
sudo apt-get install openjdk-8-jre
Now add the key and repository
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update
I like to install a specific version of Elasticsearch and then pin that version so it doesn’t update automatically. Auto-upgrading can break add-ons like SearchGuard.
apt-cache policy elasticsearch will show installable package versions from that repository. I’m choosing 6.6.1, the most current.
sudo apt-get install elasticsearch=6.6.1 logstash=1:6.6.1-1 kibana=6.6.1
Now pin those versions
sudo apt-mark hold elasticsearch logstash kibana
And set Elasticsearch, Logstash, and Kibana to auto-start
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl enable logstash.service
sudo systemctl enable kibana.service
When you’re ready to upgrade in the future, after verifying compatibility, run the following and then upgrade.
sudo apt-mark unhold elasticsearch
Optimizations
This isn’t a production worthy stack but a few optimizations can’t hurt.
The Java heap size that Elasticsearch relies on is 1 GB by default. Depending on the resources you allocated you can increase this to 50-80% of total RAM by modifying Xms1g and Xmx1g in /etc/elasticsearch/jvm.options.
My VM only has 4 GB so I’ll leave the default.
Edit the config: sudo nano /etc/elasticsearch/elasticsearch.yml
- Uncomment
cluster.nameandnode.nameand make them distinct names - Set
bootstrap.memory_lock :true - Set
network.host 0.0.0.0 - Set
discovery.type: single-node
Edit the service: sudo systemctl edit elasticsearch.service
Add the following:
[Service]
LimitMEMLOCK=infinity
Reload:
sudo systemctl daemon-reload
And restart:
sudo systemctl start elasticsearch.service
Now check to make sure Elasticsearch is responding
curl http://localhost:9200/_cat/health
You should get back something that looks similar to
1551641374 19:29:34 demo-cluster green 1 1 0 0 0 0 0 0 - 100.0%
Kibana
Edit Kibana’s config: sudo nano /etc/kibana/kibana.yml
Set Kibana to respond on its external interface
server.host: 0.0.0.0
Start the service
sudo systemctl start kibana.service
Navigate to http://<IP-OF-VM>:5601 and you should see the Kibana console.
Install Logstash
sudo apt install logstash
sudo systemctl enable logstash.service
sudo systemctl daemon-reload
We’ll set the config in the next post. Just remember that logstash isn’t currently running.
You now have Elasticsearch up and running, ready for data. More to install coming up.