In preparation for an upcoming post, I recently dove into my notes on installing the Prometheus monitoring server. My last time setting up Prometheus was on an Ubuntu server and the repository version was at least the same major revision version as the current release. This time I’m installing on Debian 9 and currently the latest Prometheus version is 2.3.2 while the Debian repository is offering 1.5.2. That’s unacceptable. While the sid repository does contain 2.3.2, I decided to take the opportunity to deploy in a cleaner (and less permanent) manner through Docker. Prometheus is well supported in Docker environments and it gave me an opportunity to brush up on my container deploying skills.
In this post we’ll cover the following:
- Installing Docker on Debian
- Deploying the latest images of prometheus, blackbox_exporter, and grafana with docker-compose
- Networking the containers together
- Checking the individual components to ensure they’re working
Installing Docker on Debian 9
Since I’m installing this on Debian the repositories tend to contain older, stable versions of software which can sometimes be many revisions out of date. So we’re going to add the official Docker repository and install an up-to-date version from there.
First we install some dependencies
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common
And make sure we don’t currently have any installed versions
sudo apt-get remove docker docker-engine docker.io
Now add the GPG key for the Docker repository
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
And add the Docker repository to apt
echo "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee -a /etc/apt/sources.list.d/docker.list
Update the apt database
sudo apt-get update
Now make sure you’re going to be installing from the correct repository
apt-cache policy docker-ce
The output should show links pointing to https://download.docker.com
Now install Docker
sudo apt-get -y install docker-ce
Check the status afterwards
sudo systemctl status docker
Now add your user account to the Docker group so you can deploy images as non-root.
sudo usermod -aG docker <username>
Install docker-compose
Since I’m deploying a total of 3 containers I decided to use docker-compose to script their creation. Docker-compose files have version numbers (v3 for now), and the docker-compose version in the default Debian repository doesn’t understand v3 at this time, so we need to install the current version. You should install docker-compose from the Docker website instructions. Since this is a download of a specific version number, you should check the instructions to determine the most up-to-date version. As of this writing that’s version 1.22 so I ran the following.
sudo curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
Make sure docker-compose --version now shows the correct version number.
Deploying Prometheus, blackbox_exporter, and Grafana
I’ll start by showing you the docker-compose.yml file in its entirety. You’ll notice that it’s just defining a network and two volumes, followed by our three containers. Much of this script was taken from here.
version: '3'
networks:
monitor-net:
driver: bridge
volumes:
prometheus_data:
grafana_data:
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention=240h'
- '--web.enable-lifecycle'
restart: unless-stopped
expose:
- 9090
ports:
- "9090:9090"
networks:
- monitor-net
blackboxexporter:
image: bitnami/blackbox-exporter:latest
container_name: blackboxexporter
volumes:
- ./blackbox/:/etc/blackbox_exporter/
- /etc/ssl/certs/:/etc/ssl/certs/:ro
command:
- '--config.file=/etc/blackbox_exporter/blackbox.yml'
restart: unless-stopped
expose:
- 9115
ports:
- "9115:9115"
networks:
- monitor-net
grafana:
image: grafana/grafana:latest
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
environment:
- GF_SECURITY_ADMIN_USER=please_change_me
- GF_SECURITY_ADMIN_PASSWORD=please_change_me
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
links:
- prometheus:prometheus
expose:
- 3000
ports:
- 3000:3000
networks:
- monitor-net
So I’ll start by discussing some general concepts of the script and then we’ll look at the details of each section. In YAML a # at the start of a line comments out that line. If it’s easier for you go ahead and comment out all of the lines starting at blackboxexporter:. Now when the script runs it will only deploy a Prometheus container. Once you’ve tested that you can uncomment either of the other two containers and test deploying gradually.
First you’ll notice that there is a single network defined and all three containers are joined to it. Next you’ll notice the port mappings. Prometheus runs on port 9090, blackbox_exporter on port 9115, and Grafana on port 3000. We’ll need to remember this later when we test services. Next we have the container names and the images they’re based on. Remember that you can have multiple containers based on the same image, but you need a unique container_name in order to identify each instance. All of the images will be downloaded from their official Docker repository the first time the script is run, with the exception of blackbox_exporter for which we will be using an image published by bitnami since no official docker image exists.
Volumes and Commands
The entries for each container under volumes are mappings in the form of local_resource:container_resource. This means an entry like ./prometheus/:/etc/prometheus/ maps a local folder name prometheus to a path of /etc/prometheus/ inside the container. Because of this I can place files inside my local prometheus folder and they will be symbolically linked inside the container at the path specified.
Prometheus
For Prometheus we need two volumes. As described above ./prometheus/:/etc/prometheus/ is a mapping and in order for us to map a local folder named prometheus, it first needs to exist. By local I mean in the same directory you will be running docker-compose from. The purpose of this directory is for us to place our prometheus.yml config file so that it will show up in the container as /etc/prometheus/prometheus.yml. So create the folder and find a sample prometheus.yml to place inside.
The second volume, prometheus_data:/prometheus maps a Docker volume named prometheus_data to a prometheus folder in the root of the container. You’ll see in the commands section that the entry --storage.tsdb.path=/prometheus' defines this as a storage location for the database. Having the database in a detachable external volume allows this data to be retained when a new image is deployed. You can run docker volume ls to see all the Docker volumes that have been created.
The remaining commands are fairly self-explanatory. A retention time of 240h is 10 days, adjust as needed.
blackbox_exporter
Blackbox_exporter also has a local folder named blackbox which needs created locally, into which we will place our blackbox.yml. Mine looks like this.
modules:
http_2xx:
prober: http
http:
preferred_ip_protocol: ip4
method: GET
tls_config:
insecure_skip_verify: false
icmp_query:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: ip4
I use blackbox_exporter for http and ping queries so those are the only modules I define. Other modules for things like DNS or TCP port testing are available. Blackbox_exporter has a preference for IPv6 over IPv4 if both are available, so I tell it to use IPv4.
Our second mapping links /etc/ssl/certs/ from the host system into the same location in the container. This is done because the container is a minimal Linux environment and this directory is one that exists in a full Linux installation but has been removed to save space inside the container. Blackbox_exporter needs access to this cert list however to verify the validity of SSL certs it encounters when scanning HTTPS sites. If you see the error x509: failed to load system roots and no roots provided in your debug logs, this mapping is probably missing or incorrect.
You’ll notice and additional :ro to this entry. This third column is optional and in this case defines the location as read-only.
Grafana
Grafana has a single Docker volume defined as the Grafana working dir. We also pass in three environment variables that Grafana will look for on first boot for its initial configuration. You specify a unique username and password to override the default of admin:admin. User registration is also disabled.
Deploying the configuration
In one directory you should now have a prometheus folder containing prometheus.yml, a blackbox folder containing blackbox.yml, and your docker-compose.yml script. To deploy the environment run docker-compose up and parse the output for any errors. Use Ctrl+C to terminate the running containers. Continue testing until all the errors are resolved.
When you’re ready for your final deployment use docker-compose up -d which will run Docker in the background.
Verification and Troubleshooting
First make sure you can access Prometheus by visiting http://<SERVER_IP>:9090
See if Prometheus has correctly read your configuration by visiting http://<SERVER_IP>:9090/targets and seeing the list of targets.
See if blackbox_exporter is correctly reading your configuration by visiting http://<SERVER_IP>:9115
From here you’ll be able to check the individual probe logs and send a debug probe to identify any issues.
The Bitnami blackbox-exporter Docker image also sends the container logs to the stdout. To view the logs run docker logs blackbox-exporter
Finally connect to http://<SERVER_IP>:3000 where you should be greeted by a login prompt. Use the username and password supplied in docker-compose. If that doesn’t work try admin:admin but that would mean your configuration wasn’t used.
Reset the environment and start over
If you find yourself completely mixed up by changes that were made it’s easy to start over with a new environment.
Run docker ps to list all running containers and their container ID. Issue a docker stop CONTAINER_ID for each running container. When all are stopped run docker ps -a to show all containers, including stopped ones. Each container should have a status of Exited. Now run docker rm CONTAINER_ID to delete each container. Run docker volume ls to list all volumes. Look specifically for the two volumes created by docker-compose, the prometheus_data and grafana_data folders. Issue docker volume rm VOLUME_NAME. The next time you run docker-compose up a new environment will be created.
Useful docker commands
See all downloaded images: docker images
See all volumes: docker volume ls
Delete a volume: docker volume rm VOLUME_NAME
Show running containers: docker ps
Show all containers: docker ps -a
Stop a running container: docker stop CONTAINER_ID
Delete a container: docker rm CONTAINER_ID
In a follow-up post I’ll describe configuring and using Prometheus.