Monitor and troubleshoot applications with Glances and InfluxDB

Imagine you are trying to determine why one of your applications is not performing well. The application logs do not show a specific issue and by the time you checked the server, the problem is gone.

Most likely, you would want to record performance for a given period of time to see what metrics deteriorated on the machine. But you don't have a dedicated agent to take care of that.

Some tools are perfect for showing the server's health in real time, like HTOP and Glances. However, what differentiates Glances from other utilities is that it can record metrics to a remote agent while showing them in real time.

You also need to share the information you learn with team members so that you can go through these metrics as a time series. That's where a database solution like InfluxDB, which integrates well with many other programs, comes into play.

[ Related reading: How to analyze time-series data with Python and InfluxDB ]

Set up a quick observability solution

The goal of this tutorial is to set up an observability solution as quickly as possible to capture details about the problem as it happens.

You will learn how to:

• Install Glances using a Python virtual environment
• Install InfluxDB using a Podman container
• Connect InfluxDB and Glances to record historical metrics

Start by installing these tools.

Install Glances and InfluxDB

Begin by installing Glances on the machine where you want to capture the metrics.

At the time of writing this article, Glances has a minor issue that causes it to crash when exporting metrics to InfluxDB. The problem has already been fixed in the upstream project, but it has not been released yet. To make sure you can follow along with this example, install Glances 3.2.5 (or newer) using a Python virtual environment.

Create a new Python virtual environment and activate it:

$ sudo dnf install python3-devel
$ python3 -m venv ~/virtualenv/glances
$ . ~/virtualenv/glances/bin/activate
(glances) $ pip install wheel

Install Glances into the virtual environment:

(glances) $ pip install --upgrade glances[all]==3.2.5
Collecting glances
  Downloading Glances-3.3.1.1-py3-none-any.whl (708 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 708.4/708.4 kB 6.7 MB/s eta 0:00:00
Collecting psutil>=5.6.7
  Downloading psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 280.2/280.2 kB 9.7 MB/s eta 0:00:00
Collecting defusedxml
  Downloading defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting ujson>=5.4.0
  Downloading ujson-5.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (52 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.8/52.8 kB 6.7 MB/s eta 0:00:00
Installing collected packages: ujson, psutil, packaging, defusedxml, glances
Successfully installed defusedxml-0.7.1 glances-3.3.1.1 packaging-23.0 psutil-5.9.4 ujson-5.7.0
...

Then install the InfluxDB Python client, which allows Glances to export the metrics directly to InfluxDB:

(glances) $ pip install influxdb-client
Requirement already satisfied: six>=1.5 in ./virtualenv/glances/lib64/python3.11/site-packages (from python-dateutil>=2.5.3->influxdb-client) (1.16.0)
Collecting typing-extensions<5.0.0,>=4.1.1
  Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Installing collected packages: typing-extensions, reactivex, influxdb-client
Successfully installed influxdb-client-1.36.1 reactivex-4.0.4 typing-extensions-4.5.0

Now, pick a machine to store Glances metrics for later analysis and run InfluxDB using a container:

josevnz@server2:~$ mkdir -p data/influxdb
josevnz@server2:~$ podman run --name mydb --detach \
  --publish 8086:8086 \
  --volume $PWD/data/influxdb:/var/lib/influxdb2:Z \
  influxdb:latest --reporting-disabled

87cd8aae2b9af65a1c10be530e699b5f54026d8ca2b66bd0890a49a30c643853

josevnz@server2:~$ podman logs mydb
Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
2023-03-18T21:57:19.388287863Z  warn    boltdb not found at configured path, but DOCKER_INFLUXDB_INIT_MODE not specified, skipping setup wrapper    {"system": "docker", "bolt_path": ""}
...

If you run this container on Fedora or Red Hat Enterprise Linux (RHEL), use option Z when mounting a volume to ensure it receives the correct SELinux labels.

[ Get the SELinux cheat sheet. ]

The next step is initializing the database. I prefer to follow the wizard:

josevnz@server2:~$ podman exec --tty --interactive mydb /bin/bash
root@cd378ef1f5c3:/# influx setup
> Welcome to InfluxDB 2.0!
? Please type your primary username josevnz
? Please type your password *********
? Please type your password again *********
? Please type your primary organization name KodeGeek
? Please type your primary bucket name glances
? Please type your retention period in hours, or 0 for infinite 0
? Setup with these parameters?
  Username:   josevnz
  Organization:KodeGeek
  Bucket:     glances
  Retention Period:  infinite
 Yes
User    Organization    Bucket
josevnz KodeGeek    glances
root@cd378ef1f5c3:/# 

While connected to the container, create an all-access token that you will use to provide access to your database:

root@cd378ef1f5c3:/# influx auth create \
  --org KodeGeek \
  --read-authorizations \
  --write-authorizations \
  --read-buckets \
  --write-buckets \
  --read-dashboards \
  --write-dashboards \
  --read-tasks \
  --write-tasks \
  --read-telegrafs \
  --write-telegrafs \
  --read-users \
  --write-users \
  --read-variables \
  --write-variables \
  --read-secrets \
  --write-secrets \
  --read-labels \
  --write-labels \
  --read-views \
  --write-views \
  --read-documents \
  --write-documents \
  --read-notificationRules \
  --write-notificationRules \
  --read-notificationEndpoints \
  --write-notificationEndpoints \
  --read-checks \
  --write-checks \
  --read-dbrp \
  --write-dbrp \
  --read-annotations \
  --write-annotations \
  --read-sources \
  --write-sources \
  --read-scrapers \
  --write-scrapers \
  --read-notebooks \
  --write-notebooks \
  --read-remotes \
  --write-remotes \
  --read-replications \
  --write-replications
ID          Description Token                                               User Name   User ID         Permissions
0ae9b2f1ea468000            F8y7eoaPX5gMkWvpxZ-b2LOnJjMO6gdH1ba1HfQV0dXmJm6oBekA7WsPiPk-3zhOxL8Y55_aJB1Ii-kRBDsH6w==    josevnz     0ae9b2131d868000    [read:orgs/b38ef4c091e3eca2/authorizations write:orgs/b38ef4c091e3eca2/authorizations read:orgs/b38ef4c091e3eca2/buckets write:orgs/b38ef4c091e3eca2/buckets read:orgs/b38ef4c091e3eca2/dashboards write:orgs/b38ef4c091e3eca2/dashboards read:orgs/b38ef4c091e3eca2/tasks write:orgs/b38ef4c091e3eca2/tasks read:orgs/b38ef4c091e3eca2/telegrafs write:orgs/b38ef4c091e3eca2/telegrafs read:/users write:/users read:orgs/b38ef4c091e3eca2/variables write:orgs/b38ef4c091e3eca2/variables read:orgs/b38ef4c091e3eca2/secrets write:orgs/b38ef4c091e3eca2/secrets read:orgs/b38ef4c091e3eca2/labels write:orgs/b38ef4c091e3eca2/labels read:orgs/b38ef4c091e3eca2/views write:orgs/b38ef4c091e3eca2/views read:orgs/b38ef4c091e3eca2/documents write:orgs/b38ef4c091e3eca2/documents read:orgs/b38ef4c091e3eca2/notificationRules write:orgs/b38ef4c091e3eca2/notificationRules read:orgs/b38ef4c091e3eca2/notificationEndpoints write:orgs/b38ef4c091e3eca2/notificationEndpoints read:orgs/b38ef4c091e3eca2/checks write:orgs/b38ef4c091e3eca2/checks read:orgs/b38ef4c091e3eca2/dbrp write:orgs/b38ef4c091e3eca2/dbrp read:orgs/b38ef4c091e3eca2/annotations write:orgs/b38ef4c091e3eca2/annotations read:orgs/b38ef4c091e3eca2/sources write:orgs/b38ef4c091e3eca2/sources read:orgs/b38ef4c091e3eca2/scrapers write:orgs/b38ef4c091e3eca2/scrapers read:orgs/b38ef4c091e3eca2/notebooks write:orgs/b38ef4c091e3eca2/notebooks read:orgs/b38ef4c091e3eca2/remotes write:orgs/b38ef4c091e3eca2/remotes read:orgs/b38ef4c091e3eca2/replications write:orgs/b38ef4c091e3eca2/replications]

root@cd378ef1f5c3:/# exit

The important piece is the fresh new token: F8y7eoaPX5gMkWvpxZ-b2LOnJjMO6gdH1ba1HfQV0dXmJm6oBekA7WsPiPk-3zhOxL8Y55_aJB1Ii-kRBDsH6w==. Keep it safe. You will use it shortly to allow Glances to connect to InfluxDB.

[ Want to test your sysadmin skills? Take a skills assessment today. ]

Connect InfluxDB and Glances

Glances expects to have its configuration in one or more locations. For this example, I will use ~/.config/glances/glances.conf with InfluxDB settings on server1 (the machine with problems):

$ mkdir ~/.config/glances/
$ /bin/cat<<EOF>~/.config/glances/glances.conf
[global]
refresh=2
check_update=false
history_size=28800
[influxdb2]
# server2 is where InfluxDB is running
host=server2
port=8086
protocol=http
org=KodeGeek
bucket=glances
# And here you put the tocket we generated on the previous step
token=F8y7eoaPX5gMkWvpxZ-b2LOnJjMO6gdH1ba1HfQV0dXmJm6oBekA7WsPiPk-3zhOxL8Y55_aJB1Ii-kRBDsH6w==
EOF

Capture the metrics

It's time to capture those metrics. Activate your virtual environment and start Glances:

$ . ~/virtualenv/glances/bin/activate
(glances) $ glances -t 5 --export influxdb2

Run the sysbench application to simulate activity:

$ sudo dnf install -y sysbench
$ for i in $(seq 10); do sysbench --test=cpu run; done

Watch the results

Glances is excellent at showing real-time metrics, but now you can see how the information looks as a time series. Using your browser, navigate to the IP address or hostname of the machine running InfluxDB on port 8086 and log in with the user you created when setting up the database. You can use the Data Explorer view to see the metrics collected from Glances.

See the following video for an example of how it looks like on my system:

As you can see, Glances created several metrics and tags in the Glances bucket you defined in the configuration.

Wrap up

• Traditional tools like System Activity Report (sar) are useful for capturing metrics, but newer utilities have more flexibility and options. Glances is written in Python and supports a server mode, which means you can write your own tools on top of it.
• Glances and InfluxDB are a great combination to monitor and troubleshoot system issues. They are easy to install, work well together, and are easy to use, which helps you focus on solving your performance issue.
• Both Glances and InfluxDB have a vibrant community. If you run into trouble with them, you will find help quickly.
• I briefly mentioned Sysbench. You can run other benchmark tools to help you find system bottlenecks.

[ Keep your most commonly used commands handy with the Linux commands cheat sheet. ]