Monitoring with Grafana & Prometheus

In this tutorial, we are going to create a grafana dashboard for our stake pool that have 3 nodes relays and 1 core node (block producer).

One of the relay nodes, the relay3, will be used as grafana host, it will be the server where installing and deploying grafana monitoring web. So for this server, the configuration steps will be a bit different, and the rest relays and core nodes will follow the same steps.

We will monitor both the cardano-node generated metrics and the general server metrics of the Prometheus node-exporter service.

Relay3 host

Setup environment

Install Grafana

Add key to apt :

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Add repository to apt:

sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

Update the packages (now we have grafana repo and verifiable with the key):

sudo apt update

Install grafana:

sudo apt install grafana

Change port where grafana web will be deployed:

sudo vim /etc/grafana/grafana.ini

Choose the port of your choice(you can use the default port):

http_port = <port>

Start grafana service and enable to start on reboot:

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Check grafana status:

sudo systemctl status grafana-server

Others options:

sudo systemctl restart grafana-server
sudo systemctl stop grafana-server

Open the port used in grafana.ini with ufw:

sudo ufw allow proto tcp from any to any port <port>

Now, everyone can access to the login grafana page throw a browser:http://<ip-address>:<port>

1est time, use 'admin' as user & password, then, change the password.

You can give a domain to the ip-address to give it a more friendly access.

Finally, install the Clock plugin:

grafana-cli plugins install grafana-clock-panel

Install Prometheus & node exporter

sudo apt-get install -y prometheus prometheus-node-exporter prometheus-alertmanager

sudo find / -name  prometheus-node-exporter.service
vim path-to/prometheus-node-exporter.service

Choose the port fo your choice, change the line from:

ExecStart=/usr/local/bin/node_exporter

To:

ExecStart=/usr/local/bin/node_exporter --web.listen-address=:<port>

Start the services and enable them to init when reboot:

sudo systemctl start prometheus
sudo systemctl start prometheus-node-exporter
sudo systemctl enable prometheus
sudo systemctl enable prometheus-node-exporter

Once all the services are installed and working, lets edit the prometheus conf file to add the relays and core metrics data source:

sudo find / -name  prometheus.yml
sudo vim <path-to>/prometheus.yml

Add metrics data sources to Prometheus: relays & core

Before continue, be sure to complete all relays and core steps.

Delete all content and add the new conf:

# global configs
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

scrape_configs:
  - job_name: 'cardano' # To scrape data from the running cardano-node
    scrape_interval: 15s
    static_configs:
      - targets: ['<ip-address1>:12798']
        labels:
          alias: 'relay1'
          type:  'cardano-node'
      - targets: ['<ip-address2>:12798']
        labels:
          alias: 'relay2'
          type:  'cardano-node'
      - targets: ['<ip-address3>:12798']
        labels:
          alias: 'core'
          type:  'cardano-node'
      - targets: ['127.0.0.1:12798']
        labels:
          alias: 'relay3-host'
          type:  'cardano-node'

  - job_name: 'node' # To scrape data from a node exporter to monitor the linux host metrics.
    scrape_interval: 15s
    static_configs:
      - targets: ['<ip-address1>:<node-exporter-port>']
        labels:
          alias: 'relay1'
          type:  'host-system'
      - targets: ['<ip-address2>:<node-exporter-port>']
        labels:
          alias: 'relay2'
          type:  'host-system'
      - targets: ['<ip-address3>:<node-exporter-port>']
        labels:
          alias: 'core'
          type:  'host-system'
      - targets: ['127.0.0.1:<node-exporter-port>']
        labels:
          alias: 'relay3-host'
          type:  'host-system'

Note that, the relay3 is the host of grafana monitoring tool, so we just need to specify the localhost ip 127.0.0.1 for this source, because they are in the same machine.

Be sure to replace <ip-address> and <node-exporter-port> with your own values.

Restart the service to load the new conf:

sudo systemctl restart prometheus

Relay1 & relay2 & core

Setup environment

Install Prometheus & node exporter

sudo apt-get install -y prometheus-node-exporter

sudo find / -name  prometheus-node-exporter.service
vim path-to/prometheus-node-exporter.service

Choose the port fo your choice, change the line from:

ExecStart=/usr/local/bin/node_exporter

To:

ExecStart=/usr/local/bin/node_exporter --web.listen-address=:<node-exporter-port>

Start the services and enable them to init when reboot:

sudo systemctl start prometheus-node-exporter.service
sudo systemctl enable prometheus-node-exporter

Cardano config: Make cardano-node metrics accessible all over the world:

Lets find the cardano-node configuration file, then edit prometheus binding ip address:

sudo find / -name  mainnet-config.json
vim path-to/mainnet-conf.json

Change field hasPrometheus from:

"hasPrometheus": [ "127.0.0.1", 12798 ]

To:

"hasPrometheus": [ "0.0.0.0", 12798 ]

After editing configuration, you must restart the cardano-node service to get the new config values.

Firewall: restrict cardano-node port 12798 just to grafana host server

Open port to the specific ip address, in this case, just the host relay can request metrics:

sudo ufw allow from <host-ip-address3> to any port 12798
sudo ufw allow from <host-ip-address3> to any port <node-exporter-port>

Congrats! we have the cardano-node generating accesible metrics from port 12798 and the server metrics exposed in <node-exporter-port> .

Grafana monitor web

Add Prometheus source:

In grafana left menu, go to Configuration > Data Sources > Add data source and select Prometheus.

Create a new stake pool dashboard from scratch:

In grafana left menu, go to + > Create > Dashboard.

Create a graph about connected peers:

1. Select Prometheus data source

2. Select a specific metric

Create a chart about connected peers:

Example output:

3. Set chart properties

Load a dashboard from file.

In grafana panel, Go to + > Create > Import > enter json content.

Example panel version 1:

Support our work delegating to Ada Booster.