How to install and configure Prometheus Node Exporter (node_exporter) as a service?

Once you setup the Prometheus server, you need to configure it to scrape metrics from the remote servers. The default exporter for collecting System level metrics is node_exporter.

The node_exporter collects all system level metrics and expose on /metrics endpoint.

The default port number of node_exporter is 9100. Once you setup and start the node_exporter on your system, you can start collecting Metrics from your IP:9100/metrics.

You can initiate a curl call to fetch the metrics and expose it. In the similar way, Prometheus will initiate a connection to this end point and scrape metrics from the server.

And now you have data in your Prometheus server, now you can start visualise it from Prometheus or from the Grafana dashboard which has the data source as your Prometheus server. In this blog post, I will explain how we can install and configure node_exporter on CentOS6 / CentOS7 servers along with the Initd/Systemd script for managing node_exporter process.

Prerequisites

• Basic idea on Prometheus working. You can read this article to get a quick overview on Prometheus and also this will explain the advantages of Prometheus.
• CentOS 6 or CentOS 7 server.
• Root / Sudo access.
• Connectivity should be enabled from Prometheus server to the end server on port number 9100 to scrape the metrics.

Install node_exporter

It’s not a big deal. Just download the tar, extract and run that’s it! Please do follow the steps below to install node_exporter for collecting system metrics:

Step 1: We need a user. Create a system user for the installation.

useradd -m -s /bin/bash node_exporter

Step 2: Log into node_exporter user and download the latest node_exporter package.

su - node_exporter

wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0-rc.1/node_exporter-0.16.0-rc.1.linux-amd64.tar.gz

HTTP request sent, awaiting response... 200 OK
Length: 5617869 (5.4M) [application/octet-stream]
Saving to: ‘node_exporter-0.16.0-rc.1.linux-amd64.tar.gz’

100%[===============================================================================================================================================>] 5,617,869    194KB/s   in 32s

2020-07-27 01:28:08 (173 KB/s) - ‘node_exporter-0.16.0-rc.1.linux-amd64.tar.gz’ saved [5617869/5617869]

Step 3: Extract the node_exporter package and rename the directory to ‘node_exporter’ for your convenience.

tar -xzvf node_exporter-0.16.0-rc.1.linux-amd64.tar.gz

mv node_exporter-0.16.0-rc.1.linux-amd64 node_exporter

Step 4: Start the node_exporter process.

cd node_exporter

./node_exporter

Now the node_exporter process is up and listening on its default port 9100.

Once you start the process you can check the connectivity from your Prometheus server on port number 9100

Step 5: Enable 9100 for your Prometheus server.

You have to open port 9100 for your Prometheus servers, so that the Prometheus can scrape the data from node_exporter. If you are using CentOS 7 server:

firewall-cmd --add-port=9100/tcp
firewall-cmd --reload

If you are using CentOS 6 server:

iptables -A INPUT -i eth0 -p tcp -s x.x.x.x --dport 9100 -m state --state NEW -j ACCEPT

x.x.x.x --> Is the Prometheus server IP address

Step 6: Create a service file

Now we just startup the process in foreground. We have run the process in background as a daemon. So we need a service file to manage the process.

CentOS 7 service file for node_exporter

Create a file /etc/systemd/system/node_exporter.service with following content.

[Unit]
Description=Node Exporter

[Service]
User=node_exporter
EnvironmentFile=/etc/sysconfig/node_exporter
ExecStart=/usr/sbin/node_exporter $OPTIONS

[Install]
WantedBy=multi-user.target

Create /etc/sysconfig/node_exporter and you can add your node_exporter binary’s environment variables through this file. Like, if you want to disable few metric set or enable more metric sets, you can manage through this file.

> sudo cat /etc/sysconfig/node_exporter

OPTIONS="--collector.nfs --log.level='fatal'"

CentOS 6 init file example

If you are using CentOS 6 init based server, you can use the below script for managing node_exporter as a service.

File: /etc/init.d/node_exporter
Command: sudo service node_exporter status|stop|start

#!/bin/bash
OPTIONS=`cat /etc/sysconfig/node_exporter`
RETVAL=0
PROG="node_exporter"
EXEC="/usr/sbin/node_exporter"
LOCKFILE="/var/lock/subsys/$PROG"
LOGFILE=/var/log/node_exporter.log
ErrLOGFILE=/var/log/node_exporter_error.log

# Source function library.
if [ -f /etc/rc.d/init.d/functions ]; then
  . /etc/rc.d/init.d/functions
else
  echo "/etc/rc.d/init.d/functions is not exists"
  exit 0
fi

start() {
  if [ -f $LOCKFILE ]
  then
    echo "$PROG is already running!"
  else
    echo -n "Starting $PROG: "
    nohup $EXEC $OPTIONS > $LOGFILE 2> $ErrLOGFILE &
    RETVAL=$?
    [ $RETVAL -eq 0 ] && touch $LOCKFILE && success || failure
    echo
    return $RETVAL
  fi
}

stop() {
  echo -n "Stopping $PROG: "
  killproc $EXEC
  RETVAL=$?
  [ $RETVAL -eq 0 ] && rm -r $LOCKFILE && success || failure
  echo
}

restart ()
{
  stop
  sleep 1
  start
}

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  status)
    status $PROG
    ;;
  restart)
    restart
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
esac
exit $RETVAL

Here also we are passing the node_exporter options through this file “/etc/sysconfig/node_exporter”

Step 7: Start the process

Once you created the service file, you can start the process:

sudo systemctl start node_exporter.service

You should have copy the binary to /usr/sbin/ location.

Jul 27 02:11:05 server.com systemd[1]: Started Node Exporter.
Jul 27 02:11:05 server.com systemd[1]: Starting Node Exporter...
Jul 27 02:11:05 server.com systemd[25256]: Failed at step EXEC spawning /usr/sbin/node_exporter: No such file or directory

cp /home/node_exporter/node_exporter/node_exporter /usr/sbin/

That’s it. Now the process is up. We are good with node_exporter side.

Step 8: Configure Prometheus to scrape metrics

Open the Prometheus configuration file and add your server IP/hostname with port, then restart the Prometheus process.

vi prometheus.yml

Under the ‘scrape_config’ line, add new job_name node_exporter by copy-pasting the configuration below.

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['ip:9100']

Save and exit. Then restart Prometheus. Step 8 will vary based on you Prometheus server configuration. This is the default basic configuration.

Now you can start visualising your system metrics