Sensor database on the Raspberry Pi

Anyone who wants to record sensor data over a long period of time is inevitably faced with a problem: How can the collected data be stored and presented?
InfluxDB and Grafana are actually tools designed for monitoring large servers, networks and websites.

This tutorial explains how this combination can be used to visualize sensor data at home.

The project

Suitable for: advanced

Time required: at least 2 h

What you need: Raspberry Pi (model 2 or newer recommended), sensors or other interesting data sources

The Tools

InfluxDB

InfluxDB is a Time Series Database (TSDB), a database which was developed especially for time series. Compared to conventional databases such as mySQL, TSDBs have several advantages:

Better performance and lower resource requirements
Special functions for the downsampling of measurement series
Better compression of data

InfluxDB also includes the program Telegraf, which collects measured values from many sources and forwards them to InfluxDB. In this tutorial Telegraf is used to collect data like the CPU usage of the Raspberry Pis. If this is not desired, the installation of Telegraf is not necessary.

Grafana

Grafana is a platform for the visualization of time series. Data can be presented e.g. with graphs, tables and heat maps. A large selection of plugins offers many possibilities for extension. If you want to have a look at the interface before installation, you can use the live demo on https://play.grafana.org.

Setup on the Raspberry Pi

Due to the higher write load on the SD card caused by the database, it is recommended to install the operating system on a better data carrier, for example a USB-SSD.
At the very least, an SD card designed for high write load should be used (for example, a card for Dashcams).

The installation of Raspbian will not be discussed further at this point, the tutorial is based on the version “Raspbian Buster Lite”.

Installing the packages

The packages in the official Raspbian repositories are outdated, so the repositories of the respective developers must be added:

```bash
curl -sL https://packages.grafana.com/gpg.key | sudo apt-key add -
curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -

echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
echo "deb  stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install -y grafana influxdb telegraf

sudo systemctl enable influxdb grafana-server telegraf
sudo systemctl start influxdb grafana-server telegraf
```

Configuration

By default Grafana starts a web server on port 3000, so the frontend can be reached with the browser under the URL http://:3000. The login is “admin”/”admin” and should be changed directly at the first login. After that the home dashboard is already displayed.

First Grafana must be connected to InfluxDB. For this purpose, a new data source is created by clicking on “Add Data Source”, in the selection afterwards “InfluxDB” must be selected. Most of the settings can be left at the default values at first, only the URL has to be entered: http://localhost:8086. In addition, the database ‘telegraf’ has to be entered under “InfluxDB Details”. A click on “Save & Test” saves the settings, the confirmation “Data source is working” should appear.

Dashboards

After that you can already create the first dashboard, this is done by clicking the “New Dashboard” button on the home dashboard. The first panel has already been created automatically, click “Add Query” to get to the panel menu.

Telegraf should already have collected some data about the Raspberry Pi since the installation, like CPU load, RAM usage etc. To display the CPU load, for example, the measurement series “cpu” must be selected in the “FROM” line under “select measurement”. Afterwards, the field “usage_system” or “usage_user” can be selected in the line “SELECT” under “value”.

Once the data has been selected, the buttons on the left are used to access further menus in which the graphical display can be configured.

Additional data sources in Telegraf

Telegraf has a multitude of “Input Plugins”, which enable the collection of various data. Most plugins are designed for servers and measure statistics about different server applications. A funny exception to this is the ‘minecraft’ plugin, which collects the scoreboards of players of a Minecraft server. A detailed list is available in the Telegraf source code. To enable plugins, the configuration file ‘/etc/telegraf/telegraf.conf’ must be modified. If you want to get deeper into Telegraf, please refer to the documentation.

Integration of sensor data

Data format

InfluxDB accepts data in the form of HTTP requests, so all network-compatible devices can easily send measured values to the database. Here, ‘curl’ is used as an example for a sensor:

```bash
curl -i -XPOST 'http://<IP-des-Pis>:8086/write?db=mydb' --data-binary 'sensor,ort=kueche temperatur=24.7'
```

This query stores a measured value in the measurement series ‘sensor’, with the additional marking ‘location=kitchen’ and the field ‘temperature’, with the value ‘24.7’. This measured value appears immediately in Grafana after setting up a dashboard.

ESP8266 / ESP32

In order to connect smaller sensors to the database, they must be sent to the database by a microcontroller. The easiest way to do this is with the ESP8266 and ESP32 controllers from Espressif, as these can communicate directly with the Raspberry via WLAN. There is also a suitable Arduino library, so the same measured value can be transmitted in a few lines of code:

```c++
InfluxData measurement("sensor");
measurement.addTag("ort", "kueche");
measurement.addValue("temperatur", 24.7);

influx.write(measurement);
```

A big disadvantage of these two controllers is the high current consumption due to the WLAN connection (about 80mA at 3.3V). For battery operation it is therefore absolutely necessary to use the sleep modes. Furthermore, the range is limited by the range of the WLAN network.

LoRaWAN

The best solution for power consumption and range is the relatively new wireless protocol LoRa (Long Range). A similar sensor with a LoRa transmitter has been tirelessly measuring the temperature in the author’s garden (with two mignon cells and every minute) for almost a year now and was able to successfully transmit data over several kilometers in range tests. However, LoRa requires special transmitters and receivers, the setup of which will be explained in upcoming How-To’s.

Further possibilities

The article showed how a Raspberry Pi with InfluxDB and Grafana can be used to collect data from sensors and present it in an appealing way. Such a setup has been used by the author for about two years. Here are some interesting applications, which have all been integrated into the system: