Monitoring a Synology Diskstation with Munin

I have been using Munin to monitor the health of my Raspberry Pi for while now. As I have more devices installed in my network I was looking for a way to monitor these devices as well. As Munin uses a client-server model you are required to install the Munin node on the device to be monitored. Every five minutes the Munin server polls its clients for the values and creates charts using RRDTool.

One of the devices is a network attached storage by Synology I use for backups as well as an external storage for movies, music and other stuff. The Diskstation is basically a server that runs a customized Linux (Diskstation Manager). In various forums I found information about how to set-up the Synology Diskstation as Munin node. An easier way is to use the Simple Network Management Protocol (SNMP). Just activate the SNMP service in the control panel of the Diskstation, give it a community name and you are all set:

Now tell the Munin master the community name, so it can fetch data from the Diskstation. Therefor create a file inside /etc/munin/plugin-conf.d/ folder with the name of the host (e.g. diskstation) containing two lines:


[snmp_diskstation_*]
env.community secret_community_name

The community string acts as password here. One Note: Using the community-based SNMP Version 2 (SNMPv2c) the data is sent in clear text (unencrypted) over the network. It should only be used inside LANs behind firewalls, not in WANs. If secure is an issue you should consider using SNMP Version 3 (SNMPv3).
The Munin master now knows the community name yet is unaware of what data to collect from the device. You can use the command munin-node-configure to discover those:

sudo -u munin munin-node-configure -snmp diskstation -snmpcommunity secret_community_name -shell

Munin now polls the device „diskstation“ using the SNMP-community „secret_community_name“ and shows the corresponding symlinks which have to be added to /etc/munin/plugins. You could either copy and execute them manually or just pipe them using | bash. After creating the symlinks add the following host to your /etc/munin/munin.conf to have the Diskstation appear as separate host. (Note: The IP address has to be the one of the Munin master which does the polling, not the IP address of the Diskstation).


[home.tafkas.net;diskstation]
address 127.0.0.1
use_node_name no

Once added the last step is restarting the Munin node

sudo /etc/init.d/munin-node restart

After a couple of minutes the Munin server fetches the first values, stores them in its RRD files and generates the charts:

SolarPi – A Flask powered photovoltaic monitor

After collecting some photovoltaic data using PikoPy and a some readings from the residential meter it was time to put everything together. The data is collected by a couple of scripts triggered by a cronjob every five minutes.

$ crontab -l
*/5 * * * * python /home/solarpi/kostal_piko.py
*/5 * * * * python /home/solarpi/collect_meter.py
*/15 * * * * python /home/solarpi/collect_weather.py

The results are then written into a SQLite database. I chose SQLite over PostgreSQL for its lighter footprint on the Raspberry Pi. And since we will have no concurrent writes SQLite should be capable serving as a database backend.

The sole purpose of the web application is to fetch data from the database and render the results. I decided to use Flask, a microframework for Python based on Werkzeug and Jinja 2. For the Frontend I used the Bootstrap based SB Admin 2 Theme.

Initially I used SQLAlchemy as an Object Relational Mapper where I defined my models in Python:

class PVData(SurrogatePK, Model):
__tablename__ = 'pv_data'
id = Column(db.Integer(), nullable=False, primary_key=True)
created_at = Column(db.Text(), nullable=False, default=dt.datetime.utcnow)
dc_1_u = Column(db.Integer(), nullable=True)
dc_1_i = Column(db.Float(), nullable=True)
ac_1_u = Column(db.Integer(), nullable=True)
ac_1_p = Column(db.Integer(), nullable=True)
dc_2_u = Column(db.Integer(), nullable=True)
.
.
.
current_power = Column(db.Integer(), nullable=True)
daily_energy = Column(db.Float(), nullable=True)
total_energy = Column(db.Integer(), nullable=True)

While using an ORM is very helpful for CUD operation I decided to keep my read queries using raw SQL instead.

The main view is a dashboard presenting all kinds of interesting KPIs like yielded, imported, exported Energy and how much money was received for the sold energy:

SolarPi Dashboard

Furthermore there are detailed charts available for each day at a five minute interval, or the last 7 days and 30 days, respectively:

<a href="https://i2.wp.com/blog.tafkas.net/wp-content/uploads/SolarPiDailyChart.png”>SolarPi Daily Chart

The project is live at: http://solarpi.tafkas.net

The code is on Github at https://github.com/Tafkas/solarpi

A Raspberry Pi photovoltaic monitoring solution

A friend of mine had a photovoltaic system (consisting of 14 solar panels) installed on his rooftop last year. As I was looking for another raspberry pi project I convinced him I would setup a reliable monitoring solution that will lead him to an access to the data in real-time data. The current setup comes with an inverter by the company Kostal.

Kostal Piko 5. 5 Inverter

The Kostal Piko 5.5 runs an internal web server showing statistics like current power, daily energy, total energy plus specific information for each string. However, the visual layout of the the web page and the bundled software look like they were designed in the late 90s and development has been abandoned:

Kostal Piko 5.5 Web Interface

The power inverter itself logs data at either a 15 minutes interval for about 100 days or at an hour interval for about 400 days. At the current setting the 15 minutes interval has been selected. Furthermore my friend took notes irregularly from the built-in liquid-crystal display.

The final plan is now set. A Raspberry Pi would fetch data from the inverter, store it in a database and a web application would render the results to a browser:

SolarPi SetUp

The project is live at http://solarpi.tafkas.net

Installing Oracle Java 7 SDK on the Raspberry Pi

Two days ago the official hard-float Oracle Java 7 JDK has been announced on the official Raspberry Pi blog. Prior to this there was only the OpenJDK implementation which was lacking performance.

Furterhmore the Raspberry Pi Foundation announced that future Raspbian images would ship with. Oracle Java by default.

If you want to give it a spin you can install the JDK with:

sudo apt-get update && sudo apt-get install oracle-java7-jdk

Using htop to monitor system processes on the Raspberry Pi

If you work a lot on the command line you are probably familiar with the top utility to see what process is taking the most CPU or memory. There’s a similar utility called htop that is much more powerful. On top of the information that top provides, htop additionally shows your usage per CPU, as well as a meaningful text graph of your memory and swap usage right at the top.

htop running on the RaspberryPi

As if that wasn’t enough, you can just use your up/down arrow keys to select a process, and then you can kill it with the F9 key if you like, or you can change the priority by using the F7 and F8 keys.

To install htop simply use
sudo apt-get install htop

Once installed you can start it using
htop

Getting the Raspberry Pi temperature from the command-line

If you are overclocking your Raspberry Pi or you just curious how hot this little guy gets, there are two ways to get the internal temperature. Assuming you are running Raspbian as your operating system.

Method 1:
$ /opt/vc/bin/vcgencmd measure_temp
This gives you the temperate in in degrees Celsius: temp=54.1'C

Method 2:
If you need the temperature to be more precise (e.g. storing it in an database or for further processing) use the following command.
$ cat /sys/class/thermal/thermal_zone0/temp
This will give you the temperature in Millidegrees Celsius: 54072

From my personal experience the temperature ranges from about 50°C to 55°C and I have never seen my Raspberry Pi running over 58°C.

RaspberryPi Temperature

Using SSH Public Key Authentication on the Raspberry Pi

If you log into your Raspberry Pi using ssh it will prompt you for a password. Having to do this multiple times a days this is very annoying. To ease the pain, and enhance security, you can use public key authentication instead. Therefor you create a pair of keys on your client, and store the public key on your Raspberry Pi. Then you set up an authentication by key. Afterwards the user can login into the Raspberry Pi using his private key.

Creating the keys

The first step is to create a pair of keys on the client using ssh-keygen. For the RSA key we choose 2048 bits:
Generate SSH Key Pair

This will generate a pair of keys and store them in the folder ~/.ssh/:
Content of dot SSH Folder

I have not set a passphrase. Actually a passphrase is good idea as it gives more security yet I do not want to enter the passphrase everytime I access the key.

Store public key on the Raspberry Pi

For the last time you have to log into your Raspberry Pi using password. Once logged in you copy your public key from the client into ~/.ssh/authorized_keys:

The file should look similar to:
authorized_keys

Optional: Deactivate password authentication

In order to increase the security you can disable the password authentication. Be aware that you cannot login into your Raspberry Pi over SSH using a password.

In your /etc/ssh_config set PasswordAuthentication no and restart your ssh daemon. Try to connect to the Raspberry Pi after the ssh daemon has restarted before you end your current session. In case something goes wrong you will not be able to connect again.

Gathering and Charting Temperatures using RRDTool and Highcharts

tl;dr Checkout the charts on my RaspberryPi

For quite a long time I was looking for a way to monitor and record th temperature and humidity at my apartment. What was missing was a convenient, preferably wireless solution. After receiving my RaspberryPi I started to look into that more intensively.

USB-WDE1 Receiver

The USB Weather Data Receiver USB-WDE1 wirelessly receives data from various weather sensors of ELV at 868 MHz. The receiver is connected to a USB port on the computer, so no additional power supply is required. The data is transmitted via a simple serial ASCII protocol, which is well documented by ELV. The RasberryPi running Raspbian is used for the data acquisition allowing very little power consumption while being completely flexible.

The USB interface of the USB WDE1 is realized by the USB-serial converter CP 2102 of Silicon Labs. The responsible kernel module CP2101 for accessing the device is included in any modern Linux distribution. When connecting the USB-WDE1 should appear in the system once the appropriate messages:

$ Dmesg
usb 1-3.1: Product: ELV USB WDE1 weather data receiver
usb 1-3.1: Manufacturer: Silicon Labs

The udev subsystem then also creates a corresponding device file, usually is the / dev/ttyUSB0. This device behaves as seen by a Linux application program such as a serial port and therefore can be accessed with any terminal program such as minicom. If you connect other USB-to-serial converter to the RaspberryPi, the device can also be called /dev/ttyUSB1 or similar. It is important to set the baud rate to 9600 bits/s.

A simple and universal way to output the data supplied by the receiver on the terminal provides to tool socat, which should also be part of any Linux distribution. You may have to re-install it via the package manager. Using

socat / dev/ttyUSB0, B9600 STDOUT
$1;1;;21,6;9,5;21,6;21,3;21,1;19,2;;;58;78;58;59;42;53;;;;;;;;0
$1;1;;21,6;9,5;21,6;21,3;21,1;19,3;;;58;78;58;59;42;53;;;;;;;;0
$1;1;;21,6;9,5;21,6;21,3;21,1;19,3;;;58;78;58;59;42;53;;;;;;;;0

Each line represents a complete data set consisting of 25 semicolon-separated fields. The first three fields are immutable, followed by the measured temperature (°C) of eight sensors and their humidity values​(%). The next fields show temperature (° C), humidity (%), wind speed (km / h), precipitation (rocker beats) and rain sensor (0/1) of the combination sensors. Since I do not have a combination sensor I won’t focus on those values. The last field with the fixed value of 0 indicates the end of the record.

Gathering Data With RRDtool

Now that I could receive temperature, as well as humidity, from the sensors I needed to come up with a way to store the information. For this I chose the RRDTool package to manage the data. Its a circular (RR in RRD stands for Round Robin) database that lets you store a predefined amount of data. After initial creation of the DB it is as big as it will ever get and just contains “unknown” data. This is a widely used open source package that has a bit of a steep learning curve on some of its aspects but gives you everything for functionality. It works on multiple platforms including Linux and Windows and has a large, active support community.

The ‘rrdtool create’ command is used to setup the database. Here’s the tmpdb.sh bash script I used to set it up:

Once I had setup the database I needed something to read all of the sensors every 5 minutes and place the data in the DB. For this I run a little script:

The script remains in an infinite loop while socat receives data from the sensors. After a complete line has been received, rrdtool updates database.

Graphs

Once you have some data in your temperatures.rrd database it is time to create some charts. rrdtool comes with a built-in graphics engine that can be utilized to easily create some charts. One drawback though is, that the generated charts do not look very appealing:

Another reason to avoid pre-generated graphics is that the creating process takes lots of cpu power, where with the RaspberryPi this is a very limited resource. In the beginning I created the charts every five minutes. Later I changed the schedule to every hour for the periods of month and year. Even then I was not totally happy with the result. After looking for alternatives on the Internet I stumbled upon Highcharts, a charting library written in pure JavaScript. This approach delegates the chart generating to the client side. Therefor I export data from the RRDTool to an xml file:

This results in a bunch of xml files. I use the jQuery.get method to get the contents of the xml files. In the success callback function, I parse the returned values, add the results to the series members of the options object, and create the chart:

The resulting chart:
Last weeks temperature charted with HighCharts

Setting up Dynamic DNS on the Raspberry Pi

Once you have set up your Raspberry Pi chances are that you want to access it from remote machine or host a little web site on it. The problem is that your provider usually gives you a dynamic IP, which changes every time you connect to the Internet. In Germany most (A|V)DSL provider reset your connection every 24h. The solution for this is a dynamic DNS (DDNS), which automatically updates the name server in the Domain Name System (DNS). Here is how you set it up using the provider DynDNS NoIP.

The first step is obtaining a free subdomain from the provider. Therefor you register an account at NoIP.com and check the little box Create my hostname later. Once you activated the account and logged into the website click on *Hosts/Redirects*. To create a new subdomain click on Add a Host and get creative. As host type choose DNS Host (A)

After setting up your host you can update your IP using curl:

curl -u $USERNAME:$PASSWORD "http://dynupdate.no-ip.com/nic/update?hostname=$HOSTNAME"

To update your IP everytime you boot your Raspberry Pi you can set-up a crontab: crontab -e

@reboot curl -u $USERNAME:$PASSWORD "http://dynupdate.no-ip.com/nic/update?hostname=$HOSTNAME"

After you have successfully updated your IP address you have to setup your router to open the desired ports and forward requests to the Raspberry Pi in your local network.

Update: DynDNS no longer offerering free accounts.