DIY IOT : XMPP Chatbot in Python

A key challenge in building the "Internet of Things" is to be able to connect a device to a computer over the internet and to use as simple and lightweight an infrastructure as possible. In this post we demonstrate how a public XMPP chat server can be used to transmit data and commands from one device to another using a chat client at one end and a python "bot" sitting on the other end. We will demonstrate the ability to INSERT data into an SQLite database, SELECT records from the same, play a variety of .wav files and execute any system commands on a "central" machine from any distant machine that supports an XMPP chat client.

Before starting on this exercise, we searched the web for prior activity in this area and we came across this webpage that suggests a similar approach for controlling devices over the internet, but the strategy explained here is simpler to code and implement.

We looked for a list of public XMPP chat servers and selected the Ad Astra service, perhaps because it was the first on the list! We next registered two userids, say x1@adastra.re and x2@adastra.re to be used on the two different machines using the web-registration form, because the in-band registration does not work!

We next installed the Pidgin chat client on two different Ubuntu machines and verified that text messages where being sent and received through the Ad Astra server without any problem.

Since working with two machines on a desktop is difficult, we configured our experimental setup as one Android phone acting as the distant machine and an Ubuntu 14.04 laptop acting as the central server. Commands transmitted from the Android phone using the Xabber chat app would be received on the server and acted upon.

For the server side we configured a chatting robot, or chatbot with Python using the xmppy library and using the code for the sample bot as a starting point. The sample bot program has a number of sample "handlers" that perform some simple tasks. These were modified to perform the following tasks

• PUSH - to insert a piece of data into an SQLite database
• PULL - to select a record from the SQLite database
• SOUND - to play a selection of .wav files using the aplay command available in Ubuntu

These tasks were to be executed by calling a shell program and passing the relevant parameters through the python subprocess command.

There were three other handlers

• EXESYS - to execute any system command on the central machine
• TEST - to echo the command received back to the sender as a debugging measure
• HELP - that lists all the possible commands.

Before the chatbot starts, we need to install SQLite, create a database and within it, an rdbms table. 

The chatbot ("chatIOT") is a python script that needs to be started with the userid@servername.com and the password as the parameters. It then listens for text messages and responds as appropriate to the six commands listed above.

The Xabber chat client on the Android phone is now used to send messages as follows

In the left hand image of a screenshot of the Xabber client on the Android phone, we see that the value PULLed from the database is 100.4. Then we PUSH 51.29 into the database and the subsequent PULL shows two values namely 100.4, 51.29 that were stored in the persistent database.

The second, right hand image, shows how the unix command ls for directory listing is executed and the data is returned. We note the three .wav files that could be played by sending the SOUND command along with a number that indicates which file is to be played.

How realistic is this setup in simulating a real IOT scenario ?

All that the distant device needs is a chat client. If it is an Arduino board then it is possible to create a chat client for the same as explained here and elsewhere. If the Arduino is connected to Raspberry Pi2, then life is even simpler because with python it is very easy to create a simple chat client - xmit.py

If one is not comfortable with messages being transmitted through a public chat server then one can build one's one chat server using the free and opensource OpenFire software and host it on a secure but publicly visible IP address. But even otherwise the usage of standard userid, password mechanism of a public chat server offers decent security.

What we have demonstrated in this exercise is how to build a secure data transport mechanism using publicly available components and with minimal programming. IOT developers and enthusiasts can focus on the behaviour of the edge devices without having to worry about the "plumbing" that will move the data from one machine to another.

---

The following files are available at Github :

• the main python "bot" script, chatIOT.py and the shell script kmd.sh that is called by chatIOT.py to execute the SQLite and sound commands.
• createIOTdata.sh and viewIOTdata.sh to create the SQLite tables and view the data
• three .wav files for the three sounds
• xmit.py - a lightweight xmpp chat client for sending text messages

MQTT with Mosquitto: A beginner's tutorial

Using MQTT on Mosquitto and Paho

In the current tutorial, I shall describe the usage of one of the most widely used protocols named “MQTT” using open source server/broker and clients. I have used my Windows 7 machine for running the tutorial.

MQTT

MQTT( Message Queue Telemetry Transport) is used for device data collection. As its name suggests, its main purpose is telemetry, or remote monitoring. Its goal is to collect data from many devices and transport that data to the IT infrastructure. It targets large networks of small devices that need to be monitored or controlled from the cloud. MQTT uses a hub-and-spoke architecture, where all the devices connect to a data concentrator server. The protocol works on top of TCP, which provides a simple, reliable stream of data. MQTT works in “publish subscribe” mode and is a very good means of low power transfer of data from devices to broker with reliability. The messages' payloads are just a sequence of bytes, up to 256MB, with a fixed header of two bytes to most messages. The clients can subscribe to these messages and get updated by the broker when new messages arrive. MQTT lets clients and brokers set a "Quality of Service" on messages basis from "fire and forget" to "confirmed delivery".

Please refer to <http://mqtt.org/faq> for more information on the specific protocol.

Paho

MQTT needs client implementations. The Eclipse Paho project is part of the Eclipse Foundation's M2M (Machine to Machine) mission to provide high quality implementations of M2M libraries and tools. Under the Paho banner, open source client libraries for MQTT are being developed.  MQTT C and Java libraries with Lua, Python, C++ and JavaScript are at various stages of development. In this article we'll be showing how to use the Paho Java MQTT libraries to publish and subscribe.

Mosquitto

In order to receive published messages and send them on to any clients who have subscribed, we need a broker. Mosquitto is one of such brokers which is easy to configure and run for MQTT messages. Mosquitto is open source, so you can download it and run it on your own system, on different operating systems like Windows, Mac OS X, Linux or many other platforms. The Mosquitto broker code is also being contributed to Eclipse as part of a new project.

Mosquitto's default configuration means it is set up to not use username/password authentication and it accepts all connections on port 1883. It also comes with two clients, mosquitto_pub and mosquitto_sub. mosquitto_pub client is used for publishing simple messages, while the later is for subscribing to a topic and printing the message that it received. It is also widely used for debugging applications. We shall see the usage of both these clients in the tutorial.

We shall test out the following configuration using mosquitto and the code snippets.

Downloads required

You need to download the following.

Eclipse IDE for Java developers: Source <https://eclipse.org/downloads/>, select the 32 bit or 64 bit version as compatible to your system.

Install eclipse as per the guidelines. Make sure a version of JRE is installed in your machine and eclipse should point to the same JRE folder for execution

Mosquitto: Source <http://mosquitto.org/download/> . I have used the binary installation as follows.

mosquitto-1.4.3-install-win32.exe (~200 kB) (Native build, Windows Vista and up)

At the time of installation, you will be prompted to install additional dlls (for win 32) as follows

libeay32.dll, ssleay32.dll  and  pthreadVC2.dll

You need to get the dlls from the URLs specified during download process (on clicking the links, you will be taken to the sources of the dlls directly). You need to copy the dlls is the directory where mosquito is installed. These dlls are mandatory for running mosquito.

Add the path of the folder where Mosquitto executables are copied using the System Properties-> Environment variables (on Windows 7). This will make sure that mosquito executable will run from any place. 

Paho : Source : <http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/

Download the following and extract in a folder (can be under eclipse). I used the “zip” version.

org.eclipse.paho.mqtt.java-1.0.2.zip  or org.eclipse.paho.mqtt.java-1.0.2.tar.gz   or org.eclipse.paho.mqtt.java-1.0.2.tar.xz  

After extraction, add the path of the downloaded files in your CLASSPATH through System Properties-> Environment variables (on Windows 7). Specifically, the following folder should be added to the CLASSPATH for the current exercise.

java-1.0.2\org.eclipse.paho.client.mqttv3

Running the application

On the Eclipse IDE

Open the Eclipse IDE, create a new Java project and then select Configure → Convert to Maven project.

Since the Paho MQTT code isn't in Maven Central, we need to include its repository. This needs to be done on the pom.xml file from Eclipse IDE.

Open the pom.xml file and after </version> add

<repositories>

<repository>

<id>paho-mqtt-client</id>

<name>Paho MQTT Client</name>

<url>https://repo.eclipse.org/content/repositories/paho-releases/</url>

</repository>

</repositories>

Then we need to add the dependency for the Mqtt-client code. On the pom.xml file, after </build>, add

<dependencies>

<dependency>

<groupId>org.eclipse.paho</groupId>

<artifactId>mqtt-client</artifactId>

<packaging>jar</packaging>

<version>0.4.0</version>

</dependency>

</dependencies>

From Command Prompt

Run Mosquitto.exe (type “mosquitto” from command prompt). One screen will open up

To make sure (or check) whether mosquitto  is running, you can open another command prompt window and run “netstat –an”.  You should be able to see the following screen with boxed line (showing Listening at port 1883).

 

Now we have made sure that MQTT broker is running and is listening to port 1883.

Open another command prompt and run the following command. With the options used, the messages are dumped on the screen.

mosquitto_sub -t "#" –v

Go back to Eclipse IDE and create a new class called PahoDemo. Add the following lines of code.

import org.eclipse.paho.client.mqttv3.MqttClient;

import org.eclipse.paho.client.mqttv3.MqttException;

import org.eclipse.paho.client.mqttv3.MqttMessage;

public class PahoDemo {

         MqttClient client;

        

         public PahoDemo() {}

         public static void main(String[] args) {

           new PahoDemo().doDemo();

         }

         public void doDemo() {

           try {

             client = new MqttClient("tcp://localhost:1883", "pahomqttpublish1");

             client.connect();

             MqttMessage message = new MqttMessage();

             message.setPayload("A single message from first program".getBytes());

             client.publish("pahodemo/test", message);

             client.disconnect();

           } catch (MqttException e) {

             e.printStackTrace();

           }

         }

       }

Output:

You will be able to see the message “pahodemo/test A single message from first program” on the mosquitto_sub screen.

Also if you open another command prompt and type the following.

mosquitto_pub -t "mosquittodemo/test" -m "Hello"

You will get the same message echoed at the mosquitto_sub screen as follows.

Thus, we managed to get a basic Paho MQTT client running. The messages are displayed both from the Java Application as well as the mosquitto_pub client

DIY IOT : Collecting Remote Data over the Internet

One of the primary requirements for IOT - the "Internet of Things" is to collect data from remote devices over a TCP/IP internet connection and use the same for  analysis. Since the number of devices is expected to be very large -- far larger than the the number of IPv4 IP addresses that are possible, it is expected that IPv6 addressing will be used. However the sluggish rollout of IPv6 enabled network devices has necessitated the usage of intermediate "broker" services that allow the collection of data from devices and then publish the same for subsequent analytics.

This post will describe the architecture of such a system and provide sample codes that can be used to get a basic IOT Sensor Data collection.

Since the data will be travelling over the internet, the edge device must be a TCP/IP enabled computing device to which the sensor is connected. The simplest possible computer would be a Raspberry Pi2 or Arduino Yún  or similar machine, running on some lightweight flavour Unix or Linux. So in this case the sensor will be simulated by a small shell program that, in this case, emits two pieces of data (a) a sensor ID and (b) a numeric value, of temperature or pressure or voltage, that has been recorded by the sensor. With a real sensor device, this program will collect, or read it, from the sensor through a device driver program.

Once the data is available to a shell program ( we will call it ISD_pushData.sh ) it must be pushed into a central server and the easiest way to do it is using the curl command that is available in any Unix distribution. Where does one find such servers?

Companies like Carriots and GroveStream offer services that allows one to define a "device" that uses curl to send data in JSON or XML format to a datastream where it is stored for subsequent analysis. Carriots in fact offers a free service in which one can connect up to ten devices from which to collect, store and display data. Simple tutorials are available through which one can learn "How to send a stream using curl" and "How to create triggers" that will initiate actions based on the value of data that is received.

After working through these tutorials it becomes very evident that a similar service can be created on a Apache/MySQL/PhP platform that is widely available from any webhosting service like x10hosting or hostinger. The free versions are good enough for our purpose. This post in tweaking4all shows how this can be done and is forms the basis of this post.

What we need is (a) MySQL table to store the data (b) a shell program that will send data using curl to a destination URL on the web server and (c) a PhP program, available at the destination URL, that will accept data passed as parameters and insert it into the MySQL table.

The SQL command to create the MySQL table looks like this : ( Create_IOT_SensorData.sql)

CREATE TABLE `IOT_SensorData` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY COMMENT 'unique ID',
`event` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Event Date and Time',
`sensorID` VARCHAR(30 ) NOT NULL COMMENT 'Unique ID of the sensor',
`Value` NUMERIC( 4,1 ) NOT NULL COMMENT 'Value of data recorded'
);

This PhP program (ISD_pushData.php) that sits on the web server and accepts the data

<?php
    // Connect to MySQL
    include("dbConnect.php");

    $con = mysql_connect($MyHostname, $MyUsername, $MyPassword) or die(mysql_error());
    if (!$con)
     {
       die("Could not connect: 1" . mysql_error());
     }
     mysql_select_db($MyDBname) or die(mysql_error());

    // Prepare the SQL statement
    $SQL = "INSERT INTO IOT_SensorData (sensorID ,Value) VALUES ('".$_GET["sensorID"]."', '".$_GET["Value"]."')";  
    
    // Execute SQL statement
    mysql_query($SQL,$con);
    
    // Send Mail
    $mailMSG = $SQL;
    $mailDEST = "someone@somewhere.com";
    if ($_GET["Value"] > 50){
      $mailSUB = "Warning : Value HIGH";
      } else {
      $mailSUB = "Value Normal";
      }
    mail($mailDEST,$mailSUB,$mailMSG);

?>

The PhP program  on the web server accepts the data from the following shell (ISD_pushData.sh) program, running on the remote Linux machine. Instead of reading a value from the sensor, the program is generating a random number and sending it.

p1="http://prithwis.x10.bz/IOT/ISD_pushData.php?sensorID="
sensorID="1003B"
p2="&Value="
Value=$(shuf -i 1-80 -n 1)   # random number being generated for Value
URL=$p1$sensorID$p2$Value
echo $URL
# -------------------------------------------------------
curl $URL

In fact, the PhP push program on the server is not only inserting the data into the MySQL table but is also acting as a trigger by sending two different types of mail depending on the value of the data that is received from the remote sensor.

Once the data is available in the MySQL table, it can be displayed (using ISD_viewData.php) as

In fact, one can run the shell script on any machine and the new value will appear when this pages is refreshed!

It is also possible to visualize the data graphically (using ISD_graphData.php)

All the codes used in creating this post are available on Github. The graph has been created with free tools available from JpGraph and the specific graph shown here is based on the sample shown here.

In this post we have demonstrated how data lying on remote Linux machine ( that is possibly connected to a physical sensor ) can be pushed into webserver and subsequently used for data analytics.