DIY IOT : Collecting Remote Data over the Internet

One of the primary requirements for IOT - the "Internet of Things" is to collect data from remote devices over a TCP/IP internet connection and use the same for  analysis. Since the number of devices is expected to be very large -- far larger than the the number of IPv4 IP addresses that are possible, it is expected that IPv6 addressing will be used. However the sluggish rollout of IPv6 enabled network devices has necessitated the usage of intermediate "broker" services that allow the collection of data from devices and then publish the same for subsequent analytics.

This post will describe the architecture of such a system and provide sample codes that can be used to get a basic IOT Sensor Data collection.

Since the data will be travelling over the internet, the edge device must be a TCP/IP enabled computing device to which the sensor is connected. The simplest possible computer would be a Raspberry Pi2 or Arduino Yún  or similar machine, running on some lightweight flavour Unix or Linux. So in this case the sensor will be simulated by a small shell program that, in this case, emits two pieces of data (a) a sensor ID and (b) a numeric value, of temperature or pressure or voltage, that has been recorded by the sensor. With a real sensor device, this program will collect, or read it, from the sensor through a device driver program.

Once the data is available to a shell program ( we will call it ISD_pushData.sh ) it must be pushed into a central server and the easiest way to do it is using the curl command that is available in any Unix distribution. Where does one find such servers?

Companies like Carriots and GroveStream offer services that allows one to define a "device" that uses curl to send data in JSON or XML format to a datastream where it is stored for subsequent analysis. Carriots in fact offers a free service in which one can connect up to ten devices from which to collect, store and display data. Simple tutorials are available through which one can learn "How to send a stream using curl" and "How to create triggers" that will initiate actions based on the value of data that is received.

After working through these tutorials it becomes very evident that a similar service can be created on a Apache/MySQL/PhP platform that is widely available from any webhosting service like x10hosting or hostinger. The free versions are good enough for our purpose. This post in tweaking4all shows how this can be done and is forms the basis of this post.

What we need is (a) MySQL table to store the data (b) a shell program that will send data using curl to a destination URL on the web server and (c) a PhP program, available at the destination URL, that will accept data passed as parameters and insert it into the MySQL table.

The SQL command to create the MySQL table looks like this : ( Create_IOT_SensorData.sql)

CREATE TABLE `IOT_SensorData` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY COMMENT 'unique ID',
`event` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Event Date and Time',
`sensorID` VARCHAR(30 ) NOT NULL COMMENT 'Unique ID of the sensor',
`Value` NUMERIC( 4,1 ) NOT NULL COMMENT 'Value of data recorded'
);

This PhP program (ISD_pushData.php) that sits on the web server and accepts the data

<?php
    // Connect to MySQL
    include("dbConnect.php");

    $con = mysql_connect($MyHostname, $MyUsername, $MyPassword) or die(mysql_error());
    if (!$con)
     {
       die("Could not connect: 1" . mysql_error());
     }
     mysql_select_db($MyDBname) or die(mysql_error());

    // Prepare the SQL statement
    $SQL = "INSERT INTO IOT_SensorData (sensorID ,Value) VALUES ('".$_GET["sensorID"]."', '".$_GET["Value"]."')";  
    
    // Execute SQL statement
    mysql_query($SQL,$con);
    
    // Send Mail
    $mailMSG = $SQL;
    $mailDEST = "someone@somewhere.com";
    if ($_GET["Value"] > 50){
      $mailSUB = "Warning : Value HIGH";
      } else {
      $mailSUB = "Value Normal";
      }
    mail($mailDEST,$mailSUB,$mailMSG);

?>

The PhP program  on the web server accepts the data from the following shell (ISD_pushData.sh) program, running on the remote Linux machine. Instead of reading a value from the sensor, the program is generating a random number and sending it.

p1="http://prithwis.x10.bz/IOT/ISD_pushData.php?sensorID="
sensorID="1003B"
p2="&Value="
Value=$(shuf -i 1-80 -n 1)   # random number being generated for Value
URL=$p1$sensorID$p2$Value
echo $URL
# -------------------------------------------------------
curl $URL

In fact, the PhP push program on the server is not only inserting the data into the MySQL table but is also acting as a trigger by sending two different types of mail depending on the value of the data that is received from the remote sensor.

Once the data is available in the MySQL table, it can be displayed (using ISD_viewData.php) as

In fact, one can run the shell script on any machine and the new value will appear when this pages is refreshed!

It is also possible to visualize the data graphically (using ISD_graphData.php)

All the codes used in creating this post are available on Github. The graph has been created with free tools available from JpGraph and the specific graph shown here is based on the sample shown here.

In this post we have demonstrated how data lying on remote Linux machine ( that is possibly connected to a physical sensor ) can be pushed into webserver and subsequently used for data analytics.