Choosing an IoT platform

Thread Starter

Ian0

Joined Aug 7, 2020
9,495
I’m currently investigating IoT platforms for remotely logging data.
Having looked at the various websites, they all seem very much like an attempt to keep out the hoi polloi. There is much talk about getting a system up and running in 5 minutes, but 5 hours later and I am still struggling through the acronyms: OTA updates? But I don’t even have an operational transconductance amplifier in circuit.
Surely sending data to the server must be trivial: serial-to-Ethernet module sending internet command followed by some form of ID, followed by variable name, and finally the new data value, so why does it seem to require the downloading of megabytes of ”libraries”?
I would have thought that these firms would need to appeal to the hardware designer and embedded programmer, rather that just the computer cognoscenti, because those are the engineers that actually have real-world-connected hardware that needs monitoring.
Does anyone know of any engineer-friendly IoT providers?
 

Ya’akov

Joined Jan 27, 2019
8,973
I think you have to decide what you mean by IoT and how many devices you intend to deploy, maintain, and manage.

IoT has different thrusts: Home Automation, Sensor Networks, Building Control, etc. Each has different protocol appropriate for scaling up. And, if you do have a fleet of thousands of nodes, OTA updates and management are actually rather important.

If you are dealing with only a few devices, don't have to manage them, and don't expect to scale up you are more free to ignore IoT specific vendors. Can you give a generalized use case description?
 

Ya’akov

Joined Jan 27, 2019
8,973
Oh, sorry—OTA is Over The Air.

As far as sending data to the server, it could be trivial but if you are running battery powered sensors or dealing with fading channels, fault tolerant protocols that can accommodate nodes not always on the air are pretty important things.
 

Thread Starter

Ian0

Joined Aug 7, 2020
9,495
It’s all hard-wired Ethernet.
I need to collect about 20 bytes every 15 minutes from a hundred or so machines at various throughout the country.

And I did work out OTA eventually, because I came across it written out in full - so after it being “wireless” it’s now “OTA”, so do we now refer to wifi as “OTA internet”?
 

Thread Starter

Ian0

Joined Aug 7, 2020
9,495
Have a look at ESP8266 and ESP32.
Also look at Onion Omega2.
I’m aware of those, and several more besides such as USR K-series, but my data is being sent around the system on CAN, so A UART-interfaced module would be perfect, as I have Spare UARTs on the processor.
But the information that is tricky to get is how to get the module to communicate with the server.
 

BobaMosfet

Joined Jul 1, 2009
2,105
I’m currently investigating IoT platforms for remotely logging data.
Having looked at the various websites, they all seem very much like an attempt to keep out the hoi polloi. There is much talk about getting a system up and running in 5 minutes, but 5 hours later and I am still struggling through the acronyms: OTA updates? But I don’t even have an operational transconductance amplifier in circuit.
Surely sending data to the server must be trivial: serial-to-Ethernet module sending internet command followed by some form of ID, followed by variable name, and finally the new data value, so why does it seem to require the downloading of megabytes of ”libraries”?
I would have thought that these firms would need to appeal to the hardware designer and embedded programmer, rather that just the computer cognoscenti, because those are the engineers that actually have real-world-connected hardware that needs monitoring.
Does anyone know of any engineer-friendly IoT providers?
I think it has to be understood up front that engineering disciplines are not easy. Anything having to do with electronics and embedded is 'sold' as easy by outfits making a quick buck and tying into that desire- but at the end of the day, there are real hurdles that must be climbed in order to actually get to where you can perform.

Electronics is not plug & play.
 

Ya’akov

Joined Jan 27, 2019
8,973
It’s all hard-wired Ethernet.
I need to collect about 20 bytes every 15 minutes from a hundred or so machines at various throughout the country.

And I did work out OTA eventually, because I came across it written out in full - so after it being “wireless” it’s now “OTA”, so do we now refer to wifi as “OTA internet”?
I think MQTT is a really well suited building block for such a network. As far as the hardware platform, I am going to suggest something a bit unusual—the Milk V Duo.

This is a very new RISC-V board, cheap (depending on your definition at $5-$9USD), with an on-board Ethernet PHY. It will run Linux(!) or an RTOS. Super low power as well.

Read about it and see if it appeals. It might not, but having a real OS can be a good thing to fall back on.

Failing that, I would probably look Espressif offerings because they are cheap, pervasive, and very well supported both by the community and Espressif.

Well, “wireless” and “OTA” aren’t synonymous. The key point about OTA updating is not the radio, it’s the unattended capability. Technically, OTA updates could be done by wire.
 

Ya’akov

Joined Jan 27, 2019
8,973
I think MQTT is a really well suited building block for such a network. As far as the hardware platform, I am going to suggest something a bit unusual—the Milk V Duo.

This is a very new RISC-V board, cheap (depending on your definition at $5-$9USD), with an on-board Ethernet PHY. It will run Linux(!) or an RTOS. Super low power as well.

Read about it and see if it appeals. It might not, but having a real OS can be a good thing to fall back on.

Failing that, I would probably look Espressif offerings because they are cheap, pervasive, and very well supported both by the community and Espressif.

Well, “wireless” and “OTA” aren’t synonymous. The key point about OTA updating is not the radio, it’s the unattended capability. Technically, OTA updates could be done by wire.
Looking back at this reply I realized I misread something far back. I apologize. I am quite sick and my brain is really not operating properly. I am going to set retreat from online interactions, possibly for and extended period as this COVID does not seem to be letting go.

Sorry for sowing confusion—I am going to go lie down.
 

Thread Starter

Ian0

Joined Aug 7, 2020
9,495
Looking back at this reply I realized I misread something far back. I apologize. I am quite sick and my brain is really not operating properly. I am going to set retreat from online interactions, possibly for and extended period as this COVID does not seem to be letting go.

Sorry for sowing confusion—I am going to go lie down.
I wish you a speedy recovery.
For appropriately themed light entertainment, have a look at the UK government’s COVID enquiry. It’s setting the bar very high for future political sit-com writers.
 

nsaspook

Joined Aug 27, 2009
12,776
I’m aware of those, and several more besides such as USR K-series, but my data is being sent around the system on CAN, so A UART-interfaced module would be perfect, as I have Spare UARTs on the processor.
But the information that is tricky to get is how to get the module to communicate with the server.
Do you want a wired or wireless connection to the network? For wired I've used the waveshare module in the past for Ethernet connectivity of a controller board. There are much cheaper integrated options.
https://www.waveshare.com/uart-to-eth.htm
https://forum.allaboutcircuits.com/...nd-sensor-node-for-canbus.189388/post-1790226

Sensor only boards and host board configured as a remote sensor CANBUS network to the host board. Host board has a TTL to Ethernet module for data transmission to a remote Linux system running a Java client to connect and download via the ETH module TCP server for formatting and sending updates to the MYSQL central sensor database.
 

Thread Starter

Ian0

Joined Aug 7, 2020
9,495
It’s wired Ethernet.
The UART-interfaced module would certainly be my favoured option for several reasons:
1. I have an abundance of UARTs on the control board. It’s the same board that outputs the CAN, so all the data is available. I just need to work out how to use the module to send MQTT. I presume that I just set the TCP options.
2. I want to keep all the software development on the main processor. I don’t particularly want to write software on a different platform for an interface Module.
3. I don’t particularly want to have to provide MAC addresses.
 

nsaspook

Joined Aug 27, 2009
12,776
I've haven't used MQTT in s sensor project but it looks pretty good as I have the software broker and clients install on a Linux server for testing.
https://developer.ibm.com/articles/iot-mqtt-why-good-for-iot/
1698948030718.png
1698948924436.png

So you would need a client on the sensor controller side or somewhere that collects data from the CANBUS network. My home CAN network has a Linux USB -> CAN node that can publish any received data to a MQTT server/broker from a C program
running on a Linux desktop PC. So if I need data from that using MQTT, I won't need a separate Ethernet module on a controller for that data. My MQTT client will be the Linux USB -> CAN node using the mosquitto client API in the C program.

For the PIC family there are few canned MQTT code libraries.
https://ww1.microchip.com/downloads/en/DeviceDoc/MPLAB® Code Configurator MQTT Library Release Notes v2.0.0.pdf
https://microchip-mplab-harmony.github.io/wireless/system/mqtt/docs/usage.html
 
Last edited:

Thread Starter

Ian0

Joined Aug 7, 2020
9,495
I like the IBM article.
The CANbus is less of a problem than you think. I want to send MQTT from the same board that sends the CAN data.
The ENC28J60 link also looks interesting.
 

nsaspook

Joined Aug 27, 2009
12,776
I hacked a simple MQTT interface for my FM80 solar monitor data in C. The Linux CANBUS node (USB connection) receives the 'sensor' data from the controller. The Linux server could be a RPi or any capable SoC running Linux.
1699034979155.png
That data is sent to a broker using this C MQTT client: https://github.com/LiamBindle/MQTT-C/tree/master
It's a very nice client example.

MQTT-C is an MQTT v3.1.1 client written in C. MQTT is a lightweight publisher-subscriber-based messaging protocol that is commonly used in IoT and networking applications where high-latency and low data-rate links are expected. The purpose of MQTT-C is to provide a portable MQTT client, written in C, for embedded systems and PC's alike. MQTT-C does this by providing a transparent Platform Abstraction Layer (PAL) which makes porting to new platforms easy. MQTT-C is completely thread-safe but can also run perfectly fine on single-threaded systems making MQTT-C well-suited for embedded systems and microcontrollers. Finally, MQTT-C is small; there are only two source files totalling less than 2000 lines.
This short interface lib was written for the Linux canbus node to use.
C:
/*
* MQTT routines for mateQ84 solar energy monitor
*/
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <pthread.h>
#include "templates/posix_sockets.h"
#include "mqtt_pub.h"

const char* addr;
const char* port;
const char* topic;
int sockfd;

struct mqtt_client client;
uint8_t sendbuf[2048]; /* sendbuf should be large enough to hold multiple whole mqtt messages */
uint8_t recvbuf[1024]; /* recvbuf should be large enough any whole mqtt message expected to be received */
const char* client_id;
uint8_t connect_flags;
pthread_t client_daemon;

/*
* init the socket connection to the broker server and start client daemon
*/
int mqtt_socket(void)
{
    //    addr = "hp8.sma2.rain.com"; // internal house server/broker

    addr = "test.mosquitto.org"; // cloud testing server
    port = "1883";
    topic = "mateq84";

    /* open the non-blocking TCP socket (connecting to the broker) */
    sockfd = open_nb_socket_mqtt(addr, port);

    if (sockfd == -1) {
        return sockfd;
    }

    /* setup a client */
    mqtt_init(&client, sockfd, sendbuf, sizeof(sendbuf), recvbuf, sizeof(recvbuf), publish_callback);
    /* Create an anonymous session */
    client_id = NULL;
    /* Ensure we have a clean session */
    connect_flags = MQTT_CONNECT_CLEAN_SESSION;
    /* Send connection request to the broker. */
    mqtt_connect(&client, client_id, NULL, NULL, 0, NULL, NULL, connect_flags, 400);

    /* check that we don't have any errors */
    if (client.error != MQTT_OK) {
        return client.error;
    }

    /* start a thread to refresh the client (handle egress and ingree client traffic) */
    if (pthread_create(&client_daemon, NULL, client_refresher, &client)) {
        return 1;
    }

    return 0;

}

/*
* send client data to the broker server
*/
int mqtt_check(uint8_t * application_message)
{
    if (strlen(application_message) < 3) {
        return -1;
    }
    /* publish the logging data */
    mqtt_publish(&client, topic, application_message, strlen(application_message) - 2, MQTT_PUBLISH_QOS_0);

    /* check for errors */
    if (client.error != MQTT_OK) {
        return client.error;
    }
    return 0;
}

/*
* cleanup before exit
*/
void mqtt_exit(void)
{
    /* exit */
    exit_example(EXIT_SUCCESS, sockfd, &client_daemon);
}

void exit_example(int status, int sockfd, pthread_t *client_daemon)
{
    if (sockfd != -1) close(sockfd);
    if (client_daemon != NULL) pthread_cancel(*client_daemon);
    exit(status);
}

void publish_callback(void** unused, struct mqtt_response_publish *published)
{
    /* not used in this example */
}

void* client_refresher(void* client)
{
    while (1) {
        mqtt_sync((struct mqtt_client*) client);
        usleep(100000U);
    }
    return NULL;
}
C:
int main(int argc, char *argv[])
{
// init code removed
    mqtt_socket(); // setup the mqtt network socket
// app code removed

// a parser for various CANBUS messages,
static void print_frame(canid_t id, const uint8_t *data, int dlc, int inc_data)
{
    int i;

    if (print_hex) {
        printf("%04x: ", id);
    }
    if (id & CAN_RTR_FLAG) {
        printf("remote request");
    } else {
        if (print_hex) {
            printf("[%d]", dlc);
        }
        for (i = 0; i < dlc; i++) {
            if (print_hex) {
                printf(" %02x", (uint8_t) (data[i] + inc_data));
            }
            if (id == EMON_ER || id == EMON_CO || id == EMON_DA) {
                full_buffer[i] = (uint8_t) (data[i] + inc_data);
            }
            if (id == EMON_SL) {
                data_buffer[i] = (uint8_t) (data[i] + inc_data);
            }
            if (id == EMON_SU) {
                data_buffer[i + CAN_MSG_LEN] = (uint8_t) (data[i] + inc_data);
            }
        }

        if (id == ) {
            fprintf(stdout, "log %s", data_buffer);
            mqtt_check(data_buffer); // <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< MQTT send
        }
        if (id == EMON_ER) {
            fprintf(stderr, "%s", full_buffer);
        }
        if (id == EMON_DA) {
            fprintf(stderr, "BLOB \r");
        }
        if (id == EMON_CO) {
            token = strtok(full_buffer, ",");
            if (token != NULL) {
                fprintf(stderr, "%s ", token);
                token = strtok(NULL, ",");
                fprintf(stderr, " relay outputs: %s\r\n", token);
            }
        }

    }
    if (print_hex) {
        printf("\n");
    }
}

}
When the canbus id is EMON_SU, that triggers a function to print to the screen and to send that same data to the remote broker server remotely on test.mosquitto.org.
1699036030671.png
This CSV data is normally used to chart daily energy factors on the solar energy system. Published on Topic "matq84" using cloud broker test.mosquitto.org

1699036400904.png
Green line is net energy received or used per sample period of 10 seconds over a week.
 
Last edited:

nsaspook

Joined Aug 27, 2009
12,776
Now that the connections code is working, it's time to convert the CSV data string into something useful for IoT like JSON encoding so we can use standard IoT MQTT apps : https://www.geeksforgeeks.org/cjson-json-file-write-read-modify-in-c/

Parse the string into it separate variables and encode to standard JSON format.
C:
#include <cjson/cJSON.h>
// code fragment

        if (id == EMON_SU) {
            fprintf(stdout, "log %s", data_buffer);
            token = strtok(data_buffer, ",");
            if (token != NULL) {
                /*
                 * parse the string for varible values
                 */
                token = strtok(NULL, ",");
                token = strtok(NULL, ",");
                token = strtok(NULL, ",");
                token = strtok(NULL, ",");
                token = strtok(NULL, ",");
                token = strtok(NULL, ",");
                /*
                 * convert this token into a double variable for the JSON data
                 */
                solar = atof(token);
                fprintf(stderr, " log variable: %s ", token);
                token = strtok(NULL, ",");
                load = atof(token);
                fprintf(stderr, " %s ", token);
                token = strtok(NULL, ",");
                token = strtok(NULL, ",");
                energy = atof(token);

                fprintf(stderr, " %s\r\n", token);
            }

            json = cJSON_CreateObject();
            cJSON_AddStringToObject(json, "name", "mateq84");
            cJSON_AddNumberToObject(json, "energy", energy);
            cJSON_AddNumberToObject(json, "load", load);
            cJSON_AddNumberToObject(json, "solar", solar);
            cJSON_AddStringToObject(json, "system", "FM80 solar monitor");
            // convert the cJSON object to a JSON string 
            char *json_str = cJSON_Print(json);

            mqtt_check(json_str);

            cJSON_free(json_str);
            cJSON_Delete(json);
        }
1699047713794.png
So now we have JSON formatted data as the output from the broker to a subscriber.

1699048459771.png1699048477766.png1699048497032.png
Graphing the 'energy and 'load' JSON variables on the phone using a IoT MQTT app.
 

nsaspook

Joined Aug 27, 2009
12,776
Thanks for a lot of useful stuff.
Having put some thought into how to wire it all up, a DIN-rail mount module fits the way we like to build things.
So I am looking at
https://www.waveshare.com/wiki/RS232/485/422_TO_POE_ETH_(B)
and
https://www.pusr.com/products/1-serial-port-etherne-device-servers-usr-n510.html
which have MQTT clients built in.
That waveshare module is pretty nice. I didn't look too hard but didn't see much about security protocols (like SSL/TLS) or setup on that device.
https://files.waveshare.com/upload/a/a6/EN-RS485-TO-ETH-B-MQTT-and-json-user-manual2.pdf

This device has the MQTT gateway with SSL/TLS security. Your usr-510 link above doesn't say it has MQTT.
https://www.pusr.com/products/1-rs485-serial-port-etherne-device-servers-usr-n510.html

For my MQTT from CANBUS data project in the Linux server (that has pretty good network security) I've completed most of the data export client C code for the broker so I can download or plot the solar sensor data from just about any online device with a MQTT app.
1699118498007.png1699119605326.png
MQTT is pretty simple and cool. I'll have to convert a few other projects now that I have a workable framework.
 
Last edited:
Top