Hi,
I am trying to develop an IoT enabled device which reads sensor data and sends it to a web server. I am using Wemos D1 mini, so that it can connect to the internet using wifi and send data to a web server. So far, I have developed the hardware part and getting readings from the sensors. Now I need to figure out a way to send this data to a database and then process it there later and access the data on website. Can you please suggest some platforms to do that or any other methods? Is it possible to setup a local webserver and connect it for testing purpose. I don't have much knowledge in php or web development. Please help. Thank you.

 

you would at least need to know something on the web server side
sending something to a server is pretty much a HTTP POST (or GET) request
https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST

if you know some java, you could use servlets to receive it
https://happycoding.io/tutorials/java-server/post

then PHP perhaps could look like this
https://www.w3schools.com/php/php_superglobals_post.asp
https://stackoverflow.com/questions/34846887/configure-server-php-to-handle-post-requests

 

Hi,
I am trying to develop an IoT enabled device which reads sensor data and sends it to a web server. I am using Wemos D1 mini, so that it can connect to the internet using wifi and send data to a web server. So far, I have developed the hardware part and getting readings from the sensors. Now I need to figure out a way to send this data to a database and then process it there later and access the data on website. Can you please suggest some platforms to do that or any other methods? Is it possible to setup a local webserver and connect it for testing purpose. I don't have much knowledge in php or web development. Please help. Thank you.

I am much more familiar with web sites on big iron, but hopefully you will find this info useful.

Much depends on where your web server will reduce. The hardware and operating solution will dictate to some degree what your options are. You will need to become familiar with the available toold to define the web server and network to which it connects.

You will need to learn HTML and (/or) a programming language. PHP or Java come to mind.

Then, you will have to pick a database and database. How to install the software, create and maintain the actual database and interface it with the chosen language.

Not impossible but not simple.

 

Hi,
I am trying to develop an IoT enabled device which reads sensor data and sends it to a web server. I am using Wemos D1 mini, so that it can connect to the internet using wifi and send data to a web server. So far, I have developed the hardware part and getting readings from the sensors. Now I need to figure out a way to send this data to a database and then process it there later and access the data on website. Can you please suggest some platforms to do that or any other methods? Is it possible to setup a local webserver and connect it for testing purpose. I don't have much knowledge in php or web development. Please help. Thank you.

You can set up your own lampp webserver (linux, MariaDB (MySQL), php, etc) all in a neat stack that pretty much sets itself up. As long as it's local and you keep it behind a firewall, you can concentrate on your project, rather than protecting the box from hackers.

 

I am going to suggest ditching the web server idea and going with MQTT which is purpose built for sensor networks, much lighter weight all the way around, and has supporting libraries in the Arduino environment.

Second, I would suggest using Node-RED which happily runs on a Raspberry PI to collect the data and dispatch it wherever you want. Node-RED uses a visual programming paradigm which makes a lot of things very easy. If you need it, Javascript is available (which it is written in) but there are so many modules you probably won't.

It also allows the creation of user dashboard applications, which may be useful.

 

I should add that any database running on Linux will work happily alongside Node-RED but exactly what you want to do with the data will determine the sensible option. If you are collecting time series data, and need to present it, there are specific tools often used with Node-RED (since it has IoT roots) ready to do that.

 

MQTT is good, but it tends to need a library as it is a protocol on its own.
HTTP (i.e. web) is ok, and for things like with esp8266 etc it tends to be 'easier'.
The protocols tend to be: first connect, then send data, and disconnect. The headers is plain text, the body depends on your content type and content length declaration.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST

It isn't very difficult really, conventionally it used to be CGI and a (e.g. apache) web server. But these days it is easier to do so with PHP (with apache server) or java (e.g. tomcat or jetty server). I think there are lots more options for a web server node js, python, go, ruby etc inn addition to PHP and java.

if one prefers something 'ready made', one could try wordpress (this is based on php)
https://wordpress.org/
and there are actually a lot more of them
https://en.wikipedia.org/wiki/List_of_content_management_systems
but these normally are for 'upload as text' data
there are normally quite a bit of setups and configuration to do as well, but those are complete apps on their own.

 

MQTT is good, but it tends to need a library as it is a protocol on its own.
HTTP (i.e. web) is ok, and for things like with esp8266 etc it tends to be 'easier'.
The protocols tend to be: first connect, then send data, and disconnect. The headers is plain text, the body depends on your content type and content length declaration.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST

It isn't very difficult really, conventionally it used to be CGI and a (e.g. apache) web server. But these days it is easier to do so with PHP (with apache server) or java (e.g. tomcat or jetty server). I think there are lots more options for a web server node js, python, go, ruby etc inn addition to PHP and java.

if one prefers something 'ready made', one could try wordpress (this is based on php)
https://wordpress.org/
and there are actually a lot more of them
https://en.wikipedia.org/wiki/List_of_content_management_systems
but these normally are for 'upload as text' data
there are normally quite a bit of setups and configuration to do as well, but those are complete apps on their own.

You are mistaken about MQTT. First of all, the libraries are readily available and actively maintained, It is the standard for sensor networks and the ESP8266 and other Esperssif MCUs are among the top, if not the top targets.MQTT is dead simple, I would say simpler than HTTP because the libraries are written to do what is needed in this case.

One of the libraries for the ESPx modules and MQTT​

MQTT's publish/subscribe model with store and forward capability would have to be reproduced which is a waste of time since it already exists. There is no reason to reinvent the wheel. Even if you don't use Node-RED, which I do strongly suggest, Mosquitto, the open source MQTT broker is much lighter weight than Apache, and Java? Why?

If I was inclined to write something from scratch it would be with modern tools and techniques. Javascript and Node.js, for example, rather than a megaton of infrastructure that brings no benefits. I wonder, too, what benefit you are expecting Wordpress to provide, or any CMS, given the description of the problem to be solved.

I think you probably haven't done something like this recently. The suggestions you are making are either overkill and roads to pain, or, unless i am missing something, not relevant.

Maybe I am missing something. Could you explain why your suggestions would be better than MQTT (or even Nginx if you must have an httpd)? Your stated reason has no basis but even if MQTT was ruled, you've chosen very odd parts to build this from.

 

I am going to suggest ditching the web server idea and going with MQTT which is purpose built for sensor networks, much lighter weight all the way around, and has supporting libraries in the Arduino environment.

Second, I would suggest using Node-RED which happily runs on a Raspberry PI to collect the data and dispatch it wherever you want. Node-RED uses a visual programming paradigm which makes a lot of things very easy. If you need it, Javascript is available (which it is written in) but there are so many modules you probably won't.

It also allows the creation of user dashboard applications, which may be useful.

Web monitor/control. Running a small aircon from the solar energy bank.

I've done HTTP based sensor networking in the past. It still works fine today after more than a decade but reconfiguration is a PITA.
https://forum.allaboutcircuits.com/...ic-controlled-battery-array.32879/post-354766

I completely agree with you that there must be a better way.

I'm currently prototyping some stuff using one of these as the standard for local service using the Linux Mosquitto Server.
https://www.microchip.com/en-us/development-tool/EV12F11A
https://github.com/MicrochipTech/PIC32MZW1_Curiosity_OOB

 

View attachment 271474
Web monitor/control. Running a small aircon from the solar energy bank.

I've done HTTP based sensor networking in the past. It still works fine today after more than a decade but reconfiguration is a PITA.
https://forum.allaboutcircuits.com/...ic-controlled-battery-array.32879/post-354766

I completely agree with you that there must be a better way.

I'm currently prototyping some stuff using one of these as the standard for local service using the Linux Mosquitto Server.
https://www.microchip.com/en-us/development-tool/EV12F11A
https://github.com/MicrochipTech/PIC32MZW1_Curiosity_OOB

I wrote quite a few lightweight http-based systems. But I had to replicate so much of the functionality already in MQTT, and I had to maintain my own stack, or else not update, which firewall systems works well enough.

I used perl because that was my "native" language. But I was able to get a relatively lean Apache + mod_perl config, or else use CGI. Perl had a nice event loop module that was helpful too. Sometime I just used a perl-based http.

I an nowhere near as fluent in Javascript as perl, but Node-RED is really great stuff and the newer web technologies are very nice. I am currently looking at PWAs for the front end. It's getting more and more likely that will be the future if Apple will step up. They started it and now the App Store is an impediment—but they started letting PWAs in the App Store, so...

 

I wrote quite a few lightweight http-based systems. But I had to replicate so much of the functionality already in MQTT, and I had to maintain my own stack, or else not update, which firewall systems works well enough.

I used perl because that was my "native" language. But I was able to get a relatively lean Apache + mod_perl config, or else use CGI. Perl had a nice event loop module that was helpful too. Sometime I just used a perl-based http.

I an nowhere near as fluent in JavaScript as perl, but Node-RED is really great stuff and the newer web technologies are very nice. I am currently looking at PWAs for the front end. It's getting more and more likely that will be the future if Apple will step up. They started it and now the App Store is an impediment—but they started letting PWAs in the App Store, so...

Most of the interactive graphics stuff was in JavaScript but the bulk was in C/XML.
https://forum.allaboutcircuits.com/threads/javascript-libc-love.157424/post-1363873

 

agreed those CMS stuff are off tracked, HTTP is still relevant as it is kind of ubiquitous. NodeRed is good stuff
https://nodered.org/
NodeRed apparently has HTTP endpoints as well
https://cookbook.nodered.org/
and it probably makes things much easier than java / etc. i.e. that is already there in NodeRed
and agreed MQTT possibly solved some problems which otherwise additional developments needs to be done for those.
for a 'simple' HTTP based sensor app, NodeRed is possibly a best fit here, it can interface MQTT as well.

 

I wrote quite a few lightweight http-based systems. But I had to replicate so much of the functionality already in MQTT, and I had to maintain my own stack, or else not update, which firewall systems works well enough.

I used perl because that was my "native" language. But I was able to get a relatively lean Apache + mod_perl config, or else use CGI. Perl had a nice event loop module that was helpful too. Sometime I just used a perl-based http.

I an nowhere near as fluent in Javascript as perl, but Node-RED is really great stuff and the newer web technologies are very nice. I am currently looking at PWAs for the front end. It's getting more and more likely that will be the future if Apple will step up. They started it and now the App Store is an impediment—but they started letting PWAs in the App Store, so...

Thank you guys for your reply. I am planning to do processing and calculations of the data in the website, because it takes a lot of time to take relevant number of samples and process it by the esp module. In addition to that I need to make a dashboard, where the user can control some devices using this esp module and display the caluclated values there. There will be hundreds of such devices sending realtime data to the server. I was thinking of using HTTP. I made a free webdomain for testing purpose using infinityfreeapp.net. I will look into MQTT and NodeRed, but is it appropriate for such an application with so many devices?

 

Thank you guys for your reply. I am planning to do processing and calculations of the data in the website, because it takes a lot of time to take relevant number of samples and process it by the esp module. In addition to that I need to make a dashboard, where the user can control some devices using this esp module and display the caluclated values there. There will be hundreds of such devices sending realtime data to the server. I was thinking of using HTTP. I made a free webdomain for testing purpose using infinityfreeapp.net. I will look into MQTT and NodeRed, but is it appropriate for such an application with so many devices?

There isn’t any reason that MQTT with Mosquitto and Node-RED couldn’t scale. MQTT is designed for sensor networks and has very little overhead with guaranteed delivery. HTTP has a lot of overhead that serves no purpose in this case and doesn’t have the ability to send data back to the sensors for command and control. You will have to duplicate a lot of MQTT‘s functionality if you use HTTP, and you can’t possible do it as well as what you can get readymade.

You can cook the data on the server, or use the ESPs as a distributed computational network and send the raw plus cooked data for the normal use of the data.

Are you depending on battery power?
What is the update interval?
What is the expected size of the sensor network?
Do you have the WiFi capacity to connect hundreds of nodes?
How widely will the network be distributed geographically?
How much reliability do you need?
Will the network grow from hundreds to thousands?
How good are your programming skills?
Is this a commercial venture?
How much security do you need?

 

using NodeRed (a fully developed app in Node.js / javascript / typescript) vs Java, Python etc is more a matter of preferences and familiarity with the languages / tools etc. MQTT is a message server, as simply defined as it gets, NodeRed has interfaces for MQTT, so do Java, Python etc using libraries.

NodeRed is an app, literally. vs say NodeJs which is a javascript server ! So if one prefers to go "bare metal" (i.e. develop the app from scratch) one could work in say NodeJs (javascript), Tomcat (java), python etc.

 

Node-RED is written using node.js and offers full access to it if you want. You can either use a function node which allows for Javascript code to handle and send messages, or you a custom node which is a node.js module you can include in a flow.

It makes absolutely no sense to reinvent the wheel repeatedly given that Node-RED can create user dashboards (a noted requirement) with displays and controls, and can integrate very well with Grafana which has excellent interactive graphing.

So MQTT is just heat and serve for the application and much better than anything a single person could specify and write; and Node-RED has code from hundreds of developers ready to use, has dashboards, and a lot of support.

It makes no sense whatever given the current requirements to write this from scratch, it’s not a matter of preference but of best practice. Also, I have no idea why you‘d mention that MQTT (actually Mosquitto) is a message broker since that was already said, it’s as if you are correcting something that nobody claimed.

It’s also good to point out that MQTT is a protocol that is an OASIS standard. It is used all over industry and particularly for IoT. The protocol is implemented in Mosquitto which is probably the most popular but not the only broker.

There is no reason to muddy the waters for a neophyte developer by mentioning random languages and options that don’t advance his project. If he mentions a requirement for which MQTT and Node-RED are not the best practice option for a single developer without an existing codebase (e.g.: must use Apache) then there is a reason to offer alternatives but rattling them off repeatedly is not helpful.

 

There isn’t any reason that MQTT with Mosquitto and Node-RED couldn’t scale. MQTT is designed for sensor networks and has very little overhead with guaranteed delivery. HTTP has a lot of overhead that serves no purpose in this case and doesn’t have the ability to send data back to the sensors for command and control. You will have to duplicate a lot of MQTT‘s functionality if you use HTTP, and you can’t possible do it as well as what you can get readymade.

You can cook the data on the server, or use the ESPs as a distributed computational network and send the raw plus cooked data for the normal use of the data.

Are you depending on battery power?
What is the update interval?
What is the expected size of the sensor network?
Do you have the WiFi capacity to connect hundreds of nodes?
How widely will the network be distributed geographically?
How much reliability do you need?
Will the network grow from hundreds to thousands?
How good are your programming skills?
Is this a commercial venture?
How much security do you need?

Not battery powered. I am planning to send the adc value directly to the server. The expected size will be 30 to 50 devices which needs to be connected to one wifi network. So now I am thinking of connecting all of these to a raspberry pi and collect all the data and store it to a database. Yes the project is a commercial one and I am good at C and Python. I have zero knowledge in web development or php or html. For testing I have only one device and I have found an article online with some php codes which store the data sent from esp device directly to an sql database. I am testing it.

 

this is a story about NodeRed, told by the developers themselves
https://nodered.org/about/

Node-RED started life in early 2013 as a side-project by Nick O’Leary and Dave Conway-Jones of IBM’s Emerging Technology Services group.

What began as a proof-of-concept for visualising and manipulating mappings between MQTT topics, quickly became a much more general tool that could be easily extended in any direction.

ESP32 apparently has MQTT client libraries
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/mqtt.html

So in a way this is a road that has been travelled, based on that NodeRed story, and that it is where NodeRed become what it is today.
Today NodeRed has a lot more public contributed Nodes, far more than a 'basic' system. It is its own ecosystem.

 

Not battery powered. I am planning to send the adc value directly to the server. The expected size will be 30 to 50 devices which needs to be connected to one wifi network. So now I am thinking of connecting all of these to a raspberry pi and collect all the data and store it to a database. Yes the project is a commercial one and I am good at C and Python. I have zero knowledge in web development or php or html. For testing I have only one device and I have found an article online with some php codes which store the data sent from esp device directly to an sql database. I am testing it.

OK, so here is my advice specifically based on experience with these things and market knowledge:

SQL is a bad choice for storing time series data. It is a general purpose RDB. A noSQL time series DB like MongoDB is designed to do the job and will be much lower overhead both for CPU and memory, and when integrating with visualization tools for charts and graphs. You should run this on a dedicated and reliable system with redundant storage if your historical data is important.

Plan on MQTT for transporting the data, it can be a reverse channel as well for command and control if that becomes useful. The most commonly used MQTT broker is Mosquitto.

Using GPS time on the sensors would be very helpful. Sending timestamps with the ADC numbers will help if there are connectivity issues. The time the data is received might not be very close to the time it ws recorded.

If the cost of the GPS receiver modules is too high, and the nodes can talk to the Internet use NTP. If the nodes are on an isolated network (much better operational security) set up either a stratum 0 (using GPS) or a stratum 1 NTP server and point them at that.

If your data is critical, plan on crash or outage recovery. Cache several measurements in a ring buffer on the sensor device. They are going to be quite small so the ESP32 should have enough RAM to keep a decent period. If the broker or network becomes unavailable the sensor device should note the position of the first entry after the failed send in the ring buffer, and keep retrying. Once it succeeds send any other records in the ring buffer collected since then.

Use Node-RED. It offers nodes (modules) to do anything you might do with IoT. It is already done. It can be used to handle any incoming data and do the DB inserts. It also provides a visual tool to create user dashboards. It has hundreds of actime developers and tens of thousands of users. It will run on a RPi just fine.

Another possible architecture is to have a dedicated program that empties out the MQTT queue and inserts records into MongoDB. This would relieve the processor on the Node-RED device from dealing with it and it could simply be a consumer of the data out of the MongoDB.

Use Grafana (as mentioned above) for visulization. It can produce interactive charts and graphs and works happily in the Node-RED environment.

Consider using the ESP-Now protocol instead of WiFi for the sensor nodes. It has better range and doesn’t require the overhead of a WiFi connection. It would require one or more ESP32 gateways but it would allow much better scaling and keep the nodes off the Internet. I consider this a very important thing to look at, but not initially.

Since this is a commercial product be sure to check the licensing of any open source software you use. You will have a few different requirements depending on the project but almost all include the requirement to say the code was used. I don’t know that anything I have mentioned requires licensing for commercial use but you can get support if you pay for it.

I know this is a lot but if you go down the road of cobbling together code because “it’s a simple project” you will be doing a disservice to both your customers and the programmer(s) who have to ass the features that were “never going to be part of it”.
That programmer may be you! At least until you burn out from constantly gluing on features and working around legacy code.

Good luck.

 

OK, so here is my advice specifically based on experience with these things and market knowledge:

SQL is a bad choice for storing time series data. It is a general purpose RDB. A noSQL time series DB like MongoDB is designed to do the job and will be much lower overhead both for CPU and memory, and when integrating with visualization tools for charts and graphs. You should run this on a dedicated and reliable system with redundant storage if your historical data is important.

Plan on MQTT for transporting the data, it can be a reverse channel as well for command and control if that becomes useful. The most commonly used MQTT broker is Mosquitto.

Using GPS time on the sensors would be very helpful. Sending timestamps with the ADC numbers will help if there are connectivity issues. The time the data is received might not be very close to the time it ws recorded.

If the cost of the GPS receiver modules is too high, and the nodes can talk to the Internet use NTP. If the nodes are on an isolated network (much better operational security) set up either a stratum 0 (using GPS) or a stratum 1 NTP server and point them at that.

If your data is critical, plan on crash or outage recovery. Cache several measurements in a ring buffer on the sensor device. They are going to be quite small so the ESP32 should have enough RAM to keep a decent period. If the broker or network becomes unavailable the sensor device should note the position of the first entry after the failed send in the ring buffer, and keep retrying. Once it succeeds send any other records in the ring buffer collected since then.

Use Node-RED. It offers nodes (modules) to do anything you might do with IoT. It is already done. It can be used to handle any incoming data and do the DB inserts. It also provides a visual tool to create user dashboards. It has hundreds of actime developers and tens of thousands of users. It will run on a RPi just fine.

Another possible architecture is to have a dedicated program that empties out the MQTT queue and inserts records into MongoDB. This would relieve the processor on the Node-RED device from dealing with it and it could simply be a consumer of the data out of the MongoDB.

Use Grafana (as mentioned above) for visulization. It can produce interactive charts and graphs and works happily in the Node-RED environment.

Consider using the ESP-Now protocol instead of WiFi for the sensor nodes. It has better range and doesn’t require the overhead of a WiFi connection. It would require one or more ESP32 gateways but it would allow much better scaling and keep the nodes off the Internet. I consider this a very important thing to look at, but not initially.

Since this is a commercial product be sure to check the licensing of any open source software you use. You will have a few different requirements depending on the project but almost all include the requirement to say the code was used. I don’t know that anything I have mentioned requires licensing for commercial use but you can get support if you pay for it.

I know this is a lot but if you go down the road of cobbling together code because “it’s a simple project” you will be doing a disservice to both your customers and the programmer(s) who have to ass the features that were “never going to be part of it”.
That programmer may be you! At least until you burn out from constantly gluing on features and working around legacy code.

Good luck.

Is it possible to access the dashboard through internet if we use NodeRed? There will numerous locations with 30-50 esp modules in each location. I need to send the adc value from each esp module to a server, process it and make calculations there. Meanwhile I need to display the calculated value on a dashboard (for each location seperately) and need to control each esp module through the dashboard. The data collected needs to be analysed later.