IoT Web server

Ya’akov

Joined Jan 27, 2019
6,035
Is it possible to access the dashboard through internet if we use NodeRed? There will numerous locations with 30-50 esp modules in each location. I need to send the adc value from each esp module to a server, process it and make calculations there. Meanwhile I need to display the calculated value on a dashboard (for each location seperately) and need to control each esp module through the dashboard. The data collected needs to be analysed later.
Sure. You could do that in a number of ways. You can run it on a VPS with enough capacity for all locations, or you could have a small instance of Node-RED on the LAN there (appliance style) and use it as both the dashboard and the gateway for the other modules.

That is, you could have the Node-RED appliance act as an MQTT gateway collecting the messages from the local sensor nodes and forwarding them on to the main broker for insertion into the TSDB (Time Series DB). Process the data and send it back for local display on the dashboard.

The answer is "yes" but even merely the scale of this project and its network dependency cries out for a serious design not just tossing pieces at the problem. I can already see that you are likely to have wireless capacity problems if you don't dedicate an SSID to the sensors (that's where ESP-Now comes in, get them off the WIFi!)

You also don't really want the nightmare of hundreds of devices free to talk over the Internet, that's scary. You also need to consider OTA updates for the ESP32s and plan for it. If oyu use ESP-Now you will have to have a way to tell the ESP32 to switch to WiFi for the OTA and then reboot back into the ESP-Now configuration.

There's a lot to get this right, and sustainable, and scalable. How many people on this project? Oh, and what is the measurement/transmit interval?
 

Thread Starter

anandhuskumar

Joined May 14, 2014
25
Sure. You could do that in a number of ways. You can run it on a VPS with enough capacity for all locations, or you could have a small instance of Node-RED on the LAN there (appliance style) and use it as both the dashboard and the gateway for the other modules.

That is, you could have the Node-RED appliance act as an MQTT gateway collecting the messages from the local sensor nodes and forwarding them on to the main broker for insertion into the TSDB (Time Series DB). Process the data and send it back for local display on the dashboard.

The answer is "yes" but even merely the scale of this project and its network dependency cries out for a serious design not just tossing pieces at the problem. I can already see that you are likely to have wireless capacity problems if you don't dedicate an SSID to the sensors (that's where ESP-Now comes in, get them off the WIFi!)

You also don't really want the nightmare of hundreds of devices free to talk over the Internet, that's scary. You also need to consider OTA updates for the ESP32s and plan for it. If oyu use ESP-Now you will have to have a way to tell the ESP32 to switch to WiFi for the OTA and then reboot back into the ESP-Now configuration.

There's a lot to get this right, and sustainable, and scalable. How many people on this project? Oh, and what is the measurement/transmit interval?
Thank you so much for your guidance. I need to do a lot of research to process all this. And right now, I am the only person working on this project. Just started the projcet few days back and now I am almost finished with the hardware design part. Since the adc of esp is not a reliable one, I used ads1115 module to read the analog signals. Each esp needs to send the adc value wirelessly. Initially I thought of sending the data directly from the esp module to a database, but when the number of devices increases, it is not possible. So, now I am planning to collect all the adc values from all the nodes continuously and store this data to a database, make the calculations realtime, display them and store these values for further analysis. I need to implement some machine learning model to analyse this data at a later stage. But, my current concern is to send adc values collected by a single esp module to a database.
 

Thread Starter

anandhuskumar

Joined May 14, 2014
25
Sure. You could do that in a number of ways. You can run it on a VPS with enough capacity for all locations, or you could have a small instance of Node-RED on the LAN there (appliance style) and use it as both the dashboard and the gateway for the other modules.

That is, you could have the Node-RED appliance act as an MQTT gateway collecting the messages from the local sensor nodes and forwarding them on to the main broker for insertion into the TSDB (Time Series DB). Process the data and send it back for local display on the dashboard.

The answer is "yes" but even merely the scale of this project and its network dependency cries out for a serious design not just tossing pieces at the problem. I can already see that you are likely to have wireless capacity problems if you don't dedicate an SSID to the sensors (that's where ESP-Now comes in, get them off the WIFi!)

You also don't really want the nightmare of hundreds of devices free to talk over the Internet, that's scary. You also need to consider OTA updates for the ESP32s and plan for it. If oyu use ESP-Now you will have to have a way to tell the ESP32 to switch to WiFi for the OTA and then reboot back into the ESP-Now configuration.

There's a lot to get this right, and sustainable, and scalable. How many people on this project? Oh, and what is the measurement/transmit interval?
Hi. Now I am trying to connect all the ESP devices (around 20-50 devices) to a hub (raspberry pi), store the data there and send it to a webserver database. Can I connect a large number of ESP devices to the hub through a single wifi router or is it possible to connect all these devices to the hub directly without connecting to the modem? I am thinking about using node red to collect the data from all these devices and then send these data to a webserver database. Is this the best way to do this or is there any better way to do it? Please help. Thank you.
 

Ya’akov

Joined Jan 27, 2019
6,035
Hi. Now I am trying to connect all the ESP devices (around 20-50 devices) to a hub (raspberry pi), store the data there and send it to a webserver database. Can I connect a large number of ESP devices to the hub through a single wifi router or is it possible to connect all these devices to the hub directly without connecting to the modem? I am thinking about using node red to collect the data from all these devices and then send these data to a webserver database. Is this the best way to do this or is there any better way to do it? Please help. Thank you.
The best way is with an MQTT broker, as described above.

I am going to suggest ditching the web server idea and going with MQTT which is purpose built for sensor networks, much lighter weight all the way around, and has supporting libraries in the Arduino environment.
MQTT provides a robust, lightweight method to quickly collect sensor data—the reason it was developed.

I would also strongly suggest using an MQTT gateway with ESP-Now protocol instead of WiFi. WiFi has a lot of useless overhead for your application and with the number of nodes you intend it might well experience capacity problems which will cost money to remediate.

These videos include an example of using a configuration of the type I am suggesting. It has some content you don't need, but I am not trying to show you a step-by-step video. There are a lot of resources for MQTT and ESP-Now on the web and a little searching will find them.

You will have to read and learn the two protocols and how they apply.

 

Thread Starter

anandhuskumar

Joined May 14, 2014
25
The best way is with an MQTT broker, as described above.



MQTT provides a robust, lightweight method to quickly collect sensor data—the reason it was developed.

I would also strongly suggest using an MQTT gateway with ESP-Now protocol instead of WiFi. WiFi has a lot of useless overhead for your application and with the number of nodes you intend it might well experience capacity problems which will cost money to remediate.

These videos include an example of using a configuration of the type I am suggesting. It has some content you don't need, but I am not trying to show you a step-by-step video. There are a lot of resources for MQTT and ESP-Now on the web and a little searching will find them.

You will have to read and learn the two protocols and how they apply.

Thank you for your reply. But I am afraid I can't ditch the webserver idea, because there will be a number of hubs like that at different locations and I need to collect the data from all of these hubs, store them, process them and analyse them before dispalying the processed data on a seperate dashboard for each of those hubs. Something like the cayenne iot platform.
 

Ya’akov

Joined Jan 27, 2019
6,035
Thank you for your reply. But I am afraid I can't ditch the webserver idea, because there will be a number of hubs like that at different locations and I need to collect the data from all of these hubs, store them, process them and analyse them before dispalying the processed data on a seperate dashboard for each of those hubs. Something like the cayenne iot platform.
I’m sorry, there is nothing you described about what you want to do that changes my answer. I can’t help you with a “webserver” solution because there isn’t one. If you are talking about using something like HTTP POST, well, as I have said previously that’s not advisable.

If I was being paid by you as a consultant, and you insisted on something other than the industry standard solution for no good reason you’ve given, I would fire you as a client to protect my own reputation and avoid the heartache of trying to solve your problem with something that isn’t a solution.

I wish you the very best of luck but I‘m afraid I can’t help you further. I believe you are out of your depth here and heading for a lot of pain. This is not a project for someone who doesn’t even understand MQTT to tackle. I hope you can prove me wrong but my experience says things are not going to go well.

If you decide that MQTT might be a good option after all, you can always ask more questions here, but I can’t handhold you through each step. As I said, best of luck with the project.
 

Thread Starter

anandhuskumar

Joined May 14, 2014
25
I’m sorry, there is nothing you described about what you want to do that changes my answer. I can’t help you with a “webserver” solution because there isn’t one. If you are talking about using something like HTTP POST, well, as I have said previously that’s not advisable.

If I was being paid by you as a consultant, and you insisted on something other than the industry standard solution for no good reason you’ve given, I would fire you as a client to protect my own reputation and avoid the heartache of trying to solve your problem with something that isn’t a solution.

I wish you the very best of luck but I‘m afraid I can’t help you further. I believe you are out of your depth here and heading for a lot of pain. This is not a project for someone who doesn’t even understand MQTT to tackle. I hope you can prove me wrong but my experience says things are not going to go well.

If you decide that MQTT might be a good option after all, you can always ask more questions here, but I can’t handhold you through each step. As I said, best of luck with the project.
I am sorry, but I think you misunderstood what I meant. I have a number of esp modules reading sensor data at a location. The sensor data is then collected by a single raspberrypi module using MQTT. There will be many such locations. I need a website as well where all the data from all these locations are collected, processed and stored. In the website, I need to create seperate dashboards for different locations, analyze and visualize the data collected from these locations. For now I am just trying to connect all these esp modules to a single raspberry pi and I am using NodeRed. Can we connect these esp modules directly to the rasberrypi without a modem? Or is it best to connect these modules through the router?
 

Ya’akov

Joined Jan 27, 2019
6,035
I am sorry, but I think you misunderstood what I meant. I have a number of esp modules reading sensor data at a location. The sensor data is then collected by a single raspberrypi module using MQTT. There will be many such locations. I need a website as well where all the data from all these locations are collected, processed and stored. In the website, I need to create seperate dashboards for different locations, analyze and visualize the data collected from these locations. For now I am just trying to connect all these esp modules to a single raspberry pi and I am using NodeRed. Can we connect these esp modules directly to the rasberrypi without a modem? Or is it best to connect these modules through the router?
We are definitely having a communications problem.

My first response, which I quoted for you, was when you were planning on using a “web server” to collect the data. “Ditching the web server“ was not about the dashboard(s).

Also, the rest of the post has very important information about how to create the network infrastructure, which you didn’t address at all. I answered your question about how to connect things together in that post: an MQTT gateway between receiving data from the sensor nodes using ESP-Now and sending them to an MQTT server via WiFi.

Connecting all of the sensor nodes via WiFi is not scaleable, and is very inefficient. The overhead of a WiFi connection is just dead weight for your application. And, don’t think of Node-RED as being the target, the target is the MQTT broker (which could be a Node-RED node or could be Mosquitto running on the same RPi.

I also pointed out that you need a TSDB and not an RDB for storage. All of this information is in my earlier posts, particularly #19 and #24. If you have questions about anything I have said in those, I am happy to answer them. I’m afraid I can’t answer random questions about your implementation choices that fall outside what is described there.

One more thing: you need to take this step-wise. First thing is to establish your sensor node network and its communication with the MQTT broker, and the insertion of the data into a TSDB.

Once you have that working, the dashboard can be addressed. The first activity—data collection—has no dependencies caused by your dashboard design choices so long as you collect all of the data you need, or can derive it from what you do collect.

So don’t even think about the dashboard at this point, just get your sensor network running and your MQTT/TSDB services operational.
 

ag-123

Joined Apr 28, 2017
238
I am sorry, but I think you misunderstood what I meant. I have a number of esp modules reading sensor data at a location. The sensor data is then collected by a single raspberrypi module using MQTT. There will be many such locations. I need a website as well where all the data from all these locations are collected, processed and stored. In the website, I need to create seperate dashboards for different locations, analyze and visualize the data collected from these locations. For now I am just trying to connect all these esp modules to a single raspberry pi and I am using NodeRed. Can we connect these esp modules directly to the rasberrypi without a modem? Or is it best to connect these modules through the router?
you can run your NodeRed on the server, nodejs is a web server after all. For anything more than that, a custom solution, you would need to develop it on your own.
 

Thread Starter

anandhuskumar

Joined May 14, 2014
25
you can run your NodeRed on the server, nodejs is a web server after all. For anything more than that, a custom solution, you would need to develop it on your own.
Thank you. I need seperate node red instances for each hub which can be accessed by users and then an administrator can access all of the data from all the hubs. But right now my concern is to connect a single esp module to a single hub and get data from it. How can I make a for loop in node red. The idea is to implement this function:
for(int i=0; i<50; i++)
{
z = ads.readADC_Differential_0_1() * 0.1875F;
x = x + z;
y = y + (z*z);
}
read 50 samples of the sensor data and make the calculations.
 

ag-123

Joined Apr 28, 2017
238
for MQTT you would probably need a library, on your esp device.
you would need a MQTT server/broker for that at the other end.
https://mosquitto.org/
Then you need another server and run your node red there.
https://nodered.org/docs/tutorials/first-flow
https://nodered.org/docs/tutorials/second-flow
it is possible to do various calculations in node red (javascript)
https://nodered.org/docs/user-guide/writing-functions
Node red can pull data (subscribe topics) from MQTT server/broker.
https://cookbook.nodered.org/mqtt/subscribe-to-topic
you can test all that on a Raspberry Pi
https://nodered.org/docs/getting-started/raspberrypi
 
Last edited:

Thread Starter

anandhuskumar

Joined May 14, 2014
25
We are definitely having a communications problem.

My first response, which I quoted for you, was when you were planning on using a “web server” to collect the data. “Ditching the web server“ was not about the dashboard(s).

Also, the rest of the post has very important information about how to create the network infrastructure, which you didn’t address at all. I answered your question about how to connect things together in that post: an MQTT gateway between receiving data from the sensor nodes using ESP-Now and sending them to an MQTT server via WiFi.

Connecting all of the sensor nodes via WiFi is not scaleable, and is very inefficient. The overhead of a WiFi connection is just dead weight for your application. And, don’t think of Node-RED as being the target, the target is the MQTT broker (which could be a Node-RED node or could be Mosquitto running on the same RPi.

I also pointed out that you need a TSDB and not an RDB for storage. All of this information is in my earlier posts, particularly #19 and #24. If you have questions about anything I have said in those, I am happy to answer them. I’m afraid I can’t answer random questions about your implementation choices that fall outside what is described there.

One more thing: you need to take this step-wise. First thing is to establish your sensor node network and its communication with the MQTT broker, and the insertion of the data into a TSDB.

Once you have that working, the dashboard can be addressed. The first activity—data collection—has no dependencies caused by your dashboard design choices so long as you collect all of the data you need, or can derive it from what you do collect.

So don’t even think about the dashboard at this point, just get your sensor network running and your MQTT/TSDB services operational.
Hi,
Thank you very much for your guidance. I have successfully sent the sensor reading to the node red running on raspberry pi and displayed the reading on a dashboard and influxdb is used to store the data. There's also a switch in the dashboard which can control the onboard led on the ESP module. The connection is made through the WiFi modem. How many such devices can be connected like this over a normal household WiFi modem? And how secure is this setup? If it is vulnerable, how can I imporve the security?
 

Ya’akov

Joined Jan 27, 2019
6,035
Hi,
Thank you very much for your guidance. I have successfully sent the sensor reading to the node red running on raspberry pi and displayed the reading on a dashboard and influxdb is used to store the data. There's also a switch in the dashboard which can control the onboard led on the ESP module. The connection is made through the WiFi modem. How many such devices can be connected like this over a normal household WiFi modem? And how secure is this setup? If it is vulnerable, how can I imporve the security?
I can’t answer your question about connection limits as it is heavily dependent on the AP/router you use. Some will periodically run out of resources (such as ARP table space) and stop routing packets from some nodes but still work with others. I have had this problem with expensive Netgear routers with ~60 devices connected. But the reliability of your infrastructure will depend on the actual device you use, and on the location since another source of trouble are neighbor APs causing co-channel interference.

As far as security goes, don’t let the sensor nodes be accessible via the Internet and you have an 80% solution. This is deceptive, though, since attacks on the nodes could be made using a device on your network that is accessible from outside. So security means making sure that anything which can be reached from the Internet is secure against attacks that allow the attacker to gain control of it as a platform for LAN based attacks.

The other 20% would be form local threats. That is, someone inside either maliciously or negligently compromising the sensor nodes as a local user. This could be someone trying to cause trouble, or someone that inserts, say, a USB drive with malware into their computer and becomes a source for infection. It really depends on your risk profile.

InfluxDB is a good option for the TSDB. It should work well. But it can only work as well as your database design. Your schema has to be well thought out from the start to ensure you are capturing normalized data to allow queries to the database to be efficient and not grow cruft as reporting requirements change in unexpected though predictable ways. You have to think carefully about the nature of th e data you are collecting and its use, and make sure you don’t end up haveing to cook the data repeatedly to get useful results.

I don‘t think your current approach is a scalable one. It has some good parts, and it is “easier” to implement than a rally scalable solution, but amount of technical debt you are accumulating is prodigious. From the description of the current scope of the project it may not take long to hit a brick wall but even if the first deployment ”succeeds”, it will take only what may well appear to the end user/specifier to be a trivial feature request to make the whole thing unworkable.

As I have said, this is not a small project, and with the size, distributed nature, and number of moving parts involved a real effort at engineeering a sustainable and scalable architecture is called for. Someone will have ot pay the technical debt—it may or may not be you. If it isn’t you, someone is going to be very unhappy with you, possibly long after you moved on.

It is an unfortunate reality that small projects need the same rigor as large projects to succeed. But with small projects you don’t get the economy of scale that makes the front-loaded effort seem economical while the smaller projects seem… small, which they are, but they will outgrow their original architecture if they don’t use the techniques of the larger ones.

I hope that things somehow go completely against my experience and your project is a rousing success, not just on go-live day but in the days, weeks, months, and years that follow. One hard lesson is the “temporary” and “small” solutions stay forever and grow.
 

Thread Starter

anandhuskumar

Joined May 14, 2014
25
I can’t answer your question about connection limits as it is heavily dependent on the AP/router you use. Some will periodically run out of resources (such as ARP table space) and stop routing packets from some nodes but still work with others. I have had this problem with expensive Netgear routers with ~60 devices connected. But the reliability of your infrastructure will depend on the actual device you use, and on the location since another source of trouble are neighbor APs causing co-channel interference.

As far as security goes, don’t let the sensor nodes be accessible via the Internet and you have an 80% solution. This is deceptive, though, since attacks on the nodes could be made using a device on your network that is accessible from outside. So security means making sure that anything which can be reached from the Internet is secure against attacks that allow the attacker to gain control of it as a platform for LAN based attacks.

The other 20% would be form local threats. That is, someone inside either maliciously or negligently compromising the sensor nodes as a local user. This could be someone trying to cause trouble, or someone that inserts, say, a USB drive with malware into their computer and becomes a source for infection. It really depends on your risk profile.

InfluxDB is a good option for the TSDB. It should work well. But it can only work as well as your database design. Your schema has to be well thought out from the start to ensure you are capturing normalized data to allow queries to the database to be efficient and not grow cruft as reporting requirements change in unexpected though predictable ways. You have to think carefully about the nature of th e data you are collecting and its use, and make sure you don’t end up haveing to cook the data repeatedly to get useful results.

I don‘t think your current approach is a scalable one. It has some good parts, and it is “easier” to implement than a rally scalable solution, but amount of technical debt you are accumulating is prodigious. From the description of the current scope of the project it may not take long to hit a brick wall but even if the first deployment ”succeeds”, it will take only what may well appear to the end user/specifier to be a trivial feature request to make the whole thing unworkable.

As I have said, this is not a small project, and with the size, distributed nature, and number of moving parts involved a real effort at engineeering a sustainable and scalable architecture is called for. Someone will have ot pay the technical debt—it may or may not be you. If it isn’t you, someone is going to be very unhappy with you, possibly long after you moved on.

It is an unfortunate reality that small projects need the same rigor as large projects to succeed. But with small projects you don’t get the economy of scale that makes the front-loaded effort seem economical while the smaller projects seem… small, which they are, but they will outgrow their original architecture if they don’t use the techniques of the larger ones.

I hope that things somehow go completely against my experience and your project is a rousing success, not just on go-live day but in the days, weeks, months, and years that follow. One hard lesson is the “temporary” and “small” solutions stay forever and grow.
Thank you so much. I couldn't have come this far without your guidance. Do you have any suggestions to connect more nodes to the network? How can I make this project scalable? The solution needs to be wireless.
 

Ya’akov

Joined Jan 27, 2019
6,035
Thank you so much. I couldn't have come this far without your guidance. Do you have any suggestions to connect more nodes to the network? How can I make this project scalable? The solution needs to be wireless.
Using the ESP-Now protocol instead of WiFi, with an ESP-Now to MQTT gateway would be my approach. I posted information about this approach above. The ESP-Now protocol eliminates the overhead of WiFi which, in your case, provides no benefit and the MQTT approach simplifies the the data collection while still allowing the command and control of the nodes.

If you use MQTT for all of the data transfer, you can have both the local Node-RED instance and a centralized TSDB server and subscribe to the relevant topic(s) to receive and process the data. MQTT is designed to do this job and offers QoS options that include guaranteed delivery, and it will act as a store and forward system for nodes that might sometimes be asleep.

As far as I am concerned, there is no sensible alternative to MQTT for this application. It embodies the thinking of many, very smart people about what you need to do this job properly; it is open source and very widely deployed meaning there are a lot of resources available to you. It also has as much security as you might need if you choose to implement it.

These are the two major components I would use as the foundation for the sensor network.
 
Top