What's going on under the hood with libraries for NodeMCU, WiFi, etc?

arduinolego611 · May 8, 2024

There are so many methods and examples on the internet that can be copied and pasted into your Arduino IDE to connect a microcontroller to WiFi and execute some code. There are a lot. But they all rely heavily libraries (which hide all the real code that's going into their functionality), so we dont really get a chance to see the steps needed to connect to WiFi. There are so many libraries out there, but I dont know how to use them in the first place, because I cant find any guides online that tell me what the steps are and in what order they need to be executed.

There's DNS, TCP, HTTP...there's network connection, parsing...so what code does the microcontroller need to access WiFi at its most basic level?

My project is just a simple LED lighting up at the push of an HTML button. Ive copied my code from the internet and it works, but when I look at the code to really understand what's going on, all the functions are referencing stuff in the library that I cant see, so I cant seem to make sense out of the code. The example code uses so many libraries, I have no idea which are being used when...

What I am looking for is a list of the bare bones necessary steps to use arduino/node MCU/any microcontroller to WiFi. Not for any project in particular, just for my own understanding of what's going on within the example codes I have been looking at...

Example:

1. connect to network
2. request port 80
...etc

geekoftheweek · May 8, 2024

Unfortunately a lot of that depends on what you are connecting to. For each type of service you want to connect to there is a specific transmission sequence that needs to be followed on both sides to make a successful and useful connection.

Post your code and we can break it down a little to maybe at least shed some light into what that particular example is doing. It's a kind of everything works the same, but it's different (if that makes any sense whatsoever).

When it comes to things like WiFi the libraries are a necessary "evil". Sure they hide a lot of what is happening behind the scenes, but without intimate knowledge of how a micro works and the steps it needs to take you're going to be beating your head against the wall trying to get things to work. Sometimes it's just easier to use the libraries.

arduinolego611 · May 8, 2024

geekoftheweek said:
Unfortunately a lot of that depends on what you are connecting to. For each type of service you want to connect to there is a specific transmission sequence that needs to be followed on both sides to make a successful and useful connection.

Post your code and we can break it down a little to maybe at least shed some light into what that particular example is doing. It's a kind of everything works the same, but it's different (if that makes any sense whatsoever).

When it comes to things like WiFi the libraries are a necessary "evil". Sure they hide a lot of what is happening behind the scenes, but without intimate knowledge of how a micro works and the steps it needs to take you're going to be beating your head against the wall trying to get things to work. Sometimes it's just easier to use the libraries.

Code:

#ifdef ESP8266
#include <ESP8266WiFi.h>
#include <ESP8266mDNS.h>
#elif defined(ESP32)
#include <WiFi.h>
#include <ESPmDNS.h>
#else
#error "Board not found"
#endif


const int LED = 5;

char webpage[] PROGMEM = R"=====(
<!DOCTYPE html>
<html>
<body>

<h1>Wireless Communication</h1>
<p>Click the buttons to toggle an LED</p>

<button onclick="window.location = 'http://192.168.4.1/led1/on'">On</button>
<button onclick="window.location = 'http://192.168.4.1/led1/off'">Off</button>

</body>
</html>
)=====";

// ipaddress/led1/on
//ipaddress/led1/off

// ipaddress/led2/on
//ipaddress/led2/off
#include <ESPAsyncWebServer.h>

AsyncWebServer server(80); // server port 80

void notFound(AsyncWebServerRequest *request)
{
  request->send(404, "text/plain", "Page Not found");
}

void setup(void)
{

  Serial.begin(115200);
  pinMode(LED, OUTPUT);

  WiFi.softAP("Home25", "6033216098");
  Serial.println("softap");
  Serial.println("");
  Serial.println(WiFi.softAPIP());


  if (MDNS.begin("ESP")) { //esp.local/
    Serial.println("MDNS responder started");
  }



  server.on("/", [](AsyncWebServerRequest * request)
  {

  request->send_P(200, "text/html", webpage);
  });

   server.on("/led1/on", HTTP_GET, [](AsyncWebServerRequest * request)
  {
    digitalWrite(LED, HIGH);
  request->send_P(200, "text/html", webpage);
  });

  server.onNotFound(notFound);

  server.begin();  // it will start webserver
}


void loop(void)
{
}

geekoftheweek · May 8, 2024

I'll try my best to expand a little without getting to carried away. I'll apologize in advance if I get into details you already know as I don't know your level of understanding and want to do the best I can.

Code:

AsyncWebServer server(80); // server port 80

What this actually does is create a socket that another socket on a different device can connect to. In the world of sockets you can have blocking or non-blocking. A blocking socket will wait at each step for a connection request, or read, or write from the connected socket before continuing in the program. For a non-blocking socket you need to create the socket, and attach a listener to alert you when there is new activity on the socket. The AsyncWebServer does all that work for you.
You can actually do a Unix style socket program on an ESPxxxx, but I'm not going to get into that here. Port 80 is the standard HTTP (web page) port that is used pretty much everywhere for public access. Some other common ones are 53 for DNS, 443 for HTTPS (secure web pages), 21 for FTP, and the list goes on. The ESP family is limited to one user defined socket per device, but a typical server or PC can have more than one socket available to connect to. It's kind of like dialing an extension in a large building, but instead it just connects to a different program.

Code:

MDNS.begin("ESP")

Normally when you enter something.com into your browser it connects to a DNS server that returns an IP address that your browser actually uses to load the page. MDNS is a way to eliminate the DNS server and have the device respond to a MDNS packet on the network instead. Most WiFi printers, smart TVs, and WiFi devices in general use this to allow you to connect without a great deal of manual configuration.

Code:

server.on("/", [](AsyncWebServerRequest * request)

This is where the socket listener I mentioned earlier comes in to play. Long story short when you put say something.com into your browser the browser creates a socket and connects to the socket on the other end. The browser then sends a HTTP request header that contains the name of the page, image, file, or whatever it is you are trying to access, along with your browser information, IP address, and some other more or less useless information (unless the server actually uses it for something). In this case you are requesting the root page of this site which is normally either index.html, index.php, index.???, or whatever the server is configured to serve up. At this point the socket connection is still active until you either send a response or return to the system loop where it is closed and cleaned up for the next go around.

Code:

request->send_P(200, "text/html", webpage);

In the previous "server.on()" the system passed a pointer to the current connection in the form of "AsyncWebServerRequest * request". Here you are sending the response through the already open connection. The 200 and "text/html" get converted to HTTP header information that will be returned to the browser before sending "webpage". The headers let your browser know if the request was successful, what kind of data it's returning, and the data itself.

I hope that cleared things up a little bit. When it comes to networking there is a lot of data that actually gets passed back and forth that you never see and that is mostly what is handled in the libraries. I tried to find some better examples and explainations online to point to, but unfortunately most are just code examples that you need to figure out how to modify to meet your requirements.

Some things to check out:
- Most common network interactions are carried out in TCP https://en.wikipedia.org/wiki/Transmission_Control_Protocol or UDP https://en.wikipedia.org/wiki/User_Datagram_Protocol packets. A lot of what happens behind the scenes is putting these packets in order, checking for errors, and parsing through the layers to find out just what exactly is supposed to be happening. I have to admit I have very limited knowledge in this area... going to deep might not have really helped anyways.

- Some basic behind the scenes happenings for HTTP (web pages) https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages

- Unix type socket program - not for ESPxxx, but just for some behind the scenes insight. https://github.com/denehs/unix-domain-socket-example/blob/master/client.c

arduinolego611 · May 10, 2024

geekoftheweek said:
I'll try my best to expand a little without getting to carried away. I'll apologize in advance if I get into details you already know as I don't know your level of understanding and want to do the best I can.

Code:

AsyncWebServer server(80); // server port 80

What this actually does is create a socket that another socket on a different device can connect to. In the world of sockets you can have blocking or non-blocking. A blocking socket will wait at each step for a connection request, or read, or write from the connected socket before continuing in the program. For a non-blocking socket you need to create the socket, and attach a listener to alert you when there is new activity on the socket. The AsyncWebServer does all that work for you.
You can actually do a Unix style socket program on an ESPxxxx, but I'm not going to get into that here. Port 80 is the standard HTTP (web page) port that is used pretty much everywhere for public access. Some other common ones are 53 for DNS, 443 for HTTPS (secure web pages), 21 for FTP, and the list goes on. The ESP family is limited to one user defined socket per device, but a typical server or PC can have more than one socket available to connect to. It's kind of like dialing an extension in a large building, but instead it just connects to a different program.

Code:

MDNS.begin("ESP")

Normally when you enter something.com into your browser it connects to a DNS server that returns an IP address that your browser actually uses to load the page. MDNS is a way to eliminate the DNS server and have the device respond to a MDNS packet on the network instead. Most WiFi printers, smart TVs, and WiFi devices in general use this to allow you to connect without a great deal of manual configuration.

Code:

server.on("/", [](AsyncWebServerRequest * request)

This is where the socket listener I mentioned earlier comes in to play. Long story short when you put say something.com into your browser the browser creates a socket and connects to the socket on the other end. The browser then sends a HTTP request header that contains the name of the page, image, file, or whatever it is you are trying to access, along with your browser information, IP address, and some other more or less useless information (unless the server actually uses it for something). In this case you are requesting the root page of this site which is normally either index.html, index.php, index.???, or whatever the server is configured to serve up. At this point the socket connection is still active until you either send a response or return to the system loop where it is closed and cleaned up for the next go around.

Code:

request->send_P(200, "text/html", webpage);

In the previous "server.on()" the system passed a pointer to the current connection in the form of "AsyncWebServerRequest * request". Here you are sending the response through the already open connection. The 200 and "text/html" get converted to HTTP header information that will be returned to the browser before sending "webpage". The headers let your browser know if the request was successful, what kind of data it's returning, and the data itself.

I hope that cleared things up a little bit. When it comes to networking there is a lot of data that actually gets passed back and forth that you never see and that is mostly what is handled in the libraries. I tried to find some better examples and explainations online to point to, but unfortunately most are just code examples that you need to figure out how to modify to meet your requirements.

Some things to check out:
- Most common network interactions are carried out in TCP https://en.wikipedia.org/wiki/Transmission_Control_Protocol or UDP https://en.wikipedia.org/wiki/User_Datagram_Protocol packets. A lot of what happens behind the scenes is putting these packets in order, checking for errors, and parsing through the layers to find out just what exactly is supposed to be happening. I have to admit I have very limited knowledge in this area... going to deep might not have really helped anyways.

- Some basic behind the scenes happenings for HTTP (web pages) https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages

- Unix type socket program - not for ESPxxx, but just for some behind the scenes insight. https://github.com/denehs/unix-domain-socket-example/blob/master/client.c

This is great! Really great! Thank you!!

I would like to do the same breakdown of a few different things, like WiFi/JSON or similar methods of connecting to internet/web pages ...or a breakdown of another esp8266 program

geekoftheweek · May 10, 2024

I'm glad that helped!! I was actually wondering if I was going down the right path or not.

When it comes to WiFi connections a lot of the same concepts are the same. Instead of web page information being passed as TCP packets it passes network information.

Pretty much all the libraries do is write blocks of data in memory that need to be transmitted, point the module to the memory location of the data to be transmitted, set other memory locations with the data size and other configuration information, and finally tell it to do it's thing. For reading it's basically the same... you set up memory pointers for the module to write the data that it read. I looked through the technical manual for something to point to, but came up short in the WiFi department.

While I can't give WiFi specifics what I can do is describe how the USB module of a PIC works... while they are not the same they work very similar from a programming context. For instance the memory address 400h is the beginning of various USB endpoint configuration bytes. First you write the data to be transmitted back to the PC at say 500h, then starting at 400h the first byte is some endpoint configuration bits that are used to configure the USB module, the next two bytes define where the data to transmit is located (500h in this case), and then one byte for the data length. After all that is set up you then wait for the PC to read the data at which time an interrupt flag is set signalling that the data was read. Next the program reads various other bytes of memory that correspond to status registers to determine if the transmission was successful and where to go from there. That of course is a gross oversimplification... my own custom library I created was well over 1000 lines for even the most basic USB setup. WiFi is going to be similar in that you build your data packet in memory, point the WiFi module to the memory that contains the packet, tell the module how big the packet it, and finally change the bits in memory needed to let the module know new data is ready. Everything else is handled in hardware.

As far as JSON goes it's pretty much just data packets that are run through and additional filter that sorts out formatting information from what I know. I have worked very little with it so I really don't know anything useful there.

It might be more than you want to get in to, but https://www.espressif.com/sites/default/files/documentation/esp8266-technical_reference_en.pdf goes into details of the inner workings of the ESP8266 and ultimately every library is just a way to manipulate the manipulate the memory locations described in the manual to make the ESP do what needs done.

Thread starter	Similar threads	Forum	Replies	Date
T	Is question 31 what's going on here!	Homework Help	26	May 24, 2024
	What's going on with Youtube ?	Off-Topic	8	Feb 4, 2024
	Key fob only works on battery and not power supply. What's going on here?	General Electronics Chat	26	Aug 23, 2021
	What's going on here?	Off-Topic	2	Jun 16, 2015
	+ and * in after logged in users in "What's Going on"	Off-Topic	0	Sep 28, 2013

What's going on under the hood with libraries for NodeMCU, WiFi, etc?

Join our Engineering Community! Sign-in with:

What's going on under the hood with libraries for NodeMCU, WiFi, etc?

arduinolego611

geekoftheweek

arduinolego611

geekoftheweek

arduinolego611

geekoftheweek

You May Also Like

Infineon Targets AI Servers With Newly Added CoolSiC MOSFETs

R&S Isolated Probe Tackles Switching Challenges of Complex Signal Types

Intel First to Install High NA EUV Lithography Scanner

Synopsys Claims Industry’s First PCI Express 7.0 IP Set