Hi,

I'm looking help to understand four key terms: TCP/IP, Web Socket, HTTP, and the OSI model, specifically the physical layer, within the context of a smart home automation scenario main part of smart home system is raspberry Pi.

This setup involves controlling lights, TV, AC, and live video streaming to Android mobile devices through an app or a web application on a laptop. All live video is stored on a remote server, and equipment on/off data is transmitted to the server. This process consists of a laptop, mobile devices, a router with an internet connection, and a smart home system with lights, TV, and cameras controlled by a driver board.

I know there are so many questions; we would tackle them one by one in the context of the OSI layer with the smart home system

I think The OSI physical layer deals with the hardware and physical connections, including Ethernet cables, Wi-Fi, and connections between devices, the router, and the server.

What would happen in data link and network layer ?

Please refer to an example of a smart home system. Avoid providing a general description since I've already gone through many documents and articles, and it's causing confusion. I believe I've provided all the essential requirements, In case I missed anything feel free to ask. Explaining in the context of a smart home system will give me a clearer understanding and help me address all the specific points I'm trying to understand.

 

Hi mate,

I have created a little information sheet for what you asked for, hope this helps!

TCP/IP is a set of protocols governing internet data transmission in a smart home. It enables devices like Raspberry Pi, laptops, and mobiles to communicate and access a remote video server.

Web Sockets establish a long-lasting, full-duplex communication channel. They're vital for real-time applications, like streaming live video to Android devices, ensuring efficient, low-latency communication between smart home devices and mobiles.

HTTP is a protocol for internet data transmission. It's used for web browsers to request and receive web pages. In a smart home, it may be used for controlling devices via a web application on laptops or mobiles.

The OSI Model encompasses different layers:

1. Physical Layer manages hardware and connections (e.g., Ethernet cables, Wi-Fi) ensuring data transmission between devices, router, and remote server.
2. Data Link Layer ensures reliable data transmission over a physical link, addressing issues like error detection and correction on the local network.
3. Network Layer routes data packets between devices in different networks, ensuring correct delivery of status updates to the remote video server.

 

I have created a little information sheet for what you asked for, hope this helps!

Thank you for the very detailed description.

With the use of only three layers - the physical layer, data link layer, and network layer - can I access my smart home system from anywhere in the world?

 

Thank you for the very detailed description.

With the use of only three layers - the physical layer, data link layer, and network layer - can I access my smart home system from anywhere in the world?

Just to clarify...

TCP (without the IP) is a connection oriented protocol used to provide a reliable connection between a client and server.
It resides at the Transport layer (layer 4) of the OSI model.

IP (without the TCP) is network layer communication protocol that provides the transmission/reception/routing of datagrams across networks.
It resides at the Network layer (layer 3) of the OSI model.

TCP/IP specifies the use of TCP over the IP protocol.

WebSockets is a bidirectional communications protocol and provides a comm service similar to that of peer-to-peer network architectures. This can be use for real time communications.
It basically resides at the applications layer (Layer 7) of the OSI model.

HTTP is a unidirectional communications protocol used for client-server applications.
It basically resides at the applications layer (layer 7) of the OSI model.

So....

If the Raspberry Pi supports all of the above, either natively, or thru the use of additional software, and is provided a connection to the internet, then yes, its possible for the device to communicate with a similar device ALMOST anywhere in the world.

 

Just to clarify...
So....

If the Raspberry Pi supports all of the above, either natively, or thru the use of additional software, and is provided a connection to the internet, then yes, its possible for the device to communicate with a similar device ALMOST anywhere in the world.

I'm still not entirely clear. It seems that while the lower layers (physical, data link, and network) are important for establishing a network connection, but accessing a smart home system from anywhere in the world would typically require the use of higher-level protocols and layers, such as HTTP or WebSockets.

I think just only using three (physical, data link, and network) and without using HTTP or WebSockets. We can't access smart home system from anywhere in the world. it's my understanding only by reading information on internet but I'm not sure.

 

The lower layers in the network stack deal with physically moving messages between one node and another -- what information those messages contain is irrelevant to those layers. If you want to create and receive those messages and do something with the information contained within them, that is the purview of the upper layers.

 

The lower layers in the network stack deal with physically moving messages between one node and another -- what information those messages contain is irrelevant to those layers. If you want to create and receive those messages and do something with the information contained within them, that is the purview of the upper layers.

Okay, let's begin by discussing the minimum OSI layers required.

I think, We need at least Four layers the physical, data, network, and application layers to enable a smart home system to send or receive data from a remote server.

Do you agree?

 

I'm still not entirely clear. It seems that while the lower layers (physical, data link, and network) are important for establishing a network connection, but accessing a smart home system from anywhere in the world would typically require the use of higher-level protocols and layers, such as HTTP or WebSockets.

I think just only using three (physical, data link, and network) and without using HTTP or WebSockets. We can't access smart home system from anywhere in the world. it's my understanding only by reading information on internet but I'm not sure.

The physical, datalink, and network layers, without HTTP or WebSockets, will only establish IP connectivity between the devices.
The devices will only be able to "ping" each other (from ALMOST anywhere in the world), but nothing else. Anything beyond that has to be provided in the upper layers of the OSI model.

 

Okay, let's begin by discussing the minimum OSI layers required.

I think, We need at least Four layers the physical, data, network, and application layers to enable a smart home system to send or receive data from a remote server.

Do you agree?

What data? Where is this data originating? Where is it ending up?

The network layer at the sending end receives data from the layer above it, does some stuff to it, and delivers a packet containing the data to the transport layer below it. It does not produce spontaneous data (except data directly related to the operation of the network layer). It certainly does not know, nor care, about your home automation stuff. Similarly, the network layer at the receiving end receives data from the transport layer below it, does some stuff to it, and delivers the resulting data to the layer above it.

Without the layer above it, the network layer is limited to being about to route packets it receives that are not destined for that node onto another node for delivery.

 

The devices will only be able to "ping" each other (from ALMOST anywhere in the world), but nothing else. Anything beyond that has to be provided in the upper layers of the OSI model.

Thank you. Now, considering requirements which involves controlling lights, TV, AC, and live video streaming to Android mobile devices through an app or a web application on a laptop, with live video stored on a remote server and equipment on/off data transmitted to the server.

We have two options or many: WebSocket and HTTP

I think we can use either WebSocket or HTTP or any other. I am trying to understand which one give much advantage for given requirements ?

 

With the use of only three layers - the physical layer, data link layer, and network layer - can I access my smart home system from anywhere in the world?

The seven layers of the OSI model are always present, they are just not always separately implemented. Sometimes the layers are combined because the implementation of the network stack makes them monoilithic.

But you are always going to need an application layer at the top and a physical layer at the bottom.

 

Thank you. Now, considering requirements which involves controlling lights, TV, AC, and live video streaming to Android mobile devices through an app or a web application on a laptop, with live video stored on a remote server and equipment on/off data transmitted to the server.

We have two options or many: WebSocket and HTTP

I think we can use either WebSocket or HTTP or any other. I am trying to understand which one give much advantage for given requirements ?

If you need real-time bi-directional communications between devices, then use WebSocket, otherwise use HTTP.

 

If you need real-time bi-directional communications between devices, then use WebSocket, otherwise use HTTP.

I've been trying to determine which option, WebSocket or HTTP, we should use for our provided requirements, but I'm finding it challenging to decide.

Could you please share your recommendation? I'm interested in your choice and the reasons you believe one might be a better fit for our scenario.

 

I've been trying to determine which option, WebSocket or HTTP, we should use for our provided requirements, but I'm finding it challenging to decide.

Could you please share your recommendation? I'm interested in your choice and the reasons you believe one might be a better fit for our scenario.

I shared my recommendation.
You have to decide the needs of the application you intend to develop.

 

Web Socket

This one is likley most understood.

A socket is formed when the source (client) connects to a listening port on destination (server).
You can have the IPv6 stack service all connections (IPv4 and IPv6) to a service port (for various OS's).

How can 1,000 web browsers all connect to one server port TCP-443? By way of creating sockets.

 

I shared my recommendation.
You have to decide the needs of the application you intend to develop.

It seems WebSocket would good choice.

The primary reason is that WebSockets are designed for real-time, bidirectional communication.

I see the need for real-time response and bidirectional communication in requirements . For example, when we turn on a light, we want an immediate response without delay. We can send commands from the client to the server (e.g., turn off the AC), and the server can send updates or responses confirming the AC's status.

 

What

It seems WebSocket would good choice.

The primary reason is that WebSockets are designed for real-time, bidirectional communication.

I see the need for real-time response and bidirectional communication in requirements . For example, when we turn on a light, we want an immediate response without delay. We can send commands from the client to the server (e.g., turn off the AC), and the server can send updates or responses confirming the AC's status.

If you want an immediate response without delay, then you need to get rid of the network altogether, particular a network with anything resembling a layered protocol stack. They all introduce delay and if you are wanting to be able to access this via the Internet, then that delay can be significant and highly variable since the lower layers are introducing latency in order to achieve a reliable, error-free communications link based on an unreliable and error-laden fabric.

So instead of just throwing around terms line "real-time response", you need to determine what that means for YOUR application. How much latency is tolerable and which approaches, if any, can meet that requirement?

 

(This is very long and may seem discursive but I promise it is the result of a lot of experience with, and time teaching about, networks and related technology. At the end there are a lot of links to material to read about emerging standards in the IoT networking domain—but it would be a shame if you skipped the material above it, which I believe will give you a little of the perspective that I have on this subject.)

I think one of the problems you are having is that instead of looking at what exists you are trying to imagine what might exist and the resulting naïve conception becomes the baseline that must be debunked. This is a terrible way to learn and very frustrating to anyone who might be trying to teach something. You are in effect constraining the instruction to telling you what is wrong with your imaginary solution instead of answering the question you purport to be asking: "how is an IoT node connected to its network?"

Before you can know even the right questions to ask, you have to research things in an iterative progression from middle, to high, low level abstractions and back up and down. This topic is incredibly mature and well beyond anything you have talked about or can currently imagine. There are at least tens of thousands of people working on this, and hundreds of companies—including some of the largest in the world.

While there are a variety of ways to accomplish Command & Control (C&C) of IoT nodes, and many types of nodes, the specific situation will determine what is selected. Because standards are a practical necessity, and each layer of the interaction has its own layered model, you can't just choose simple minded options at arbitrary layers and call that a "solution" that comports with the reality of IoT networks.

If you really want to understand IoT networks, their components, and how they interact you have to look at the practical reality of modern IoT implementations. The arbitrary and naïve scenarios you construct place meaningless constraints on answers that can be made to your questions, and so 80% of the effort of answering is debunking the assumptions—very frustrating.

Here is an answer to the question(s) you haven't asked, because you can't, but the answers to those questions is what you ned to know to go further. It is complicated and extremely extensive, but it is the answer to the question(s) inferred for your's, when an informed point of view is applied.

How do I establish an IoT network for Home Automation?
While there are many competing methods, based on standards (e.g. TCP, HTTP, MQTT, etc.) for IoT networking, there is an emerging overarching standard and set of best practices that will, in one form or another, prove to be the general basis for interoperable IoT networking.

This methodology is based on technical excellence, practical considerations (including interoperability, availability, scope of adoption by hardware and software companies, simplicity for the developer and user both, and the possibility of future enhancements concerning hardware and software innovation that we can predict will happen but can't predict the form it will take.

The world of IoT, and home automation IoT is evolving rapidly. The evolutionary pressures will favor some arbitrary choices due to convenience, but as we are seeing, unlike the unguided natural selection of biological evolution, will be increasingly targeted by immediate intelligent selection by humans.

So while IoT started evolving only a short time ago, we've gone well beyond the convenience drive stage and we are now seeing a stable platform upon which new products and innovation is standing. For example, in the earl(ier) days of IoT, a developer, trying not to reinvent the wheel would choose something already developed to bolt on to her project/product leading to, for example, broad adoption of:

1. Wi-Fi, Ethernet, Bluetooth
2. IP
3. TCP, UDP
4. HTTP, MQTT
5. &c

Some of these layers were a given—they represented established infrastructure that, for practical reasons, had to be retained. It makes no sense, for example, to replace physical layer protocols with something new. This would require the implementation of bespoke hardware and software when the existing and established networks were already sufficient—if not "ideal" in some useless sense.

Of course, if you say "cost and time are no object" you've just removed yourself from the real world, and stopped being concerned about reality. It is not a useful exercise if you want to understand how to build an IoT network if you first explore how to replace the physical network that is well established in theory and practice.

At one point it would have been worth considering. When I first started dealing with wireless networking there was no established standard. It was a time when standards were appearing and competing for status as the consensus. In 1997, the Wi-Fi Alliance was formed to promulgate a set of interoperability standards for a wireless networking protocol. The first public appearance of what would become the most important force in wireless networking was at the COMDEX trade show in Las Vegas.

Not completely coincidentally, at the same COMDEX I chaired a panel on the future of wireless networked made up of representatives of various companies who were building proprietary wireless network adapters and competing to become a de facto standard. I had no representative of the Wi-Fi Alliance on the panel because Wi-Fi was only a proposal at the time.

While the members of may panel argued about why their product would achieve hegemony in the wireless networking world, the Wi-Fi alliance manned their modest booth on the show floor explaining why the world needed them. When I was interviewed at the time concerning the future of wireless networking I was asked about Wi-Fi—which was the subject of much skepticism. I said that the alliance was a very good idea but only time could say if it would become a good thing.

But, I also said that the potential was much greater than any of the existing proprietary standards because Wi-Fi was starting out as a cooperative venture to create a shared platform that could support competition and innovation while making sure there was a constant basis for it, a common understanding of what a wireless network is.

The choice of IP networking (and its connected—TCP—and connectionless—UDP—protocols) seems very obvious today but at the time Wi-Fi was ascending, IP was only for nerds and nerd networks. ARPANET, developed in 1973 at MIT, was the platform for the evolution and eventual standardization of TCP/IP.

a logical map of ARPANET in 1977—it was not running TCP/IP at the time, note "NAMES SHOWN ARE IMP NAMES..."​

Eventually, the extremely limited Internet Message Processor (IMP) was replaced by TCP/IP. The original ARPANET innovated the ideas of internetworking and platform independence when most if not all other networks were proprietary and lacked the idea of routing.

As soon as the need for internet(work) routing—that is connecting a Local Area Network (LAN) to other LANs, creating a Wide Area Network (WAN)—the adoption of TCP/IP began apace. LANs were originally, by and large, islands. And if an organization was spread out over a wide geographic area they often just used a bridge which sent all of the same traffic to all sites creating a giant LAN, rather than the much more efficient router which selectively sent traffic based in the network address of the remote target.

And, eventually, even ARPANET itself, with its established physical infrastructure was adopted* as a way of getting immediately to business.
*yes, Al Gore was actually pivotal in that though he didn't "create it" nor did he ever claim he did (I am not a special fan of Gore, but he's gotten a bum rap on the "I created the Internet" myth—many scientists and engineers, and even Newt Gingrich of all people, tried to set the record straight with little success.

This is the point of choosing established technology and standards. To eliminate the need for wheel reinvention and move the innovation up a level in abstraction. That is why today, save very low latency, and some other requirements which may exist*. The coverage of Ethernet, Wi-Fi, Zigbee, and Bluetooth will be the overwhelming choices for their respective functions. It makes no sense to choose something else.
*for example very low power combined with low overhead (where ESP-Now might be an option and if long range is more important the perhaps LoRa—the main principle is that if an existing option provides 80% or more, making your own is wheel reinvention and probably can't be justified

So, established standards make the most sense—if and only if—they actually satisfy the requirements. And example of this is MQTT. MQTT was developed by IBM for very low power sensor networks. It provides key facilities such as a very low overhead protocol for reduced power usage, store and forward capabilities for nodes that may not always be awake, or otherwise connected all the time, and an established protocol definition. As it was adopted more broadly it became an OASIS standard and F/OSS options for servers and client libraries became readily available.

MQTT is still widely used in IoT and where it is appropriate it is a great solution—but as Home Automation IoT has matured, the shortcomings of MQTT for that application have become more evident. So, today, starting from scratch, MQTT is not the obvious default is was only a few years ago. You might still choose to use it for its merits, especially if you don't need more and quick compatibility is important, but that may well be a short-sighted choice.

So you get the picture, choosing a technological platform has many dimensions, and a very important one is choosing something that will make it compatible and leverage the work of others. This is why so many commercial products today, from large companies, have disclosure statements concerning the F/OSS licenses that are compliant with. Open Source is an example of not succumbing to wheel reinvention, and getting free labor to boot. It is almost always a two way street as well since companies depending on F/OSS have a stake in the security and effectiveness of that software and so contribute time and/or money to those projects.

So where does that leave your Home Automation IoT node?
Today, the emerging standards for Home Automation IoT interoperability and functionality have all but settled on a few standards. Those are:

1. IEEE 802.15.4 (Zigbee hardware), Wi-Fi, Ethernet, and Bluetooth
2. Thread (OpenThread)
3. IP, with its attendant TCP and UDP
4. Matter (Matter on Github)
5. CoAP (Constrained Application Protocol, RFC7252)

Each of these emerging and established standards has a rôle to play at the level of its layer. There is a lot of material there. You've been doing a lot of reading. I suggest you read through the material at the links. Some is descriptive and some explanatory. If you read something about each one of these, you will have a basis for asking cogent and meaningful questions about both the abstract idea of layered network stacks and how to put it into practice, and the demands of the various protocols you will choose among to create a practical IoT network.

The real questions that arise from this research will be both more possible to answer and more useful when it is. I spent time on this I hope is productive for you. If you choose to ignore it, that's your choice but it will certainly also be your loss.

Best of luck whatever you choose.

 

What

If you want an immediate response without delay, then you need to get rid of the network altogether,

I believe that real-time response may not be an absolute necessity in our case. As long as the system can reliably receive instructions from the server and turn on the lights within a reasonable timeframe, whether that's 1 second, 5 seconds, or even 30 seconds, it should suffice. However, for an optimal user experience, I would consider a maximum acceptable delay of up to 1 minute

 

The real questions that arise from this research will be both more possible to answer and more useful when it is. I spent time on this I hope is productive for you. If you choose to ignore it, that's your choice but it will certainly also be your loss.

Best of luck whatever you choose.

Thank you for taking the time to provide such a detailed explanation. My main concern is when faced with multiple options, choosing the one that best fits my specific requirements. Making this decision requires a strong understanding, and that's why I presented my requirements and two options (although I didn't want to overwhelm with choices). I'm more interested in understanding the communication process between servers and clients. I've spent a considerable amount of time researching online, reading books, and various materials to grasp how communication occurs in this context, including the communication protocols used. That's why I shared my requirements and asked for opinions on which option to choose, along with the reasoning behind those choices. I also provided my own perspective on why I might lean towards a particular option. As I mentioned in my initial post, I didn't want a general description, as that information is readily available online. I asked these questions because I find it challenging to comprehend these concepts within the framework of real-world requirements. I'm focusing on practical requirements, especially since I'm considering a smart home system. While some may disagree with my approach, but it works best for me because it allows me to gather diverse opinions and refine my understanding by comparing it with others

I am here getting useful answers on smart home automation IoT project that I wanted to understand