Hi,
I have 2x PCBs BASE REMOTE with a radio link between them.
I'm trying to find the best way to send messages between them in order.
E,G SEND->SEND->SEND->SEND->SEND-><-SEND SEND->SEND->SEND->SEND->SEND-><-SEND

At the moment there is a 'TOFRO' variable in every 5th TX and every TX, that once READ and PARSED, can be used to 'ECHO' a message back.

There is a second option: Both BASE and REMOTE have a GPS, which has a timing signal. This could be used to time all ECHOS.

There is a problem, that if any messages or GPS gets lost, then what happens.

Any thoughts?
Cheers, Camerart.

 

What are you using for the RF link? What is the data you are sending? Why are you inventing your own protocol?

 

What are you using for the RF link? What is the data you are sending? Why are you inventing your own protocol?

Hi Y,
It's a long story: https://forum.allaboutcircuits.com/threads/remote-control-by-location-pic-in-oshonsoft.148795/

It's a radio control system, where the BASE controls the REMOTE via radio link, and the REMOTE sends back DATA.
C

 

So what is the radio link technology? Why are you trying to create a new protocol instead of using a standard?

 

So what is the radio link technology? Why are you trying to create a new protocol instead of using a standard?

Hi Y,
I'm using a pair of HC-12 radios using UARTs.

What is a standard?
C

 

Hi Y,
I'm using a pair of HC-12 radios using UARTs.

What is a standard?
C

TCP/IP for example. Not suggesting it as a solution for you, just as an example of a standard protocol, and a very good one at that.

 

So, here’s the thing. You are using sub-optimal hardware for your purpose though if you are using PICs and BASIC you might not be able to do better. Nonetheless, you should work out your protocol based on a specification not as-you-go.

I mean a specification that you write. You need to decide what attributes are important to you. For example, it appears you need guaranteed delivery. That is, you need to get a frame¹ to the target even if there is interference (a very, very likely thing in the LDP433 band you are using!)

This aspect of your protocol needs to deal both completely lost, but also damaged frames. You need to be able to detect if a frame contains an error, with a checksum, for example, and perhaps you will also want to use ECC (Error Correction Code) to implement FEC (Forward Error Correction) so that some errors can be repaired on the receiver side without retransmission.

You also appear to have time critical needs. You have definite time during which a frame is useful and so the overhead of the protocol must be managed to reduce latency. You have to consider what to do if a frame can’t be transmitted in the specified time (when it becomes “stale”, and should be discarded). That will be depend on what the messages you are sending do.

One of the things you will have in your frame is a timestamp, which will help decide if the frame is valid, and also a sequence number which will tell you if you missed something. For example if you receive a frame with number 3487, then one with 3489, you know that 3488 has gone missing. At that point you can decide is you need to send a message to retry the trans mission or something else.

So, as in the examples above, think, in the abstract about what your protocol will need to accomplish. Sometimes these things are mutually exclusive to varying degrees and you will need to optimize between them. Other times it will turn out that one thing make another redundant. The key is to figure out what attributes your messages must have and include those.

The layer above presentation is the application layer. The protocols at this layer are things like HTTP, FTP, SMTP, &c. You probably need a bit of this, at least in your thinking if not formally implemented as functions (though depending on how efficient your BASIC is, that would be a great idea).

You can add a field to your frame to indicate a frame type. This is a small number (in your case I can’t imagine needing more than 8 bits) which tells the receiver the type of content’s in the frame’s payload. The payload is the message itself without the protocol overhead.

By implementing different frame types you can decide on different handling for that frame. For example, let’s say you have sensor data which is non-critical. That is, if you miss a sample it doesn’t hurt anything. So such a payload could be a frame that is marked with a type that is discardable.

Take the scenario of needing a time continuous record of sensor data, that is you need each sample you collect to be newer than the last. When a sensor data frame arrives and is processed, the sequence number is recorded. If a new (to the receiver) frame arrives and its sequence number is less than the last recorded frame, your program knows to discard the data. This is possible without comparing timestamps thanks to the sequence number.

Once you have the attributes worked out, you can decide on a frame structure, which starts with a header. This is where the fields discussed above live. This is the structure of an IP (packet) header:

​

Bear in mind we are wantonly conflating some aspects of frames and packets here. So this represents the sort of thing that will be on the front end of our frame. Not all these fields make sense for you, and some would be used differently but serve a very similar function.

The version field is very useful. You should probably include one. That way, if you change how things work you can be sure you know if you are talking to a new or older version of the protocol. I am almost certain that a couple of bytes would cover you for a lifetime.

HL is the header length field. This allows the receiver to know when the header ends and the payload begins. This is prretty much mandatory as long as you want to be able to vary the length of the header. Alternatively, you can allocate a particular number of bytes for the header and make it invariant. This may be a good idea since it greatly simplifies ingesting the frame.

Type of Service in an IP header indicates the QoS (Quality of Service) requested for the packet. In your case you could choose to do it that way (e.g.: guaranteed delivery, time critical delivery, low priority delivery, &c) or you can have the type field I mentioned above and use your knowledge of the requirements of the frame type to manage those things. It is my opinion that you can ignore this field and use the follow protocol field as the type field instead.

Total Length this works in conjunction with the HL field, or of you choose fixed length headers, that number, to tell the reciever how much data represents a full frame. This allows you to know when you have it all.

The Identification field is going to be sequence number for you. As I said we are mashing up OSI layers here and the sequence number in TCP/IP is located in the TCP header. IT doesn’t matter for our purposes that we are being far from rigorous in this design.

The next four fields are irrelevant to this discussion.

The Header Checksum is a data integrity measure that lets the reciever now they‘ve gotten the correct information. To be honest, I don’t know whether the HC-12 would ever deliver damaged data, this is something you should find out. Because if the HC-12 is already implementing network layer checksums or other integrity checks, adding them to your protocol would be redundant.

Source and Destination addresses are probably not useful to you considering you are using a point to point connection that must be made before sending anything. On the other hand, they could be used to decide a “soft” endpoint. For example if the frame is sensor data of a particular type, it could be addressed to the function that is meant to handle it. Similarly if the remote device had different logical subsystems, frames of the same type could be sent to different functions for processing.

The above is all speculative and only you can say if it seems to have any utility.

The Options field can be ignored.

Padding is there because the IP header must end on a 32-bit boundary, so bytes are added to make that happen. This idea may be useful to you because it simplifies processing but it is probably more useful for C than BASIC.

The next thing in the frame would be the payload. The payload’s length can be derived by the header information. You can also decide to make frame length invariant if you can so formalize your messages that they can always fit in a specific length. This would prohibit arbitrary messages so it is probably not a good idea.

In the payload will be your commands and the responses. This is would be the next layer (application) protocol. You won‘t need a header for it, but you should have commands and parameters, like MOVE <coordinates>, REPORT <conditions>, and the like as well as SENSOR_DATA <data>, EXCEPTION <exception report>, &c.

I have completely made up the above commands and message types, what it should be is base doh what you are trying to do.

I don’t have any idea if this is at all helpful, If oyu do something like it you will never paint yourself into a corner, and you will have much more reliable performance than just cobbling together messages and patching them until they “work”.

Unfortunately, if you were starting from scratch I surely would have suggested different hardware that can easly use TCP/IP and something like MQTT which is both standard faire and already very polished.

Please let me know if you have questions or need some help with these ideas.


—
1. A frame is a PDU (Protocol Data Unit) which is a logical data “chunk” that is used in a protocol. It is different from a packet in that resides at the data link layer instead of the network layer. We are concerned with frames because while in sone cases you can optimize things by aligning frames with packets, reducing fragmentation (using multiple packets for a single frame), we don’t really have any way to do that in your case. So frames are the basic units we deal with for you.

(An aside: the frames we will work with are going to be transparently encapsulated in whatever the HC-12 is doing to actually move the data. Since we are working around limitations, it is not going to be possible to strictly follow the OSI model’s definitions of layer functions.)

 

Hi Y and B,
I'm afraid what you outline is far beyond my skills ( I started late in life), but thanks.

At the moment the message exchange works fine. Each message has a particular length, and if the begining and end and length are correct then it is used otherwise not. There is a time which could be counted as you mention, which I monitor when testing.

Assuming the messages are ok, how would you toggle them in order as I showed in #1?
C.

 

Hi Y and B,
I'm afraid what you outline is far beyond my skills ( I started late in life), but thanks.

At the moment the message exchange works fine. Each message has a particular length, and if the begining and end and length are correct then it is used otherwise not. There is a time which could be counted as you mention, which I monitor when testing.

Assuming the messages are ok, how would you toggle them in order as I showed in #1?
C.

I am not clear about your question. Why would then ever not be in order?

 

I am not clear about your question. Why would then ever not be in order?

Hi Y,
The BASE and REMOTE messages would be independant, if not controlled in some way.
From what I can see, there are two choices.
1/ Send a return signal to receive an 'echo' from both modules.
2/ Timed from either the GPS or perhaps the XTL would do it. Being corrected by the BASE when needed.

I'm afraid that even though I've been programming for many years, I improve only at a snails pace, occassionally going backwards, so please don't spend too much effort, in case I waste it.
This project has been developed, as you say, bit by bit patching bits on and getting them to work. It's the only way I can work, and it's far far beyond my skills, but members from this another forums, have got me this far, which is a miracle.
Cheers, C.

 

OK, so you probably need to establish a master-slave relationship.

It is impossible to synchronize them externally. Instead, you should clock it from one side or the other.

The basic protocol would be something like:

BASE buffers n CMD (Command) messages each with a serial number, then sends them to REMOTE, then an EOT (End Of Transmission) which includes the total number of CMD messages sent.

REMOTE receives the the string of CMD messages, recording each serial number, and buffering the MSG content. When it receives the EOT message, it answers with either ACK (Acknowledged) if the correct number have been received, and processes them in serial order, clearing the buffer, or sets a RETRY flag storing the state so it knows it is waiting for a RETRY, then sends a NAK (Not Acknowledged) message which includes a list of CMD serial numbers received and waits for retransmission of the missing CMD messages.

BASE receives the reply. If it is an ACK, it clears the CMD message buffer, fills it with a new set, then sends another set of CMD messages as at the start. If it is a NAK it checks the list of received CMDs and sends the missing CMD messages preceded by RETRY. if BASE doesn't receive a message within some timeout, it resends the messages as at first.

If remote is in a RETRY state but does not receive a RETRY message, it ignores any new CMD messages and resends the NAK message.

That's the basics, I am sure I have omitted some error state handling but if you spot a problem I can fill in. The main idea is the BASE is the master, and REMOTE is the slave. The state of the exchange and the current data is stored so it can be recovered. The serial numbers insure that any retries are placed in the right sequence in the CMD queue and executed in order.

If the communication breaks down it should be recoverable, and while the recovery is happening you have to decide what the REMOTE side will do (probably just stand still) but slick would be if REMOTE had received say 3 of 5 CMDs in order, it would execute those then do the RETRY, as opposed to a situation where it received messages not starting with the next serial number is in the sequence

 

OK, so you probably need to establish a master-slave relationship.

It is impossible to synchronize them externally. Instead, you should clock it from one side or the other.

The basic protocol would be something like:

BASE buffers n CMD (Command) messages each with a serial number, then sends them to REMOTE, then an EOT (End Of Transmission) which includes the total number of CMD messages sent.

REMOTE receives the the string of CMD messages, recording each serial number, and buffering the MSG content. When it receives the EOT message, it answers with either ACK (Acknowledged) if the correct number have been received, and processes them in serial order, clearing the buffer, or sets a RETRY flag storing the state so it knows it is waiting for a RETRY, then sends a NAK (Not Acknowledged) message which includes a list of CMD serial numbers received and waits for retransmission of the missing CMD messages.

BASE receives the reply. If it is an ACK, it clears the CMD message buffer, fills it with a new set, then sends another set of CMD messages as at the start. If it is a NAK it checks the list of received CMDs and sends the missing CMD messages preceded by RETRY. if BASE doesn't receive a message within some timeout, it resends the messages as at first.

If remote is in a RETRY state but does not receive a RETRY message, it ignores any new CMD messages and resends the NAK message.

That's the basics, I am sure I have omitted some error state handling but if you spot a problem I can fill in. The main idea is the BASE is the master, and REMOTE is the slave. The state of the exchange and the current data is stored so it can be recovered. The serial numbers insure that any retries are placed in the right sequence in the CMD queue and executed in order.

If the communication breaks down it should be recoverable, and while the recovery is happening you have to decide what the REMOTE side will do (probably just stand still) but slick would be if REMOTE had received say 3 of 5 CMDs in order, it would execute those then do the RETRY, as opposed to a situation where it received messages not starting with the next serial number is in the sequence

Hi Y,
I think it is kind of set up in a very similar way to your suggestion.
One thing I noticed is the ACK answer! I feel this is a waste of a message, and I think the return message is ACK enough.
If messages are missed, then do the best with what is received, perhaps by averaging or similar.
I realise that I'm a bit unconventional in my programming, and it's a good job, I'm not doing anything important
Let's see the next tests results, and if they are miles out, then I'll read your 'long' messages a bit closer, with someone who understands the jargon.
Thanks I'll report back later.
Cheers, C.

 

Hi,
I have 2x PCBs BASE REMOTE with a radio link between them.
I'm trying to find the best way to send messages between them in order.
E,G SEND->SEND->SEND->SEND->SEND-><-SEND SEND->SEND->SEND->SEND->SEND-><-SEND

At the moment there is a 'TOFRO' variable in every 5th TX and every TX, that once READ and PARSED, can be used to 'ECHO' a message back.

There is a second option: Both BASE and REMOTE have a GPS, which has a timing signal. This could be used to time all ECHOS.

There is a problem, that if any messages or GPS gets lost, then what happens.

Any thoughts?
Cheers, Camerart.

Sounds like it is a simple point to point two node networks without other traffic and you are only dealing with possible interference or lost reception on your radio links.
Problems - lost reception Or interference from the base or remote unit

Permutations:
- lost reception while sending (from either unit)
(these cases, you can't know until you don't receive an acknowledgement from the other)
(You will not know which sent blocks made it though if this is the BASE)

- lost reception while receiving
(You have to make sure your code doesn't get hung up waiting for the message to come if connection is lost and how long to wait. And how to tell the sender that an error occurred). If this is the REMOTE, you have to know when you can start replying that an error occurred because the BASE is sending several blocks.
As per timing accuracy, do you have a way to measure how accurate your clocks are on the PICs? You can use some very long, essentially low baud rate, steps to send each bit of data if timing has low accuracy.

You simply need a way to say,
A) got it, everything worked
B) didn't get it exactly right, resend please
C) nothing is working and the base fires some three tone blasts and listens then repeats until a connection is established with a three time blast from the remote only after the remote hears a three tone blast. Then start over.

this is kind of like old school smoke signals or Morse code. An old public library book (1970s era) in elementary digital communications might be a good place to start.

 

Hi Y,
I think it is kind of set up in a very similar way to your suggestion.
One thing I noticed is the ACK answer! I feel this is a waste of a message, and I think the return message is ACK enough.
If messages are missed, then do the best with what is received, perhaps by averaging or similar.
I realise that I'm a bit unconventional in my programming, and it's a good job, I'm not doing anything important
Let's see the next tests results, and if they are miles out, then I'll read your 'long' messages a bit closer, with someone who understands the jargon.
Thanks I'll report back later.
Cheers, C.

The potential problem with omitting the ACK of the ACK/NAK is that is it possible the return message could be missed as well, that means the whole system is in an unknown state with each side uncertain of the state of the other. It seems unlikely the overhead of the ACK is costly enough to do without it.

But, empirical testing well tell you if that‘s the case. So try and see.

The main concern I have is the operating band of the radios is extremely congested an interference is more likely than not. TYhat’s why a robust protocol is a good idea. But, again, test and see.

 

The potential problem with omitting the ACK of the ACK/NAK is that is it possible the return message could be missed as well, that means the whole system is in an unknown state with each side uncertain of the state of the other. It seems unlikely the overhead of the ACK is costly enough to do without it.

But, empirical testing well tell you if that‘s the case. So try and see.

The main concern I have is the operating band of the radios is extremely congested an interference is more likely than not. TYhat’s why a robust protocol is a good idea. But, again, test and see.

Hi Y and S,
Just had a thought! When I said a message could be sent instead of an ACK/NACK. If BASE sent the TIME, and REMOTE added the time into it's return message, this would be a useful message also an ACK, or if the time is too different then a NACK.
C

 

Hi Y and S,
Just had a thought! When I said a message could be sent instead of an ACK/NACK. If BASE sent the TIME, and REMOTE added the time into it's return message, this would be a useful message also an ACK, or if the time is too different then a NACK.
C

If yo have sequence numbers you can just put an ACK/NAK flag and the number of the last received message. Nut if you don't care about retrying then it would be fine.

 

If yo have sequence numbers you can just put an ACK/NAK flag and the number of the last received message. Nut if you don't care about retrying then it would be fine.

Hi Y,
I'm not fully sure I understand what you're saying, but:
There appear to be 2x choices:
Either time co-ordinated messages using GPS to correct timings at each end, or in a similar way to what you and I are saying.
Which is best?
C

 

Hi Y,
I'm not fully sure I understand what you're saying, but:
There appear to be 2x choices:
Either time co-ordinated messages using GPS to correct timings at each end, or in a similar way to what you and I are saying.
Which is best?
C

The issue with using clock time is that synchronization is extremely hard to guarantee. It is a much harder problems than you are likely to expect. There are methods for dealing with time stamps in distributed systems. For example Lamport Clocks but I would think that’s too complex for your use.

Sequence numbers are guaranteed unique, and you can think of them as a ind of timestamp. If need be, you can record the sender side clock time with sequence number so you can know when the message was sent, if that matters at all.

If you do use sequence numbers, don’t forget to handle rollover when you run out of bits. Or, you can use a large clock time index, for example year + day of the year number as the first part and allocate a big enough integer variable to the sequence number that you can’t run out of the number for any given day.

You can certainly try timestamps using clock time, but pair in mind your resolution needs to be larger than the sync error, which may mean you will have duplicates. It that happens, you can add a sequence number (though you will need far fewer bits) at the seconds level (e.g.: 065930005 would be the fifth message sent during the second starting at 6:59:30).

It might help you decide if this is necessary by testing how well you can sync the nodes over a period between GPS updates. It would be worth testing it at a long interval, and then a shorter one.

You never mentioned the expected message rate, and that‘s important too.

Just to be clear, no matter how often you update time via GPS, the timestamps generated by each of BASE and REMOTE will only agree to a finite precision. That limit is where you start adding the sequence numbers of you need to get below it.

 

The issue with using clock time is that synchronization is extremely hard to guarantee. It is a much harder problems than you are likely to expect. There are methods for dealing with time stamps in distributed systems. For example Lamport Clocks but I would think that’s too complex for your use.

Sequence numbers are guaranteed unique, and you can think of them as a ind of timestamp. If need be, you can record the sender side clock time with sequence number so you can know when the message was sent, if that matters at all.

If you do use sequence numbers, don’t forget to handle rollover when you run out of bits. Or, you can use a large clock time index, for example year + day of the year number as the first part and allocate a big enough integer variable to the sequence number that you can’t run out of the number for any given day.

You can certainly try timestamps using clock time, but pair in mind your resolution needs to be larger than the sync error, which may mean you will have duplicates. It that happens, you can add a sequence number (though you will need far fewer bits) at the seconds level (e.g.: 065930005 would be the fifth message sent during the second starting at 6:59:30).

It might help you decide if this is necessary by testing how well you can sync the nodes over a period between GPS updates. It would be worth testing it at a long interval, and then a shorter one.

You never mentioned the expected message rate, and that‘s important too.

Just to be clear, no matter how often you update time via GPS, the timestamps generated by each of BASE and REMOTE will only agree to a finite precision. That limit is where you start adding the sequence numbers of you need to get below it.

Hi Y,
First, there is a slight complication! Both BASE and MASTER receive messages from each other plus a GPS sentence into the UART RX, so have a switch between them. This could be eliminated if I could get the GPS to use SPI, which I've so far not succeeded in doing.

Both modules have a GPS which is set to 5Xmessages/sec, and also have a 1/sec output, which are in Sync, so could be used easily.

A GPS message looks like this: $GNRMC,123456.00,A,1234.76019,N,01234.93717,W,0.301,,150620,,,A*7E I have used the same format for my messages, so a $ to a W, at a known length, with comma separated values within. The begining of the GPS sentence is a time, which I could use, and would be rolled over as time does.

I hope to send messages as app 3xsec.

I think it's best to tidy up what we've got with all of these things in mind, and see what tests show.
Cheers, C.