Oshonsoft programs with INTERRUPTS and PARSE

Thread Starter

camerart

Joined Feb 25, 2013
3,830
3 channels - Nice!

I think the misalignment in the terminal readout is due to the radio being half-duplex i.e. can't transmit and receive at the same time. The HC-12 has some delay required to change directions which is detailed in the datasheet. It takes some time after the end of a message in one direction to be ready to send in the other direction. It's also possible that it is reporting the results of one of the other channels when the PC decides to send the $REMOTE message. That will also cause some losses. For diagnostics, it's just annoying. If you actually require 2 way comms in the final code, you'll have to figure out a way to coordinate the sending of $REMOTE messages with the receiving back from the PIC. If that's a real requirement, we can cogitate on that. Otherwise, I'd live with it.

strbuf = "" is necessary where it is. You need to start with a clean string before you start building a new one. The best place to init something is just before you use it. If you want to generate errors, add a piece of code to poke bad characters into the string after building it or set a breakpoint and modify the string manually.

Not sure what you mean. The code just reports what it receives. It might be faster because it isn't burdened with a bunch of blocking delays but if your GPS(?) is sending the time string at 2/sec then you are guaranteed to get time reports with the the same time value.

Carry on.
Hi JT,
I'll live with the radio anomaly, no problem.

I mentioned STRBUF = "" because once a sentence has been PARSED, all of STRBUF left until cleared has bad characters really, so adding different ones won't help. The error only happened in SIM, when I was sending messed up sentences, but I didn't see it LIVE, so perhaps not a real error. Generally the messages are pure, apart from starting in the middle of a sentence, but the system takes care of this. I'll leave it for now.

All of the 3x messages are transmitting at 5/SEC. I was happy to see the GPS Time at 2/SEC, which shows that the whole sequence went round 2x that second.

So it is looking pretty good. Once I add it into the MAIN program along with the SPI INPUTS, I can see what STRINGs are being saved.

Have you now finished? If so, I'll add it into the MAIN program, but then it is much harder to work on.
Cheers, C.
 
Last edited:

Thread Starter

camerart

Joined Feb 25, 2013
3,830
Hi JT,
I was shown how to measure times by 'J' a while ago, so I thought I'd try it for your INTERRUPT routine.

I hope I've got it correct?
Add a BREAK just after the SAVE SYSTEM in the INTERRUPT, and another just before RESUME.

Run the SIM, and add a $Sentence into hardware UART. When the program BREAKs, click the REAL TIME DURATION and it will set to Zero, then speed the rate up again, until the next BREAK, and read the Duration.
I make the INTERRUPT 18.5uS then remove the bottom BREAK and run again, this will give the time between INTERRUPTs. I make that 58.5uS, which then gives the fraction of the digit READ the INTERRUPT takes. Approx 1/4.
C.
 

JohnInTX

Joined Jun 26, 2012
4,787
18.5uS to process one interrupt is believeable and within reason. I don't think 58.5uS between interrupts is correct unless we have problems. The interrupt is designed to happen ONCE per character. At 9600baud, it takes at least 1.04 msec per character so that would be the interrupt rate, all things being working as expected. Thanks to some timely advice from some other members, I've found that you can change the simulated transmit rate to speed it up. I would look there first.
When I can, I'll see what it says on my end.

I would recommend that you resolve the known deficiency in 'parse' re: decoding the misspelled $QEID tag. Maybe adding the checks on the length of the received buffer would be sufficient.
 

JohnInTX

Joined Jun 26, 2012
4,787
Measuring myself I get 46-51uS execution time for the interrupt routine depending on where in the stream it is. About half of that is the context save and restore the the compiler adds which might explain the difference between our readings. Even so, that's about 5% of the max possible interrupt rate at 9600baud so that's OK.

Time between interrupts is not valid in the sim as it is directly influenced by the UART Transmit/Receive Time setting. In the real world it will be at least 1.04ms because of the baud rate.

I'd say punch up 'parse' as we discussed, do a cleanup pass and you're ready to proceed. Depending on the final data types you want from the $messages, you might be able to streamline some of those conversions but if it's keeping up with what you need speed-wise, you can defer that too.
 

jjw

Joined Dec 24, 2013
823
Hi JT,
I was shown how to measure times by 'J' a while ago, so I thought I'd try it for your INTERRUPT routine.

I hope I've got it correct?
Add a BREAK just after the SAVE SYSTEM in the INTERRUPT, and another just before RESUME.

Run the SIM, and add a $Sentence into hardware UART. When the program BREAKs, click the REAL TIME DURATION and it will set to Zero, then speed the rate up again, until the next BREAK, and read the Duration.
I make the INTERRUPT 18.5uS then remove the bottom BREAK and run again, this will give the time between INTERRUPTs. I make that 58.5uS, which then gives the fraction of the digit READ the INTERRUPT takes. Approx 1/4.
C.
Save System and Resume belong to the interrupt, so the Breaks should be before Save System and after Resume.
 

JohnInTX

Joined Jun 26, 2012
4,787
@camerart
Looking at 'parse' we have a couple of problems. I recall that you and others were wrestling with some of this earlier but I'm just getting down to it..
The RX buf is an array of bytes, not a string. The MidStr actions in parse opeate on a string variable. As a quick fix, I wrote a loop to copy the buf byte by byte to strbuf. As a string, now you could use MidStr et.al to operate on a copy of the buffer contents.
Two big problems:
It takes an insane amount of time to use normal string functions to copy bytes from the RX buf into strbuf - 8.483msec for the 47 chars in $GNRMC. :eek: After that it 'only' took and additional 1.137ms to decode and parse the string into all the other sub-strings. Wow. I rewrote the copy function and it's a much better 644uS. But that brings up problem 2, is one you've seen before i.e. to do parse like you have it you have to store the whole message in a string making ALL strings 45+ chars and I believe you were having RAM problems there.

Since parse needs work anyway I propose we don't make a string copy of the buffer. That means no MidStr but I'm pretty sure you can extract selected data from the raw buffer into shorter strings and add a string terminator to the end. That way, you can have shorter strings overall while keeping a big char array buffer and you don't spend all day copying data.

As it currently is, 'parse' clobbers strbuf because it copies more characters than your default string size of 40 characters so....

I think I'll play with it awhile. Just know that it's not ready for prime time.
 

Thread Starter

camerart

Joined Feb 25, 2013
3,830
@camerart
Looking at 'parse' we have a couple of problems. I recall that you and others were wrestling with some of this earlier but I'm just getting down to it..
The RX buf is an array of bytes, not a string. The MidStr actions in parse opeate on a string variable. As a quick fix, I wrote a loop to copy the buf byte by byte to strbuf. As a string, now you could use MidStr et.al to operate on a copy of the buffer contents.
Two big problems:
It takes an insane amount of time to use normal string functions to copy bytes from the RX buf into strbuf - 8.483msec for the 47 chars in $GNRMC. :eek: After that it 'only' took and additional 1.137ms to decode and parse the string into all the other sub-strings. Wow. I rewrote the copy function and it's a much better 644uS. But that brings up problem 2, is one you've seen before i.e. to do parse like you have it you have to store the whole message in a string making ALL strings 45+ chars and I believe you were having RAM problems there.

Since parse needs work anyway I propose we don't make a string copy of the buffer. That means no MidStr but I'm pretty sure you can extract selected data from the raw buffer into shorter strings and add a string terminator to the end. That way, you can have shorter strings overall while keeping a big char array buffer and you don't spend all day copying data.

As it currently is, 'parse' clobbers strbuf because it copies more characters than your default string size of 40 characters so....

I think I'll play with it awhile. Just know that it's not ready for prime time.
Hi JT,
I'll need to read this a couple of times, to follow it.

Regarding LEFT STRING MID STRING and RIGHT STRING. While I was testing programs a while ago, I found LEFT and RIGHT stopped working, which may be an issue with Oshonsoft. Oshonsoft may have fixed it, but it seemed to be intermittent, so beware.

Enjoy playing, no hurry.
C.
 

JohnInTX

Joined Jun 26, 2012
4,787
Hi J,
Thanks for the correction, I'll try again tomorrow.
C
Try this one.
I added a few utilities that extract raw characters from the RX buf and build OSH strings - like MidStr() but the buf doesn't have to be a string. That means that you don't have to run a default string size big enough for the RX buf.
I also added a raw buf to string compare function 'MatchString' that compares raw characters in the buf to a fixed string like $REMOTE or $QEIDEG. That way your bogus $QEID string will not be taken as valid.
Still no string length comparisons yet but we can add that.
Parse runs much faster than before: from 317uS to 924uS depending on the $sentence. Very snappy.
Your longitude in
$GNRMC,123456.80,A,1234.77326,N,12345.94201,W
is extracted as 2345.942 (missing the leading '1'). If that is not what you intended you can modify the buffer index parameter in the call to AddStr in parse.
Misc. cleanup.

Have fun.
 

Attachments

Last edited:

Thread Starter

camerart

Joined Feb 25, 2013
3,830
Hi JT,
Try this one.
I added a few utilities that extract raw characters from the RX buf and build OSH strings - like MidStr() but the buf doesn't have to be a string. That means that you don't have to run a default string size big enough for the RX buf.
I also added a raw buf to string compare function 'MatchString' that compares raw characters in the buf to a fixed string like $REMOTE or $QEIDEG. That way your bogus $QEID string will not be taken as valid.
Still no string length comparisons yet but we can add that.
Parse runs much faster than before: from 317uS to 924uS depending on the $sentence. Very snappy.
Your longitude in
$GNRMC,123456.80,A,1234.77326,N,12345.94201,W
is extracted as 2345.942 (missing the leading '1'). If that is not what you intended you can modify the buffer index parameter in the call to AddStr in parse.
Misc. cleanup.

Have fun.
Hi JT,
The example GPS, REMOTE and QEIDEG are for SIM, but all change each LIVE READing, I hope this is what you understand. I don't understand the missing '1' as that is the equivalent to the '1' in e,g 12:00 oclock, so 12hours 34mins and 45 seconds.
4am so back to bed :)
C.
 

Thread Starter

camerart

Joined Feb 25, 2013
3,830
Try this one.
I added a few utilities that extract raw characters from the RX buf and build OSH strings - like MidStr() but the buf doesn't have to be a string. That means that you don't have to run a default string size big enough for the RX buf.
I also added a raw buf to string compare function 'MatchString' that compares raw characters in the buf to a fixed string like $REMOTE or $QEIDEG. That way your bogus $QEID string will not be taken as valid.
Still no string length comparisons yet but we can add that.
Parse runs much faster than before: from 317uS to 924uS depending on the $sentence. Very snappy.
Your longitude in
$GNRMC,123456.80,A,1234.77326,N,12345.94201,W
is extracted as 2345.942 (missing the leading '1'). If that is not what you intended you can modify the buffer index parameter in the call to AddStr in parse.
Misc. cleanup.

Have fun.
Hi JT,
I tried in SIM, and it's working well. I can't try LIVE yet, until the DATASWITCH is switching.

In previous programs, if 1x of the $sentences is not there, for example out of radio range or no GPS signal, then the other sentences are still processed, this is necessary, which is better if possible.
The reason: First the BASE (The one we're working on) is almost the same as the REMOTE, apart from e,g, BASE has joysticks and a compass knob. e,g, REMOTE has motor control and a battery sensor.
The BASE sends a location, altitude and direction, and hopefully the REMOTE goes there. 'Say' the GPS signal is lost then the REMOTE should stay at that location until further orders, but still may be receiving sentences from the BASE, that may be 'return' home, which would envoke a return home routine.

Your latest 'no STRING' routines have left me behind, and will take me a long time in the SIM to 'get' them, which I'll have to do once it's finalised, but excellent stuff :)

C.
 

JohnInTX

Joined Jun 26, 2012
4,787
DATASWITCH should work OK if you uncomment the call to next_channel:
Code:
    'Gosub next_channel  TESTS: don't change channel, stay on REMOTE <<<---------------- UNCOMMENT THIS
    Gosub setUARTchannel  'start UART on channel(datasw), begins looking for sentence
Goto main
In previous programs, if 1x of the $sentences is not there, for example out of radio range or no GPS signal, then the other sentences are still processed, this is necessary, which is better if possible.
I wondered about that myself. I'd consider an interrupt driven timer to guard the messages. What did you use? Maybe we can incorporate that into this code.

At this point the 'No String' helper routines can be treated as black boxes. As long as you understand the how to call them. I tried to comment them so that even if you were unfamiliar with the proc and function constructs you could still see how they work.
 

Thread Starter

camerart

Joined Feb 25, 2013
3,830
DATASWITCH should work OK if you uncomment the call to next_channel:
Code:
    'Gosub next_channel  TESTS: don't change channel, stay on REMOTE <<<---------------- UNCOMMENT THIS
    Gosub setUARTchannel  'start UART on channel(datasw), begins looking for sentence
Goto main
I wondered about that myself. I'd consider an interrupt driven timer to guard the messages. What did you use? Maybe we can incorporate that into this code.

At this point the 'No String' helper routines can be treated as black boxes. As long as you understand the how to call them. I tried to comment them so that even if you were unfamiliar with the proc and function constructs you could still see how they work.
Hi JT,
Ah yes, I forgot the uncomment.

At this stage, I don't think it matters about the 3x channels, as long as we address it sometime. I think it was just the route the CODE took that allow a channel to be missing.

Ok, I'll just accept your 'black boxes' for now. Later, once the CODE has settled down a bit, I can step though it in the SIM.
Cheers, C.
 

Thread Starter

camerart

Joined Feb 25, 2013
3,830
Hi JT,
Just tried a LIVE test and everthing worked, but the memory battery on the GPS went flat, so the GPS READings aren't as fast as before. It's a rechargeable cell, so this shouldn't happen.
I'll have to re-program it.
C.
 

JohnInTX

Joined Jun 26, 2012
4,787
Hi JT,
Just tried a LIVE test and everthing worked, but the memory battery on the GPS went flat, so the GPS READings aren't as fast as before. It's a rechargeable cell, so this shouldn't happen.
I'll have to re-program it.
C.
There is an old programmer's adage that says 'You know when you're getting close to good software when the hardware begins to fail'. So, good news, I guess ;)
 

JohnInTX

Joined Jun 26, 2012
4,787
Here's hopefully the penultimate update. It's a cleanup pass plus:
Channel switching is ON.
Adds length checking to the 3 $entences. See parse for list of values used and modify those values if you need to.
Re-ordered some init stuff fix some issues notably in IO and UART inits. Starts with GPS channel and is ready to receive before the LEDs test so you don't lose a message while delaying for LEDs.
Relocated the UART channel switching to right after parse. That way, it can start on the next sentence while you are finishing the processing of the parsed data from the previous one.
Shortened default string size to 20. It looks like we have a lot of short strings only. Change as req'd
Deleted old variables, renamed some others to better describe their functions.
Added a parse GOOD flag for future reception monitoring. See next.

About all that is left is to decide how/if you want to handle missing or corrupted receptions. Retry, guard timer, ??? I agree that you need something to keep it from locking up waiting on a failed channel.
When that's in the bag, you'll have a solid little receiver/decoder and can add whatever magic you want from there.
Whew!
 

Attachments

Thread Starter

camerart

Joined Feb 25, 2013
3,830
Here's hopefully the penultimate update. It's a cleanup pass plus:
Channel switching is ON.
Adds length checking to the 3 $entences. See parse for list of values used and modify those values if you need to.
Re-ordered some init stuff fix some issues notably in IO and UART inits. Starts with GPS channel and is ready to receive before the LEDs test so you don't lose a message while delaying for LEDs.
Relocated the UART channel switching to right after parse. That way, it can start on the next sentence while you are finishing the processing of the parsed data from the previous one.
Shortened default string size to 20. It looks like we have a lot of short strings only. Change as req'd
Deleted old variables, renamed some others to better describe their functions.
Added a parse GOOD flag for future reception monitoring. See next.

About all that is left is to decide how/if you want to handle missing or corrupted receptions. Retry, guard timer, ??? I agree that you need something to keep it from locking up waiting on a failed channel.
When that's in the bag, you'll have a solid little receiver/decoder and can add whatever magic you want from there.
Whew!
Hi JT,
I'll fix the GPS module before checking, but thanks.

I replaced the GPS cell, re-programmed it, and left it overnight. It forgot the settings. i,e, 5x/SEC and only 'see' the NMEA $GNRMC sentence. There's 7x others. I'll re-try in case I didn't actually SAVE it, otherwise perhaps the GPS RAM has conked.

I wonder if the HSEROUTs could be done while UART is OFF, which you may have fixed above, not sure. Which will hopefully tidy up the Terminal view. Perhaps not important!

As this is the BASE Transmitter, the GPS should not block, or there's something really bad, so perhaps we could use the TIME to switch the DATASWITCH if it locks. I'm testing here indoors, and if it's cloudy, sometimes the reception struggles a bit.
C.
 

Thread Starter

camerart

Joined Feb 25, 2013
3,830
Hi JT,
GPS at 5x/SEC again. I may have missed a tick somewhere?

Result from #269:
It's working pretty well.

I notice the REMOTE doesn't always update, but I'll see what happens after today's program.
Cheers, C.
 

Attachments

JohnInTX

Joined Jun 26, 2012
4,787
Busy today but Hserout can transmit with the UART ‘off’. We only shut down the RX. Maybe the new parse tests are rejecting corrupt Remote messages?

Agreed on the need for a guard timer on the messages.
 
Top