Data Serialization & Protocol Buffers

Thread Starter

ActivePower

Joined Mar 15, 2012
155
I need to send consecutive batches of data over USB (to a PC or another uC) and I was wondering if there is a known and efficient way to do so. Right now I use a make-shift software flow control with an ACK and STOP bits but it feels clunky and inefficient. I am programming in Python on the PC and C on the uC.

I'm plotting data over USB onto a PC software so I don't want to miss data in the process while keeping up a good speed.

1. How do you send a bunch of data over serial? In other words, what is a good, lightweight way to communicate between two nodes, say a PC and a uC?

2. I was googling and came across the terms 'serialization' and 'framing' of packets which seem related but I'm not sure. I was wondering if someone could explain those.

3. Does it make sense to have the highest possible baud rate at the uC end of the loop to ensure high speed data transfer, especially over a UART-USB bridge? In other words, what is the best/optimum way to keep the data transfer speed to a good value.

4. What actually decides the working throughput of a communication channel, say USB? I don't have much background in communication systems, any links to a relevant resource(s) would be great.

5. I also came across Google's Protocol Buffers, which seem to be built for this kind of thing (for transmission of data in a network, they quote). I found an embedded port for it in NanoPB which looks promising. Is it relevant at all?

I am trying to figure out all the nuances of setting up a reliable communication link. I'be previously gotten by with half-baked adhoc protocols and stuff but I was looking for something better this time.

Any help would be much appreciated.

Thanks!
 
Last edited:

djsfantasi

Joined Apr 11, 2010
9,156
What is the software in your μC written in? Same question for your PC?

Why invent your own protocols if the standard serial communications work? I asked about the software, because I would expect that both sides would have libraries of code available to do this already.

I've done limited communications from a PC with USB to a μC RS232 port in this manner. I defined the data format as desired, and formatted/parsed it in software.

(Here is a link to the software: http://back2basic.phatcode.net/?Issue_#6:BASIC_programming_for_Multi-Tasking_Control )
 

GetDeviceInfo

Joined Jun 7, 2009
2,192
3. Does it make sense to have the highest possible baud rate at the uC end of the loop to ensure high speed data transfer, especially over a UART-USB bridge? In other words, what is the best/optimum way to keep the data transfer speed to a good value.
Drop the bridge and use a uC with USB.
 

Thread Starter

ActivePower

Joined Mar 15, 2012
155
Drop the bridge and use a uC with USB.
I would have if not for the task of writing custom device driver for my hardware. I need to transfer bulk of data and from what it looks like the CDC class would have to be customized, doesn't it?

In any case, I don't have access to an affordable USB-enabled uC I could prototype with (I do have a FTDI board to spare). So, I figured I'd stick with the bridge at least for the initial designs.

@djsfantasi:
Should I just send a packet with a couple of bytes of data with a giant 'switch' controlling the flow?
 

djsfantasi

Joined Apr 11, 2010
9,156
I don't know what you mean by a "giant switch"? What are you switching? I thought one PC to one uC which doesn't require switching. I thought switching between TX and RX, but don't see that as necessary as serial comms can be duplex.

I would just send the bytes of data necessary. You control the protocol at both ends, so you can code whatever validation, verification and acknowledgement you need. By this I mean, you aren't communicating with a device that REQUIRES a specific protocol.

I apologize if I am being vague, but I've done this myself many times. If you are connecting via a serial comms port with built in software functions, then as far as your programs are concerned, all they see is a string of data, just as if they read it from ROM. I know you said you wanted to use something better than the half-baked schemes you've used in the past, but what are you defining as "better"?
 

Thread Starter

ActivePower

Joined Mar 15, 2012
155
@djsfantasi:
I'm sorry I must have been more clear in my writing. I meant to imply the C switch statement. Guess everyone is not a mind-reader after all :)

I have done the type of serial communication sending predefined ASCII to indicate the start and stop of packets and commands necessary in the past too. I was just wondering if there is a more err.. standardized way to do so. I'm sorry I was a bit too vague regarding my question in the OP but I am working on a class project now and I thought maybe I could learn 'how everyone else does it' through the project.

Anyways, I think I'll stick to a homebrew protocol for now. I read somewhere that software state machines could be used for parsing frames/packets in a protocol. Is that the way you do it? Do you have any code pointers I could look at?

Thanks for all the help :)
 

THE_RB

Joined Feb 11, 2008
5,438
What's the difficulty? Just split the data into strings of X bytes, add a header byte pair before the string (hopefully a byte pair that does not appear often in the data) and a checksum after the string.

Your only real issue is if receiving starts during the middle of a string transmission, but it should resync fine on the start of the next string.

I'm not sure what you see as the potential problem with sending/receiving the data, so if you outline that then we can suggest good solutions to that problem.
 
Top