Generating a Unique Code in Data Stream

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
I want a simple method to generate a unique code within a data stream. This code would frame certain data, like packet size. Here is my attempt to do that:

The code would simply be 0x00_01
Now, this code would certainly appear in the data stream natively, so as not to transmit the code where it's not needed, I propose to do this:

If the code 0x00_01 appears in the data stream, I add a byte to make the code 0x00_02_01. The receiver removes the 0x02 byte and recovers the original data.

But, the code 0x00_02 might be in the stream as well, and so I repeat the process and generate: 0x00_03_02.... I repeat this as many times as needed to encode the stream and advoid sending the unique code. For example, if the data stream contains 0x00_03_02_01, then I transmit:

0x00_04_03_03_02_02_01

To decode the data, I slide along the stream testing each byte pair for the condition of decrementing values, until that condition is false, then I save the last value, thus:

Rich (BB code):
0x00_04_03_03_02_02_01
         T  F  T  F  T
If I save all 'F' bytes, I have the original data 0x00030201 -- the lase byte "01" is actually T, but I save it anyway, since it's the last byte.

So, does this make sense? Does anyone spot a flaw in the logic? Is there a better or easier way to accomplish this?

Thanks!
 

Markd77

Joined Sep 7, 2009
2,806
You could use a similar scheme to html. The & character is used to display special characters like μ (&micro) but to display & it uses &amp. If you want to display &amp, you just use &ampamp.
<ed> To elaborate, you encode any real 0x00 in the data as 0x00_01, and your unique code you use 0x00_02. </ed>
 
Last edited:

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
I like it. That might actually be a little safer. I think I'll run tests on both methods. Any else have any ideas?
 

nsaspook

Joined Aug 27, 2009
7,356
I like it. That might actually be a little safer. I think I'll run tests on both methods. Any else have any ideas?
There are few other methods to send out of band data. One method is to send breaks and use a framing error/break detect to signal the next codes will be out of band, another method is the use 9 bit serial transmission. If the 9th bit is low then it's normal digital data but if the 9th bit is high then the 8 bits will contain out of band data. (an address or anything else)

http://ww1.microchip.com/downloads/en/appnotes/00774a.pdf
http://electronicdesign.com/article/communications/an-xon-xoff-like-in-band-flow-control-method-for-b
 

THE_RB

Joined Feb 11, 2008
5,438
Format your data into small packets of a fixed size with a control byte at the start of each packet.

Then the control byte tells whether that packet contains data or some control code or other structure. It's fast, reliable and professional.

Just sticking something anywhere in the middle of a "data stream" to me sounds like the wrong way to do it.
 

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
That's what I'm trying to do ( except my packets are vairable length ) I only want to know that my control codes doesn't show up in the data stream as regular data.
 

nsaspook

Joined Aug 27, 2009
7,356
Packet or hardware methods all depend on your program logic knowing when regular data or signal data is expected. Your control codes showing up in normal data will not be a problem if the program is in the "all codes are normal data" mode during the regular message part of the packet or data stream. Once a full message is received the program just needs to switch to "all codes are signal data" mode to look for the next sequence.

Start in control code mode
BOM: control mode, begin packet
LEN: control mode, length of TXT, now switch to "all codes are normal data" for LEN bytes
TXT: data, at LEN bytes switch back to control code mode
EOM: control, end of packet

If everything is perfect then you don't need to worry about data latency, data corruption, timeouts, retransmissions, CRC, preemption and a lot of other messy details.

http://en.wikipedia.org/wiki/Packetized_elementary_stream
 

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
I've worked with packetized elementary streams before. I had to parse out various start codes from the stream. There is no "all codes" or "all data" modes to the programs. Header information is simply perpended to the data within the packet, and the start codes are unique codes within the stream, and there is never any chance that those codes will be in the data area of the stream. My purpose in this project is to encode a data stream in a similar manner. It's not MPEG, but rather a message data stream for a military customer.

Thanks!
 
Last edited:

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
I have no desire to reinvent anything. I am only looking for a simple way to insert a unique code into a data stream.

ED: SLIP sounds alot like the suggestion from post #2.

If
a data byte is the same code as END character, a two byte sequence of
ESC and octal 334 (decimal 220) is sent instead. If it the same as
an ESC character, an two byte sequence of ESC and octal 335 (decimal
221) is sent instead.
 
Last edited:

Thread Starter

Brownout

Joined Jan 10, 2012
2,390
There are few other methods to send out of band data. One method is to send breaks and use a framing error/break detect to signal the next codes will be out of band, another method is the use 9 bit serial transmission. If the 9th bit is low then it's normal digital data but if the 9th bit is high then the 8 bits will contain out of band data. (an address or anything else)

http://ww1.microchip.com/downloads/en/appnotes/00774a.pdf
http://electronicdesign.com/article/communications/an-xon-xoff-like-in-band-flow-control-method-for-b
I missed this post before ( IE is really acting up for some reason ) I like the idea of using the 9th bit for OOB data.
 

nsaspook

Joined Aug 27, 2009
7,356
I've worked with packetized elementary streams before. I had to parse out various start codes from the stream. There is no "all codes" or "all data" modes to the programs. Header information is simply perpended to the data within the packet, and the start codes are unique codes within the stream, and there is never any chance that those codes will be in the data area of the stream. My purpose in this project is to encode a data stream in a similar manner. It's not MPEG, but rather a message data stream for a military customer.

Thanks!
So you want to superimpose/piggback another data stream into the data part of the packet only using the codes that are valid for data by using character or byte stuffing?

http://www.cs.umd.edu/~shankar/417-F01/Slides/chapter5c-aus/sld016.htm

Two flag bytes in the data section could mark the start and end of packets in the data section.

transmit encoded
TXT: data text
SOM: if a flag byte is in the stream send twice for data
SOM: two flag bytes makes one data byte, data text
TXT: data text
SOM: flag byte for start of packet in the data section
TXT: packet text
SOM: if a flag byte is in the stream send twice for data
SOM: two flag bytes makes one data byte, packet text
TXT: packet text
EOM: if a flag byte is in the stream send twice for data
EOM: two flag bytes makes one data byte, packet text
TXT: packet text
EOM: flag byte end packet
TXT: data text

receive decoded
TXT: data text
SOM: data text
TXT: data text
TXT: packet text
SOM: packet text
TXT: packet text
EOM: packet text
TXT: packet text
TXT: data text
 

nsaspook

Joined Aug 27, 2009
7,356
I missed this post before ( IE is really acting up for some reason ) I like the idea of using the 9th bit for OOB data.
I've used that method before to send binary C structure data via a RS-232 link from PIC to PIC uC. Nothing in life is free so there is a reduction in the baud (the rate of actual information) rate.


PIC C18 snippet
Rich (BB code):
    TXSTA1bits.TX9D = LOW; // clear bit 9 for data
    P1wait();
    TXREG1 = *mbmcflag.data_ptr; // send data
    mbmcdata_count++;
    P1wait();
    mbmcflag.data_pos++; // move the data pointer
    if (mbmcflag.data_pos >= mbmcflag.data_len) { // buffer has been sent
        TXSTA1bits.TX9D = HIGH; // set bit 9 for command
        P1wait();
        TXREG1 = HOST_ACK; // send ack
        mbmcdata_count++;
        P1wait();
        TXSTA1bits.TX9D = LOW;
        mbmcflag.cmd_timeout = FALSE; // there is another system
        mbmcflag.mbmc_cmd = HOST_REQ; // set to default cmd
        HOST_COMM = FALSE; // clear the transmit flag
        HOST_BUSY = FALSE; // clear the command flag
    } else {
        mbmcflag.data_ptr++; // move the buffer pointer position
    }
PIC32 code snippet
Rich (BB code):
void __ISR(_UART2_VECTOR, ipl2) IntUart2Handler(void) {
    static WORD rx9data;
    // Is this an RX interrupt?
    if (INTGetFlag(INT_SOURCE_UART_RX(UART2))) {
        rx9data = ReadUART2();
        mbmc_status.received++;

        if ((rx9data & MASK9BIT) != 0) { // check for bit 9
            mbmcflag.rx_9bit = TRUE; // set command  flag
            mbmc_status.cmdreceived++;
        } else {
            mPORTDToggleBits(BIT_2); // data LED
            mbmcflag.data_timeout = 0; // reset timer
            mbmcflag.rx_9bit = FALSE; // clear command flag

            if (!mbmcflag.mbmc_done) { // we need more data
                mbmcflag.data_ptr[mbmcflag.data_pos] = (BYTE) rx9data; // move received data into buffer
                if (mbmcflag.data_pos++ >= mbmcflag.data_len) { // the data buffer is filled to data_len
                    mbmcflag.mbmc_done = TRUE; // data block has been received
                    // set a place keeper buffer
                    mbmcflag.data_ptr = (BYTE*) & dd.data_default;
                    mbmcflag.data_len = 4;
                    mbmcflag.data_pos = 0;
                } else { // nothing for now
                }
            }

        }

        if (mbmcflag.rx_9bit) {
            if (rx9data == mbmcflag.host_ack) {
                mbmc_status.ackreceived++;
                mPORTDToggleBits(BIT_1); // command LED
                mbmcflag.host_done = TRUE;
                mbmcflag.data_timeout = 0; // reset timer
            }
        }
        // Clear the RX interrupt Flag
        INTClearFlag(INT_SOURCE_UART_RX(UART2));

    }

    // We don't care about TX interrupt
    if (INTGetFlag(INT_SOURCE_UART_TX(UART2))) {
        INTClearFlag(INT_SOURCE_UART_TX(UART2));
    }
}
 
Top