Generating a Unique Code in Data Stream

Discussion in 'Programmer's Corner' started by Brownout, Mar 20, 2012.

  1. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    I want a simple method to generate a unique code within a data stream. This code would frame certain data, like packet size. Here is my attempt to do that:

    The code would simply be 0x00_01
    Now, this code would certainly appear in the data stream natively, so as not to transmit the code where it's not needed, I propose to do this:

    If the code 0x00_01 appears in the data stream, I add a byte to make the code 0x00_02_01. The receiver removes the 0x02 byte and recovers the original data.

    But, the code 0x00_02 might be in the stream as well, and so I repeat the process and generate: 0x00_03_02.... I repeat this as many times as needed to encode the stream and advoid sending the unique code. For example, if the data stream contains 0x00_03_02_01, then I transmit:

    0x00_04_03_03_02_02_01

    To decode the data, I slide along the stream testing each byte pair for the condition of decrementing values, until that condition is false, then I save the last value, thus:

    Code ( (Unknown Language)):
    1.  
    2. 0x00_04_03_03_02_02_01
    3.          T  F  T  F  T
    4.  
    If I save all 'F' bytes, I have the original data 0x00030201 -- the lase byte "01" is actually T, but I save it anyway, since it's the last byte.

    So, does this make sense? Does anyone spot a flaw in the logic? Is there a better or easier way to accomplish this?

    Thanks!
     
  2. Markd77

    Senior Member

    Sep 7, 2009
    2,803
    594
    You could use a similar scheme to html. The & character is used to display special characters like μ (&micro) but to display & it uses &amp. If you want to display &amp, you just use &ampamp.
    <ed> To elaborate, you encode any real 0x00 in the data as 0x00_01, and your unique code you use 0x00_02. </ed>
     
    Last edited: Mar 20, 2012
  3. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    I like it. That might actually be a little safer. I think I'll run tests on both methods. Any else have any ideas?
     
  4. nsaspook

    AAC Fanatic!

    Aug 27, 2009
    2,908
    2,169
    There are few other methods to send out of band data. One method is to send breaks and use a framing error/break detect to signal the next codes will be out of band, another method is the use 9 bit serial transmission. If the 9th bit is low then it's normal digital data but if the 9th bit is high then the 8 bits will contain out of band data. (an address or anything else)

    http://ww1.microchip.com/downloads/en/appnotes/00774a.pdf
    http://electronicdesign.com/article...n-xoff-like-in-band-flow-control-method-for-b
     
  5. THE_RB

    AAC Fanatic!

    Feb 11, 2008
    5,435
    1,305
    Format your data into small packets of a fixed size with a control byte at the start of each packet.

    Then the control byte tells whether that packet contains data or some control code or other structure. It's fast, reliable and professional.

    Just sticking something anywhere in the middle of a "data stream" to me sounds like the wrong way to do it.
     
  6. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    That's what I'm trying to do ( except my packets are vairable length ) I only want to know that my control codes doesn't show up in the data stream as regular data.
     
  7. nsaspook

    AAC Fanatic!

    Aug 27, 2009
    2,908
    2,169
    Packet or hardware methods all depend on your program logic knowing when regular data or signal data is expected. Your control codes showing up in normal data will not be a problem if the program is in the "all codes are normal data" mode during the regular message part of the packet or data stream. Once a full message is received the program just needs to switch to "all codes are signal data" mode to look for the next sequence.

    Start in control code mode
    BOM: control mode, begin packet
    LEN: control mode, length of TXT, now switch to "all codes are normal data" for LEN bytes
    TXT: data, at LEN bytes switch back to control code mode
    EOM: control, end of packet

    If everything is perfect then you don't need to worry about data latency, data corruption, timeouts, retransmissions, CRC, preemption and a lot of other messy details.

    http://en.wikipedia.org/wiki/Packetized_elementary_stream
     
  8. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    I've worked with packetized elementary streams before. I had to parse out various start codes from the stream. There is no "all codes" or "all data" modes to the programs. Header information is simply perpended to the data within the packet, and the start codes are unique codes within the stream, and there is never any chance that those codes will be in the data area of the stream. My purpose in this project is to encode a data stream in a similar manner. It's not MPEG, but rather a message data stream for a military customer.

    Thanks!
     
    Last edited: Mar 22, 2012
  9. Papabravo

    Expert

    Feb 24, 2006
    10,144
    1,790
    Look at the RFC for "SLIP". That's the Serial Link Internet Protocol. I believe it is RFC #1055 by Romkey. When you need something for passing binary data in a transparent fashion it is hard to beat. Why reinvent the wheel?

    In fact here it is:
    http://www.ietf.org/rfc/rfc1055.txt
     
  10. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    I have no desire to reinvent anything. I am only looking for a simple way to insert a unique code into a data stream.

    ED: SLIP sounds alot like the suggestion from post #2.

     
    Last edited: Mar 22, 2012
  11. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    I missed this post before ( IE is really acting up for some reason ) I like the idea of using the 9th bit for OOB data.
     
  12. nsaspook

    AAC Fanatic!

    Aug 27, 2009
    2,908
    2,169
    So you want to superimpose/piggback another data stream into the data part of the packet only using the codes that are valid for data by using character or byte stuffing?

    http://www.cs.umd.edu/~shankar/417-F01/Slides/chapter5c-aus/sld016.htm

    Two flag bytes in the data section could mark the start and end of packets in the data section.

    transmit encoded
    TXT: data text
    SOM: if a flag byte is in the stream send twice for data
    SOM: two flag bytes makes one data byte, data text
    TXT: data text
    SOM: flag byte for start of packet in the data section
    TXT: packet text
    SOM: if a flag byte is in the stream send twice for data
    SOM: two flag bytes makes one data byte, packet text
    TXT: packet text
    EOM: if a flag byte is in the stream send twice for data
    EOM: two flag bytes makes one data byte, packet text
    TXT: packet text
    EOM: flag byte end packet
    TXT: data text

    receive decoded
    TXT: data text
    SOM: data text
    TXT: data text
    TXT: packet text
    SOM: packet text
    TXT: packet text
    EOM: packet text
    TXT: packet text
    TXT: data text
     
  13. Brownout

    Thread Starter Well-Known Member

    Jan 10, 2012
    2,375
    998
    ^- yeah I get it. All good stuff.
     
  14. nsaspook

    AAC Fanatic!

    Aug 27, 2009
    2,908
    2,169
    I've used that method before to send binary C structure data via a RS-232 link from PIC to PIC uC. Nothing in life is free so there is a reduction in the baud (the rate of actual information) rate.


    PIC C18 snippet
    Code ( (Unknown Language)):
    1.     TXSTA1bits.TX9D = LOW; // clear bit 9 for data
    2.     P1wait();
    3.     TXREG1 = *mbmcflag.data_ptr; // send data
    4.     mbmcdata_count++;
    5.     P1wait();
    6.     mbmcflag.data_pos++; // move the data pointer
    7.     if (mbmcflag.data_pos >= mbmcflag.data_len) { // buffer has been sent
    8.         TXSTA1bits.TX9D = HIGH; // set bit 9 for command
    9.         P1wait();
    10.         TXREG1 = HOST_ACK; // send ack
    11.         mbmcdata_count++;
    12.         P1wait();
    13.         TXSTA1bits.TX9D = LOW;
    14.         mbmcflag.cmd_timeout = FALSE; // there is another system
    15.         mbmcflag.mbmc_cmd = HOST_REQ; // set to default cmd
    16.         HOST_COMM = FALSE; // clear the transmit flag
    17.         HOST_BUSY = FALSE; // clear the command flag
    18.     } else {
    19.         mbmcflag.data_ptr++; // move the buffer pointer position
    20.     }
    PIC32 code snippet
    Code ( (Unknown Language)):
    1. void __ISR(_UART2_VECTOR, ipl2) IntUart2Handler(void) {
    2.     static WORD rx9data;
    3.     // Is this an RX interrupt?
    4.     if (INTGetFlag(INT_SOURCE_UART_RX(UART2))) {
    5.         rx9data = ReadUART2();
    6.         mbmc_status.received++;
    7.  
    8.         if ((rx9data & MASK9BIT) != 0) { // check for bit 9
    9.             mbmcflag.rx_9bit = TRUE; // set command  flag
    10.             mbmc_status.cmdreceived++;
    11.         } else {
    12.             mPORTDToggleBits(BIT_2); // data LED
    13.             mbmcflag.data_timeout = 0; // reset timer
    14.             mbmcflag.rx_9bit = FALSE; // clear command flag
    15.  
    16.             if (!mbmcflag.mbmc_done) { // we need more data
    17.                 mbmcflag.data_ptr[mbmcflag.data_pos] = (BYTE) rx9data; // move received data into buffer
    18.                 if (mbmcflag.data_pos++ >= mbmcflag.data_len) { // the data buffer is filled to data_len
    19.                     mbmcflag.mbmc_done = TRUE; // data block has been received
    20.                     // set a place keeper buffer
    21.                     mbmcflag.data_ptr = (BYTE*) & dd.data_default;
    22.                     mbmcflag.data_len = 4;
    23.                     mbmcflag.data_pos = 0;
    24.                 } else { // nothing for now
    25.                 }
    26.             }
    27.  
    28.         }
    29.  
    30.         if (mbmcflag.rx_9bit) {
    31.             if (rx9data == mbmcflag.host_ack) {
    32.                 mbmc_status.ackreceived++;
    33.                 mPORTDToggleBits(BIT_1); // command LED
    34.                 mbmcflag.host_done = TRUE;
    35.                 mbmcflag.data_timeout = 0; // reset timer
    36.             }
    37.         }
    38.         // Clear the RX interrupt Flag
    39.         INTClearFlag(INT_SOURCE_UART_RX(UART2));
    40.  
    41.     }
    42.  
    43.     // We don't care about TX interrupt
    44.     if (INTGetFlag(INT_SOURCE_UART_TX(UART2))) {
    45.         INTClearFlag(INT_SOURCE_UART_TX(UART2));
    46.     }
    47. }
    48.  
     
Loading...