Getting a file with HEX values off of an SD card and dumping it serially out with an Arduino

Discussion in 'Embedded Systems and Microcontrollers' started by programmer6502, May 19, 2015.

  1. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    Hi,

    I like to interface with vintage computers and processors and would like to start using my Arduino UNO R3 for my upcoming projects. What I've been wanting to do for a long time now is take a file with HEX values (like a ROM) from an SD card and dump it out of the Arduino serially into whatever I need such as shift registers or UARTS.

    What's the practical way to accomplish this? In a nut shell I assume the file would need to be opened, read one line or hex value at a time, converted into binary, and outputted through the appropriate pin repeating this order until the end of the file. What sounds most complicated to me is opening the file properly so I get the HEX values instead of garbage like you do when opening a ROM with Windows notepad instead of an HEX editor.

    If anyone could explain how this is done and get me going in the right direction that would be awesome!

    Thank you!
     
  2. MrChips

    Moderator

    Oct 2, 2009
    12,421
    3,355
    What makes you think that the data are HEX values?
     
  3. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    Excuse me, I believe I misused the word value. What I'm working with are ROMs for 8-bit computers and they're in HEX. I know they are because I've written Eproms with them.
     
  4. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    I'm sure you get what I mean...
     
  5. Papabravo

    Expert

    Feb 24, 2006
    10,136
    1,786
    I'm not sure I understand what you are up to. If you can open a file on the SD card and read the data you can send it out of a serial port. What happens after that is anybody's guess.
    You can also read the contents of a ROM chip and convert those contents to a file.
     
  6. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    You said it yourself, it's as simple as that! All I'm trying to figure out is how to read the file and send it out and I'm not wanting to know anything past that. If my assumption of how it's done in my first post is correct, the real question is how to open the file correctly so that the Arduino can read the HEX and not garbage like I mentioned earlier because of various file formats.
     
    Last edited: May 19, 2015
  7. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    For example, my HEX files are in a .rom format and if I open them in Windows Notepad, it'll display garbage. But if I use a HEX editor to open them, it displays in HEX like I want. Wouldn't the Arduino do the same thing as Notepad if I try to read them because of the format?
     
  8. Papabravo

    Expert

    Feb 24, 2006
    10,136
    1,786
    It kind of depends on what tools you use and how much effort you're willing to put into learning how this is done. In the Arduino library there should be a function called fopen() which is used to open files. If you open them in ascii mode you'll get a result that treats the characters as line oriented text which is not what you want. If you open the file in binary mode you'll get the value of each and every byte. Now you need a function that converts a byte into two characters from the set {0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}. There may or may not be a library function to do this, but if not it is a simple function to write.
     
  9. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    Thank you!! That was the missing piece of my puzzle. I should be able to get what I want now!
     
  10. MrChips

    Moderator

    Oct 2, 2009
    12,421
    3,355
    Guess what? ROMs do not contain HEX. ROMs store BINARY.

    I covered this before in another thread. Computers do not store HEX. Computers store BINARY.
    You thought you were writing a HEX file to a ROM or a computer file. HEX is a shorthand notation that humans use.
    What you see on your computer screen are a bunch of alphanumeric characters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F because they look "pretty" to your eyes (maybe). The underlying data that the computer sees are a bunch of zeros and ones, i.e, binary.

    Trust me. I teach this stuff at university level.
     
  11. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    I'm aware of that which is why I included "convert into binary" in my past assumption on how to do what I wanted in my first post. I was heavily lectured by you on one of my early threads and while I'm now grateful for it, there's no need to be so **** rude!

    But thank you for taking the time to help me.
     
  12. MrChips

    Moderator

    Oct 2, 2009
    12,421
    3,355
    Here is a quick lesson on how computers manipulate data.

    All data is binary. The text you see on this computer screen is made up of binary pixels, ones and zeros.
    Text is binary. ASCII is binary. HEX is binary. Data transmitted via the internet is binary.

    Let us start from scratch. Suppose you have a ROM chip and would like to retrieve the data and save as a backup. Let us assume this is an 8-bit ROM, i.e. the data is stored as 8-bit bytes. We call this raw binary. In order to read this data from your ROM you will need some additional hardware such as a microcontroller.

    Next, you can transmit single 8-bit raw binary via a UART to another device. The problem with doing so is that the receiver has no way of knowing how many bytes are to be transmitted, when the last byte is received and any way of checking for errors in transmission.

    This is where protocol comes in.

    protocol, protocol, protocol

    All reliable means of data transmission use a protocol. In other words, the binary data is packeted using defined rules. All data transmitted via the internet is in the form of a packet.

    Intel created the Intel HEX format. By creating a protocol we can reserve a specific binary pattern for a specific meaning such as <START OF MESSAGE> and <END OF MESSAGE> and we can eliminate certain binary patterns. The raw binary data is packeted in whatever format we choose. Using the hexadecimal characters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F is simply one of many ways to format the data.

    So the UART now transmits raw binary, not the same raw binary that was read from the ROM but a formatted message. The receiving end can receive and then interpret the data according to the rules of the protocol. What you happen to see on the computer screen is entirely up to the programmer who wrote the interpreter. The file to be written to the PC hard drive can use the same Intel HEX format or it can be a completely different protocol altogether.

    If you want to go from a PC disk file to a ROM, you simply reverse the process. So before you begin, understand the underlying protocol to be used at every step along the way. HEX is simply one protocol or one representation of raw binary, which is what the computer has to work with.

    (btw, who is being rude?)
     
    programmer6502 likes this.
  13. Papabravo

    Expert

    Feb 24, 2006
    10,136
    1,786
    The other confusion in this thread is is the overloading of the term "HEX", or "Hex", or "hex". In all capital letters I usually understand it to mean "an ASCII text file in Intel HEX format" which was invented in the early 1970's to allow the contents of memory devices to be specified in a standard format. When written as "hex" or "Hex", I understand it to mean "a hexadecimal number".
     
  14. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    Yep, my latest microcontroller project was an IBM 5150 PS/2 keyboard adapter because I didn't have the keyboard for that machine and proud to say it works! I built it completely from scratch by my own design and source code without the assistance of others. It was the first time I had ever had to follow a protocol and getting the timing correct was obviously key. I didn't even have analyzer to see what it was doing so it took a lot of trial and error, but I had fun doing it! And it certainly payed off because I can now use the IBM!

    I'm sorry for my rudeness in my past threads and hope that we can set aside our contentions in the future. You're an effective source for help and I admire your knowledge in this field.
     
  15. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    Wow, okay. Well there's a first for everything :)
     
  16. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    By the way, the reason I come to this forum is because I know I'm not an expert. I now come here as a last resort to my problems hoping that you guys can get me out of the mud and back on the road in a matter of speaking. And one way or another, it happens lol.

    Thanks again
     
  17. John P

    AAC Fanatic!

    Oct 14, 2008
    1,632
    224
    I'd have thought that the main issue would be that SD cards store data in various formats, similar to the ways a disk drive is formatted. Just to say "take a file with HEX values (like a ROM) from an SD card" ought to lead to a discussion of how a file would be stored, how you identify it and how the processor can get it off the card. It's not a trivial process, and actually not much like a ROM at all. With a memory chip, you'd just be concerned with the address range that the data is stored at, but with a disk/card you have to worry about far more--forgive me for being vague, but I've never had to do this although I know it's a challenge! Here's a paper about reading an SD card, written for a particular processor but it talks about the physical interface in detail:
    http://alumni.cs.ucr.edu/~amitra/sdcard/Additional/sdcard_appnote_foust.pdf
    Or this seems to talk about Arduinos and the FAT16/FAT32 storage protocol:
    http://www.arduino.cc/en/Reference/SD

    You can worry about hex and binary later, but that's when you've got some data to send.
     
  18. djsfantasi

    AAC Fanatic!

    Apr 11, 2010
    2,795
    830
    The other confusion that I have, is that the values in ROM are actually binary, but are represented as hex numbers for readability. The file may contain a byte 0b10111001 (represented in binary), which can also be represented as 0xB9. Now in an Intel HEX file, this byte would be represented by two bytes containing the ASCII characters "B" and "9"

    To confuse things even more, purely for educational purposes, the bytes containing the hex value 0xB9 in the Intel HEX file would be 0x42 or equivalently 0b01000010.

    Actually, what you need to do is fairly simple - as long as you keep the various data representations straight. You only have to concern yourself with the input value and the output format/value.
     
    programmer6502 likes this.
  19. MrChips

    Moderator

    Oct 2, 2009
    12,421
    3,355
    Here is the key:

    ... are actually binary, but are represented as hex numbers for readability.

    Readability by whom?

    Only by humans.
     
  20. programmer6502

    Thread Starter Member

    Feb 1, 2014
    126
    6
    Glad you mentioned this! I vaguely started to realize it when I picked up on my research last night. This must be why other people's source codes that accomplish a similar task are so sophisticated due to the formats.

    The PDF looks very helpful, thanks!
     
Loading...