Getting a file with HEX values off of an SD card and dumping it serially out with an Arduino

Thread Starter

programmer6502

Joined Feb 1, 2014
132
Hi,

I like to interface with vintage computers and processors and would like to start using my Arduino UNO R3 for my upcoming projects. What I've been wanting to do for a long time now is take a file with HEX values (like a ROM) from an SD card and dump it out of the Arduino serially into whatever I need such as shift registers or UARTS.

What's the practical way to accomplish this? In a nut shell I assume the file would need to be opened, read one line or hex value at a time, converted into binary, and outputted through the appropriate pin repeating this order until the end of the file. What sounds most complicated to me is opening the file properly so I get the HEX values instead of garbage like you do when opening a ROM with Windows notepad instead of an HEX editor.

If anyone could explain how this is done and get me going in the right direction that would be awesome!

Thank you!
 

Papabravo

Joined Feb 24, 2006
21,094
I'm not sure I understand what you are up to. If you can open a file on the SD card and read the data you can send it out of a serial port. What happens after that is anybody's guess.
You can also read the contents of a ROM chip and convert those contents to a file.
 

Thread Starter

programmer6502

Joined Feb 1, 2014
132
I'm not sure I understand what you are up to. If you can open a file on the SD card and read the data you can send it out of a serial port. What happens after that is anybody's guess.
You can also read the contents of a ROM chip and convert those contents to a file.
You said it yourself, it's as simple as that! All I'm trying to figure out is how to read the file and send it out and I'm not wanting to know anything past that. If my assumption of how it's done in my first post is correct, the real question is how to open the file correctly so that the Arduino can read the HEX and not garbage like I mentioned earlier because of various file formats.
 
Last edited:

Thread Starter

programmer6502

Joined Feb 1, 2014
132
For example, my HEX files are in a .rom format and if I open them in Windows Notepad, it'll display garbage. But if I use a HEX editor to open them, it displays in HEX like I want. Wouldn't the Arduino do the same thing as Notepad if I try to read them because of the format?
 

Papabravo

Joined Feb 24, 2006
21,094
For example, my HEX files are in a .rom format and if I open them in Windows Notepad, it'll display garbage. But if I use a HEX editor to open them, it displays in HEX like I want. Wouldn't the Arduino do the same thing as Notepad if I try to read them because of the format?
It kind of depends on what tools you use and how much effort you're willing to put into learning how this is done. In the Arduino library there should be a function called fopen() which is used to open files. If you open them in ascii mode you'll get a result that treats the characters as line oriented text which is not what you want. If you open the file in binary mode you'll get the value of each and every byte. Now you need a function that converts a byte into two characters from the set {0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}. There may or may not be a library function to do this, but if not it is a simple function to write.
 

Thread Starter

programmer6502

Joined Feb 1, 2014
132
It kind of depends on what tools you use and how much effort you're willing to put into learning how this is done. In the Arduino library there should be a function called fopen() which is used to open files. If you open them in ascii mode you'll get a result that treats the characters as line oriented text which is not what you want. If you open the file in binary mode you'll get the value of each and every byte. Now you need a function that converts a byte into two characters from the set {0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}. There may or may not be a library function to do this, but if not it is a simple function to write.
Thank you!! That was the missing piece of my puzzle. I should be able to get what I want now!
 

MrChips

Joined Oct 2, 2009
30,618
Excuse me, I believe I misused the word value. What I'm working with are ROMs for 8-bit computers and they're in HEX. I know they are because I've written Eproms with them.
Guess what? ROMs do not contain HEX. ROMs store BINARY.

I covered this before in another thread. Computers do not store HEX. Computers store BINARY.
You thought you were writing a HEX file to a ROM or a computer file. HEX is a shorthand notation that humans use.
What you see on your computer screen are a bunch of alphanumeric characters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F because they look "pretty" to your eyes (maybe). The underlying data that the computer sees are a bunch of zeros and ones, i.e, binary.

Trust me. I teach this stuff at university level.
 

Thread Starter

programmer6502

Joined Feb 1, 2014
132
Guess what? ROMs do not contain HEX. ROMs store BINARY.

I covered this before in another thread. Computers do not store HEX. Computers store BINARY.
You thought you were writing a HEX file to a ROM or a computer file. HEX is a shorthand notation that humans use.
What you see on your computer screen are a bunch of alphanumeric characters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F because they look "pretty" to your eyes (maybe). The underlying data that the computer sees are a bunch of zeros and ones, i.e, binary.

Trust me. I teach this stuff at university level.
I'm aware of that which is why I included "convert into binary" in my past assumption on how to do what I wanted in my first post. I was heavily lectured by you on one of my early threads and while I'm now grateful for it, there's no need to be so **** rude!

But thank you for taking the time to help me.
 

MrChips

Joined Oct 2, 2009
30,618
Here is a quick lesson on how computers manipulate data.

All data is binary. The text you see on this computer screen is made up of binary pixels, ones and zeros.
Text is binary. ASCII is binary. HEX is binary. Data transmitted via the internet is binary.

Let us start from scratch. Suppose you have a ROM chip and would like to retrieve the data and save as a backup. Let us assume this is an 8-bit ROM, i.e. the data is stored as 8-bit bytes. We call this raw binary. In order to read this data from your ROM you will need some additional hardware such as a microcontroller.

Next, you can transmit single 8-bit raw binary via a UART to another device. The problem with doing so is that the receiver has no way of knowing how many bytes are to be transmitted, when the last byte is received and any way of checking for errors in transmission.

This is where protocol comes in.

protocol, protocol, protocol

All reliable means of data transmission use a protocol. In other words, the binary data is packeted using defined rules. All data transmitted via the internet is in the form of a packet.

Intel created the Intel HEX format. By creating a protocol we can reserve a specific binary pattern for a specific meaning such as <START OF MESSAGE> and <END OF MESSAGE> and we can eliminate certain binary patterns. The raw binary data is packeted in whatever format we choose. Using the hexadecimal characters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F is simply one of many ways to format the data.

So the UART now transmits raw binary, not the same raw binary that was read from the ROM but a formatted message. The receiving end can receive and then interpret the data according to the rules of the protocol. What you happen to see on the computer screen is entirely up to the programmer who wrote the interpreter. The file to be written to the PC hard drive can use the same Intel HEX format or it can be a completely different protocol altogether.

If you want to go from a PC disk file to a ROM, you simply reverse the process. So before you begin, understand the underlying protocol to be used at every step along the way. HEX is simply one protocol or one representation of raw binary, which is what the computer has to work with.

(btw, who is being rude?)
 

Papabravo

Joined Feb 24, 2006
21,094
The other confusion in this thread is is the overloading of the term "HEX", or "Hex", or "hex". In all capital letters I usually understand it to mean "an ASCII text file in Intel HEX format" which was invented in the early 1970's to allow the contents of memory devices to be specified in a standard format. When written as "hex" or "Hex", I understand it to mean "a hexadecimal number".
 

Thread Starter

programmer6502

Joined Feb 1, 2014
132
Here is a quick lesson on how computers manipulate data.

All data is binary. The text you see on this computer screen is made up of binary pixels, ones and zeros.
Text is binary. ASCII is binary. HEX is binary. Data transmitted via the internet is binary.

Let us start from scratch. Suppose you have a ROM chip and would like to retrieve the data and save as a backup. Let us assume this is an 8-bit ROM, i.e. the data is stored as 8-bit bytes. We call this raw binary. In order to read this data from your ROM you will need some additional hardware such as a microcontroller.

Next, you can transmit single 8-bit raw binary via a UART to another device. The problem with doing so is that the receiver has no way of knowing how many bytes are to be transmitted, when the last byte is received and any way of checking for errors in transmission.

This is where protocol comes in.

protocol, protocol, protocol

All reliable means of data transmission use a protocol. In other words, the binary data is packeted using defined rules. All data transmitted via the internet is in the form of a packet.

Intel created the Intel HEX format. By creating a protocol we can reserve a specific binary pattern for a specific meaning such as <START OF MESSAGE> and <END OF MESSAGE> and we can eliminate certain binary patterns. The raw binary data is packeted in whatever format we choose. Using the hexadecimal characters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F is simply one of many ways to format the data.

So the UART now transmits raw binary, not the same raw binary that was read from the ROM but a formatted message. The receiving end can receive and then interpret the data according to the rules of the protocol. What you happen to see on the computer screen is entirely up to the programmer who wrote the interpreter. The file to be written to the PC hard drive can use the same Intel HEX format or it can be a completely different protocol altogether.

If you want to go from a PC disk file to a ROM, you simply reverse the process. So before you begin, understand the underlying protocol to be used at every step along the way. HEX is simply one protocol or one representation of raw binary, which is what the computer has to work with.

(btw, who is being rude?)
Yep, my latest microcontroller project was an IBM 5150 PS/2 keyboard adapter because I didn't have the keyboard for that machine and proud to say it works! I built it completely from scratch by my own design and source code without the assistance of others. It was the first time I had ever had to follow a protocol and getting the timing correct was obviously key. I didn't even have analyzer to see what it was doing so it took a lot of trial and error, but I had fun doing it! And it certainly payed off because I can now use the IBM!

I'm sorry for my rudeness in my past threads and hope that we can set aside our contentions in the future. You're an effective source for help and I admire your knowledge in this field.
 

Thread Starter

programmer6502

Joined Feb 1, 2014
132
The other confusion in this thread is is the overloading of the term "HEX", or "Hex", or "hex". In all capital letters I usually understand it to mean "an ASCII text file in Intel HEX format" which was invented in the early 1970's to allow the contents of memory devices to be specified in a standard format. When written as "hex" or "Hex", I understand it to mean "a hexadecimal number".
Wow, okay. Well there's a first for everything :)
 

Thread Starter

programmer6502

Joined Feb 1, 2014
132
By the way, the reason I come to this forum is because I know I'm not an expert. I now come here as a last resort to my problems hoping that you guys can get me out of the mud and back on the road in a matter of speaking. And one way or another, it happens lol.

Thanks again
 

John P

Joined Oct 14, 2008
2,025
I'd have thought that the main issue would be that SD cards store data in various formats, similar to the ways a disk drive is formatted. Just to say "take a file with HEX values (like a ROM) from an SD card" ought to lead to a discussion of how a file would be stored, how you identify it and how the processor can get it off the card. It's not a trivial process, and actually not much like a ROM at all. With a memory chip, you'd just be concerned with the address range that the data is stored at, but with a disk/card you have to worry about far more--forgive me for being vague, but I've never had to do this although I know it's a challenge! Here's a paper about reading an SD card, written for a particular processor but it talks about the physical interface in detail:
http://alumni.cs.ucr.edu/~amitra/sdcard/Additional/sdcard_appnote_foust.pdf
Or this seems to talk about Arduinos and the FAT16/FAT32 storage protocol:
http://www.arduino.cc/en/Reference/SD

You can worry about hex and binary later, but that's when you've got some data to send.
 

djsfantasi

Joined Apr 11, 2010
9,155
The other confusion in this thread is is the overloading of the term "HEX", or "Hex", or "hex". In all capital letters I usually understand it to mean "an ASCII text file in Intel HEX format" which was invented in the early 1970's to allow the contents of memory devices to be specified in a standard format. When written as "hex" or "Hex", I understand it to mean "a hexadecimal number".
The other confusion that I have, is that the values in ROM are actually binary, but are represented as hex numbers for readability. The file may contain a byte 0b10111001 (represented in binary), which can also be represented as 0xB9. Now in an Intel HEX file, this byte would be represented by two bytes containing the ASCII characters "B" and "9"

To confuse things even more, purely for educational purposes, the bytes containing the hex value 0xB9 in the Intel HEX file would be 0x42 or equivalently 0b01000010.

Actually, what you need to do is fairly simple - as long as you keep the various data representations straight. You only have to concern yourself with the input value and the output format/value.
 

MrChips

Joined Oct 2, 2009
30,618
Here is the key:

... are actually binary, but are represented as hex numbers for readability.

Readability by whom?

Only by humans.
 

Thread Starter

programmer6502

Joined Feb 1, 2014
132
I'd have thought that the main issue would be that SD cards store data in various formats, similar to the ways a disk drive is formatted. Just to say "take a file with HEX values (like a ROM) from an SD card" ought to lead to a discussion of how a file would be stored, how you identify it and how the processor can get it off the card. It's not a trivial process, and actually not much like a ROM at all. With a memory chip, you'd just be concerned with the address range that the data is stored at, but with a disk/card you have to worry about far more--forgive me for being vague, but I've never had to do this although I know it's a challenge! Here's a paper about reading an SD card, written for a particular processor but it talks about the physical interface in detail:
http://alumni.cs.ucr.edu/~amitra/sdcard/Additional/sdcard_appnote_foust.pdf
Or this seems to talk about Arduinos and the FAT16/FAT32 storage protocol:
http://www.arduino.cc/en/Reference/SD

You can worry about hex and binary later, but that's when you've got some data to send.
Glad you mentioned this! I vaguely started to realize it when I picked up on my research last night. This must be why other people's source codes that accomplish a similar task are so sophisticated due to the formats.

The PDF looks very helpful, thanks!
 
Top