Been researching how to use an SD card in SPI mode and it seems that it works in blocks of 512 bytes so you can't just access single bytes at time. Additionally, I never knew that SD cards have a little intelligence and the host device is required to send commands to it and await responses, as those things are tiny enough just for the amount of memory they can store!

I found a considerably more advanced Arduino SD card library called SdFat and it appears to have all the work done for you including a class called SdFile that provides binary functions. I have yet to figure out how to use them but at least I'm heading inthe right direction. The nice thing is once I accomplish the task I'm wanting, I can just adapt the source code for whatever device/chip I'm interfacing with!

 

One thing I recall seeing about microcontroller interfaces with SD cards is that you need a certain amount of onboard memory, which rules out the smallest processors. That's probably because of that 512-byte block size.

 

What shield are you using for the SD card? Just curious.

 

I got mine from Radio Shack and I'm pretty sure it's this one. I bought it for early bread board staged circuits and plan to make my own for later more permanent stages.

 

The Arduino SD library provides a method which reads from a file on the SD card, one byte at a time. I use the standard SD library, and have a line of code which looks like this:

  byte inputchar;
  inputchar='\0';
  boolean Done=false;
  while ((! Done) && (file_ACode.available())) {
    inputchar=file_ACode.read();
   // processing code
 }

Internally, the library reads from the SD card in chunks of 512 bytes; however, your program reads it in a byte at a time. In fact, the only input method in the SD library reads a byte at a time.

 

The 512 byte block specification needs to be put into context. There's no reason why the entire block of 512 bytes has to be in the microcontroller's RAM before writing or after reading.

programmer6502 simply wants to read data from the card and push it out the microcontroller's serial port. This is certainly doable with much less than 512 bytes of RAM. I've done it plenty of times, on PICs with 256 bytes (and didn't even use all of it). For each byte read from the SD card you simply push it out the serial port without holding it in RAM.

 

I really appreciate everyone's input so far! I would like to start experimenting with the SD but I'm away from all my stuff until the start of next week and will have to wait until then.

 

The 512 byte block specification needs to be put into context. There's no reason why the entire block of 512 bytes has to be in the microcontroller's RAM before writing or after reading.

programmer6502 simply wants to read data from the card and push it out the microcontroller's serial port. This is certainly doable with much less than 512 bytes of RAM. I've done it plenty of times, on PICs with 256 bytes (and didn't even use all of it). For each byte read from the SD card you simply push it out the serial port without holding it in RAM.

While you are putting it into context, remember the context of an Arduino. The OP is using one, and the SD library has a 512 byte internal buffer. So that's a given memory requirement... Unless you want and can write your own library from scratch.

 

The Arduino SD library reference shows that the read() function returns one byte. Indeed their own example shows reading a file one byte at a time and sending it to the serial port.

http://www.arduino.cc/en/Tutorial/ReadWrite

 

The Arduino SD library reference shows that the read() function returns one byte. Indeed their own example shows reading a file one byte at a time and sending it to the serial port.

http://www.arduino.cc/en/Tutorial/ReadWrite

Which is what I've already said. Can you clarify your point. Are you trying to refute that a 512 byte buffer is required?

I understand that the library returns data one byte at a time. That is at the high level of your program. But at the hardware level, the library is reading 512 bytes, per the SD Card specification. See section 1.7.2... Under the covers, the library takes one byte at a time from the internal buffer and returns it to the calling program. When 512 bytes have been read, the library reads another 512 bytes into the internal buffer.

The standard SD library uses more memory than SDFat. SDFat's code size is smaller, so the OP's decision to use SDFat is likely a good thing.

 

What I meant was that 512 bytes aren't absolutely required to work with SD cards. I've done it with much less. I didn't realize the Arduino library was caching a whole block. But a programmer who is up to the task can certainly hack the library to use less than 512 bytes.

I'm confused, though. If programmer6502 is using an Arduino, and there are SD libraries written for it, why are we fretting over the library's 512 byte requirement?

 

I'm confused, though. If programmer6502 is using an Arduino, and there are SD libraries written for it, why are we fretting over the library's 512 byte requirement?

For me, it was just for educational purposes. Because I studied up on SD cards and because of the explanations I received here, I now have a much much better understanding of the libraries and frankly the cards to begin with. Also, my AVR has 32K of ram so I never was too concerned about though I like things to be clean and efficient.

(Correction, that's the flash memory capacity! The RAM isn't even close, more like 2K)

 

Well, like I said, you don't absolutely need 512 bytes. People assume just because SD cards use 512 byte blocks that you need 512 bytes of RAM for some reason. Nope. Libraries do it because it makes things easy. And if you have 32K of RAM, no worries.

 

So I take it the SD.open() command from Arduino's library is perfectly capable of opening binary files? This along with the read() (or stream.read) command that returns the next byte(s) of a file, I think I'm in business.

 

Which is what I've already said. Can you clarify your point. Are you trying to refute that a 512 byte buffer is required?

I understand that the library returns data one byte at a time. That is at the high level of your program. But at the hardware level, the library is reading 512 bytes, per the SD Card specification. See section 1.7.2... Under the covers, the library takes one byte at a time from the internal buffer and returns it to the calling program. When 512 bytes have been read, the library reads another 512 bytes into the internal buffer.

The standard SD library uses more memory than SDFat. SDFat's code size is smaller, so the OP's decision to use SDFat is likely a good thing.

The older MMC/SD cards could be byte accessed directly on the device (read_block_len could be 1 but it was messy) but new SDHC spec changed to fixed blocks to fit the needed amount of blocks into the csd structure.

Some old MMC/SD/SDHC card driver code...YMMV

int16_t MMC_get_volume_info(void)
//****************************************************************************
{
   SDC0.sddetect = FALSE;
   if (!SDC0.sdinit) {
     if (!SDC0.sdinit) return ERR1; // no card after second try
   }
   vinf = &MMC_volume_Info; //Init the pointer;
   if (SDC0.sdtype == SDT6) { // SDHC size calc
     vinf->sector_count = (((uint32_t) csd[7] & 0x3F) << SHIFT16) | ((uint32_t) csd[8] << SHIFT8) | (uint32_t) csd[9];
     vinf->size_MB = (vinf->sector_count + 1) * SDBUFFERSIZE;
   } else { // SD/MMC size calc
     vinf->read_block_len = (uint32_t) 1 << (csd[5] & 0x0F); // sector size

     vinf->sector_count = (uint32_t) csd[6] & 0x03;
     vinf->sector_count <<= SHIFT8;
     vinf->sector_count += (uint32_t) csd[7];
     vinf->sector_count <<= SHIFT2;
     vinf->sector_count += (uint32_t) ((uint32_t) csd[8] & 0xc0) >> SHIFT6;

     vinf->sector_multiply = (uint32_t) csd[9] & 0x03;
     vinf->sector_multiply <<= SHIFT1;
     vinf->sector_multiply += (uint32_t) ((uint32_t) csd[10] & 0x80) >> SHIFT7;
     // work out the MBs
     // mega bytes in u08 == C_SIZE / (2^(9-C_SIZE_MULT))
     vinf->size_MB = (vinf->sector_count >> (9 - vinf->sector_multiply)) * (vinf->read_block_len / SDBUFFERSIZE);
   }

   vinf->serial = ((uint32_t) cid[9] << SHIFT24) + ((uint32_t) cid[10] << SHIFT16) + ((uint32_t) cid[11] << SHIFT8) + (uint32_t) cid[12]; // Serial Number
   // get the name of the card
   vinf->name[0] = cid[3];
   vinf->name[1] = cid[4];
   vinf->name[2] = cid[5];
   vinf->name[3] = cid[6];
   vinf->name[4] = cid[7];
   vinf->name[5] = 0x00; //end flag

   SDC0.sddetect = TRUE;
   return NULL;
}

Edit: added text about device operations.

 

So I take it the SD.open() command from Arduino's library is perfectly capable of opening binary files? This along with the read() (or stream.read) command that returns the next byte(s) of a file, I think I'm in business.

The .open() method only concerns itself whether you are going to read or write to the file (read is the default). It is agnostic as to the contents of the file. A byte is a byte is a byte. What the binary value represents is external to the read operation.

http://www.arduino.cc/en/Reference/SDopen​

 

The .open() method only concerns itself whether you are going to read or write to the file (read is the default). It is agnostic as to the contents of the file. A byte is a byte is a byte. What the binary value represents is external to the read operation.

http://www.arduino.cc/en/Reference/SDopen​

Yeah, that totally makes sense. Well sweet, I can't wait to get back to my stuff and play around! I've had so many plans that would now be possible because of this and I can't believe I was held back so long for something so simple!

Thanks again guys!! I'll report back when the time comes.

 

The older MMC/SD cards could be byte accessed (read_block_len could be 1 but it was messy) but new SDHC spec changed to fixed blocks to fit the needed amount of blocks into the csd structure.

They ALL can be byte accessed. What you are referring to is the size of the block. Yes, it's now 512 bytes...So? You don't have to read it all into RAM. You start the process and send the bytes out the serial port as soon as you read each one. I could do this on a PIC with less than 100 bytes of RAM.

And if you're writing a block there's no reason why you have to have the entire thing in RAM before you start writing. Want to write one byte?...Start the write process for a new block, write your byte, and finish the block with 511 bytes of 0xFF.

If you want to add one byte to an existing block, sure, you can't do it because you'd need to read in the block and modify it before sending it back. But so?...Lots of projects don't absolutely need that functionality. And there's no limit on how much time can pass between sending bytes. Just keep the block open until it's full.

 

They ALL can be byte accessed. What you are referring to is the size of the block. Yes, it's now 512 bytes...So?

Correct, I modified my earlier statement to be about device level operation not SPI stream operations.

You don't have to read into the whole block_buffer to get a fraction of it.

int16_t mmc_read_block(uint32_t block_number)
{
   static int16_t retry, i, nsec, ssm;
   static uint8_t tmp;
   static uint32_t block_tmp;

   SDSAFE = S_OFF;
   SDEBUGM('R');
   ClrWdt(); // reset the WDT timer
   block_tmp = block_number;
   if (SDC0.sdtype != SDT6) { // use byte style blocks for SD cards
     block_tmp = block_tmp << SHIFT9;
   }

   DESELECT();
   send_dummys();
   SELECT();
   retry = 0;
   do {
     if (SDC0.sdtype == SDT6) {
       tmp = send_cmd(CMD17, block_tmp); // send sdhc read single block command
       SDEBUGM('6');
     } else {
       tmp = send_cmd(CMD18, block_tmp); // send sdhc read single block command
       SDEBUGM('2');
     }
     retry++;
     if (retry == 1000) {
       DESELECT();
       SDEBUGM('E');
       return ERR1; // write command error
     }
     ClrWdt(); // reset the WDT timer
   } while (tmp != (uint8_t) NULL);

   retry = 0;
   do {
     tmp = rcvr_spi();
     retry++;
     if (retry == 1000) {
       DESELECT();
       SDEBUGM('E');
       return ERR1; // receive command error
     }
   } while (tmp != (uint8_t) 0xfe); // got data token

   if (SDC0.sdtype == SDT6) {
     for (i = 0; i < SDBUFFERSIZE; i++) {
       block_buffer[i] = rcvr_spi(); // send data to card
     }
     rcvr_spi(); // CRC16 bytes that are not needed
     rcvr_spi();
     DESELECT(); // set SS = 1 (off)
     send_dummys(); // give mmc the clocks it needs to finish off
     SDEBUGM('r');
     SDSAFE = S_ON;
     return NULL;
   } else {
     ssm = vinf->read_block_len / SDBUFFERSIZE;
     for (nsec = 1; nsec <= ssm; nsec++) {
       for (i = 0; i < SDBUFFERSIZE; i++) {
         block_buffer[i] = rcvr_spi(); // send data to card
       }
       rcvr_spi(); // CRC16 bytes that are not needed
       rcvr_spi();
     }
     // send stop tran
     tmp = send_cmd(CMD12, block_tmp); // send sdhc read single block command
     rcvr_spi();
     while (rcvr_spi() == (uint8_t) NULL); // wait
     DESELECT(); // set SS = 1 (off)
     send_dummys(); // give mmc the clocks it needs to finish off
     SDEBUGM('r');
     SDSAFE = S_ON;
     return NULL;
   }

 

Well I'm happy to report that it works! Using that piece of code from djsfantasi and some touches of my own, I setup a program to dump one of my ROM files onto a serial monitor. It prints in decimal and by using a calculator to convert to hex, I can see that it matches up correctly when comparing the ROM in a hex editor!

Thank you all once again for the help, you have no idea what this means to me!