Downloading file using HTTP GET & Saving to SD card using Arduino MKR1400 GSM

Thread Starter

h.yakub

Joined Mar 11, 2019
97
Hi All,

I am using Arduino MKR1400 board with 2 GB SD card. I need to perform OTA update on the board, so far I have used <SDU.h> library provided by arduino that updates the code by uploading .Bin file on SD card when rebooted.
Now, I have a http server containing the .bin file ready to download. I just want your guidance in how to download the file from http server and save it directly to SD card.
My attempts to download the file from http server was a success, but I am stuck in saving this file to SD card. I am attaching the code, please review it and let me know the changes....:)..Waiting for your earliest responses....

C:
Arduino Code:
#include <MKRGSM.h>
#include <SD.h>
#include <SPI.h>
#include <ArduinoHttpClient.h>

const char PINNUMBER[]     = "";
// APN data
const char GPRS_APN[]      = "";
const char GPRS_LOGIN[]    = "";
const char GPRS_PASSWORD[] = "";

// initialize the library instance
GSMClient client;
GPRS gprs;
GSM gsmAccess;

File myFile;

// URL, path and port (for example: example.org)
char server[] = "103.9.13.150";
int port = 80; // port 80 is the default for HTTP

HttpClient http(client, server, port);

void setup() {
  // initialize serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
  Serial.print(F("Initializing SD card..."));
  if (!SD.begin(4)) {
    Serial.println(F(" initialization failed!"));
    return;
  }
  Serial.println(F(" initialization done."));
  Serial.println("Starting Arduino web client.");
  // connection state
  bool connected = false;

  // After starting the modem with GSM.begin()
  // attach the shield to the GPRS network with the APN, login and password
  while (!connected) {
    if ((gsmAccess.begin(PINNUMBER) == GSM_READY) &&
        (gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD) == GPRS_READY)) {
      connected = true;
    } else {
      Serial.println("Not connected");
      delay(1000);
    }
  }

  Serial.println("connecting...");
  http.beginRequest();
  Serial.println("connected");
  // Make a HTTP request:
  // Check to see if the file exists
  if (SD.exists("UPDATE.bin")) {
    Serial.println("UPDATE.bin exists.");
    Serial.println("Removing UPDATE.bin...");
    SD.remove("UPDATE.bin");
  } else {
    Serial.println("UPDATE.bin doesn't exist.");
  }
  Serial.println("Creating UPDATE.bin...");
  myFile = SD.open("UPDATE.bin", FILE_WRITE);

  if (myFile)
  {
    Serial.print("File Created...");
    http.get("/UPDATE.bin");
    http.sendHeader("Content-Type", "application/octet-stream");
    http.sendHeader("Content-Disposition", "attachment; filename=UPDATE.bin");
    http.sendHeader("Connection", "close");
    http.endRequest();
    // close the file:
    myFile.close();
    Serial.println("");
    Serial.println("done.");
  }// if you get a connection, report back via serial:
}

void loop()
{
  // if there are incoming bytes available
  // from the server, read them and print them:

}
 

ericgibbs

Joined Jan 29, 2010
8,723
hi,
This is my Arduino file, changed extension to .txt to upload.
The program receives a message string via a HC12 receiver on the UART, displays the data on a LCD and writes the string to the an SD card.
The SD card code section may give you an idea, its been working for over 2 years as weather data logger.

E
 

Attachments

Thread Starter

h.yakub

Joined Mar 11, 2019
97
Hi E.G,
Thanks for the response.
I am already familiar with this functioning of SD card and it works if you create a new file on SD card.
I am interested in saving a whole file onto SD card which is downloaded through http server using GSM module. The format of my file is .bin which is a bit tricky to read.
 

djsfantasi

Joined Apr 11, 2010
5,585
I’m not sure I understand...

Nowhere do you output to the output file?

I would expect at least one myfile.print() statement.,

The fact that the input file is a .bin file is irrelevant. You read it one character at a time or one block at a time. Then write it to your output file. If the .bin is a sticking point, rename it to .txt Does this simplify the problem for you? If it does, then the operation is the same no matter what the file extension. You are just doing a file copy, byte for byte.
 

Thread Starter

h.yakub

Joined Mar 11, 2019
97
I’m not sure I understand...

Nowhere do you output to the output file?

I would expect at least one myfile.print() statement.,

The fact that the input file is a .bin file is irrelevant. You read it one character at a time or one block at a time. Then write it to your output file. If the .bin is a sticking point, rename it to .txt Does this simplify the problem for you? If it does, then the operation is the same no matter what the file extension. You are just doing a file copy, byte for byte.
Hi djsfantasi,
Thanks for the response.
One of the problem is how do I read the downloaded file....?. Right now, I am sure about the link being hit and once the link is hit the file is downloaded. But, I have no idea how to perform actions (like read/write) on that downloaded file.
 

djsfantasi

Joined Apr 11, 2010
5,585
First, while you perform a get of update.bin, you actually never access its contents. Once you perform the http get, you need to read the bytes of the file. This article shows how.

As you read in each byte, print it out to your local copy on the SD card. After the last byte is read, close the file and http connection.
 

Thread Starter

h.yakub

Joined Mar 11, 2019
97
First, while you perform a get of update.bin, you actually never access its contents. Once you perform the http get, you need to read the bytes of the file. This article shows how.

As you read in each byte, print it out to your local copy on the SD card. After the last byte is read, close the file and http connection.
Hi djsfantasi,
Thanks for your advice. Now I am able to read and download the file to my SD card successfully. :)
I am attaching the output file image.

Here's the code for the same:
#include<SDU.h>
#include <MKRGSM.h>
#include <SPI.h>
#include <SD.h>

const char PINNUMBER[] = "";
// APN data
const char GPRS_APN[] = "";
const char GPRS_LOGIN[] = "";
const char GPRS_PASSWORD[] = "";


GSMClient client;
GPRS gprs;
GSM gsmAccess;

File myFile;

char server[] = "103.9.13.150";
char path[] = "/UPDATE.bin";
int port = 80;

void setup() {
// initialize serial communications and wait for port to open:
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}

Serial.print("Initializing SD card...");
if (!SD.begin(4)) {
Serial.println("Initialization failed!");
while (1);
}
Serial.println("Initialization done.");

Serial.println("Starting Arduino web client.");
bool connected = false;
while (!connected) {
if ((gsmAccess.begin(PINNUMBER) == GSM_READY) &&
(gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD) == GPRS_READY)) {
connected = true;
} else {
Serial.println("Not connected");
delay(1000);
}
}
Serial.println("connecting...");

if (client.connect(server, port)) {
Serial.println("connected");
// Make a HTTP request:
client.print("GET ");
client.print(path);
client.println(" HTTP/1.1");
client.print("Host: ");
client.println(server);
client.println("Connection: close");
client.println();
} else {
// if you didn't get a connection to the server:
Serial.println("connection failed");
}

if (SD.exists("UPDATE.bin")) {
Serial.println("UPDATE.bin exists.");
// delete the file:
Serial.println("Removing UPDATE.bin...");
SD.remove("UPDATE.bin");
}
else {
Serial.println("UPDATE.bin doesn't exist.");
}
myFile = SD.open("UPDATE.bin", FILE_WRITE);
}
void loop()
{
if (client.available()) {
char c = client.read();
Serial.print(c);
myFile.print(c);
}
// if the server's disconnected, stop the client:
if (!client.available() && !client.connected()) {
Serial.println();
Serial.println("disconnecting.");
client.stop();
myFile.close();
Serial.println("done.");
// do nothing forevermore:
for (;;)
;
}

}
 

Attachments

Thread Starter

h.yakub

Joined Mar 11, 2019
97
Although there is a small issue, I am facing. While downloading the file, the file also saves the server details from which it is downloaded, that changes my actual .bin file.
This is the extra content printed in my .bin file that corrupts it (I might not be using the corrupt word proper in this context). :p

HTTP/1.1 200 OK
Date: Tue, 23 Jul 2019 09:00:38 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Wed, 17 Jul 2019 11:00:45 GMT
ETag: "6fb0-58dde67b124a5"
Accept-Ranges: bytes
Content-Length: 28592
Connection: close
Content-Type: application/octet-stream

Can you suggest a way to avoid this lines from being saved in actual file......
 

djsfantasi

Joined Apr 11, 2010
5,585
Although there is a small issue, I am facing. While downloading the file, the file also saves the server details from which it is downloaded, that changes my actual .bin file.
This is the extra content printed in my .bin file that corrupts it (I might not be using the corrupt word proper in this context). :p

HTTP/1.1 200 OK
Date: Tue, 23 Jul 2019 09:00:38 GMT
Server: Apache/2.4.6 (CentOS)
Last-Modified: Wed, 17 Jul 2019 11:00:45 GMT
ETag: "6fb0-58dde67b124a5"
Accept-Ranges: bytes
Content-Length: 28592
Connection: close
Content-Type: application/octet-stream

Can you suggest a way to avoid this lines from being saved in actual file......
Sure...

Read the file in two loops.

The first loop builds strings for each line in the file. Check the input strings for:
Content-Type: application/octet-stream​
When you find this line, start the second loop.

This loop reads the bytes and outputs them to your file.

Basically, skip the crap and write the good stuff only.
 

Thread Starter

h.yakub

Joined Mar 11, 2019
97
Sure...

Read the file in two loops.

The first loop builds strings for each line in the file. Check the input strings for:
Content-Type: application/octet-stream​
When you find this line, start the second loop.

This loop reads the bytes and outputs them to your file.

Basically, skip the crap and write the good stuff only.
I have followed your steps and it works like a charm now.
The only thing is the .bin file is of 90 KB in size and my SDU.h libraryfor SAMD21 is not able to upload in on to the microcontroller. As far as small size files are concerned it works fine.
Do you have any experience working with SDU.h library on SAMD21. Please provide your suggestion for it.
 

Thread Starter

h.yakub

Joined Mar 11, 2019
97
New Update:

I have solved the issue by adding the SDU.h header file to the code whose .bin file is to be downloaded.

Thanks to @djsfantasi & @ericgibbs for your support. Looking forward to next challenges..:)

This thread is closed from my side now.
 
Top