To continue, you are going to add code to receive the web server’s response and save the information to a file on the SD card.
Immediately after the line myClient.println() in the sketch’s loop() function, add the following code:
if (SD.begin(4)) {
if (SD.exists("DOWNLOAD.TXT")) SD.remove("DOWNLOAD.TXT");
Serial.print("Saving response… ");
digitalWrite(LED, HIGH);
File dd = SD.open("DOWNLOAD.TXT", FILE_WRITE);
while (myClient.connected()) { if (myClient.available() > 0) { dd.write(myClient.read());
} }
dd.close();
delay(500);
Serial.println("OK");
digitalWrite(LED, LOW);
} else {
Serial.println("No SD card detected!");
}
If you are not familiar with working with the Arduino Ethernet Shield’s built-in SD card socket and the SD.h library, you may wish to read Using SD Cards.
Three new methods of the EthernetClient class are included in the code above:
Method Description
connected() Returns true if the connection to the server is still active, or false if it is no longer usable.
available() Returns the number of bytes that are currently in the Ethernet Shield’s buffer, waiting to be read.
read() With no arguments, read() fetches a single byte from the server’s response and returns it.
After creating a new file to hold the data, the code enters a while loop that waits until the server drops the HTTP connection. Since there is no guarantee that all of the data will be sent immediately, this loop waits for available data and then writes it to the SD card when it does arrive.
Using HTTP/1.0, the server should end the connection once it is finished sending its response. Your HTTP request includes the field “Connection: close” to help ensure that this happens when communicating with HTTP/1.1 servers. However, one of the most common causes of problems is web servers not closing
connections. Downloading files from the Internet through the Ethernet Shield can be slow, but if the LED remains lit for a very long period of time then it may mean that the server has not closed the connection.
In the next project, Project 2 – Scraping Webpages to Retrieve Information, you will see how to implement timeouts so that the sketch can recover if the server stops sending information but does not close the connection. For now, if you
encounter this problem, try downloading a different file from a different web server.
Once the Arduino has downloaded the file and the LED is turned off, you can turn off the Arduino and remove the SD card. Insert the SD card into your PC and open up the file DOWNLOAD.TXT in a text editor such as Notepad, TextEdit, or Vi. The file contains HTML code for the webpage you requested, but with several lines of information before that.
Figure 6. An HTTP response from a web server
Beginning with “HTTP/1.1” and ending with the blank line, this is the server’s HTTP response header and it is not a part of the webpage. The first line of this response is the most important, and it is called the status line.
Even though your code requests that the server use HTTP/1.0, this particular web server responds with HTTP/1.1. However, it does not use any of the HTTP/1.1 features that could make it difficult for you to process the response.
After the protocol version, the server sends an HTTP status code (200) and a reason phrase that describes the code. A full list of HTTP status codes is shown in Appendix A, section 9. Status Code Definitions. However, for the purpose of
checking that the HTTP request completes successfully, there are only a few status codes that you need to be aware of:
HTTP Status Code
Description
200 OK. The request completed successfully.
400–
499
Indicates that the request failed because of a problem caused by the client. For example, requesting a file that is not on the server (404) or making an invalid request.
500–
599 Indicates that the request failed because of a problem on the server.
Regardless of the status code, most web servers return an HTML page after the HTTP response header. This page may not be the file you requested and, instead, may be a page that describes the error that has occurred.
In this project you will only modify the sketch so that it ignores the HTTP header and does not write it to the file. This means that the file saved to the SD card could be an error page.
Tip: You can use a web-based tool such as Rex Swain’s HTTP Viewer (http://www.rexswain.com/httpview.html) to verify how a web server responds to an HTTP/1.0 request.
Before the while(myClient.connected()) loop, add the following lines:
char lc;
while (myClient.connected()) { if (myClient.available()) { char nc = myClient.read();
if ((lc == 10) && (nc == 13)) { while (myClient.available() == 0);
myClient.read();
break;
} else lc = nc;
} }
This is not pretty code, but it works. The loop skips characters from the server’s response until it finds a line feed (10) followed by a carriage return (13) – it
matches the middle part of the sequence that is used to mark the end of the HTTP response header. The code then skips the next character (the final line feed) and terminates the loop.
The sketch can now write the remaining characters from the server’s response to the SD card. In this project, the LED is lit while it writes the file so that you know not to remove the card in the middle of the process.