curl -o MSIE.html -A 'Mozilla/4.0 (compatible; MSIE 7.0;
Windows NT 6.0; SLCC1; .NET CLR 2.0.50727;
Media Center PC 5.0; .NET CLR 3.0.04506)' http://www.example.com/
# Firefox 2.0.0.15 on MacOS X
curl -o FFMac.html -A 'Mozilla/5.0 (Macintosh; U;
Intel Mac OS X; en-US; rv:1.8.1.3)
Gecko/20070309 Firefox/2.0.0.15' http://www.example.com/
# "Blazer" web browser on PalmOS devices
curl -o Palm.html -A 'Mozilla/4.0 (compatible; MSIE 6.0; Windows 98;
PalmSource/hspr-H102; Blazer/4.0) 16;320x320' http://www.example.com/
7.6.3. Discussion
There is no rhyme or reason to User-Agent strings, except the vestigial "Mozilla" at the beginning of the string—a reminder of the browser wars. There are many databases and websites that collect these strings, but as a tester, you want to gather them differently. You want to find out from the developers or from the source code itself which user agents the code responds to (if any). That way you can determine how many different kinds of tests you need to do.
You may want to talk to operations staff to get some of your web server logs and look at what User-Agents you're seeing in the wild.
If you want to browse around interactively, impersonating another device, take a look at Section 7.7. By poking around interactively, you may discover that your application does react to the User-Agent, and, therefore, you need to make some test cases based on this recipe.
7.6.3.1. Providing customized content
Yahoo! is a major website that reacts to the User-Agent string. If you choose something it doesn't recognize, it will send a very small web page (and one that has very little JavaScript and fewer advertisements). If your User-Agent is
recognizable as Internet Explorer, Firefox, or another well-known browser, Yahoo! will deliver customized
content—including JavaScript that is carefully tuned to execute correctly in your web browser. One of the reasons Yahoo! does this is to provide a good-looking interface to new devices that they have never heard of before. The first person to visit http://www.yahoo.com/ with a Nintendo Wii or an Apple iPhone got a generic page that probably rendered pretty well, but did not have all the features of Yahoo! when viewed in a browser. Eventually, as Yahoo!
becomes aware of the capabilities of the Wii or the iPhone, they will change their site to react differently, based on the User-Agent.
7.6.3.2. Reacting to User-Agent is rare
Most web applications don't react to browsers at all. You only need to consider this testing technique if you know for a fact that your application behaves this way. Note that many sites and applications that use complex cascading style sheets (CSS) or asynchronous JavaScript and XML (AJAX) will have a lot of complex JavaScript code that loads differently in the browser depending on which browser it is. This is not the same as the User-Agent string and having the server perform different operations based on what browser requests the page. Many sites send JavaScript that will be executed differently depending on the browser. Few look at the User-Agent string at run time.
Realize that, if you're one of the lucky few who has software that responds differently to different User-Agents, this will increase your test matrix significantly. Tests for vulnerabilities like cross-site scripting (XSS), SQL injection, or session fixation will have to be done with representatives of various different kinds of browsers to be sure that all the code is tested.
Recipe 7.7. Interactively Impersonating Another Device 7.7.1. Problem
If testing with cURL shows that your site responds to the User-Agent string (see Section 7.6), you might want to just probe around interactively and see what your website looks like when a search engine (like Google, Yahoo!, or MSN) sees it.
7.7.2. Solution
Use Chris Pederick's User Agent Switcher extension for Firefox. It can be found at
http://chrispederick.com/work/useragentswitcher/. It is installed like any Firefox extension (see Section 2.2).
Once installed, it provides an option on the Tools menu, as shown in Figure 7-1. From there you can easily choose another User-Agent. Firefox will continue to masquerade as that user agent until you choose something else.
Figure 7-1. User Agent Switcher menu option
To change your User-Agent to Googlebot, for example, simply select Tools User Agent Switcher Googlebot.
To add a user agent, go to Tools User Agent Switcher Options Options... and then choose the "Agents"
option on the left. Figure 7-2 shows the dialog box where you can manage your existing User-Agent strings and add new ones.
Figure 7-2. User Agent Switcher agents dialog
7.7.3. Discussion
There are several online databases of User-Agent strings:
http://www.useragentstring.com/pages/useragentstring.php http://www.tnl.net/ua/
http://www.user-agents.org/
As a quick reference, Table 7-1 lists several popular web browsers and their User-Agent strings, for use in your tests.
Note that these strings are pretty long, and they will be presented across multiple lines. In actuality, they are single strings, with no line breaks or special characters in them.
Table 7-1. Popular User-Agent strings
Web browser User-Agent String
Internet Explorer 6.0 on Windows XP SP2
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
Safari 2.0.4 on MacOS X 10.4.9 Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/419 (KHTML, like Gecko) Safari/419.3
Firefox 2.0.0.3 on Windows XP Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3
Blackberry 7210 BlackBerry7210/3.7.0 UP.Link/5.1.2.9 Treo 600 Smartphone ("Blazer" web
browser)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 95; PalmSource; Blazer 3.0) 16;160x160
Motorola RAZR V3 MOT-V3/0E.40.3CR MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.0 Googlebot (Google's search spiders) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) cURL on MacOS X 10.4.9 curl/7.15.4 (i386-apple-darwin8.9.1) libcurl/7.15.4 OpenSSL/0.9.7l zlib/1.2.3
The User Agent Switcher dialog box will prompt you for a variety of things: appversion, description, platform, useragent, vendor, and vendorsub. These things roughly correspond to the historical components of the User-Agent header. You
don't need to worry about them, however. You can simply put the entire string in the useragent field and it will work as you expect.
Some developers will wrongly view cURL as a "hacker tool" and will want to recognize its User-Agent and deny access to anyone using cURL. This is a misguided security effort, as you should realize from reading this recipe. Anyone using cURL (or wget, or fetch, or a Perl script) can change their User-Agent to impersonate anything they want. Rejecting requests from cURL doesn't really keep a competent hacker out at all.
Recipe 7.8. Imitating a Search Engine with cURL 7.8.1. Problem
Your web application reacts to the User-Agent header, and you want to see how the web page looks when Google, Yahoo!, MSN, or some other robot crawls your site. This may be necessary, especially from a security standpoint, to be sure that no confidential information is being leaked when a robot crawls the site or application.
7.8.2. Solution
See Example 7-6.Example 7-6. Fetching a page as googlebot