Web Log Data Collection

Data gathered from web servers is placed into special files called logs and can be used for web usage mining. Usually this data is called web log data as all visitors activities are logged into this file (Kosala et al. 2000). In real life web log files are huge source of information. For example, the web log file generated running online information site2 produces log with the size of 30 – 40 MB in one month time, another advertising company3 collects the file of the size of 6 MB during one day.

There are many commercial web log analysis tools (Angoss; Clementine; MINEit; NetGenesis). Most of them focus on statistical information such as the largest number of users per time period, business type of users visiting the web site (.edu, .com) or geographical location (.uk, .be), pages popularity by calculating number of times they have been visited and etc. However, statistics without describing relationships between visited pages consequently leave much valuable information undiscovered (Pitkow et al. 1994a; Cooley et al. 1997a). This lack of depth of analytic scope has stimulated web log research area expand to an individual research field beneficial and vital to e-business components.

The main goals which might be achieved mining web log data are the following:

• Web log examination enables to restructure the web site to let clients access the desired pages with the minimum delay. The problem of how to identify usable structures on the WWW related with understanding what

2

http://www.munichfound.de/

3

facilities are available for dealing with this problem and how to utilize them (Pirolli et al. 1996).

• Web log inspection allows improving navigation. This can manifest itself by organizing important information into the right places, managing links to other pages in the correct sequence, pre-loading frequently used pages.

• Attracting more advertisement capital by placing adverts into the most frequently accessed pages.

• Interesting patterns of customer behaviour can be identified. For example, valuable information can be gained by discovering the most popular paths to the specific web pages and paths users take upon leaving these pages. These findings can be used effectively for redesigning the web site to better channel users to specific web pages.

• Turning non-customers into customers increasing the profit (Faulstich et al. 1999). Analysis should be provided on both groups: customers and non- customers in order to identify characteristic patterns. Such findings would help to review customers’ habits and help site maintainers to incorporate these observed patterns into the site architecture and thereby assist turning the non-customer into a customer.

• From empirical findings (Tauscher et al. 1997) is observed that people tend to revisit pages just visited and access only a few pages frequently. Humans browse in small clusters of related pages and generate only short sequences of repeated URLs. This shows that there is no need to increase number of information pages on the web site. More important is to concentrate to the efficiency of the material placed and accessibility of these clusters of pages.

General benefits obtained from analysing Web logs are allocating resources more efficiently, finding new growth opportunities, improving marketing campaigns, new product planning, increasing customer retention, discovering cross selling opportunities and better forecasting.

The common log format

Various web servers generate different formatted logs: CERF Net, Cisco PIX, Gauntlet, IIS standard/Extended, NCSA Common/Combined, Netscape Flexible, Open Market Extended, Raptor Eagle. Nevertheless, the most common log format is common log format (CLF) and appears exactly as follows (see Fig 1.7):

Fig 1.7. Example of the Common Log Format: IP address, authentication (rfcname and

logname), date, time, GTM zone, request method, page name, HTTP version, status of the page retrieving process and number of bytes transferred

host/ip rfcname logname [DD/MMM/YYYY:HH:MM:SS -

0000]"METHOD /PATH HTTP/1.0" code bytes

• host/ip - is visitor’s hostnameor IP address.

• rfcname - returns user’s authentication. Operates by looking up specific TCP/IP connections and returns the user name of the process owning the connection. If no value is present, a "-" is assumed.

• logname - using local authentication and registration, the user's log name will appear; otherwise, if no value is present, a "-" is assumed.

• DD/MMM/YYYY:HH:MM:SS - 0000 this part defines date consisted of the day (DD), month (MMM), years (YYYY). Time stamp is defined by hours (HH), minutes (MM), seconds (SS). Since web sites can be retrieved any time of the day and server logs user’s time, the last symbol stands for the difference from Greenwich Mean Time (for example, Pacific Standard Time is -0800).

• method - methods found in log files are PUT, GET, POST, HEAD (Savola et al. 1996). PUT allows user to transfer/send a file to the web server. By default, PUT is used by web site maintainers having administrator’s privileges. For example, this method allows uploading files through the given form on the web. Access others then site maintaining is forbidden. GET transfers the whole content of the web document to the user. POST sends information to the web server that a new object is created and linked. The content of the new object is enclosed as the body of the request and is transferred to the user. Post information usually goes as an input to the Common Gateway Interface (CGI) programs. HEAD demonstrates the header of the “page body”. Usually it is used to check the availability of the page.

• path stands for the path and files retrieved from the web server.

• HTTP/1.0 defines the version of the protocol used by the user to retrieve information from the web server.

• code identifies the success status. For example, 200 means that the file has been retrieved successfully, 404 - the file was not found, 304 - the file was reloaded from cache, 204 indicates that upload was completed normally and etc.

• bytes number of bytes transferred from the web server to another machine.

Fig 1.8. CLF followed by additional data fields: web page name visitor gets from -

referrer page, browser type and cookie information

It is possible to adjust web server’s options to collect additional information such as REFERER_URL, HTTP_USER_AGENT and HTTP_COOKIE (Fleishman 1996). REFERER_URL defines URL names where from visitors came. HTTP_USER_AGENT identifies browser’s version the visitors use. HTTP_COOKIE variable is a persistent token which defines visitors identification number during browsing sessions. Then CLF is a form depicted in Fig 1.8.