LogLogic Blue Coat ProxySG
Log Configuration Guide
Document Release: September 2011 Part Number: LL600012-00ELS100001
This manual supports LogLogic Blue Coat ProxySG Release 1.0 and later, and LogLogic Software Release 5.1 and later until replaced by a new edition.
© 2011 LogLogic, Inc. Proprietary Information
This document contains proprietary and confidential information of LogLogic, Inc. and its licensors. In accordance with the license, this document may not be copied, disclosed, modified, transmitted, or translated except as permitted in writing by LogLogic, Inc.
Trademarks
LogLogic and the LogLogic logo are trademarks or registered trademarks of LogLogic, Inc. in the United States and/or foreign countries. All other company or product names are trademarks or registered trademarks of their respective owners.
Notice
The information contained in this document is subject to change at any time without notice. All warranties with respect to the software and accompanying documentation are set our exclusively in the Software License Agreement or in the Product Purchase Agreement that covers the
documentation.
LogLogic, Inc. 110 Rose Orchard Way, Suite 200 San Jose, CA 95134 Tel: +1 408 215 5900 Fax: +1 408 774 1752 U.S. Toll Free: 888 347 3883 http://www.loglogic.com
Preface
About This Guide . . . . 5
Technical Support . . . . 5
Documentation Support . . . 5
Conventions. . . 6
Chapter 1 – Configuring Blue Coat ProxySG and the LogLogic Appliance Introduction to Blue Coat ProxySG . . . 7
Prerequisites . . . 8
Configuring Blue Coat ProxySG . . . 8
Enabling Access Logging. . . 8
Setting Up Access Log Uploads . . . 10
Enabling the LogLogic Appliance to Capture Log Data . . . 16
Configuring the LogLogic Appliance for File Collection . . . 16
Adding a Blue Coat ProxySG Device. . . 17
Importing the LogLogic SSL Certificate into the Blue Coat ProxySG Device . . . 21
Creating File Transfer Rules . . . 24
Verifying the Configuration . . . 26
Chapter 2 – How LogLogic Supports Blue Coat ProxySG How LogLogic Captures Blue Coat ProxySG Log Data . . . 27
Supported Blue Coat ProxySG Log Data Formats . . . 29
LogLogic Real-Time Reports . . . 30
Chapter 3 – Troubleshooting Verifying that Log Messages are Uploaded to the LogLogic Appliance . . . 31
Appendix A – Blue Coat ProxySG Best Practices Blue Coat ProxySG Best Practices . . . 33
Optimized and Non-Optimized Report Fields . . . 33
Blue Coat Best Practices Reporting Package . . . 36
Appendix B – Format Reference LogLogic Support for Blue Coat ProxySG . . . 39
Contents
Preface
About This Guide
The LogLogic® Appliance-based solution lets you capture and manage log data from all types of log sources in your enterprise. The LogLogic support for Blue Coat® ProxySG enables LogLogic Appliances to capture audit logs from machines running Blue Coat ProxySG.
Once the logs are captured and parsed, you can generate reports and create alerts on Blue Coat ProxySG’s operations. For more information on creating reports and alerts, see the LogLogic Users Guide and LogLogic Online Help.
Technical Support
LogLogic is committed to the success of our customers and to ensuring our products improve customers' ability to maintain secure, reliable networks. Although LogLogic products are easy to use and maintain, occasional assistance might be necessary. LogLogic provides timely and comprehensive customer support and technical assistance from highly knowledgeable,
experienced engineers who can help you maximize the performance of your LogLogic Appliances. To reach LogLogic Customer Support:
Telephone: Toll Free—1-800-957-LOGS Local—1-408-834-7480
EMEA or APAC: + 44 (0) 207 1170075 or +44 (0) 8000 669970 Email: support@loglogic.com
You can also visit the LogLogic Support website at: http://www.loglogic.com/services/support. When contacting Customer Support, be prepared to provide:
Your name, email address, phone number, and fax number
Your company name and company address
Your machine type and release version
A description of the problem and the content of pertinent error messages (if any)
Documentation Support
Your feedback on LogLogic documentation is important to us. Send e-mail to
DocComments@loglogic.com if you have questions or comments. Your comments will be reviewed and addressed by the LogLogic technical writing team.
In your e-mail message, please indicate the software name and version you are using, as well as the title and document date of your documentation.
Conventions
LogLogic documentation uses the following conventions to highlight code and command-line elements:
A monospace font is used for programming elements (such as code fragments, objects, methods, parameters, and HTML tags) and system elements (such as filenames, directories, paths, and URLs).
A monospace bold font is used to distinguish system prompts or screen output from user responses, as in this example:
username: system
home directory: home\app
A monospace italic font is used for placeholders, which are general names that you replace with names specific to your site, as in this example:
LogLogic_home_directory\upgrade\
Straight brackets signal options in command-line syntax. For example: ls [-AabCcdFfgiLlmnopqRrstux1] [-X attr] [path ...]
Chapter 1 – Configuring Blue Coat ProxySG and the
LogLogic Appliance
This chapter describes the tasks you must perform to properly configure the LogLogic Appliance and Blue Coat ProxySG device to log web usage information such as HTTP traffic and Instant Messenger traffic. In addition, this chapter covers how to configure your systems to work with or without SSL authentication enabled.
Introduction to Blue Coat ProxySG . . . 7
Prerequisites . . . 8
Configuring Blue Coat ProxySG . . . 8
Enabling the LogLogic Appliance to Capture Log Data . . . 16
Verifying the Configuration . . . 26
Introduction to Blue Coat ProxySG
The Blue Coat ProxySG device is designed to integrate protection and control functions for Internet and intranet traffic without sacrificing performance and employee productivity. The device assists you in managing internet abuse by increasing security, limiting liability, and managing bandwidth usage.
The LogLogic Appliance enables you to capture log data and report on critical points of your Blue Coat ProxySG internet access control solution including web cache usage. LogLogic provides an additional level of support by enabling you to generate reports and run searches on data to improve your ability to manage your Blue Coat ProxySG activity. Blue Coat ProxySG log data can be transferred to the LogLogic Appliance using one of the following methods:
Continuous Uploads – Data is uploaded directly to the Appliance using HTTP or HTTPS Periodic Uploads – Data is uploaded to a Host Server and then transferred to the Appliance using FTP(S), SFTP, or SCP
The configuration procedures for Blue Coat ProxySG and the LogLogic Appliance depend upon the method you select for your environment. For more information, see How LogLogic Captures Blue Coat ProxySG Log Data on page 27 and Defining the Upload Schedule on page 10.
Prerequisites
Prior to configuring the Blue Coat ProxySG and LogLogic Appliance, ensure that you meet the following prerequisites:
Blue Coat ProxySG installed
Proper access permissions to make configuration changes LogLogic Appliance running Release 5.1 or later installed Administrative access on the LogLogic Appliance
3rd-party FTP, FTP(S), HTTP(S), CIFS, SCP, and/or SFTP server software installed for any platform that does not have these capabilities by default. For more information, see
Configuring the LogLogic Appliance for File Collection on page 16.
Note: If you are using a periodic upload schedule for Blue Coat ProxySG, you must configure a Host Server with the proper 3rd-party file transfer software. For more information, see Defining the Upload Schedule on page 10.
Configuring Blue Coat ProxySG
This section describes the configuration steps setting up general access logging on Blue Coat ProxySG.
Enabling Access Logging
Access logging enables you to monitor web traffic for your environment. The Blue Coat ProxySG device can be set up to generate real-time or schedule logs and reports. Once you set up the Blue Coat ProxySG device for access logging, you must enable it to send the logs to the LogLogic Appliance or a remote Host Server (e.g., SFTP Server).
Note: The following steps walk you through the basic procedure for enabling access logging. For more information on setting up the Blue Coat ProxySG for access logging, see the Blue Coat ProxySG Product Documentation.
To enable access logging:
1. In the Blue Coat Management Console navigation menu, select Access Logging > General. The Default Logging tab appears. This window might appear differently depending on the version of Blue Coat ProxySG you are running.
Figure 1 Blue Coat ProxySG Access Logging > General > Default Logging Tab
2. Select the Enable Access Logging checkbox.
3. For each of the following protocols, click Edit, edit the entry as noted, then click OK. HTTP/HTTPS—main
FTP—main SOCKS—main TCP-Tunnel—main ICP—main
Instant Messaging—IM Windows Media—streaming Real Media/QuickTime—streaming
Setting Up Access Log Uploads
After you enable access logging on Blue Coat ProxySG, you must define an upload schedule. The upload schedule dictates how and when logs are transferred to the LogLogic Appliance or a remote Host Server (e.g., SFTP Server).
Defining the Upload Schedule
The Blue Coat ProxySG device provides you the ability to upload messages continuously (near real-time) at schedule intervals (using a batch process). Messages sent continuously are available near real-time because of the processing time and network throughput between the Blue Coat ProxySG device and the LogLogic Appliance. If you intend to send messages continuously, then you must send the logs via HTTP or HTTPS using the HTTP Client. For more information, see Setting Up Logging for the HTTP Client on page 12.
Messages sent periodically, such as hourly or once a day, are first batched, then saved to disk, and then uploaded as scheduled. If you intend to send messages periodically, then you can must send the logs via a 3rd-party FTP(S), SFTP, or SCP remote Host Server.
Caution: If you choose to use periodic uploads, then you must ensure that the time between schedule uploads is greater than the amount of time it takes to upload the batch file containing multiple log entries. The batch file may become large over time, so uploading this file may not complete before the next scheduled upload.
To define the upload schedule:
1. In the Blue Coat Management Console navigation menu, go to Access Logging > Logs > Upload Schedule tab.
Figure 2 Blue Coat ProxySG Access Logging - Upload Schedule tab.
2. In the Log drop-down menu, select main. To enable continuous logging, go to Step 3. To enable periodic logging, go to Step 4.
1. Complete the following steps to modify the upload schedule settings for continuous logging:
a.In the Upload type area, in the Upload access log section, select the continuously radio button.
b.Set the timing for Wait between connection attempts and Time between keep-alive packets text fields.
c.In the Rotate the log file section, set the scheduled time to Every 1 hour.
LogLogic recommends that you specify the log rollover time to be greater than or equal to 1 hour so that the logs can be parsed and can be viewed in real-time reports. IMPORTANT! If you accumulated large legacy access log files, you must specify a large value in the Rotate the log file section to transfer files to your LogLogic Appliance. If the value selected is too small, then the access log feed is restarted before the transfer of the legacy data to the LogLogic Appliance is complete.
2. Complete the following steps to modify the upload schedule settings for periodic logging:
a.In the Upload type section, select the periodically radio button.
Caution: If you select periodically, you do not see any logs from the Blue Coat ProxySG device on the Host Server (i.e., SFTP Server) until the time set in the Rotate the log file section.
b.Set the timing for the Wait between connection attempts text field.
c.In the Rotate the log file area, set the scheduled time to upload the log file.
d.Click Apply to save the changes.
The Blue Coat ProxySG device establishes a new connection when data is available. This connection is maintained for as long as the Blue Coat ProxySG device is sending data. The Blue Coat ProxySG access log data is parsed as soon as it is received by the LogLogic Appliance.
Setting Up Logging for the HTTP Client
Using HTTP Client for access log uploads enables the LogLogic Appliance to receive Blue Coat ProxySG messages near real-time (i.e., continuously). You can create an HTTP or an HTTPS upload client through the HTTP Client dialog.
Note: If you are using a periodic upload schedule, then you do not need to configure the HTTP Client.
To set up access logging for HTTP Client:
1. From the Blue Coat Management Console, click Access Logging > Logs. The Logs tab appears.
Figure 3 Blue Coat ProxySG Access Logging - Upload Client Tab.
3. In the Log drop-down menu, select main.
4. In the Client Type drop-down menu, select HTTP Client.
5. Complete the following steps to configure the HTTP Client settings:
a.In the Upload Client area, click Settings.
Figure 4 Blue Coat ProxySG HTTP Client settings: Log main window
b.In the HTTP Client settings: Log main window, set the following for HTTP server connection information:
Settings for—Select Primary HTTP Server from the drop-down menu Host—IP address of the LogLogic Appliance.
Note: If you select the Use secure connections (SSL) checkbox, the hostname must match the hostname in the certificate presented by the server.
Port—Use port number 4433
Path—Directory path where the access log facility is uploaded to the server Username—User name to log into the LogLogic Appliance
Change Password—Password to log into the LogLogic Appliance Filename—Leave the default entry
c.For SSL only, select the Use secure connections (SSL) checkbox. If you are not using SSL, proceed to Step 6.
You must associate an SSL CA Certificate from the Host machine (i.e., LogLogic Appliance). For more information, see Importing the LogLogic SSL Certificate into the Blue Coat ProxySG Device on page 21.
6. Click OK.
7. In the Upload Client tab, click Apply.
Note: If you run Test Upload, you might receive a failure message. However, this does not mean that you have configured the Blue Coat ProxySG device and LogLogic Appliance incorrectly.
Verifying Access Log Formats
After setting up the Blue Coat ProxySG device to upload messages to the LogLogic Appliance or a Host Server, verify the access log formats.
To verify access log formats:
1. From the Blue Coat Management Console navigation menu, go to Access Logging > Formats.
The Formats page appears.
Figure 5 Blue Coat Access Logging - Formats Page
2. Verify that the Log Formats in the listing are correct and make changes as necessary. For more information on log formats, see the Blue Coat ProxySG product documentation.
Enabling the LogLogic Appliance to Capture Log Data
The following sections describe how to enable the LogLogic Appliance to capture Blue Coat ProxySG log data.
Configuring the LogLogic Appliance for File Collection
The LogLogic Appliance captures Blue Coat ProxySG logs using file pull functionality via a file transfer rule. The deployment method you use to collect Blue Coat ProxySG file-based data depends on the upload schedule (i.e., periodic or continuous) you selected during the configuration procedure. For more information, see Setting Up Access Log Uploads on page 10.
Blue Coat ProxySG File Collection Using a Periodic Upload Schedule
If you defined a periodic upload schedule (see Defining the Upload Schedule on page 10), then you need to use the following deployment method for file collection:
1. Configure a remote Host Server with file transfer capability to capture log files from the Blue Coat ProxySG host machine.
The following procedure explains, at a high-level, how to configure your environment to capture file-based log messages via SFTP. LogLogic recommends using SFTP for
Windows-based systems, or SCP for Unix-based systems, to securely transfer files to the LogLogic Appliance from your log source. However, you can use any of the
LogLogic-supported protocols in your environment (i.e., FTP(S), HTTP(S), SCP, etc.). Note: For more information on each supported protocol, including whether a Public Key Copy is needed and what search methods (i.e., CSV, Wildcard) are available, see the LogLogic
Administration Guide.
a.Make sure that a destination directory (i.e., log directory) exists and is accessible on the host machine where Blue Coat ProxySG is installed.
The destination directory should contain the original log files that Blue Coat ProxySG generates.
b.Transfer the Blue Coat ProxySG log files to a separate publishing directory on a remote Host Server.
You can use a script or 3rd-party software that makes a copy of or moves the log files from the destination directory (i.e., log directory) to the publishing directory. In addition, if you are using a script, you can specify the schedule for when the script runs (e.g., hourly, daily, or weekly).
Note: LogLogic recommends that you define a clean-up process to handle old log files that accumulate over time.
2. On the LogLogic Appliance, add Blue Coat ProxySG to the Appliance as a new device. For more information, see Adding a Blue Coat ProxySG Device on page 17.
3. Create a file transfer rule and specify SFTP as the Protocol. For more information, see Creating File Transfer Rules on page 24.
IMPORTANT! SCP and SFTP have limitations in their ability to pull a large number of files (100 or more). LogLogic recommends that you compress the files into a single file (such as .tar or tar.gz) before the files are pulled by the LogLogic Appliance.
Blue Coat ProxySG File Collection Using a Continuous Upload Schedule
If you defined a continuous upload schedule (see Defining the Upload Schedule on page 10), then you need to use the following deployment method for file collection:
1. Properly configure the HTTP Client on Blue Coat ProxySG (see Setting Up Logging for the HTTP Client on page 12).
2. On the LogLogic Appliance, add Blue Coat ProxySG to the Appliance as a new device. For more information, see Adding a Blue Coat ProxySG Device on page 17.
3. Create a file transfer rule and specify HTTP or HTTPS as the Protocol. For more information, see Creating File Transfer Rules on page 24.
Adding a Blue Coat ProxySG Device
LogLogic captures Blue Coat ProxySG log files using the file pull method. You must add the server as a new device so LogLogic can properly handle the log file data to make it available through reports and searching. Once you have successfully added an Blue Coat ProxySG device, you must configure file transfer rules for file collection. For more information, see Configuring the LogLogic Appliance for File Collection on page 16.
To add Blue Coat ProxySG as a new device:
1. Log in to the LogLogic Appliance.
2. From the navigation menu, click Management > Devices. The Devices tab appears.
Figure 6 Management > Devices > Devices Tab
3. On the Devices tab, click Add New. The Add Device tab appears.
Figure 7 Add Device Tab
4. On the Add Device tab, complete the following information:
Name—Name for the Blue Coat ProxySG device. For example, BlueCoat_LX102, where LX102 is the LogLogic Appliance platform and last three digits of the IP address for the appliance.
Description (optional)—Description of the Blue Coat ProxySG device Device Type—Select Blue Coat ProxySG from the drop-down menu
Host IP—IP address of the machine hosting the Blue Coat ProxySG log data (this can be a remote Host Server or a machine with Blue Coat ProxySG installed)
Enable Data Collection—Select the Yes radio button
Refresh Device Name through DNS Lookups (optional)—Select this checkbox to enable the Name field to be automatically updated. The name is obtained using a reverse DNS lookup on the configured refresh interval. The DNS name overrides any manual name you assign.
Use User Authentication (optional)—Select this checkbox to enable user name and password authentication for the Blue Coat ProxySG device.
Use SSL (optional)—If you are using an SSL Certificate to enable encryption of transferred files, then proceed with Step 5. If you are not using SSL Certificates, then go to Step 6.
Figure 8 Add Device tab populated with Example Content
1. (Optional) Complete this step only if you are using SSL to encrypt transferred files. You must enable SSL in the LogLogic Appliance, and then export the auto-generated SSL Certificate to the Blue Coat ProxySG device.
a.Select the Use SSL checkbox.
b.Click anywhere in the SSL Certificate text box.
c.Press the Ctrl + A keys to select all text for the certificate, and then press the Ctrl + C keys to copy the text to the clipboard.
2. Verify that all fields are populated correctly, and then click Add to add the device. The Devices tab appears.
3. Verify that the device is in the list of devices. Click the device name to check the values you entered for the device.
4. If you are using SSL, then go to Importing the LogLogic SSL Certificate into the Blue Coat ProxySG Device on page 21 and complete the steps to import the LogLogic SSL Certificate into the Blue Coat ProxySG device.
Importing the LogLogic SSL Certificate into the Blue Coat
ProxySG Device
This section only applies if you are using SSL certification to encrypt transferred files between the Blue Coat ProxySG device and the LogLogic Appliance. If you are using SSL, then you must import the SSL Certificate from the LogLogic Appliance into Blue Coat ProxySG.
To import the LogLogic Appliance SSL Certificate into the Blue Coat ProxySG device:
1. Log in to the Blue Coat ProxySG web proxy appliance.
a.Open a web browser using https and port 8082.
For example, https://10.1.1.102:8082.
b.Enter the user name and password.
The Blue Coat ProxySG web interface appears. Figure 9 Blue Coat ProxySG Web Interface
2. Click the Management Console link.
Figure 10 Blue Coat Management Console
3. From the Blue Coat Management Console, click SSL > CA Certificates. The CA Certificates tab appears.
Figure 11 Blue Coat ProxySG SSL - CA Certificate Tab
4. In the CA Certificates tab click Import. The Import CA Certificate window appears.
5. Click in the CA Certificate text box, and then press and hold the Ctrl + V keys to paste in the LogLogic SSL Certificate.
If you do not have the LogLogic SSL Certificate on your clipboard, return to the LogLogic Appliance and navigate to the Blue Coat ProxySG device details page. In the LogLogic interface, click Administration > Manage Devices, then click the Blue Coat ProxySG device name in the list. Use the Ctrl + A and Ctrl + C keys to select and copy all of the SSL Certificate text.
Figure 12 Example Import CA Certificate Entries.
6. Click OK to import the LogLogic SSL Certificate into the Blue Coat ProxySG device. Note: A dialog box may prompt you for the user name and password of the Blue Coat ProxySG device.
7. Verify that the LogLogic SSL Certificate appears in the list on the CA Certificates tab.
8. Click Apply to save changes.
Creating File Transfer Rules
After you have added your Blue Coat ProxySG device, you can create a File Transfer Rule for the access log files. File Transfer Rules enable the LogLogic Appliance to pull files from the Host Server publishing the Blue Coat ProxySG log files.
LogLogic supports the following wildcards: * (asterisk), ? (question mark), and [...] (open and close brackets) using directory queries. If you use wildcards, you must enable directory listing on your Host Server.
Examples: file
/foo/file, /bar/*.log
/foo?/bar*/*.aud, /foo1/file1.tar.gz, /foo1/file2.Z /foo[2-8]/bar*/net*.log
LogLogic can pull and decompress archive files, extract individual files from the archive files, and then process the individual files. The following file types are supported: .tar.bz2, .tar.gz, tar.Z, .tgz, .taz, .tar, .gz, .z, .Z, .zip, .ZIP. For more information, see the LogLogic Administration Guide.
To create a file transfer rule:
1. Log in to the LogLogic Appliance.
2. From the navigation menu, select Management > Devices.
3. Select the File Transfer Rule tab.
4. Add a rule for the Blue Coat ProxySG log files you want to capture by completing the following steps:
a.From the Device Type drop-down menu, select the machine where Blue Coat ProxySG is installed.
b.From the Device drop-down menu, select the appropriate Blue Coat ProxySG device.
Note: If you have added only one Blue Coat ProxySG device, the device name is automatically added.
c.Click Add New then enter the appropriate information for the following required fields:
Rule Name—Name of the transfer rule (e.g., Blue Coat ProxySG log files) Protocol—Specify the appropriate protocol (e.g., HTTP(S) if you are using a continuous upload schedule from ProxySG, SFTP, FTP(S), SCP, etc. if you are using a periodic upload schedule)
Note: LogLogic recommends using a secure file transfer protocol, such as SFTP for Windows-based devices or SCP for UNIX-based devices. If you are using SFTP or SCP, you must copy the
Appliance’s public key to the machine where the logs are located. For more information, see Configuring the LogLogic Appliance for File Collection on page 16 and the LogLogic Administration Guide.
User ID—Specify only if the protocol requires a User ID
Password/Verify Password—Specify only if required for the User ID
Files—Full path (after the IP address) to the Blue Coat ProxySG machine or the Host Server where the Blue Coat ProxySG log files are located. For example:
/log/file_name.log
To capture all logs in a specific directory specify the asterisk (*) wildcard. For example:
/log/*.zip
The server can be the host machine where the device is installed or a remote Host Server with file transfer functionality. For more information, see Configuring the LogLogic Appliance for File Collection on page 16.
File Format—Select W3C from the drop-down menu
Collection Time—Specify the time you want to retrieve the log file from the Host Server (i.e., a Host Server or the Blue Coat ProxySG machine)
File Transfer History—Click this button if you want to see the file transfer history. Use Advanced Duplication Detection—Select the Yes radio button if you want the LogLogic Appliance to check for duplicate data while capturing the Blue Coat ProxySG logs.
Enable—Select the Yes radio button to enable the File Transfer Rule
Verifying the Configuration
The section describes how to verify that the configuration changes made to Blue Coat ProxySG and the LogLogic Appliance are applied correctly.
To verify the configuration:
1. Log in to the LogLogic Appliance.
2. From the navigation menu, select Dashboards > Log Source Status. The Log Source Status tab appears.
3. Locate the IP address for each Blue Coat ProxySG device.
If the device name (Blue Coat ProxySG) appears in the list of devices, then the configuration is correct. If the device does not appear in the Log Source Status tab, check the Blue Coat ProxySG logs for events that should have been sent. If events were detected and are still not appearing on the LogLogic Appliance, verify the Blue Coat ProxySG configuration and the LogLogic Appliance configuration.
You can also verify that the LogLogic Appliance is properly capturing log data from Blue Coat ProxySG by trying to view the data in the reports. LogLogic recommends checking the reports to make sure that the data obtained is valid and matches expectations. For more information, see LogLogic Real-Time Reports on page 30.
If the device name appears in the list of devices but event data for the device is not appearing within your reports, you need to verify that the LogLogic Appliance is actually capturing the logs properly. For more information, see Verifying that Log Messages are Uploaded to the LogLogic
Chapter 2 – How LogLogic Supports Blue Coat
ProxySG
This chapter describes LogLogic’s support for Blue Coat ProxySG. LogLogic enables you to track web usage information captured by the Blue Coat ProxySG device in real-time or on a scheduled basis.
How LogLogic Captures Blue Coat ProxySG Log Data . . . 27
Supported Blue Coat ProxySG Log Data Formats . . . 29
LogLogic Real-Time Reports . . . 30
How LogLogic Captures Blue Coat ProxySG Log Data
Access logs are made available in real-time (i.e., continuously) or at scheduled intervals (i.e., periodically) and contain log records that can be made specific to each protocol. For example, you can track the network traffic flowing through your environment and append an access log at the end of each transaction for a specified protocol such as HTTP or HTTPS.
Access logs are directed to specified log facilities. Log facilities associate logs with a configured format and upload schedule. Log facilities can also be encrypted and digitally signed before they are uploaded. All log data stored in log facilities can be uploaded to a LogLogic Appliance for analysis and future archiving.
Blue Coat ProxySG log data can be uploaded directly to the LogLogic Appliance using HTTP or HTTPS if you are uploading files continuously. Log data can also be uploaded to a remote Host Server and then transferred to the Appliance using FTP(S), SFTP, or SCP if you are uploading files periodically. For more information, see Defining the Upload Schedule on page 10.
Figure 13 on page 28 provides a deployment example for capturing Blue Coat ProxySG access log messages continuously from the HTTP Client. Figure 14 on page 28 provides a deployment example for capturing Blue Coat ProxySG access log messages periodically from a remote Host Server. An SFTP server is used as a remote Host Server in the periodic upload example. For more information, see Configuring the LogLogic Appliance for File Collection on page 16.
Figure 13 Continuous File Upload Functionality (using HTTP(S)) and Connectivity for Blue Coat ProxySG and a LogLogic Appliance
Figure 14 Periodic File Upload Functionality (using SFTP) and Connectivity for Blue Coat ProxySG and a LogLogic Appliance
All Blue Coat ProxySG log data captured by the LogLogic Appliance is parsed and made available to the LogLogic Agile Report Engine and search. The Agile Report Engine provides report templates that can be run as-is or modified to create customized reports targeting specific
information. For example, you can create customized reports that display the top cache server and status for a given time frame, the most requested file types for a given time frame, or the top URLs requested for a given time frame. For more information, see LogLogic Real-Time Reports on page 30. In addition, the data is made available so you can create alerts that notify you of issues on your Blue Coat ProxySG device. For more information on creating reports and alerts, see the LogLogic User Guide and LogLogic Online Help.
Note: Blue Coat ProxySG records log data using GMT time while the LogLogic Appliance reports data using local time. In LogLogic releases prior to 3.2.3 Hotfix 2, you must consider the time difference of when log data was captured by the Blue Coat ProxySG device and the current time of the LogLogic Appliance. If you have LogLogic 3.2.3 Hotfix 2 or later, then you do not have to calculate the time difference.
LogLogic Appliance Reports Alerts Database log data Internet
Intranet InternetInternet
---End User End
User Blue Coat
ProxySG Website ------ ---HTTP/S Upload (HTTP Client) LogLogic Appliance Reports Alerts Database log data Internet
Intranet InternetInternet
---End User End
User Blue Coat
ProxySG Website --- ---Host Server SFTP Upload
Supported Blue Coat ProxySG Log Data Formats
LogLogic supports access logs in various formats such as ELFF, NCSA/Common, SmartReporter, SQUID, SurfControl, and Websense. Each log format applies to a specific type of log data. For example, ELFF (W3C Extended Log File Format) is a general format with specific implementations such as IM, Main, P2P, and streaming, NCSA/Common contains only basic HTTP access
information, and SQUID contains cache statistics.
Table 1 provides a list of the Blue Coat ProxySG fields that LogLogic parses. Table 1 Blue Coat ProxySG Fields Parsed by LogLogic
bytes s-ip c-dns s-port c-ip sc-bytes cached sc-status cs(Content-Type) time cs(Referer) time-taken cs(User-Agent) x-action cs(X-Forwarded-For) x-connect-time cs-auth-group x-hiercode cs-auth-groups x-play-time cs-method x-product cs-types x-protocol cs-uri x-sc-contentlength cs-uri-query x-smartfilter-categories cs-uri-stem x-smartfilter-result current-time x-status
date x-timestamp r-dns x-transaction r-ip x-username r-port x-virus-details rs(Content-Type) x-virus-id s-dns
LogLogic Real-Time Reports
LogLogic provides pre-configured Real-Time Reports for Blue Coat ProxySG log data. The following Real-Time Reports are available:
All Unparsed Events – Displays data for all events retrieved from the Blue Coat ProxySG log for a specified time interval
Web Cache Activity - Displays locally-stored web cache information served during a specified time interval
To access LMI 5 Real-Time Reports:
1. In the top navigation pane, click Reports.
2. Click Network Activity.
The following Real-Time Reports are available: Web Cache Activity
3. Click Operational.
The following Real-Time Reports are available: All Unparsed Events
You can create custom reports from the existing Real-Time Report templates. For more information, see the LogLogic User Guide and LogLogic Online Help.
Chapter 3 – Troubleshooting
This chapter contains troubleshooting information regarding the configuration and/or use of log collection for Blue Coat ProxySG.
Verifying that Log Messages are Uploaded to the LogLogic Appliance . . . 31
Verifying that Log Messages are Uploaded to the LogLogic
Appliance
If you set messages to be uploaded continuously, then you can view the results of your tests immediately. If you set messages to be uploaded periodically, then the messages are sent at the schedule time. However, you can force uploads if necessary.
It is possible to test that messages are being uploaded from the Blue Coat ProxySG device to the LogLogic Appliance in many ways. The following steps represent one example of how you can verify that you have set up your environment properly.
Note: The Blue Coat ProxySG upload client uses GMT to when uploading data. In LogLogic versions prior to 3.2.3 Hotfix 2, on the LogLogic Appliance, you must calculate the GMT time equivalent to your local time when searching and reporting on data. For example, if your Blue Coat ProxySG device is located in New York (EST), and you want to run a report on data that occurred from 13:00 to 15:00 (1pm to 3pm), then you must specify GMT time in your reports and searches as 17:00 to 19:00. If you have LogLogic 3.2.3 Hotfix 2 or later, then you do not have to calculate the time difference.
To verify that Log Messages are uploaded to the LogLogic Appliance:
1. On a Microsoft Windows machine, open an Instant Messenger window (e.g., Yahoo! Instant Messenger).
2. Send a message.
3. Log in to the LogLogic Appliance to view the test message sent by the Blue Coat ProxySG device.
4. From the navigation menu, select Summary Reports > Top Web Activity > Web Cache Activity.
The Web Cache Activity tab appears.
5. In the Device Type drop-down menu, select Blue Coat ProxySG.
6. In the Source Device drop-down menu, select All Blue Coat ProxySG.
7. Set the Time Interval to capture the test message you sent. For example, set the specific time to be from 1 hour before you sent the message to the current time.
8. Click Run.
The Web Cache Activity tab is populated with the appropriate data for the device.
9. View the test message and details to verify the message was sent properly.
b.Drill down on a listed item to view the Blue Coat ProxySG message details. The Web Cache Activity tab is populated with the appropriate data.
10. Verify that the test message was sent.
Note: If you do not see the test message in the Web Cache Activity tab, then verify that you completed the steps properly.
Appendix A – Blue Coat ProxySG Best Practices
This appendix describes best practices recommended by LogLogic for use with Blue CoatProxySG. It also provides a description of LogLogic’s Blue Coat Best Practices Reporting Package. Blue Coat ProxySG Best Practices . . . 33
Optimized and Non-Optimized Report Fields . . . 33
Blue Coat Best Practices Reporting Package . . . 36
Blue Coat ProxySG Best Practices
LogLogic recommends adhering to the following best practices while configuring Blue Coat ProxySG and the LogLogic Appliance:
Continuous Log Collection vs. Periodic Log Collection – It is possible to gather and transfer logs using continuous or periodic collection methods. If you intend to transfer over 100 messages per second, LogLogic recommends using the periodic method. For more information, see Defining the Upload Schedule on page 10.
If you use the periodic collection method, make sure that each “chunk” of log data that is transferred from Blue Coat ProxySG to the Host Server is less than 2 Gigabytes (GB) (1.8 GB maximum).
Regardless of the log collection method used (i.e., periodic or continuous), make sure that the network connection between Blue Coat ProxySG, the Host Server (if applicable), and the LogLogic Appliance is as fast and stable as possible. The maximum recommended Blue Coat ProxySG log collection rate, per LogLogic Appliance per day, should be less than 60 million records (or be approximately 21 GB in size).
All LogLogic platform families (i.e., LX, ST, MX) support Blue Coat ProxySG log data. However, for better performance, LogLogic recommends using the LX 2010 platform. LogLogic recommends running Release 4.2 Hotfix 1 or later on the LogLogic Appliance.
Optimized and Non-Optimized Report Fields
Blue Coat ProxySG log data can be viewed within the LogLogic Appliance’s Web Cache Activity report. For more information, see LogLogic Real-Time Reports on page 30. In order to improve reporting performance, LogLogic attempts to pre-summarize log data as it is collected on the Appliance. The log data is stored in a manner that helps optimize the speed at which the data can be retrieved during reporting.
By default, LogLogic summarizes or optimizes specific fields of parsed data collected for the Web Cache Activity report. So, each report field (i.e., Advanced Option) is either optimized or
non-optimized. As a result, reports that exclusively use optimized fields will run faster than reports that include non-optimized fields.
For example, if you ran a report that contained one non-optimized field and three optimized fields, that report’s run time will be constrained by the non-optimized field. Therefore, the report will run slower than a report that contained only optimized fields. Table 2 on page 34 provides a listing of all the optimized and non-optimized fields available within the Web Cache Activity report.
Note: When using periodic collection, it may take approximately one hour for every 8 Gigabytes of batched log data (i.e., approximately 25 million messages collected) to complete the automatic parsing and aggregation processes needed before report generation is possible.
Table 2 Web Cache Activity Report - Advanced Options
Advanced Option Optimized Non-Optimized
Source Device
Source User
Source IP
Source Host
Domain Name
Destination IP
Destination Port
Peer IP
Peer Host
Peer Status
Method
URL
Cache Code
Status
Type
Size
Filter Category
Filter Result
User Agent
Referred By
X Protocol
X Action
X Status
X Product
Virus ID
Virus Details
User Group
Caution: When you drill-down on the Web Cache Activity report's results there is no default sort-by selection. The drill-down results are generally in order by time. If you specify a sort-by selection for this report's drill-down, performance in generating the drill-down results is slower.
The default timeout period for reports is 15 minutes. Some reports might require a run time that exceeds 15 minutes. If you run into this issue, consider the following workarounds:
Shorten the Time Interval for the report
Modify the Advanced Options to include fewer fields
Change the default timeout period on the LogLogic Appliance:
Caution: LogLogic does not recommend changing the default timeout period. However, if you do change the default, you should only use this command on the Appliance where you will run the report.
1. Log in to the LogLogic Appliance’s Command Line Interface (CLI).
2. Type in the following command:
mysql logappconfig -e 'update Settings set maxIdleDBQuery=####;' Where the maxIdleDBQuery value (i.e., ####) is the timeout period (in seconds) for reports. The default is 900 seconds (15 minutes). For example, if you wanted to change the timeout to 7200 seconds (2 hours), the command would be:
mysql logappconfig -e 'update Settings set maxIdleDBQuery=7200;'
3. To save the configuration change on the Appliance, type in the following command: /loglogic/bin/loadsettings
Blue Coat Best Practices Reporting Package
LogLogic also provides a Blue Coat Best Practices Reporting Package that includes pre-defined custom reports that allow you to monitor data regarding user activity, policy violations, etc.
Note: The Blue Coat Best Practices Reporting Package is not part of a Log Source Package (LSP). This reporting package is available upon request from LogLogic’s Account team.
To access Blue Coat Best Practices Reports:
1. Log in to the LogLogic Appliance.
2. From the navigation tree, select Custom Reports.
3. Select BlueCoat Best Practices. The following Custom Reports are available:
Domains by Unique User – Domains visited by unique users yesterday sorted by count Files Downloaded via the Web – All Web-based downloads yesterday greater the 5 Kilobytes (KB)
Files Uploaded via the Web – All Web-based uploads in the past day
Filter Results by Count – Blue Coat filter results for the past day sorted by count
Peer Servers and Status – Yesterday's peer web servers providing data for cache servers and status
Top Bandwidth – Total number of bytes transferred yesterday
Top Categories Visited – Blue Coat filter categories visited yesterday sorted by count Top Domains Accessed – Domains visited yesterday sorted by count
Top Spyware Src IPs – Spyware Source IP addresses sorted by the number of bytes transferred during the past day
Top Src IPs by Bandwidth – Source IP addresses sorted by the number of bytes transferred in the past day
Top URLs Visited – Visited URLs in the past day sorted by count Top Users by Bandwidth – Top users by bandwidth for the past day
Top Users by Page Views – Top users by number of total page views for the past day Web Access to Applications – Access to Web-based applications in the past day
The performance time for each report depends upon the fields used and the amount of data being displayed. Table 3 on page 37 provides a breakdown of each report, the Advanced Options (i.e., fields) used within the report by default, and an estimated run time in minutes. The run time estimate in minutes is based upon log data containing approximately 35 million messages (or be approximately 12.25 GB in size).
Note: There is a direct correlation between longer report run times and reports that make use of non-optimized fields. For more information, see Optimized and Non-Optimized Report Fields on page 33 and Table 2 on page 34.
Note: The Advanced Options listed here do not represent all of the options available within a report. The options listed only represent the fields that are selected to be displayed by default. Table 3 Best Practices Reports, Advanced Options, and Run Times
Report Default Advanced Options Run Time (mins)
Domains by Unique User Source Device, Source User, Domain Name, Count 10.76 Files Downloaded via the Web Source Device, Source IP, Domain Name, Method, Status, Size, Count 8.21 Files Uploaded via the Web Source Device, Source IP, Method, Status, Size, Count 0.04 Filter Results by Count Source Device, Filter Result, Count 0.12 Peer Servers and Status Source Device, Peer IP, Peer Host, Peer Status, Size, Count 12.42 Top Bandwidth Source Device, Size 0.01 Top Categories Visited Source Device, Filter Category, Count 0.13 Top Domains Accessed Source Device, Domain Name, Count 9.91 Top Spyware Src IPs Source Device, Source IP, Size, Filter Category, Count 0.02 Top Src IPs by Bandwidth Source Device, Source IP, Size, Count 0.04 Top URLs Visited Source Device, Domain Name, URL, Count 62.10 Top Users by Bandwidth Source Device, Source User, Size 0.01 Top Users by Page Views Source Device, Source User, Count 0.01 Web Access to Applications Source Device, Source User, Domain Name, URL, Status, Count 20.00
Appendix B – Format Reference
This appendix describes the log format and report supported by Blue Coat ProxySG.
LogLogic Support for Blue Coat ProxySG
Table 4 LogLogic Support for Blue Coat ProxySG
Supported
Log Format date time time-taken c-ip sc-status s-action sc-bytes cs-bytes cs-method cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-username cs-auth-group s-hierarchy s-supplier-name rs(Content-Type) cs(Referer) cs(User-Agent) sc-filter-result cs-categories x-virus-id s-ip
Agile Reports Web Cache Activity
Sample Log 2006-03-29 16:42:21 31 192.168.0.150 200 - 81 199 CONNECT tcp 192.168.0.253 8082 / - - - NONE 192.168.0.253 - - "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1" PROXIED "none" - 192.168.0.253
Field Description date - The date on which the activity occurred.
time - The time, in coordinated universal time (UTC), at which the activity
occurred.
time-taken – Time taken for transaction to complete in seconds. c-ip - The IP address of the client that made the request. sc-status - The HTTP status code.
s-action – What type of action did the appliance take to process this request. sc-bytes - The number of bytes that the server sent.
cs-bytes - The number of bytes that the server received. cs-method - The requested action, for example, a GET method.
Blue Coat ProxySG logs can be classified according to the values in the
cs-method field.
HEAD GET POST PUT DELETE TRACE OPTIONS
cs-uri-scheme – Scheme from the 'log' URL. cs-host - The host header name, if any. cs-uri-port – Port from the 'log' URL.
cs-uri-path – Path from the 'log' URL. Does not include query.
cs-uri-query – The query, if any, that the client was trying to perform. A
Universal Resource Identifier (URI) query is necessary only for dynamic pages.
cs-username – The name of the authenticated user who accessed your
LogLogic Support for Blue Coat ProxySG -- continued
Field Description (continued)
cs-auth-group – One group that an authenticated user belongs to. If a user
belongs to multiple groups, the group logged is determined by the Group Log Order configuration specified in VPM. If Group Log Order is not specified, an arbitrary group is logged. Note that only groups referenced by policy are considered.
s-hierarchy – How and where the object was retrieved in the cache
hierarchy.time-taken – Time taken for transaction to complete in seconds.
cs-supplier-name – Hostname of the upstream host (not available for a
cache hit).
rs(Content-Type) - Response header: Content-Type.
cs(Referer) – The site that the user last visited. This site provided a link to
the current site.
cs(User-Agent) – The browser type that the client used.
sc-filter-result – Content filtering result: Denied, Proxied, or Observed. cs-categories – All content categories of the request URL
x-virus-id – Identifier of a virus if one was detected
