Tech Note--Audit Support for
McAfee Web Gateway
Copyright statement
Copyright (c) Broadcom. All Rights Reserved.
The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries.
Broadcom, the pulse logo, Connecting everything, and Symantec are among the trademarks of Broadcom.
The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. For more information, please visit www.broadcom.com.
Broadcom reserves the right to make changes without further notice to any products or data herein to improve reliability, function, or design. Information furnished by Broadcom is believed to be accurate and reliable. However, Broadcom does not assume any liability arising out of the application or use of this information, nor the application or use of any product or circuit
described herein, neither does it convey any license under its patent rights nor the rights of others.
Table of Contents Introduction
Supported Web Gateway firewall version Supported log formats
Specifying custom log file headers Default log file header format Mandatory fields
Adding additional properties to Logs
Configuring McAfee Web Gateway for auto log push to SpanVA Sample access log
References Revision history
Introduction
This Tech Note describes how the CloudSOC Audit application supports log files from McAfee Web Gateway devices.
Supported Web Gateway firewall version
McAfee Web Gateway minimum supported version is 7.x
Supported log formats
CloudSOC Audit App only supports logs from McAfee Web Gateway in space delimited values format.
Based on how you have configured your McAfee Web Gateway, it can generate the logs with or without a header row. Different log file format and corresponding configuration is described below.
Note: In general all files uploaded for the datasource must have the same log format.
The preferred option is to configure the Web Gateway to embed the headers inside the log file. The headers field is included as the first row in the log file starting with a ‘#’ symbol. If a header row is available, the CloudSOC Audit application parses the fields in the log file(s) based on the field names and ordering as specified in the header row.
An example of embedded headers is shown in the snippet below, which shows first two rows of a log file.
#time_stamp "auth_user" src_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res"
[27/May/2014:23:59:43 +0530] "800069682" 58.2.97.194 304 "GET
http://economictimes.indiatimes.com/photo/19319380.cms HTTP/1.1" "General News"
"Minimal Risk" "" 307 "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)" "" "0"
Specifying custom log file headers
If your log files do not have the header row as the first row, and the order of the fields in the log files do not match the defaults described in Default Log File Header Format, use the Custom Headers tools in Audit to specify the custom headers that apply to your McAfee Web Gateway. Otherwise CloudSOC cannot process the logs correctly for use in the Audit application.
For full procedures on uploading device logs to CloudSOC, see the CloudSOC Tech Note
Uploading Device Logs to CloudSOC Audit.
Important: Do not put any spaces between field names in the Custom Headers specification. Audit considers a space to be part of the delimiter, causing it to parse such headers incorrectly in your logs.
Default log file header format
The Audit application assumes the following default log format for log files without embedded header fields, and where you have not specified custom headers as described in Specifying Custom Log File Headers:
time_stamp "auth_user" src_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client bytes_from_client "user_agent" "virus_name" "block_res" "application_name" If your logs do not adhere to this format, you must either embed the headers or specify custom headers in CloudSOC.
Mandatory fields
The following fields must be present in the logs uploaded to the CloudSOC Audit application: ● time_stamp
● bytes_to_client ● bytes_from_client ● req_line or url
Adding additional properties to Logs
We recommend that you configure the Web Gateway to include the following additional fields if practical:
● server_ip - Identifies destination locations of the traffic.
● uuid - Identifies the gateway device and when multiple devices are sending logs to CloudSOC. You can filter traffic on this identifier in the Audit app.
If you have multiple McAfee Web Gateway devices in your network, you can either define multiple datasource entries in CloudSOC, one per device, or you can send logs from all those devices using the same CloudSOC datasource entry. The latter approach is simpler, and if you choose to do so, you can add an additional attribute to the logs to identify the device that is sending the logs. If CloudSOC sees any field in the log header named as device_id, uuid or hostname, it uses the first such field’s value as the identifier of the gateway sending the logs. That lets you use a single datasource to collect logs from all your gateways while still retaining the ability to drill down to logs from a single gateway.
To configure a Web Gateway to include the server_ip and uuid in its logs:
1. In the Web Gateway console, navigate to Policy > Rule Sets > Log Handler > Access Log as shown below.
2. Click on the currently enabled rule, then click copy and paste. The console creates a copy of the rule as shown below.
3. Choose the new rule and click Edit.
4. On the Edit Rule box, enter a new name for the rule as shown below.
5. In the Steps area of the Edit Rule box, click Events and click in the text box to select it as shown below.
6. On the Events toolbar, click Edit to open the Edit Set Property box as shown below.
7. On the "To concatenation of these strings:" toobar, click Add to add a new string.
8. On the Enter a String box, type a double-quote followed by a space as shown below, then click OK. This action creates a separator that CloudSOC requires in order to parse the logs.
9. On the Edit Set Property box, choose Filter > Type > IP as shown below.
10. In the "To concatenation of these strings:" toolbar, click Add. 11. Mark the Parameter Property radio button.
12. In the "Type to filter properties" box, type ip, then choose IP.ToString(IP) from the list as shown below.
14. Mark the radio button for Parameter property, then choose URL.Destination.IP as shown below.
15. Click OK to add the destination IP to the string.
16. Click OK on the Enter a String box to add the string with the destination IP to the property. 17. Click OK on the Edit Set Property box to add the property to the rule.
18. On the Edit Rule box, click in the text box, then click Edit.
19. On the Events toolbar, click Edit to open the Edit Set Property box again.
20. On the "To concatenation of these strings:" toobar, click Add to add a new string. 21. On the Enter a String box, type a double-quote followed by a space, then click OK. This
action creates a new separator. 22. Add to add a new string.
23. On the Enter a String box, mark Parameter property and search for "System.UUID" as shown below.
24. Click System.UUID to highlight it, then click OK on the Enter a String box. 25. On the Edit Set Property box, click OK.
26. On the Edit Rule box, click Finish.
27. On the Web Gateway console, mark the checkboxes to disable the previously active rule, and enable your new copy of the rule as shown below.
28. On the Web Gateway console, navigate to Policy > Settings > Engines > File System
29. In the Log header box, add a space and then "server_ip uuid" to the end of the log
header string as shown below.
30. On the console toolbar, click Save Changes.
You have now updated the Log Header to include the uuid field in the end of the default Access Log format:
time_stamp "auth_user" src_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client bytes_from_client "user_agent" "virus_name" "block_res" "application_name" server_ip uuid
You can also configure the Web Gateway to automatically push logs to SpanVA, as described in the next section.
Configuring McAfee Web Gateway for auto log push to SpanVA
McAfee Web Gateway can automatically push logs periodically or on rotation to an external monitoring device over FTP, HTTP or HTTPS. Typically, you setup and configure a local server to collect logs from firewall and proxy devices and then write a script to periodically transfer these logs to CloudSOC servers over SFTP.
A simpler alternative is to use the CloudSOC SpanVA log collector appliance to collect logs from all your network devices including Web Gateways. Your gateways can then push logs directly to SpanVA which optionally anonymizes, compresses, and transfers the logs to CloudSOC for processing. We recommend this approach because it simplifies your job. This Tech Note does not go into details of configuring SpanVA but focuses on how you can configure your Web Gateway.
1. In CloudSOC, create a SpanVA datasource for the Web Gateway as described in the CloudSOC Tech Note Installing and Configuring SpanVA.
For a SpanVA datasource of type SCP/SFTP/FTP/HTTPS Server, note the destination directory, username, and password shown on the SpanVA Datasource Details panel. 2. In the Web Gateway console, navigate to Policy > Settings > File System Logging >
Access Log Configuration as shown below.
Note: Do not change the logging settings in Configuration > Log File Manager. Those settings apply globally, and changing them may have unintended consequences. 3. In the Settings for Rotation, Pushing, and Deletion area, mark the checkbox for Enable
specific settings for user defined log as shown below.
4. In the Auto Pushing area, mark the checkbox for Enable Auto Pushing.
5. In the Destination box, combine the SpanVA hostname or IP address and the destination directory from the SpanVA Datasource Details panel, for example:
https:///ds_mycompanyco/484848041ce7c1829f5c85508
6. In the Username and Password boxes, enter the username and password from the SpanVA Datasource Details panel as shown below.
7. Configure log rotation and deletion to suit your needs as shown in the example below.
8. On the Web Gateway console toolbar, click Save Changes.
The next time your McAfee Web Gateway rotates the log file, it sends it to SpanVA using a HTTP PUT request.
For more information, see the CloudSOC Tech Notes Using the Audit Application and Installing
and Configuring SpanVA.
Sample access log
#time_stamp "auth_user" src_ip status_code "req_line" "categories" "rep_level" "media_type" bytes_to_client "user_agent" "virus_name" "block_res"
[27/May/2014:23:59:43 +0530] "800069682" 58.2.97.194 304 "GET
http://economictimes.indiatimes.com/photo/19319380.cms HTTP/1.1" "General News" "Minimal Risk" "" 307 "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1;
Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)" "" "0"
[27/May/2014:23:59:43 +0530] "800069682" 58.2.97.194 304 "GET
http://www.google-analytics.com/ga.js HTTP/1.1" "Internet Services" "Minimal Risk" "" 252 "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)" "" "0"
[27/May/2014:23:59:43 +0530] "703081901" 124.4.34.159 200 "GET
http://genpact.myhmm.org/service/remoting/resource/remote.js HTTP/1.1"
"Education/Reference" "Minimal Risk" "" 32441 "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)" "" "0" [27/May/2014:23:59:43 +0530] "" 58.2.64.237 200 "GET
052544&_=1401215381409&buddylist=1&initialize=0¤ttime=0×tamp=1&typin gto=0&blh=undefined&status=&updateconv=1401214919 HTTP/1.1" "" "-" "" 304 "Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36" "" "0"
References
● http://www.mcafee.com/sg/resources/data-sheets/ds-web-gateway.pdf ● https://community.mcafee.com/docs/DOC-4929Revision history
Date Version Description
2014 1.0 Initial release
30 October 2015 1.1 Minor revisions 6 December 2016 1.2 Fix typo
20 January 2017 2.0 Add procedure for configuring Web Gateway to include device ID and server IP in logs