Analys och spårning
Text based logs
How to read and examine logs?
• We can usually open the log as a text file, but not convenient
in general (due to the information size)
• We can write our own code to examine – Perl and Python are
the common languages used for this
–
Advantages: flexible, answer your needs (if you got the skills)
• We can use dedicated software specialized in log analysis
• Logs are the collection of basic events
–
One basic event is often not really important but several events
can lead to interesting conclusions
–
Sometimes it is the only reliable source of information left
• Cross-analyze log files may be useful
• Statistical analysis is also important
Common Log Format
• The Common Log Format is a standardized text file format used by web servers which may be analyzed by a variety of analysis programs, example: • Apache access.log
• Each line in a file stored in the Common Log Format has the following
syntax: host ident auth-user date request status bytes
• A "-" in a field indicates missing data
• 127.0.0.1 is the IP address of the client (remote host) which made the request to the server
• - RFC 1413 identity of the client, more info: http://tools.ietf.org/html/rfc1413 • frank is the user id of the person requesting the document
• [10/Oct/2000:13:55:36 -0700] is the date, time, and time zone when the server finished processing the request
• "GET /apache_pb.gif HTTP/1.0" is the request line from the client. The method GET, /apache_pb.gif the resource requested, and HTTP/1.0 the HTTP protocol
• 200 is the HTTP status code returned to the client. 2xx is a successful response, 3xx a redirection, 4xx a client error and 5xx a server error
Combined Log Format
• Another commonly used format string is called the Combined Log Format • This format is exactly the same as the Common Log Format, with the
addition of two more fields
– Referer (html page where apache_pb.gif originated) and User-agent (the client)
• Apache error.log format
• The first item in the log entry is the date and time of the message
• The second item lists the severity of the error being reported depending on the configured LogLevel
• The third item gives the IP address of the client that generated the error
• Beyond that is the message itself, a very wide variety of different messages can appear
• In this case a client was denied to access /export/home/live/ap/htdocs/test
Windows XP IIS Logs
• Microsoft web server is called Internet Information Services (IIS)
• Detailed logging is enabled by default
• Most common and default format is W3C Extended Log File Format
• Log timestamps are GMT
• Default location: %SystemRoot%\System32\Logfiles\W3SVC1\
• Log per day in format exyymmdd.log, where yy=year, mm=month
and dd=day
Windows Vista/7 IIS 7.5 Logs
W3SVC1
and
Windows XP FTP Logs
• Microsoft FTP Server
• Detailed logging enabled by default
• Most common and default format is W3C Extended Log File Format
• Log timestamps are GMT
• Default location: %SystemRoot%\System32\Logfiles\MSFTPSVC1\
• Log per day in format exyymmdd.log, where yy=year, mm=month
and dd=day
Microsoft DHCP Server Logs
• Dynamic Host Configuration Protocol (DHCP) service in which
IP address assigned dynamically upon request by host
• Microsoft servers provide this services
• IP address loaned for a short period and thus which machine
had which IP address is based on particular point in time
• Logs record host to which IP was assigned
• Time is local system time zone!
• Default location for log is: %SystemRoot%\System32\DHCP\
• Logs stored in one file per day basis
DHCP Log example
•
Event ID
- see table,
Date
,
Time
(Local system time zone)
•
Description
- action,
IP address
- IP assigned
Windows XP Firewall Logs
• Firewall added to XP with SP 2
• Firewall on by default
• Good logging utility, however, it is off by default
• Enabling is buried deep in user interface
– Don’t expect to find it enabled often, except in domain settings with good administrator!
• Default location of firewall logs is: %SystemRoot%\pfirewall.log
• Always look for it anyway
Microsoft Port Reporter
• Port Reporter is a logging service which runs on Microsoft
Windows 2000, XP, Server 2003 and newer...?
• Useful for troubleshooting, security, application profiling,
application development, and so on...
• Port Reporter logs
– Ports that are used and the time they are used – Processes that use the ports
– Whether a process is a service
– All the modules that each process has loaded – The user account that each process runs under
• Also logs TCP/IP port usage data and port changes
– Increase or decrease in connections, port state changes etc.
• Port Reporter comes from MS PortrQry used in local mode
Port Reporter Service Log files
• The service creates 3 log files with a name which uses date
and time in 24-hour format (the *) when the file was created
– PR-INITIAL-*.log
• Contains data about the ports, processes and modules running on system when the service started up
– PR-PORTS-*.log
• Contains summary data about TCP and UDP port activity on computer listed using comma-separated value (.csv) format:
– date, time, protocol, local port, local IP address, remote port, remote IP address, PID, module, user context
– PR-PIDS-*.log
• Contains detailed information about ports, processes, related modules and user account process uses to run
• Each line in PR-PORTS log has a corresponding entry in the PR-PIDS log
• In summary the 3 log files provide
– Snapshot of port usage when service starts – Summary data on ongoing port usage
Microsoft Port Reporter Parser
MicroSoft Log Parser (free)
• As an application developer you often need to write some logs for
your application
– There is many logging framework to choose among: Log4net, Log4j, Microsoft Logging Application Block, etc.
– But when it come to read those logs, search for data, create reports, extract statistics or perform some alert/action on them, things become harder
• Log Parser performs SQL queries against a variety of log files and
other system data sources
– You can query any log and data sources (database, event log, IIS logs, file system, registry, etc.) with a complex SQL query!
– On the down side, using it from the command line become quickly unpractical as you need to type your SQL query in a DOS prompt
• logparser -i:EVT "SELECT TOP 20 * FROM Security WHERE EventID=5032 ORDER BY TimeGenerated DESC" -o DATAGRID
Log Parser Architecture
• Swiss Army knife for processing Windows logs of all types (and others). The world is your database with Log Parser!
• Input Formats are generic record providers
– Input Formats can be thought of as SQL tables containing the data you want to process
– Manage .evtx (Vista/7) event logs as well
• A SQL-Like Engine Core processes the records generated by an Input Format
– SQL language (SELECT, WHERE, GROUP BY, HAVING, ORDER BY etc.)
– Aggregate functions (SUM, COUNT, AVG, MAX, MIN etc.)
– A rich set of functions (e.g. SUBSTR, CASE, REVERSEDNS, etc.)
• Output Formats are generic consumers of records
Log Parser Lizard
SQALP (Simple Query Analyzer for Log Parser)
DATAGRID
batch file alternative (%filename% in sql)
echo off cls
MicroSoft Log Parser, events etc.
• Log Parser download
– http://www.microsoft.com/technet/scriptcenter/to ols/logparser/default.mspx
• Visual Log Parser GUI (SQALP)
http://en.serialcoder.net/logiciels/visual-logparser.aspx
• Log Parser Help File
– Very good resource!
• Book with loads of scripts and queries
http://www.elsevierdirect.com/companion.jsp?ISBN=9781932266528
• Microsoft log events
– http://eventlogs.blogspot.com
– http://eventid.net (what does it mean?)
• Forensic Log Parsing with Microsoft's Log
Parser
– http://www.securityfocus.com/infocus/1712
Microsoft System Center Operations Manager
2007 R2 and Syslog (RFC 3164) alternatives
• Microsoft System Center Operations Manager är ett händelse- och
prestandaövervakningsverktyg som innehåller en mängd funktioner
för att reducera den tid det tar att konfigurera ett system eller en
tillämpning
• Course and other white papers
– http://www.microsoft.com/systemcenter/operationsmanager/en/us/default.aspx
• End-to-End Service Monitoring
• Client Monitoring
• Audit Collection
• Syslog - GNU/Linux setup
– http://www.aboutdebian.com/syslog.htm
• Other (Windows) Syslog servers
– http://en.wikipedia.org/wiki/Syslog
• Convert Windows log to Syslog
– http://www.syslogserver.com
• Setting up Syslog to redirect
logging to separate log server is very easy
Log2timeline -
http://log2timeline.net/
• A framework for automatic creation of a super timeline. The main purpose is to provide a single tool to parse various log files and artifacts found on
suspect systems (and supporting systems, such as network equipment) and produce a timeline that can be analysed by forensic investigators/analysts • The tool is written in Perl for Linux but has been tested using Mac OS X
(10.5.7+ and 10.6.+). Parts of it should work natively in Windows as well (with ActiveState Perl installed)
• "Mastering the Super Timeline With log2timeline" can be downloaded here
– http://www.sans.org/reading_room/whitepapers/logging/mastering-super-timeline-log2timeline_33438
SIMILE: http://www.simile-widgets.org/timeline/
+25 formats +9
Common Linux log file names and usage
• Most of the logs are located in /var/log or /var/log/<foldername>/*
• Usually in ASCII format – any text editor/script will do it
• Examples of logs, there may be some distribution name differences
– /var/log/auth.log: Authenication logs – /var/log/cron.log: Crond logs (cron job) – /var/log/kern.log: Kernel logs
– /var/log/message: General message and system related stuff – /var/log/boot.log : System boot log
– /var/log/mail/*: Mail server logs (more files inside this directory) – /var/log/apache/*: Apache access and error logs directory
– /var/log/samba/*: SMB server logs
– /var/log/utmp or /var/log/wtmp : Login records file
• utmp, wtmp and lastlog (who, last, lastb, lastlog, w, etc.)
– Are binary files (utmp structure), lastlog may be distribution specific
• logrotate /etc/logrotate.conf
utmp.h structure (Ubuntu 9.04)
struct utmp {
short ut_type; /* Type of record */ pid_t ut_pid; /* PID of login process */
char ut_line[UT_LINESIZE]; /* Device name of tty - "/dev/" */ char ut_id[4]; /* Terminal name suffix, or inittab(5) ID */ char ut_user[UT_NAMESIZE]; /* Username */
char ut_host[UT_HOSTSIZE]; /* Hostname for remote login, or kernel version for run-level messages */ struct exit_status ut_exit; /* Exit status of a process marked as DEAD_PROCESS;
not used by Linux init(8) */
/* The ut_session and ut_tv fields must be the same size when compiled 32- and 64-bit.
This allows data files and shared memory to be shared between 32- and 64-bit applications. */ #if __WORDSIZE == 64 && defined __WORDSIZE_COMPAT32
int32_t ut_session; /* Session ID (getsid(2)), used for windowing */ struct {
int32_t tv_sec; /* Seconds */
int32_t tv_usec; /* Microseconds */ } ut_tv; /* Time entry was made */ #else
long ut_session; /* Session ID */
struct timeval ut_tv; /* Time entry was made */ #endif
int32_t ut_addr_v6[4]; /* Internet address of remote host; IPv4 address uses just ut_addr_v6[0] */ char __unused[20]; /* Reserved for future use */
IM
(Instant
Message) chats and
other internet
artifacts
What is Maltego?
• Maltego is an information gathering tool that allows you to visually see
relationships. Maltego allows you to enumerate network and domain
information like
– Domain Names, Whois Information, DNS Names – Netblocks, IP Addresses
• Maltego also allows you to enumerate People information like
– Email addresses associated with a person's name – Web sites associated with a person's name
– Phone numbers associated with a person's name
– Social groups that are associated with a person's name
– Companies and organizations associated with a person's name
• Maltego also allows you to
– Do simple verification of email addresses – Search blogs for tags and phrases
– Identify incoming links for websites
Maltego transforms
• All the information gathering "processes" that Maltego does are called
"Transforms," and unfortunately not all of them are documented. But different transforms query different types of information. The full list is here:
Maltego resources
• Maltego Part I - Intro and Personal Recon
–
http://www.ethicalhacker.net/content/view/202/24/
• Maltego Part II - Infrastructure Enumeration
–
http://www.ethicalhacker.net/content/view/251/24/
• Data Mining Tony Hawk’s Twitter Hunt with Maltego
–
http://www.securityg33k.com/blog/?p=180
• Maltego: Transform & Correlate
–
https://www.issa.org/Library/Journals/2009/December/McRee-toolsmith.pdf
• Maltego
WebSite-Watcher 1
• Automatically check web pages for updates and
changes
• Automate your daily routine, boost your productivity
• Features
–
Monitor web pages
–
Monitor password protected pages
–
Monitor forums for new postings and replies
–
Monitor RSS feeds, Newsgroups and local files
–
Highlight changes in a page
–
Powerful filter system to ignore unwanted content
–
Many more features to stay up-to-date!
