Troubleshooting Firewalls

(1)

Troubleshooting Firewalls

Eric Stuhl

Senior Network Consultant

Chesapeake NetCraftsmen

(2)

Agenda

• Packet Flow

• Understanding the Architecture

• Failover

• Troubleshooting

• Case Studies

• Tools

• Best Practices

(3)

(4)

Understanding the Packet Flow

• To effectively troubleshoot a problem, one must first understand the

packet path through the network

• Attempt to isolate the problem down to a single device

• Then perform a systematic walk of the packet path through the

device to determine where the problem could be

• For problems relating to the Cisco ASA/PIX

®

_{/FWSM, always}

–

Determine the flow: SRC IP, DST IP, SRC port, DST port,

–

Determine the flow: SRC IP, DST IP, SRC port, DST port,

and protocol

–

Determine the interfaces through which the flow passes

Note: All firewall issues can be simplified to two interfaces (ingress and egress) and the rules tied to both

(5)

Example Flow

• Flow

– SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP

– DST IP: 198.133.219.25 DST Port: 80

• Interfaces

– Source: Inside Destination: Outside

Accounting

With the Flow Defined,

Examination of

Configuration Issues

Boils Down to Just the

Two Interfaces: Inside

and Outside

Eng Client: 10.1.1.9 Se rv e rs O u ts id e

Packet Flow

(6)

Understanding the Packet Flow

• Once the device and flow have been identified,

walk the path of the packet through the device

• The packet path through the firewall is illustrated in

the next several slides

• For troubleshooting, pay careful attention to where

the packet can be dropped in the decision-making

process

(7)

Packet Processing Flow Diagram

• The diagram below will be referenced on the

following slides; it is shown here enlarged for

reference

(8)

Packet Processing: Ingress Interface

• Packet arrives on ingress interface

• Input counters incremented

• Software input queue is an indicator of load

• “No buffers” indicates packet drops, typically due to bursty traffic

ASA-5540# show interface gb-ethernet1

interface gb-ethernet1 "inside" is up, line protocol is up

Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214 IP address 10.1.1.1, subnet mask 255.255.255.0

MTU 1500 bytes, BW 1 Gbit full duplex

5912749 packets input, 377701207 bytes, 0 no buffer

Received 29519 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 286298 packets output, 18326033 bytes, 0 underruns

input queue (curr/max blocks): hardware (0/25) software (0/0)

(9)

Packet Processing: Locate Connection

• Check first for existing connection

• If connection exists, flow is matched; bypass ACL check

• If

no

existing connection

• If

no

existing connection

–

TCP non-SYN packet, drop and log

–

TCP SYN or UDP packet, pass to ACL checks

Established Connection:

ASA-5540# show conn

TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO

Syslog Because of

No

Connection, and Non-SYN Packet:

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside

(10)

Packet Processing: ACL Check

• First packet in flow is processed through interface ACLs

• ACLs are

first

match

• First packet in flow matches ACE, incrementing hit count by

one

• Denied packets are dropped and logged

Packet Permitted by ACL:

ASA-5540B# show access-list inside

access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)

Syslog When Packet Is Denied by ACL:

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access-group "inside"

(11)

Packet Processing: Match Translation

• First packet in flow must match a translation rule*

• A quick route lookup is done only to determine egress interface

• Translation rule can be to NAT, or

not

to NAT

• NAT order of operations dictates what happens with overlapping translation

rules

• Once translation rule is matched, connection is created

Translation Exists:

ASA-5540# show xlate debug

NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00

Syslogs When No Translation Rule Found: (305005—No NAT; 305006—No Global)

ASA-3-305005: No translation group found for tcp src inside:10.1.1.9/11039 dst outside:198.133.219.25/80

ASA-3-305006: regular translation creation failed for tcp src inside:10.1.1.9/11040 dst outside:198.133.219.25/80

(12)

Translation and NAT Order of Operations

1. nat 0 access-list (nat-exempt)

2. Match existing xlates

3. Match static commands (Cisco ASA/PIX first match;

FWSM best match)

– Static NAT with and without access-list – Static PAT with and without access-list

4. Match nat commands

– nat <id> access-list (first match)

– nat <id> <address> <mask> (best match) • If the ID is 0, create an identity xlate • Use global pool for dynamic NAT • Use global pool for dynamic PAT

F

irs

t M

a

tc

h

(13)

Packet Processing:

Inspections/Sec Checks

• Inspections are applied to ensure protocol compliance

• (Optional) Customized AIC inspections

• NAT embedded IPs in payload

• Additional security checks are applied to the packet

• (Optional) Packets passed to Content Security and Control

(CSC) Module

Syslog from Packets Denied by Security Check:

ASA-4-406002: FTP port command different address:

10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside

ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP

(14)

Packet Processing: NAT IP Header

• Translate the IP address in the IP header

• Translate the port if performing PAT

• Update checksums

• (Optional) Following the above, pass packet to IPS

(AIP) module

(15)

Packet Processing: Egress Interface

• Packet is “virtually” forwarded to egress interface (i.e., not

forwarded to the driver yet)

• Egress interface is determined

first

by translation rules

• Egress interface is determined

first

by translation rules

• If translation rules do not specify egress interface (e.g.,

outbound initial packet) the results of a global route lookup

are used to determine egress interface

• Example:

static (inside,outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0 static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0

D M Z Inside Outside 172.16.0.0/16 _{172.16.12.0/24} 172.16.12.4 Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics

(16)

Packet Processing: L3 Route Lookup

• Once on egress interface, an interface route

lookup

is performed

• Only routes pointing out the egress interface are

eligible

• Remember: translation rule can forward the packet

to the egress interface, even though the routing

table may point to a different interface

Syslog from Packet on Egress Interface with No Route Pointing Out Interface:

(17)

Packet Processing: L2 Address Lookup

• Once a Layer 3 route has been found, and next

hop identified, Layer 2 resolution is performed

• Layer 2 rewrite of MAC header

• If Layer 2 resolution fails—no syslog

• show arp

will not display an entry for the L3 next

hop

• debug arp

will indicate if we are not receiving an

(18)

Packet Processing: Transmit Packet

• Packet is transmitted on wire

• Interface counters will increment on interface

• Output hardware and software queues indicate

buffering at driver level, interface is busy

ASA-5540# show interface gb-ethernet0

interface gb-ethernet0 "outside" is up, line protocol is up

Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626c IP address 172.18.124.64, subnet mask 255.255.255.0

MTU 1500 bytes, BW 1 Gbit full duplex

3529518 packets input, 337798466 bytes, 0 no buffer Received 32277 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

5585431 packets output, 359059032 bytes, 0 underruns

input queue (curr/max blocks): hardware (0/25) software (0/0)

(19)

Agenda

• Packet Flow

• Understanding the Architecture

• Failover

• Troubleshooting

• Case Studies

• Tools

• Best Practices

(20)

Cisco ASA/PIX—Understanding the

Architecture

• Cisco ASA/PIX platforms process all packets in

software (via the central CPU)

–

All packets are processed first in…usually also first out

• No software limits on the number of ACEs (rules)

that can be configured.

–

Each ACE takes a minimum of 212 bytes of RAM.

• Cisco ASA platforms have software imposed

connection limits; Cisco PIX platforms do not

(bound by RAM)

(21)

Classifier in Multimode

• FWSM has a single MAC address for all interfaces

• Cisco ASA/PIX has single MAC for ‘shared’

interfaces (physical interfaces have unique MACs)

–

Cisco ASA/PIX 7.2 introduces an option to change this

• When the firewall receives a packet, it must

‘classify’ it to determine where to send the packet

• Packets are classified based on the following

–

Unique ingress interface/VLAN

(22)

Classifier in Multimode

• Inbound traffic is ‘classified’ to context CTX3,

based on the global IP in the static

Inside FWSM .1 DST IP SRC IP

Example

V L A N 3 —1 0 .1 4 .3 .x Inside 10.1.2.2 10.1.1.2 Inside 10.1.3.2 Inbound Packet Outside VLAN 4 VLAN 5 VLAN 6 CTX1 CTX2 CTX3 MSFC .1 .2 .3 10.14.3.89 192.168.5.4 static (inside,outside) 10.14.3.89 10.1.3.2 Shared Interface

(23)

Classifier in Multimode

• If the firewall is unable to classify a packet,

the following syslog message is generated in the

Admin context*

%FWSM-6-106025: Failed to determine security context for

packet: vlan3 tcp src 192.168.5.4/1025 dest 10.14.3.25/80

(24)

Agenda

• Packet Flow

• Understanding the Architecture

• Failover

• Troubleshooting

• Case Studies

• Tools

• Best Practices

(25)

Failover Basics

• Active/standby vs.

primary/secondary

• Serial vs. LAN failover

• Stateful failover (optional)

• A failover only occurs

when either firewall

Internet

when either firewall

determines the standby

firewall is healthier than the

active firewall

• Both firewalls swap MAC

and

IP addresses when a

failover occurs

• Level 1 syslogs will give

reason of failover

Secondary

(Standby) Primary_(Active) LAN/Serial

Stateful

(26)

PIX(config)# show failover Failover On

Cable status: N/A - LAN-based failover enabled Failover unit Primary

Failover LAN Interface: Failover Ethernet5 (up) Unit Poll frequency 1 seconds, holdtime 3 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds Monitored Interfaces 2 of 250 maximum

Version: Ours 7.2(3), Mate 7.2(2)

Last Failover at: 18:30:43 UTC Apr 12 2007

Verifying Failover Configuration

Interface

Monitoring

Last Failover at: 18:30:43 UTC Apr 12 2007

This host: Primary - Active

Active time: 5371 (sec)

Interface outside (10.36.8.36): Normal Interface inside (10.5.5.144): Normal Other host: Secondary - Standby Ready

Active time: 0 (sec)

Interface outside (10.36.8.37): Normal Interface inside (10.5.5.145): Normal Stateful Failover Logical Update Statistics

Link : Failover Ethernet5 (up)

Stateful Obj xmit xerr rcv rerr General 86 0 73 0 sys cmd 74 0 73 0

(27)

What Triggers a Failover?

• Power loss/reload (this includes crashes) on the

active firewall

• SSM interface/module failure

• The standby becoming “healthier” than the active

firewall

(28)

What Triggers a Failover?

• Two consecutive “hello” messages missed on

any

monitored interface forces the interface into

testing

mode

• Both units first verify the link status on the

interface

• Next, both units execute the following tests

–

Network activity test

–

ARP test

–

Broadcast ping test

• The first test passed causes the interface on that

unit

to be marked “healthy”; only if all tests “fail” will

the interface be marked “failed”

(29)

What to Do After a Failover

• Always check the syslogs to determine root cause

• Example: switch port failed on inside interface of

active firewall

Syslogs from Primary (Active) Firewall

ASA-4-411002: Line protocol on Interface inside, changed state to down ASA-1-105007: (Primary) Link status ‘Down’ on interface 1

ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier

ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active

Syslogs from Primary (Active) Firewall

(30)

ASA# show failover state

What to Do After a Failover

• Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the

reason for failover is saved in the failover state

• This information is not saved across reboots

ASA# show failover state

State Last Failure Reason Date/Time This host - Primary

Failed Ifc Failure 12:56:00 UTC May 6 2007 Inside: Failed

Other host - Secondary

Active None ====Configuration State===

Sync Done

====Communication State=== Mac set

(31)

Agenda

• Packet Flow

• Understanding the Architecture

• Failover

• Troubleshooting

• Case Studies

• Tools

• Best Practices

(32)

Troubleshooting Tools

• Syslogs

• Debug commands

• Show commands

• Packet capture

• Packet tracer

(33)

Uses of Syslogs

• Primary mechanism to record traffic

to

and

through

the firewall

• The best troubleshooting tool available

Archival Purposes

Debugging Purposes

Buffered

SSH Client

Internet Syslog Server SNMP Server

Console

Trap .Syslog

(34)

ASA Syslog Level vs. Number of Messages

Log

Level

Description

Number of Messages (SUM)

Ver. 6.3

Ver. 7.0

Ver. 7.2

Ver. 8.0

Ver. 8.1

0 Emergencies

0

1 Alerts

41 (41)

62 (62)

77 (77)

78 (78)

87 (87)

1 Alerts

41 (41)

62 (62)

77 (77)

78 (78)

87 (87)

2 Critical

21 (62)

29 (91)

35 (112)

49 (127)

50 (137)

3 Errors

74 (136)

274 (365)

334 (446)

361 (488)

363 (500)

4 Warnings

56 (192)

179 (544)

267 (713)

280 (768)

281 (781)

5 Notifications

21 (213)

161 (705)

206 (919)

216 (984)

218 (999)

6 Informational

95 (308)

234 (939)

302 (1221)

335 (1319)

337 (1336)

7 Debugging

15 (323)

217 (1156)

258 (1479)

266 (1585)

267 (1603)

(35)

FWSM Syslog Level vs. Number of Messages

Log

Level

Description

Number of Messages (SUM)

Ver. 2.3

Ver. 3.1

Ver. 3.2

Ver. 4.0

0 Emergencies

0

1 Alerts

58 (58)

67 (67)

1 Alerts

58 (58)

67 (67)

2 Critical

21 (79)

29 (96)

3 Errors

94 (173)

305 (401)

306 (402)

318 (414)

4 Warnings

131 (304)

194 (595)

196 (598)

199 (613)

5 Notifications

26 (330)

167 (762)

169 (767)

178 (791)

6 Informational

116 (446)

245 (1007)

248 (1015)

255 (1046)

7 Debugging

23 (469)

225 (1232)

225 (1240)

226 (1272)

(36)

What Are Modifiable Syslog Levels?

• Modifiable syslog levels

–

Allows one to move any syslog message to

any level

• Problem

[no] logging message <syslog_id> level <level>

Levels

0—Emergency

1—Alert

• Problem

–

You want to record what exec commands are

being executed on the firewall; syslog ID

111009 records this information, but by

default it is at level 7 (debug)

%PIX-7-111009: User ‘johndoe’

executed cmd: show run

The problem is we don’t want to log all 1602

other syslogs that are generated at debug

level

1—Alert

2—Critical

3—Errors

4—Warnings

5—Notifications

6—Informational

7—Debugging

(37)

How to Create Modifiable Syslog Levels

• Lower syslog message 111009 to level 3 (error)

–

ASA(config)# logging message 111009 level 3

–

Or

–

ASA(config)# logging message 111009 level error

[no] logging message <syslog_id> level <level>

Solution

• Now our syslog looks as follows

–

%ASA-

3 -111009: User ‘johndoe’ executed cmd: show run

• To restore the default syslog level

–

ASA(config)# no logging message 111009 level error

–

Or

–

ASA(config)# logging message 111009 level 7

http://www.cisco.com/en/US/docs/security/asa/asa80/system/m

essage/logmsgs.html

(38)

Debug Commands

1. Debugs should not be the

first

choice to

troubleshoot

a problem

2. Debugs can negatively impact the CPU of the box,

and also the performance of it; use with caution

3. Debugs are not conditional*

4. Know how much traffic, of the specified type, is

passing through the firewall

before

enabling the

respective debug

(39)

Debug ICMP Trace

• Valuable tool used to troubleshoot connectivity issues

• Provides interface and translation information to

http://www.cisco.com

Internet

• Provides interface and translation information to

quickly

determine flow

• Echo-replys must be explicitly permitted through ACL,

or ICMP inspection must be enabled

ICMP echo-requestfrom inside:10.1.1.2 to 198.133.219.25ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.1.1.2to outside:209.165.201.22

ICMP echo-reply from outside:198.133.219.25to 209.165.201.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2

(40)

Logging Debugs to Syslog

• Problem

–

Log only debug output to syslog

• Solution

–

Create a logging list with only syslog ID 711001

–

Enable debug output to syslogs

–

Log on the logging list

–

Log on the logging list

ASA(config)#

logging list

C-MUG

message 711001

ASA(config)#

logging debug-trace

(41)

Show Output Filters

• Use output filters to filter the output of show command

to only the information you want to see

• To use them, at the end of show <Command>, use the

pipe character “|” followed by

show <cmd> |

begin|include|exclude|grep [-v] <regular_exp>

pipe character “|” followed by

–

begin

Start displaying the output beginning at the first

match of the RegEx, and continue to display the remaining

output

–

include

Display any line that matches the RegEx

–

exclude

Display any line that does not match the RegEx

–

grep

Same as include

(42)

Example: Show Output Filters

Examples

• Display the interface stats starting with the ‘inside’ interface

–

show interface | begin inside

• Display the access-list entries that contain address 10.1.1.5

show <cmd> |

begin|include|exclude|grep [-v] <regular_exp>

– show access-list | grep 10.1.1.5

• Display the config, except for the access-lists

– show run | exclude access-list

• Display only access-list entries that have non-zero hitcounts

– show access-list | grep –v hitcnt=0

• Display a count of the number of connections each host has

– show local-host | include host|count/limit

Note: You must include a space on either side of the pipe for the command to be accepted; also, trailing spaces are counted

(43)

Show CPU Usage

• Under normal conditions the CPU should stay

below 50% (baseline as per network); if the CPU

reaches 100% the firewall will start dropping

packets

• FWSM CPU is used for limited traffic processing;

during ACL compilation CPU is expected to be near

100% until ACL is compiled

• The

show cpu usage

command displays the CPU

over time as a running average

pixfirewall#

show cpu usage

CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%

(44)

Show Traffic

• The

show traffic

command displays the traffic

received and transmitted out each interface of the

firewall

ASA#

show traffic

outside:

received (in 124.650 secs):

295468 packets

167218253 bytes

295468 packets

167218253 bytes

2370 pkts/sec

1341502 bytes/sec

transmitted (in 124.650 secs):

260901 packets

120467981 bytes

2093 pkts/sec

966449 bytes/sec

inside:

received (in 124.650 secs):

261478 packets

120145678 bytes

2097 pkts/sec

963864 bytes/sec

transmitted (in 124.650 secs):

294649 packets

167380042 bytes

2363 pkts/sec

1342800 bytes/sec

(45)

Show Xlate and Show Xlate Debug

• The

show xlate

command displays information

about the translations through the firewall

• You can limit the output to just the local or global

IP

ASA#

show xlate

2 in use, 2381 most used

Global 172.18.124.68 Local 10.1.1.9

ASA#

show xlate debug

2 in use, 2381 most used

Flags: D - DNS, d - dump, I - identity, i - inside, n - no random,

o - outside, r - portmap, s - static

NAT from

inside

:10.1.1.9 to

outside

:172.18.124.68

flags -

idle 0:02:03 timeout 3:00:00

TCP PAT from inside:10.9.9.3/4101 to outside:172.18.124.65/1024

flags r idle 0:00:08 timeout 0:00:30

Global 172.18.124.68 Local 10.1.1.9

PAT Global 172.18.124.65(1024) Local 10.9.9.3(4101)

“debug” Adds interface

names, idle and xlate

(46)

ASA# show conn

2 in use, 64511 most used

TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO

UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags –

Show Conn and Show Conn Detail

ASA# show conn detail

“detail” Adds uptime and

_{timeout in 7.2(4), 8.0(4)}

Idle Time,

Bytes Transferred

Connection

Flags

‘real’ Interface

names added in

7.2(4), 8.0(4)

ASA# show conn detail

2 in use, 64511 most used

Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump, E - outside back connection, F - outside FIN, f - inside FIN,

G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, i - incomplete, J - GTP, j - GTP data, K - GTP t3-response k - Skinny media, M - SMTP data, m - SIP media, n - GUP

O - outbound data, P - inside back connection, q - SQL*Net data, R - outside acknowledged FIN,

R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS, X - inspected by service module

TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101,

flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127 UDP outside:172.18.124.1/123 dmz:10.1.1.9/123,

(47)

Example—Connection Build Up

1. Firewall receives an initial SYN packet from the inside; the SYN is

permitted by the access-list, a translation (xlate) is built up, and the

connection is also created with the flags “saA”

2. The outside device responds to the SYN packet with a SYN+ACK; the

connection flags are updated to reflect this, and now show “A”

3. The inside device responds to the SYN+ACK with an ACK and this

completes the TCP three-way handshake, and the connection is now

considered “up” (U flag)

3 ACK

5 Data

1 SYN+ACKSYNData 42

considered “up” (U flag)

4. The outside device sends the first data packet; the connection is

updated and an “I” is added to the flags to indicate the firewall

received Inbound data on that connection

5. Finally, the inside device has sent a data packet and the connection

is updated to include the “O” flag

U

saA

A

UIO

UI

Connection Flags Client Server Outside Inside

(48)

Example—Connection Teardown

1. Firewall receives a FIN packet from the inside; as the FIN passes

through

the firewall, it updates the connection flags by adding an “f” to

indicate that the FIN was received on the Inside interface

2. The outside device immediately responds to the FIN packet with a

FIN+ACK; the connection flags are updated to reflect this, and now

show “UfFR”

3. The inside device responds to the FIN+ACK with a final ACK and the

1 FIN+ACKFIN 2

Uf

UfFR

3 ACK

UfFRr

3. The inside device responds to the FIN+ACK with a final ACK and the

firewall tears down the connection; thus, there are no more

connection flags, because the connection no longer exists

Connection Flags

Client Server

Outside Inside

(49)

Outbound Connection

Inbound Connection

(50)

TCP Connection Termination Reasons

• If a TCP connection is built through the firewall, it

will

always

have a teardown reason

• The TCP teardown syslog is logged at level 6

• If you are having problems with connections

abnormally closing, temporally increase your

logging level (or move the syslog down), and check

the teardown reason

ASA-6-302014: Teardown TCP connection number for

intf_name:real_IP/real_port to intf_name:real_IP/real_port

(51)

TCP Connection Termination Reasons—

Quick Reference

Reason Description

Conn-Timeout Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout

Deny Terminate Flow Was Terminated by Application Inspection

Failover Primary Closed The Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit

FIN Timeout Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout

Flow Closed by Inspection Flow Was Terminated by Inspection Feature Flow Terminated by IPS Flow Was Terminated by IPS

Flow Reset by IPS Flow Was Reset by IPS Flow Terminated by

TCP Intercept Flow Was Terminated by TCP Intercept Invalid SYN SYN Packet Not Valid

Idle Timeout Connection Timed Out Because It Was Idle Longer Than the Timeout Value

IPS Fail-Close Flow Was Terminated Due to IPS Card Down SYN Control Back Channel Initiation from Wrong Side

(52)

TCP Connection Termination Reasons—

Quick Reference (Cont.)

Reason Description

SYN Timeout Force Termination After Two Minutes Awaiting Three-Way Handshake Completion

TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission TCP Fins Normal Close Down Sequence

TCP Invalid SYN Invalid TCP SYN Packet

TCP Reset-I TCP Reset Was Sent From the Inside Host TCP Reset-I TCP Reset Was Sent From the Inside Host TCP Reset-O TCP Reset Was Sent From the Outside Host TCP Segment Partial Overlap Detected a Partially Overlapping Segment TCP Unexpected Window

Size Variation

Connection Terminated Due to a Variation in the TCP Window Size

Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down Uauth Deny Connection Denied by URL Filtering Server Unknown Catch-All Error

(53)

show local-host

• A local-host entry is created for any IP tracked through

the firewall

• It groups the xlates, connections, and AAA information

• Very useful for seeing the connections terminating on

servers

ASA# show local-host 10.1.1.9 detail

Interface inside: 1131 active, 2042 maximum active, 0 denied local host: <10.1.1.9>,

TCP connection count/limit = 1/unlimited TCP embryonic count = 0

TCP intercept watermark = 50

UDP connection count/limit = 0/unlimited AAA:

user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10) absolute timeout: 0:05:00

inactivity timeout: 0:00:00 Xlate(s):

Global 172.18.124.69 Local 10.1.1.9 Conn(s):

(54)

show service-policy

• The

show

service-policy

command is used to quickly

see what

inspection

policies are applied and the packets

matching them

ASA# show service-policy

Global policy:

Service-policy: global_policy Class-map: inspection_default

Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0 Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0 Inspect: ftp, packet 43, drop 0, reset-drop 0

Inspect: h323 h225, packet 0, drop 0, reset-drop 0 Inspect: h323 ras, packet 0, drop 0, reset-drop 0 Inspect: http, packet 562, drop 0, reset-drop 0 Inspect: netbios, packet 0, drop 0, reset-drop 0 Inspect: rsh, packet 0, drop 0, reset-drop 0 Inspect: rtsp, packet 0, drop 0, reset-drop 0

Inspect: skinny, packet 349, drop 0, reset-drop 0 Inspect: esmtp, packet 0, drop 0, reset-drop 0 ...

Interface outside: Service-policy: VoIP

Class-map: voice_marked Priority:

(55)

show service-policy flow

• Use to determine what policies a given flow will

match in the Modular Policy Framework (MPF)

• Eventually all policies will be in MPF

ASA# show service-policy flow tcp host 10.0.0.2 host 10.1.1.2 eq 23

Global policy:

Service-policy: global_policy Interface outside:

Service-policy: outside_policy Class-map: inbound_class

Match: access-list telnet_inbound

Access rule: permit tcp host 10.1.1.2 host 10.0.0.2 eq telnet Action:

(56)

show asp drop

• Packets dropped in the Accelerated Security Path (ASP) will

increment a counter

• Frame drop counters are per packet, flow drops are per flow

• Some counters have corresponding syslogs

ASA# show asp drop

Frame drop:

Invalid encapsulation (invalid-encap) 10897 Invalid tcp length (invalid-tcp-hdr-length) 9382 Invalid udp length (invalid-udp-length) 10 No valid adjacency (no-adjacency) 5594 No route to host (no-route) 1009 Reverse-path verify failed (rpf-violated) 15 Flow is denied by access rule (acl-drop) 25247101 First TCP packet not SYN (tcp-not-syn) 36888 Bad TCP flags (bad-tcp-flags) 67148 TCP option list invalid (tcp-bad-option-list) 731 TCP MSS was too large (tcp-mss-exceeded) 10942 Bad TCP Checksum (bad-tcp-cksum) 893

(57)

Packet Capture

• Capture command first introduced in Cisco PIX 6.2;

FWSM 2.3; it deprecates the “debug packet” command

capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>]

[circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>] [match <prot> {host <sip> | <sip> <mask> | any}

[eq | lt |gt <port>] {host <dip> | <dip> <mask> | any} [eq | lt | gt <port>]]

[real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]

FWSM 2.3; it deprecates the “debug packet” command

• 7.2(3) and 8.0(3) added a ‘real-time’ option

• ASDM 6.0 adds a ‘capture wizard’

• Capture sniffs packets on an interface that match

an ACL, or match line

• Key steps

–

Create an ACL that will match interesting traffic

–

Define the capture and bind it to an access-list and interface

(58)

Packet Capture (Cont.)

• Traffic can be captured both before and after it

passes through the firewall; one capture on the

inside interface, one capture on the outside

interface

• Capture buffer saved in RAM (default size 512KB)

• Default is to stop capturing when buffer is full

• Default packet length is 1518 bytes

• Copy captures off via TFTP or HTTPS

(59)

Where Packets Are Captured in

Packet Flow

Ingress Packets

Captured

Egress Packets

Captured

• Packets are captured at the first and last points

they can be in the flow

• Ingress packets are captured

before

any packet

processing has been done on them

• Egress packets are captured

after

all processing

(excluding L2 source MAC rewrite)

(60)

Capture Command: Example

• Problem: User on the inside with an IP of 10.1.3.2 is having a

problem accessing

www.cisco.com

(197.133.219.25); the

user is getting PATed to 192.168.2.2

Capture In Capture Out

Internet

www.cisco.com Outside Inside

_Internet

198.133.219.25 10.1.3.2 10.1.3.2 192.168.2.2

Step 1: Create ACL for Both Inside and Outside Interface

Step 2: Create Captures on Both Inside and Outside Interface

Step 3: Have Inside User Access www.cisco.com

Step 4: Copy the Captures Off to a TFTP Server

Step 5: Analyze Captures with Sniffer Program

(61)

Capture Command: Example

• Step 1: Create ACL for both inside and outside interface

– ! Outside Capture ACL

Access-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80 Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2 ! Inside Capture ACL

Access-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80 Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2

• Step 2: Create captures on both inside and outside interface

– capture out access-list 100 interface outside packet-length 1518

– capture out access-list 100 interface outside packet-length 1518 capture in access-list 101 interface inside packet-length 1518

• Step 3: Have inside user access

www.cisco.com

• Step 4: Copy the captures off to a TFTP server

– ! ASA ver 7.0+ / FWSM 3.0+ copy capture

copy /pcap capture:out tftp://10.1.3.5/out.pcap copy /pcap capture:in tftp://10.1.3.5/in.pcap ! PIX ver 6.x / FWSM 2.3 copy capture

copy capture:out tftp://10.1.3.5/out.pcap pcap copy capture:in tftp://10.1.3.5/in.pcap pcap

–

Or copy using https:

(62)

Packet Capture: Example

• Step 5: Analyze captures with sniffer program

Outside CAP

(63)

Capturing Packets Dropped by the ASP

• Capture all packets dropped by the ASP

–

ASA# capture drops type asp-drop

all

• Capture on a specific drop reason

–

ASA# capture drops type asp-drop

invalid-tcp-hdr-length

ASA# capture drop type asp-drop ?

acl-drop Flow is denied by configured rule all All packet drop reasons

bad-crypto Bad crypto return in packet bad-ipsec-natt Bad IPSEC NATT packet

bad-ipsec-prot IPSEC not AH or ESP bad-ipsec-udp Bad IPSEC UDP packet bad-tcp-cksum Bad TCP checksum bad-tcp-flags Bad TCP flags

(64)

Packet Tracer

• Packet tracer is the

future

of troubleshooting

configuration issues (

and many other issues

)

• Introduced in version 7.2 and ASDM 5.2

• A packet can be traced by:

–

Defining the packet characteristics via the CLI

–

Capturing the packets using the ‘trace’ option

(65)

Packet Tracer: Overview

• A packet tagged with the ‘trace’ option is injected

into the interface, and processed in the data-plane

• Each action taken on the packet is recorded in the

packet itself

• When the packet reaches the egress interface,

or is dropped, it is punted to the control-plane

• The control-plane reads and displays the actions

taken on the packet, along with the associated

lines

(66)

Packet Tracer: Creating Packet via CLI

• From the CLI, define the input interface along with

source and destination IPs and ports

packet-tracer input <intf> <proto> <src_ip> <src_port> <dst_ip> <dst_port>

• Example—Trace the flow from inside host 10.1.1.2 to

http://www.cisco.com

(198.133.219.25)

(67)

Packet Tracer: Example Output

ASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80

Phase: 1 Type: FLOW-LOOKUP Subtype: Result: ALLOW Config: Additional Information:

Found no matching flow, creating a new flow Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Result: ALLOW Config:

access-group in in interface inside

access-list in extended permit tcp any any eq www Additional Information: Phase: 3 Type: INSPECT Subtype: np-inspect Result: ALLOW Config:

class-map match-all inspection_default match default-inspection-traffic

policy-map global_policy class inspection_default

inspect http

(68)

Packet Tracer: Example Output (Cont.)

... Phase: 10 Type: NAT Subtype: Result: ALLOW Config: nat (inside) 1 10.1.1.0 255.255.255.0 Additional Information:

Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255 ...

Phase: 15

Type: ROUTE-LOOKUP

Subtype: output and adjacency Result: ALLOW

Config:

Additional Information:

found next-hop 209.165.201.1 using egress ifc outside adjacency Active

next-hop mac address 000a.f331.83c0 hits 0 >>>>Packet successfully forwarded to fast path<<<<

(69)

Packet Tracer: Tracing Captured Packet

• Create a capture using the ‘trace’ option

• Find the packet in the capture you want traced

ASA# capture inside access-list web interface inside trace .

ASA# show capture inside

68 packets captured

• Then select that packet to be traced

ASA# show capture inside trace packet-number 4 . 68 packets captured 1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S 2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack 3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack 4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack 5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack ...

(70)

Packet Tracer: ASDM

• ASDM includes a nice GUI front-end to the packet

tracer tool

• It is located off the

Tools

• Input the packets characteristics in the top half

• Actions taken on the packet are shown in the

bottom half, along with associated config and links

back to modify that config entry in ASDM

(71)

Define

Packet

Packet Tracer: ASDM (Screen Shot)

Link Back to

Edit Rule

Matching

Config

Action

Final Result

(72)

Agenda

• Packet Flow

• Understanding the Architecture

• Failover

• Troubleshooting

• Case Studies

• Tools

• Best Practices

(73)

Case Study

(74)

Case Study: Intermittent Access to Web Server

Problem

• Most external clients are not able to load

company’s web page

10.1.1.50 Internet ASA-5510 HTTP Requests to 192.168.1.50 Clients Web Server NATed to 10.1.1.50

(75)

Case Study: Intermittent Access to Web

Server

(76)

Case Study: Intermittent Access to Web Server

• show perfmon

indicates high number of embryonic

connections

ASA-5510# show perfmon

PERFMON STATS: Current Average Xlates 0/s 0/s Connections 2059/s 299/s Connections 2059/s 299/s TCP Conns 2059/s 299/s UDP Conns 0/s 0/s URL Access 0/s 0/s URL Server Req 0/s 0/s TCP Fixup 0/s 0/s TCP Intercept Established Conns 0/s 0/s TCP Intercept Attempts 0/s 0/s TCP Embryonic Conns Timeout 1092/s 4/s HTTP Fixup 0/s 0/s FTP Fixup 0/s 0/s AAA Authen 0/s 0/s AAA Author 0/s 0/s AAA Account 0/s 0/s VALID CONNS RATE in TCP INTERCEPT: Current Average

N/A 95.00% ASA-5510#

(77)

Case Study: Intermittent Access to Web Server

• Issue

show conn

to see ‘who’ is creating the

connections

ASA-5510# show conn

54764 in use, 54764 most used

TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB

Random Sources

Embryonic Conns

TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB TCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aB TCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aB TCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB . . .

(78)

Case Study: Intermittent Access to Web

Server

Traffic

Permitted

Connection Count

Jumps

Permitted

SYN Flood

Detected

(79)

Case Study: Intermittent Access to Web Server

• Apply TCP Intercept to stop the SYN flood attack

access-list 140 extended permit tcp any host 192.168.1.50 eq www !

!

class-map protect

description Protect web server from attacks match access-list 140

!

policy-map interface_policy class protect

set connection embryonic-conn-max 100

!

(80)

Case Study: Intermittent Access to Web

Server

Few clients represent

TCP Intercept

applied

Few clients represent

50+ % of traffic

(81)

Case Study: Intermittent Access to Web Server

• Apply

per-client-max

option to limit the number of

connections any single client can establish

access-list 140 extended permit tcp any host 192.168.1.50 eq www !

!

class-map protect

description Protect web server from attacks match access-list 140

!

policy-map interface_policy class protect

set connection embryonic-conn-max 100 per-client-max 25

!

(82)

Case Study: Intermittent Access to Web

Server

per-client-max

(83)

Case Study: Intermittent Access to Web

Server

Attacks Being

Mitigated by ASA

Attacks Still

Occurring

(84)

Case Study

(85)

Case Study: Poor Voice Quality

Problem

• Poor Outbound Voice Quality at SOHO sites

Outbound RTP Stream

100 Mbps 100 Mbps _Cable Modem

2 Mbps WAN ASA-5505

(86)

Case Study: Poor Voice Quality

Solution: Traffic Shaping

• What is Traffic Shaping, and why is it needed

here?

• Why won’t Policing work?

• Why won’t Priority Queuing alone work?

100 Mbps 100 Mbps Cable Modem 2 Mbps WAN ASA-5505 Shape to 2 Mbps

(87)

Case Study: Poor Voice Quality –

Configuration Example (Traffic Shaping)

class-map voice-traffic match dscp af13 ef

!

policy-map qos_class_policy

Solution

Prioritize voice traffic and shape all traffic down to 2

Mbps on the outside interface.

policy-map qos_class_policy class voice-traffic priority ! policy-map qos_outside_policy class class-default shape average 2000000 service-policy qos_class_policy !