Appendix B: Presentation Slides
DO NOT REPRINT
© FORTINET
This lesson is about troubleshooting concepts.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
In this lesson, we will review troubleshooting strategies. We will also introduce some of the troubleshooting tools available in the FortiGate GUI and CLI.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
Let’s being by reviewing some troubleshooting concepts and strategies.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
Good administrators know their network well before any problem happens. That includes an understanding of the normal behavior related with traffic volume, network applications, traffic flows and devices' CPU and memory utilization. So, when a problem happens, good administrators identify quickly what is behaving abnormally. This information speeds up the troubleshooting process and helps to isolate the cause of the problem.
Many tools can be used to gather statistics and information while the network is operating normally:
SNMP, logging, sFlow, and the monitors located in the FortiGate GUI.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
It is also important to keep the network documentation up-to-date. Network diagrams should include physical connections, interface names and subnets. Good network documentation also includes change control records to track any change in the network: Who did the change? When was done?
What was changed?
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
If a problem happens, the first step is to define it well. For example, if the problem definition is “web filtering is not working”, the scope of the problem is too imprecise. Too many things could cause this.
This makes troubleshooting slow. So, we must ask questions to understand the details: Is the problem happening with one web site? Is it happening with all users? Is it happening randomly? How can you reproduce the problem?
After answering the right questions, we can define the problem with details. For example: “the web filtering is not blocking the web site X for the user Y”. This provides a better place to start the troubleshooting.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
A general approach for troubleshooting network issues is to follow the TCP/IP model and work the problem either from the highest layer to the bottom or from the lowest layer to the top.
In the first method you check the physical layer first. If a layer operation is ok, you move to the upper layer, until you find the layer where the problem is happening.
In the second method you check the application layer first, if a layer is not working properly you move to the layer below to rule out issues in the lower layers.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
During the second part of this lesson, we will review some of the troubleshooting tools available in the FortiGate GUI.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
The dashboard is the FortiGate GUI welcome screen. Some of its widgets contain information useful for troubleshooting, such as the system resources and the alert message console widgets.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
Remember that the dashboard is customizable. Widgets can be added, removed and customized.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
In addition to the dashboard’s widgets, the GUI includes some monitor screens for specific features.
For example, the IPsec monitor displays the status of each IPsec tunnel. The firewall monitor shows the list of authenticated users. Another example is the routing monitor, which lists the routes that have been loaded (active) the routing table.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
Another important tool for troubleshooting is the FortiGate logs. The log viewer includes a filter setting that is used to display only the logs entries related with one specific user name, IP address, URL or event type.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
This table illustrates the expected log behavior depending on the different logging settings.
The first column shows the possible values for the log setting in the firewall policies: no log, log security events, or log all sessions.
The second column indicates if the AV, web filtering or antispam profile log setting is enabled or disabled. Remember, DLP and IPS profiles always generate logs in the security log section.
The last column shows the behavior. If you enable any profiles on your policy and logging is not enabled, you will not get logs of any kind—even when profile is blocking traffic. So if you apply a security profile, it’s important to consider the logging settings.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
From the information stored in the logs, a FortiGate device with hard disk can generate reports, either manually or on an automatic schedule.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
Reports in FortiGate devices are fully customizable.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
Let's now introduce some CLI troubleshooting tools.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
The real time debug commands generate information in real time about what a specific FortiGate process or feature is doing.
The debug level is a bitmask value that specifies which types of messages are displayed. This depends on each process. For all the cases, although, a debug level of ‘0’ means no output (disabled), and a debug level of '-1' means enabling all possible message types.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
For example, this slide shows the two commands for enabling all the IPsec real time debug output.
You can also enable the option to prepend the system time to each debug line.
It is important to disable any real time debug after using it. They consume FortiGate resources and some could be CPU intensive.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
On the other hand, the application layer test commands do not display information in real time, but statistics and configuration information about a feature or process. Some of these commands can also be used to restart a process or execute a change in its operation.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
Some CLI debug commands generate a lot of output. If we know that the required information is contained in one specific line of the output, and if we know a keyword to find that line, we can use the GREP utility. This utility displays only the lines from the output that match a text string. For example, in this slide we are using the GREP utility to find the IP address 10.0.0.7 in the ARP table. We use the debug command ‘diagnose ip arp list’ to get the ARP table, and then we append the GREP utility to display only the information for one IP address. In this way, the output is only one line (with exactly the information we need), instead of a long list of entries.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
When displaying the FortiGate configuration via CLI, you can use the GREP utility with the option –f.
It will display only the configuration sections (or tables) where the text string matches at least one value. This is useful, for example, to find all the references in the configuration to a specific object. In this slide, we are using the –f option to find all the references to the wan1 interface. The output shows only the two tables where wan1 is referenced: the definition of the interface itself, and a firewall policy where wan1 is the destination interface.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
If you need to expand your FortiGate skills, or if you need more information about troubleshooting a FortiGate issue, these are some of the resources to get more information and help.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
To review, these are the topics that we just talked about.
Troubleshooting Concepts
DO NOT REPRINT
© FORTINET
In this lesson, we will show you how to troubleshoot system resources on FortiGate.
System Resources
DO NOT REPRINT
© FORTINET
You will learn some debug commands for troubleshooting FortiGate system problems, such as high CPU or memory usage. The lesson also covers new firmware installations from the BIOS, memory optimization and crashlog diagnostics.
System Resources
DO NOT REPRINT
© FORTINET
Let’s see first some general system troubleshooting commands.
System Resources
DO NOT REPRINT
© FORTINET
This is usually one of the first debug commands when troubleshooting. The output shows the firmware version, FortiGuard database versions, license status, operation mode, number of VDOMs and system time.
System Resources
DO NOT REPRINT
© FORTINET
The command 'get system performance status' shows the overall memory and CPU utilization. It also shows session creation rate, number of viruses caught and number of attacks blocked by the IPS. The last line displays the system uptime. This output gives a quick overlook at how much traffic the unit is handling.
System Resources
DO NOT REPRINT
© FORTINET
During this part of the lesson you will learn how to check the use of he FortiGate memory and CPU.
System Resources
DO NOT REPRINT
© FORTINET
To understand how a FortiGate uses its memory, we need to understand the architecture of FortiOS.
The hearth of FortiOS is its kernel. Here, the unit takes some of the most important and basic decisions, such as how to route a packet, or when to offload a session to a NPU processor.
FortiOS runs over hardware. The device drivers bridge the kernel with the hardware.
Above the kernel, there is the user space. Several application processes or daemons run at this level.
Above all that, there is the configuration layer, which is basically composed of two modules: the command line interface and the graphical user interface.
System Resources
DO NOT REPRINT
© FORTINET
How the memory is segmented depends on the FortiGate model. There are two cases: first, models running 32-bit FortiOS with more than 1 GB of memory; second, models running 64-bit FortiOS and models running 32-bit FortiOS with less than 1 GB of memory.
Let’s see the first case.
When the FortiOS is 32 bits and the system memory is more than 1 GB, the kernel cannot access directly the whole memory space. So, memory paging is used to reach the portion of the memory that cannot be accessed directly. The part of the memory that the kernel can access directly is called low memory. The part of the memory that is accessed using memory paging is called high memory. The command 'diagnose hardware sysinfo memory' displays:
- The total amount of low memory (LowTotal) - The amount of free low memory (LowFree) - The total amount of high memory (HighTotal) - The amount of free high memory (HighFree)
- The total amount of system memory (MemTotal = LowTotal + HighTotal) - The total amount of free memory (MemFree = LowFree + HighFree)
System Resources
DO NOT REPRINT
© FORTINET
For models running 32-bit FortiOS with less than 1 GB, and for models running 64-bit FortiOS, the kernel doesn't need to use memory paging to access the whole memory space. In those cases the command 'diagnose hardware sysinfo memory' shows a size of 0 kB for the total high memory and free high memory. All the memory space is considered low memory.
System Resources
DO NOT REPRINT
© FORTINET
FortiGate allocates memory for five main purposes:
- Kernel memory slabs - System I/O cache - Buffers
- Shared memory - Process memory
We will see each of these in the next slides.
System Resources
DO NOT REPRINT
© FORTINET
The kernel memory slabs are collections of objects with a common purpose. They are used by the kernel to store information in the low memory.
This slide shows example of some slabs. There are slabs for storing information about the TCP sessions.
The entries in the route cache are also stored in low-memory slabs.
System Resources
DO NOT REPRINT
© FORTINET
(slide contains animation)
To check how much memory is being allocated to kernel slabs, we use the command 'diagnose hardware sysinfo slab'.
The first column shows the slab name. The second column the total amount of active objects, (click)
then the amount of available objects, (click)
and the size of each object.
The total amount of memory allocated to each slab type can be calculated by multiplying the number of available objects by their size.
System Resources
DO NOT REPRINT
© FORTINET
The system I/O cache is used to speed up the access to information stored in the hard and flash disk memories. Some processes, such as logging, WAN optimization, and explicit proxy, store information in the hard disk, so they get the performance boost provided by this memory allocation.
An I/O cache page is labeled as active when it has been recently accessed. It goes to inactive state after not been used for some time. An inactive page can be reclaimed by the kernel if needed.
System Resources
DO NOT REPRINT
© FORTINET
(slide contains animation)
The command 'diagnose hardware sysinfo memory' displays the total amount of memory allocated for I/O cache.
(click)
It also lists the amount of memory allocated for buffers.
System Resources
DO NOT REPRINT
© FORTINET
As we explained, above the kernel layer there are multiple application processes or daemons running. The operating system allocates separated blocks of memory for each process. One process can access the memory that was allocated to it, but it cannot access any memory that was allocated to a different process.
So, a process cannot share information with another process by reading or writing data into the memory allocated to that other process. For that purpose, the operating system dynamically allocates shared memory (SHM). The SHM can be accessed by multiple processes, allowing the sharing the information among them.
System Resources
DO NOT REPRINT
© FORTINET
We can check how much memory space is being used by each process. The command 'diagnose sys top'shows that information in the last column. The command also displays, for each process: its ID number, its state, and the amount of CPU using. You can specify the refresh frequency and the number of lines to display.
While the command is running, you can press <shift-P> to sort the processes by CPU usage, of <shift-M>
to sort them by memory usage. To stop the command, use <ctrl-C>.
System Resources
DO NOT REPRINT
© FORTINET
Another useful command for displaying information about process memory usage is 'diagnose sys top-summary'. The –h option displays the different options available for this command.
System Resources
DO NOT REPRINT
© FORTINET
(slide contains animation)
For example we can use the option –s mem to sort the processes by memory usage; the option –i 60 to refresh it every 60 seconds; and the option –n 10 to display the top 10 processes.
One advantage of this command over the previous one ('diagnose sys top') is that it shows the total amount of memory allocated by all forked or child processes, including shared memory. In the case of 'diagnose sys top', each child process is displayed separated. Here, we have an aggregated view of all of them.
(click)
The output also shows the amount of file descriptors allocated. If the amount of any of the descriptors keeps constantly increasing, it might indicate that there is a memory leak problem.
(click)
If a process has forked, the number of child processes is displayed after its name.
System Resources
DO NOT REPRINT
© FORTINET
This table shows some of the most common processes.
System Resources
DO NOT REPRINT
© FORTINET
This other table shows more about the most common processes.
System Resources
DO NOT REPRINT
© FORTINET
The command 'diagnose sys top' shows the state of each process. A process can be in one of four states: Sleeping (S), running (R), do not disturb (D) or zombie (Z).
The S and R states are normal. It is also normal if a process goes briefly to D state.
The Z state is not normal. Also, it is not normal if a process stays in D state for a long time. These usually indicate that the process is not working properly.
System Resources
DO NOT REPRINT
© FORTINET
During this part of the lesson you will learn about conserve mode.
System Resources
DO NOT REPRINT
© FORTINET
There are 2 different types of conserve modes:
- Kernel conserve model is triggered when the amount of free low memory is running low
- Proxy conserve mode (also called system conserve mode) is triggered when the amount of free overall memory is running low.
The command 'diagnose hardware sysinfo shm' can be used to determine the conserve mode status. If the field conservemode is 1, the unit is in proxy conserve mode. If it is 2, the unit is in kernel conserve mode.
System Resources
DO NOT REPRINT
© FORTINET
Two margins or thresholds determine when the FortiGate enters and exists the kernel conserve mode. The margins depend on the total amount of low memory.
When a FortiGate is in kernel conserve mode, any proxy inspection is bypassed and administrators cannot change the unit configuration.
System Resources
DO NOT REPRINT
© FORTINET
These are the entries generated in the crashlog, event logs and alert message console when a FortiGate enters to and leaves the kernel conserve mode.
System Resources
DO NOT REPRINT
© FORTINET
Similarly to kernel conserve mode, two margins or thresholds determine when the FortiGate enters and exists the proxy conserve mode. The margins depend on the total amount of overall memory.
System Resources
DO NOT REPRINT
© FORTINET
These are the entries generated in the crashlog and event logs when a FortiGate enters to and leaves the proxy conserve mode.
System Resources
DO NOT REPRINT
© FORTINET
av-fail-openis the CLI setting that controls FortiGate’s behavior while it is in proxy conserve mode.
System Resources
DO NOT REPRINT
© FORTINET
An option related to fail-open, is av-failopen-session. This is a setting that kicks in not during a high memory situation, but when a proxy on the FortiGate runs out of available sessions to process the traffic.
If av-failopen-session is enabled, then FortiGate will act according to the av-failopen setting. Otherwise, by default, it will block new sessions until proxy connections become available.
System Resources
DO NOT REPRINT
© FORTINET
Additionally, FortiGate has one more mechanism to free memory when there is not much available. If the kernel cannot allocate more memory pages, it deletes the oldest sessions. The command diagnose sys session statshows the numbers of sessions that has been deleted by the kernel due to this mechanism.
System Resources
DO NOT REPRINT
© FORTINET
During this part of the lesson you will learn how to optimize the memory usage.
System Resources
DO NOT REPRINT
© FORTINET
Many FortiGate processes, such as DLP or AV scanning, are memory intensive. So, memory optimization is important, especially in small units, to guarantee that they will stay away from conserve mode. This slide shows some recommendations for optimizing the memory usage. These tips might increase significantly the available memory in a device that is frequently going to conserve mode.
The first and most logical step is to disable the features that are not required. For example, if the network
The first and most logical step is to disable the features that are not required. For example, if the network