Outblaze Messaging Solutions
Technical White Paper
Anti-spam & Boundary Services
Version: 1.0
Last Updated: January 2006 By: Gideon Sheps
Introduction
Outblaze hosts webmail services for thousands of domains with a user base of over 40 million active mailboxes. The large user base and wide reach of Outblaze services clearly pose some significant challenges in terms of viruses, junk mail (spam) and other malicious uses of email. This document will outline some of the measures Outblaze takes to address these problems.
Outblaze is committed to preventing abuse of Internet services by malicious elements. With respect to the specific services that Outblaze offers, the company strives to perform two key duties: to protect Outblaze users from incoming threats, and to prevent Outblaze users from acting as sources of threats. To achieve these goals, Outblaze invests heavily in equipment and man hours with the goal of keeping the Internet commons as free of spam and other undesirable email as possible.
Solution Components
Outblaze Network Level Anti-spam
The first line of defence is the firewall. Professional and persistent spammers and many other types of attack can be effectively blocked at the firewall level by restricting protocols and blacklisting email originating from identified sources. Outblaze updates its firewalls every 15 minutes, adding and removing over 12,000 entries per day!
At the gateway to the Outblaze systems a set of machines, collectively known as the Outblaze SPAM Filter (OBSPF) Stack, provides the second line of defence after the firewalls. This set of machines applies the text analysis and other filters that remove spam email based on its content or structure at the same time, the OBSPF also has the capability to authenticate the delivery address allowing early and inexpensive blocking of dictionary attacks. Spam email that comes from otherwise legitimate sources (and thus cannot simply be blocked by the firewall) is removed here.
For more details see the section “Inbound AS Filtering” starting on page 7 below.
The Network Level approach filtering applied by Outblaze is unique, and is one of the core strengths of the Outblaze approach. Because the OBSPF Stack is positioned before email passes through the second layer of firewalls into the mail delivery architecture it isolates anti-spam operations from the bulk of (legitimate) processing for both inbound and outbound email. Unlike other email systems this means that the overhead of AS operations does not impact webmail, SMTP, POP, IMAP and the end user experience. Even a large scale spam-attack (a daily event) will have a smaller impact on an Outblaze powered client than anyone else can claim.
Outblaze Network Anti-virus
As the SPAM Filter Stack applies its anti-spam analytics, a search for well known virus patterns is carried out. Since many of the viruses in circulation are well known and easy to identify, they can be removed at this level before any further processing time is wasted.
Outblaze Sentry Anti-spam
Sentry Anti-spam is second layer of content filtering that is optionally applied to email that has passed successfully through the Network Level OBSPF filtering. Unlike the Network Level filters which are always on, this layer is usually offered as a premium service and our client may elect to offer it to their end users or not.
The content filter checks the message header, body, and pattern using different filter standards. Using these criteria, the system assigns a score to the e-mail received. If the filter score is high, it is likely to be
filter content analysis and the individual account holders Filter Sensitivity settings. For example, if the score is high and they have set the filter to High Sensitivity, the e-mail is likely to be sent to the Junk Mail Box.
If the score is high and the user has set the Filter Sensitivity level to Low, the message is sent to the Inbox. Having these different levels of filtering ensures that messages that reach each mailbox are most likely legitimate. Individual users can turn this level of filtering off, or set it to Zero Tolerance so that only mail from their own white listed address list is delivered to their inbox.
Outblaze Sentry Anti-virus
The Outblaze AV engine is an Outblaze engineered component that is designed to accept third party AV engine plug ins to benefit from advanced expertise in the anti-virus field from the worlds leading companies. Outblaze has partnered with Central Command and has an unlimited license to use their Vexira AV engine. For more on Vexira see page 5 below. Like the Sentry Anti-spam service, Sentry AV is an optional service.
Click & Scan
Click and Scan is an option available only within the Outblaze web mail client which scans attachments as they are opened. Click and Scan can serve two purposes: since it only scans email “on demand” it can be implemented as a cheaper option to Sentry Anti-virus (which scans every email prior to delivery to the mail box) for clients who use the webmail interface; it can also be used to augment Sentry AS services by providing yet another layer of protection. Because there will be a delay between the initial inbound Sentry AV scan and the opening of the email, it is possible that the virus pattern files will have been updated. In the event of a fast moving new virus, a user may have emails in their inbox which despite having been scanned on delivery, may still be infected. Although Vexira has excellent “zero hour” heuristic detection algorithms, the possibility of an as-yet-unknown virus slipping through always exists. When a global virus alert has been issued in the media, an astute user with Click & Scan needs only wait until the patterns have been updated before opening any suspicious email. For a webmail client this affords the type of protection a desktop mail client (e.g. Outlook) user would gain by having their own locally installed AV software.
Assembling the Perfect Solution
As described in the diagram below, the various components described above can be assembled to fit the needs of any client. Because of the overall flexibility of the Outblaze email solution, the various pathways described below can even be applied to different end user accounts within the domain depending on their needs.
Inbox
Always on.
located in Internet facing "spam filter stack"
Domain level settings. Configured per service package based on client needs.
Level 1 Anti-spam and Anti-virus (aka Network Level AS/AV)
Level 2 Anti-Virus (Vexira) (aka Sentry AV)
Level 2 Anti-spam (aka Sentry AS)
Webmail 2 5 POP / IMAP 1 E-Ma il 4 4 3 1 1 3
(Path 1) Network Level Scanning + Full Sentry Service: Mail scanned by Vexira for viruses and subject
to spam filtering according to personal Sentry AS preference settings in the Webmail client. This path represents the highest level of protection.
(Path 2) Network Level Scanning: This path has all emails scanned by the Outblaze boundary services.
(Path 3) Network Level Scanning + Level 2 Anti-spam: filtering according to personal Sentry AS
preference settings in the webmail client. Email not scanned by Level 2 anti-virus.
(Path 4) Network Level Scanning + Level 2 anti-virus only: All messges scanned by Vexira. No
additional anti-spam filtering.
(5) "Click & Scan" enabled in webmail client: Attachments are scanned by Vexira when opened. This
option can be used in conjunction with any other. Click and Scan applies within the webmail client only. Network Level Scanning can be bypassed but Outblaze will normally only allow this if a suitable alternate spam filter is in place. This has been done for corporate "split services" where inbound mail is received via the corporate mail servers only and has passed through the company's spam filters before being routed to Outblaze for delivery.
Vexira Anti-Virus
Outblaze Anti-Virus System is a strong virus defense system specializing in e-mail virus protection. It defends e-mail boxes from receiving virus infected files, and prevents users from sending virus infected files to other e-mail recipients.
Outblaze integrate Vexira Anti-virus technology seamlessly into our anti-virus system. The application can intercept TCP/IP port 25 (SMTP) connections or can be started using the Internet superdaemon (inetd). It spools all inbound and outbound e-mail messages and scans them using the Vexira’s virus scanner. Messages that are found to be virus free are immediately forwarded to the recipients, whilst infected messages are deleted or quarantined with optional sender, recipient, and mail administrator notification. Presently Outblaze follow industry best practice and simply deletes virus bearing emails with no notification.
¾ Features of Outblaze Anti-Virus System include:
¾ Virus detection, quarantine, removal
¾ Virus scans all in-bound and out-bound e-mail
¾ Real-time virus interception
¾ Capability in process extremely high volumes of e-mail
¾ Scalable to the maximum capacity of the server's processing ability
¾ Configurable warning notifications to sender, recipient or postmaster
¾ Advanced heuristic virus detection technology
¾ Automatic update of scan engine and virus definition file
¾ Support virus scanning within file archives (ZIP, RAR, LHA, ARJ etc...)
¾ Detection of malformed messages
¾ Blocked access to infected files
¾ Fast and efficient
¾ Simple and transparent
¾ Per-user notification
Vexira Antivirus is updated weekly on schedule and daily as needed for fast spreading viruses. The configurable built-in updater can check for updates as often as desired. It is intelligent so that it updates both the virus scanning engine plus the virus databases and will only update when a new update exists.
How much spam do we actually stop?
The diagram below is a plot of one (1) minute’s activity, drawn at random from the logs of a selection of firewalls and anti-spam filter machines that are responsible for network anti-spam across the Outblaze clusters. The green area plots valid emails that were accepted for further processing, and the red area plots emails rejected by that machine. An email will travel through a number of machines, and be scanned by various processes from detailed anti-spam and virus detection to checking against your personal black list. A lot of processing effort and cost can be saved by identifying and rejecting spam as early in the process as possible.
The vertical axis markings on the graph below are units of 20,000 emails. The highest peak shows 80,950 emails processed in one minute by one machine, of which 9,621 were accepted and 71,329 rejected – a reject rate of 86%. Roughly similar average rejection rates can be found for each machine.
Given that Outblaze currently delivers between 200 and 250 million legitimate emails each day, this means a staggering 1.7 billion spam emails rejected every day!
Spammers shift IP addresses and change their methods as fast as they can. In addition, many viruses are designed to turn a desktop PC into a spam sending “zombie”. In order to keep up Outblaze uses a range of tools and tactics to keep track of new developments and updates its IP blocks every 15 minutes (96 times a day). In a normal day, we add 12,000 new IP addresses. We update our programmatic filters once per day, typically adding one or two new filter rules per day. Several of these tools and tactics are discussed in further detail below.
Anti-spam in Detail
Preventing Abuse of Our Systems
Anti-Mail Relay
Each Outblaze MTA can be configured to turn off message relay to remote MTAs. This feature is useful in configurations where some MTAs are outside the firewall and do not support relaying (to prevent non-customers from using an ISP’s machines for spamming), and to have other MTAs inside the firewall that do relay (to enable people inside a system to get to the outside world).
Mail Relay Prevention
Mail relay can be restricted to prevent or allow message relays from or to designated locations. Criteria that can be specified for filtered senders include the domain or IP address of the machine receiving the relay, the domain or IP address of the server to which the client is connected, and the e-mail address of the sending client.
Connection Blocking
The server can refuse incoming SMTP connections when they originate from designated IP addresses and IP networks. This feature is useful for preventing denial-of-service attacks, in which a site opens up multiple connections to a server in an attempt to prevent others from using it. The MTA can also be configured to limit the number of simultaneous SMTP connections from a given client IP, which can be specified globally or by individual IP address or subnet.
SMTP Authentication
To prevent the forgery of e-mail, SMTP Authentication verifies the accuracy of the e-mail source address. Outblaze uses the SASL protocol and the LOGIN AUTH command in the ESMTP specification.
Acceptable User Policy
Outblaze maintains and enforces a zero tolerance Acceptable Usage Policy (AUP), which may be viewed in full at http://spamblock.outblaze.com/massmail.html. The enforcement of the AUP is the responsibility of the postmaster’s team.
Preventing Inbound and Outbound Spam
Sender Blocking
Mail can be refused or the connection dropped by the server on the basis of senders designated by their complete address, their domain, or their username. Thus, the granularity of filtering is adjustable: from any sender in a specified domain to an individual account. Sender addresses (the SMTP MAIL FROM address) are compared against these blocking rules for an exact match, which will cause the message to be refused or the connection dropped. While the sender blocking list is maintained by the ISP, if desired the ISP can allow individual users to select whether or not they want to apply the sender blocks to their accounts.
Sidelining Suspicious Messages
Messages identified as potential spam may be sidelined, or queued for examination and approval by the Outblaze administrator. After examination, this sidelined mail may be deleted or reintroduced into the MTA as deferred mail and subsequently delivered. The criteria that can be used to sideline mail include configurable thresholds for number of recipients, number of connections (by IP address), and number of recipients of a message from a “null” address. The MTA logs each of these events.
Sendmail Filters
Filters can be optionally run on incoming mail to reject, bounce, sideline, forward, throw away, or proceed with normal delivery based on message content. Filter actions can be attuned to the presence or absence of certain headers, the content of headers or the body, the sender, or the recipients. Message content can be tested by searching for exact string matches or more flexible pattern recognition. Outblaze logs any filtering actions and maintains frequency-of-occurrence statistics for administrator review.
Inbound AS Filtering
The First Line of Defence
Outblaze implements a multi-pronged messaging control strategy. The first line of defence is a set of IP based filtering systems, which include packet filters and a large group of DNS based Block Lists (DNSBLs). To prevent dependence on external resources, these DNSBLs are locally mirrored and their contents validated against a local rule-set designed to reduce false positives as much as possible.
The machines responsible for accepting incoming mail are referred to as the Outblaze Spam Filters. These machines employ several tiers of mail blocking and filtering when handling mail:
1) External DNS based Block Lists (DNSBLs)
The external DNS based Block Lists used by Outblaze are ORDB, RSL, SBL, CBL, Blitzed OPM, and Sorbs DUHL, as detailed at http://spamblock.outblaze.com/spamchk.html. Outblaze maintains regularly updated local mirrors of these lists.
2) Local DNS based Block List (DNSBL)
A local checklist of spam sending netblocks is also used, called the OBSL (Outblaze Spammers List). This includes known spam sources not otherwise listed in the various other DNSBLs utilized, as well as open relays that deliver spam to the filtering servers. This list also includes known dialup and other dynamic IP pools to prevent spam from being sent by spammers via trojaned systems, people running bulkmail software on their PCs, and viruses with their own SMTP engines like Sircam, Klez and Swen.
3) Local Name based Block List
This is a list of known spammer domains and other spam sources. The list is compiled and maintained using spam reports from Outblaze users who click the “report as spam” button, log monitors that alert us when irregular activity takes place (such as when a newly registered domain starts sending volumes of spam), “honey pot” accounts maintained by Outblaze, and reports from various anti-spam lists.
Completely shutting spammers out with a firewall
In some cases insistent spam sources that are prohibited from sending messages to Outblaze nonetheless persist in their attempts to send massive quantities of spam to Outblaze mail systems, and may even have an operational impact on normal mail delivery; in these cases Outblaze may elect to reject spam sources at the firewall level. The firewall blocks anything from specific IP addresses from reaching the Spam Filters (the machines on which the above spam blocking measures are implemented), and is maintained by the Outblaze Systems Team, Outblaze Security (Antispam) Operations, and other involved Outblaze staff.
The Second Line of Defence
If a remote host is allowed to connect to the Outblaze servers (i.e., if it is not blocked by the spam filters outlined above), then it faces a battery of tests that comprise the second line of defence employed by Outblaze against junk mail.
1) HELO checks, you are not what you seem to be
A check is performed on the EHLO / HELO greetings of hosts that make SMTP connections to Outblaze systems. SMTP connections that claim to be from free email domains like Yahoo! or Hotmail but do not originate from Yahoo! or Hotmail systems are rejected by this test, along with messages displaying other characteristics of spoofed HELO patterns (which are techniques employed almost exclusively by spammers or viruses).
2) Nonexistent domains are not allowed
Mail from non-existent domains or irresolvable domains (domains that do not have MX and/or A records in DNS), is rejected by Outblaze mail servers. This test essentially checks to see if the domain is capable of accepting mail from Outblaze.
3) Forged headers are not allowed
Faked header patterns are recognized as false and the message is added to a much more complex filter for content analysis, which sees the entire email before making the decision to accept or reject the email.
The Third Line of Defence
The third line of defence is analysis of the system logs on our mail systems. Due to the sheer volume of the logs (several gigabytes a day), scripted detecting of spam patterns is required. The scripts analyze the logs using different parameters, including mail connection counts, envelope senders used, invalid recipient count, past logging history, matching sender domain and reverse DNS of the connected host for select large email providers, etc.
Outblaze also maintains a large number of spam traps (also widely known as “honey pots”), which are email accounts that do nothing except receive spam. Since the addresses of spam traps are never released to anyone outside Outblaze, the only way for spam traps to receive mail is via spam (e.g., in a “dictionary attack”), so any host sending mail to them is automatically flagged as a spammer.
Systems that fail the reverse DNS and domain-matching test for the commonly spoofed domains, those that send spoofed HELOs, those that send spam to Outblaze spam traps, and those about which user complaints are received are all tested to determine whether they are insecure and open to third party relaying or proxying of email, in which case they get added to the Outblaze local block list (the Outblaze Spammers List, or OBSL).
Servers that fail the statistical analysis test and servers sending mail to large numbers of unknown recipients are identified as probable spam senders or compromised servers. These may, at the discretion of the postmaster team, be pre-emptively blocked until the remote server’s administrator requests removal of the block. A flood of bounces from a mail server leads to it being flagged for testing as a potential open relay or proxy.
Newly registered domains that send large volumes of email to Outblaze servers are pre-emptively blocked until an administrator can verify them as legitimate.
The Fourth Line of Defence
The fourth line of defence is the human component, consisting of the postmaster and the security & abuse team. This team is responsible for dealing with Outblaze users who complain about spam sent to their mailboxes, and with the general public about abuse originating from Outblaze users.
The postmaster team looks at spam reports from Outblaze users and figures out what hosts are used for spamming. These hosts are then added to local block lists as spamming systems, unless the DNS and reverse DNS entries for that host indicate that this is a major ISP’s mail server.
Preventing Outbound Spam
Outblaze has implemented a number of measures to prevent its users from spamming, and to stop permanently those users who make the decision to spam from Outblaze run mail systems.
Limit on Mail Recipients
The first step is to limit the number of recipients per email. This is an elementary, but quite useful, step to deter bulk mailers from abusing Outblaze mail servers.
Automated Recognition and Disabling of Spam Accounts
In the case of Outblaze accounts that start sending out spam, outbound mail servers are filtered with a combination of Spamassassin and other filters, some open source, some developed in house. These filters are set up to terminate automatically the spammer’s account once sufficient levels of spam are detected. Several “Nigerian 419” scam artists and “make money fast using the net” spammers get caught by this technique, sometimes within a few minutes of creating an Outblaze email account.
Drop-Boxing/Open Proxy Prevention
Some spammers are clever enough to spam from elsewhere (say through open relays or open proxies), and use an account on our service in the “from” or return path of their spam message. When a new Outblaze account starts receiving large numbers of bounce notifications, the account is assumed to have been used for spamming and deactivated. If an established (not new) account starts getting a flood of bounces, it is assumed that the recipient name was spoofed by a virus or spammer, and mail delivery to the account is temporarily suspended until such time as the flood of bounces stops.
External Relationships and Discourse
The postmaster team members routinely correspond and maintain good relations with other postmasters and mail system administrators in order to discuss spam related issues of mutual interest. Members of this team also monitor and participate in several newsgroups and mailing lists devoted to finding a solution to the spam problem. This helps in ensuring that Outblaze always has current information about the state of the spam problem, as well as maintaining good relationships with anti-spam activists around the world.
Involvement in the Internet Community
To ensure that this complex system works, Outblaze ensures that people hired are amongst the best available. Outblaze also routinely assists other ISPs in improving mail and anti-spam systems at technical and policy levels. Outblaze also sponsors its staff to train postmasters and abuse desk administrators of other ISPs at international conferences and events such as APCAUCE (http://www.apcauce.org).
Other Techniques Employed
Textual Analysis
Spam often come with certain contextual/semantic/discourse characteristics. Outblaze Content Filter constantly updates its rules to detect those ever-changing characteristics and spam signatures, such as
• Disclaimers and confidential texts: to mislead recipients that the junk mail are delivered because of the recipients' earlier subscription or enquiry.
• Unsolicited commercial e-mail: for promotional and marketing purposes.
• Pornography: sexually explicit spam.
• Scams and Frauds: such as "get rich quick" schemes, pyramid schemes, fraud attempts, risky propositions, and not-so-wise investment offers.
• Insensitive materials: such as racist propaganda, sexist jokes, culturally offensive insults, religious slurs.
• Too Good To Be True: incredible-sounding offers to capture your attention-not to mention your time and money.
Text Manipulation Analysis
Text manipulation pertains to methods adopted by spammers to pass textual analysis (machine rules) while the message is still comprehensible by recipients (human beings). This is a matter of creativity with unlimited possible variations as long as spammers can invent. For example:
• Using visually similar characters: pi11 instead of pill (use the number one to replace the lower case of L), \/\/atch (use \ / \ / to replace W), P0RN (use the number zero to replace O)
• Separating words or characters to cripple textual on the whole word: F-R-E-E S-E-X
• Joining words or characters to cripple textual on the whole word: GET*RID.OF_DEBT and hack^^^people
• Using symbols to represent a word: Get $ soon
• Using auditory similar symbols: blonde2U and MyBody4U
• Using willfully misspelt words: free passwrd,
• Adding randomly generated character strings: Force Me To Suck It rngiyqefu and Filthy Animal Loving gtxykaqws
Heuristic Analysis
Heuristic comes from the Greek "heuriskein" meaning "to discover". It refers to the process of gaining knowledge or performing some desired results by intelligent guesswork instead of following pre-established formula.
Outblaze heuristic analysis identifies spam based on junk mail characteristics through feature-matching rules that are gained from experience. These are some examples of heuristic analysis:
• unbalanced HTML tags (e.g. embedding junk HTML tags like <!D>h<!i>e<!T>a<!B>l<!D>th<!G> c<!Y>o<!F>v<!D>era<!d>ge)
• mixed foreign character sets
• different encoding methods
• the presence of illegal characters in base64 attachment
• message written in an undesired language
• attempt to disguise pornographic words
Web Beacon
Also known as Web bug, single-pixel tag, or clear GIF, Web beacon is used by professional spammers to identify live and active e-mail addresses, to track user behavior.
Used in combination with cookies, scripts and sophisticated HTML commands, Web beacon often comes in form of one-pixel transparent GIF placed in an HTML-formatted e-mail, like:
<img src="http://www.spammersite.com/[email protected]" border="0" width="0" height="0">
The HTML code in the spam message instructs the recipient e-mail client or Webmail system to retrieve the image; as a result the recipient's IP address, viewing time and duration, browser type, as well as previously set cookie values are all exposed to the spammer.
Outblaze anti-spam system scans messages for this kind of spy-natured Web beacons. Outblaze Webmail System can further allow user to block Web beacons from the e-mail message.
URL Classification
Classifying URL links embedded in a message is an effective approach to spot spam. Almost all spam messages contain URL links to direct recipients to a site for contact information and purchasing instructions. Without these call-to-action links, the spam messages would serve no practical values. URL Classification compares these links to a list of known spam URLs.
Outblaze anti-spam system can detect visible links, dynamically downloaded links and also deliberately disguised links. Its URL Classification offers accurate junk mail detection with low false-positive rate.
The Outblaze 24x7 Anti-Spam Team
The Outblaze anti-spam team consists of specialists in Hong Kong and India working 24x7, updating and resolving spam issues. Within an hour of a spam attack, external or internal, the team is able to notice, analyze, and identify the spam sources. In most cases Outblaze is able to alert the unsuspecting ISP or mail service provider of an external abuse. For internal cases Outblaze is able to immediately suspend or block internal users from continuing spam activity at either the application level or, if security is breached, by blocking IP access.
The team also handles complaints from end users regarding:
• e-mails that were blocked / filtered by Outblaze Anti-Spam System but were regarded as legitimate
• e-mail sent by valid users managed by Outblaze but blocked by external parties.
The anti-spam team negotiates with other ISPs and e-mail service providers to remove external blocks to Outblaze-managed domains and SMTPs, as well as exchanging spam research and block lists.
Outblaze Anti-Spam specialists are also responsible for updating and tuning the Anti-Spam engine and filters in regular basis according to the latest statistical findings as well as user feedback.
Research is being conducted to seek state-of-the-art solutions for clients.
A word on SPF and other proposed Sender Authentication Schemes
After initially implementing SPF the Outblaze Anti-spam Operations Team has come to realize that there are serious flaws in the SPF model that result in unacceptable levels of lost legitimate mail and the restriction of useful features such as mail forwarding. Outblaze has removed, and no longer publishes, SPF records for the domains that it manages.
Outblaze blocks port 25 and requires Authenticated SMTP through port 587 (the message submission port, as defined in RFC 2476) which effectively eliminates anonymous spam injection into an SMTP server. Outblaze recommends that all ISPs adopt this protocol.
Outblaze continues to carefully evaluate the long term implications of various reputation and authentication technologies, such as domainkeys, and CSV, and will choose one when it is sufficiently mature.