Best Practices for Evaluating
Anti-spam Solutions
Nathan Turajski Jamz Yaneza
Benchmarking Validation
• Methodologies
– Accurate – Comprehensive – Fair• Filtering Techniques
– Pattern matching, Heuristics, IP blocking, Whitelist/Blacklist, Challenge/Response, Community …..
Anti-spam Solutions
• Current Solutions
– Software – Appliance – Services – Legislation• Methods
– Catch rate (effectiveness) – Error rate (accuracy)
Content Defined
• Spam
– UCE, commercial bulk mail – Consumers: well defined – Enterprise: borderline
• Non-spam
– Appropriate, predictable, traceable
• Graymail
– Inappropriate to environment – Requires exception capability
Factors for Evaluating Solutions
• Primary
– Effectiveness – Accuracy – Resiliency• Secondary
– Administration – IntegrationTesting Failures
• Confused spam type classification
• Non real-world environment
• Short-term testing cycle
• Fixed regional origins
• Fixed language type
• Non-relative industry
• …. Etc.
Spam Trends
• Estimates vary, but the total amount was usually agreed to have passed 40% by the beginning of 2002
• Email was 50% SPAM by January of 2003 • 65% of all email was SPAM by 2004
• Almost 80% of all email is currently either unwanted advertising or virus-ridden
Evaluation Guidelines
•
Valid vs. illegitimate mail
– sampling over time period
30% Monthly Spam Growth (2005)
Total Spam Mails Received
163,425 202,867 183,269 232,194 0 50000 100000 150000 200000 250000
Evaluation Guidelines
English vs. Non-english New Spam Mails Received
March, 34% April, 49% June, 54% March, 66% April, 51% May, 62% May, 38% June, 46% English Non-English
•
Predominant language
Evaluation Guidelines
What Country does Spam like the Most?
21.42% 10.34% 2.21% 2.05% 1.84% 2.77% 2.97% 3.83% 3.94% 21.78% China United States Republic of Korea Brazil Japan France Spain Taiwan Israel Germany Spam Countries France 5% Brazil 4% Switzerland 4% Japan 3% Spain 3% Germany 2% Poland 2% Republic of Korea 31% China
•
Point of origin
– broad mixed samplingEvaluation Guidelines Spam Categories 19% 8% 14% 36% 18% 23% 3% 24% 10% 8% 4% 1% Adult Bad Samples Commercial Financial Health Others Non-English
•
Industry definitions
Chinese Language (traditional)
• Summary:
– 38% commercial offers, 23% work related, 22%
Traditional Chinese (s na ps ho t) Work 23% Sexual 7% Health 4% Financial22% Commercial 38% Other 2% Spiritual 0.3% Education4% Work Spiritual Sexual Health Financial Education Commercial Other
Chinese Language (simplified)
• Summary:
– 69% commercial offers, 17% financial, 7% sex
Simplified Chinese (s na ps ho t) Spiritual 0.04% Education 4% Commercial 69% Health 2% Sexual 5% Other 2% Work 1% Financial 17% Work Spiritual Sexual Health Financial Education Commercial Other
German Language
• Summary:
– 15% sex related, 12% commercial, 71% mixed offers
German (s na ps ho t) Sexual 15% Other 71% Financial 1% C ommercial 12% Health 1% Sexual Health Financial Commercial Other
Evaluation Guidelines
•
Timeliness
– update frequency
– distribution strain on network/system – correction efficiency
Evaluation Guidelines
• Summary
– Efficiency and accuracy dependent on spam classification and audience
– Used testing samples to be valid and fixed – Overall results used for evaluation
– False positive graymail vs. legitimate mail – Unmodified message delivery
Other Considerations
• Product configuration and tuning
– Out of the box state– Vendor recommended tuning
– Tolerance rating based on audience target – Long-term testing timeframe
Other Considerations
• Filter technique testing
– Signature matching• Focus: catch efficiency and update timeliness – Heuristic rules
• Focus: false positive rate and mitigation tools – Hybrid techniques
• Focus: accuracy and update timeliness – IP filtering
Other Considerations
• Performance
– Deployment time
– Management reporting tools – Update overheard
SUMMARY
• Comprehensive evaluation includes
– scalability and resiliency– long term performance – customer specific goals – exception handling
Mass-mailing malware spam
• Summary:
– 2003, due to Mimail, Blaster, and Sobig
Malware Tracking Center
0 500,000 1,000,000 1,500,000 2,000,000 Janu ary Febr uary Marc h April May June July Augu st Sept emberOctobe r Nove mbe r Dec ember Bulk-mailing Malware # o f re p o rte d in fe c ti o n s (u ni qe) 20022003 2004
