Text Categorization
Moshe Koppel
Lecture 10: Spam Detection
Obligatory Scare Slide
• There‟s lots of spam
• The proportion of spam is growing – it will
soon exceed 100% of all email sent
• It costs the world gazillions of dollars
• Spam is BAD
• (Actually, lately it looks like spam email
has been mostly defeated.)
Kinds of spam
• Active spam – ads and scams
– chatbots
– commentbots
• Passive spam – websites
– link farms for SEO
– adsense parking lots
Special Issues
Spam detection is basically a text cat problem,
but there are some special issues:
• Collecting data – non-spam email is private
• Asymmetry – must never class good mail as spam
Collecting Data
• Standard collections
– SpamAssassin Corpus
– TREC corpora
• Use your own email
– Might not reflect world
• gmail has user feedback
– LOTS of examples
– Haphazardly labeled
Problem of False Positives
• False positives more costly than false negatives
• Research must report recall-precision curves;
key point is precision ~ 1
Adversarial Problem
• Spammers reverse engineer global filters;
use nasty tricks to circumvent them
• This is what makes spam detection an
interesting problem
Basic Spam
• Let‟s start with some garden variety spam
• This is easily detected by standard text cat
It cost you nothing (Yes! $0) to give Us a call, We will contact You back Absolutely No exams/Tests/classes/books/Interviews
No Pre-School qualification Needed! ---
Inside USA: 1-718-989-5XXX 0utside USA: +1-718-989-5XXX ---
Degree, Bacheelor, masteerMBA, PhDD available in the field of your choice that's Right, You can even become a doctor & receive all the benefits That omes With it!
Please Leave Below 3 INFO in voicemail: 1) your Name
2) your Country
3) your Phone No. (with Countrycode)
Most Honorable Sir,
I am Ehud Olmert, formerly the Prime Minister of Israel. I URGENTLY REQUIRE YOUR ASSISTANCE IN A MOST DISCRETE MATTER. As a result of certain events in my country, it has become necessary for me to transfer a considerable sum of cash to a foreign bank account. I turn to you as a MOST HONORABLE AND TRUSTED PERSON for your discrete assistance.
The total amount involved is THIRTY MILLION NEW ISRAELI SHEKELS only [30,000.000.00 NIS] and we wish to transfer this money into safe foreigners account abroad. I am only contacting you as a foreigner because this money cannot be
approved to a local person here, but to a foreigner who has information about the account, which I shall give to you upon your positive response. I am revealing this to you with believe in God that you will never let me down in this business, you are the FIRST AND THE ONLY PERSON that I am contacting for this business, so please reply urgently so that I will inform you the next step to take urgently.
At the conclusion of this business, you will be given 40% of the total amount, 50% will be for us while 10% will be for the expenses both parties may incurred during this transaction. PLEASE, TREAT THIS PROPOSAL AS TOP SECRET.
Early Work
Sahami et al „98
• Learner: Naïve Bayes
• Feature Set: Words, Phrases, Structural Features
• Feature Selection: top 500 infogain
• Evaluation Data: ~1700 Messages, ~88% Spam
Early Work
Sahami et al „98
Hand Crafted Features
– 35 Phrases
• „Free Money‟
• „Only $‟
• „be over 21‟
– 20 Domain Specific Features
• Domain type of sender (.edu, .com, etc)
• Sender name resolutions (internal mail)
• Has attachments
• Time received
Later Studies
• The early work was followed by the usual
stream of extended feature sets and fancier
learning methods (e.g. SVM)
• It is now common to use over 100,000
features
• Learning methods for huge data sets must
be very efficient (online algorithms)
How to Beat an Adaptive Spam Filter
Graham-Cumming „04• Use machine learning to discover words that beat
an adaptive filter
– Take a message that is near spam threshold
– Send it to the target filter 10,000 times each time
adding 5 random words
– Train an „evil‟ filter to learn which messages beat the
target filter
– Use „evil‟ filter to modify new spam messages
• Found single word additions to get new spam by
the filter
Other Tricks
• Fill messages with real text taken from
books, sites, etc.
• Can even generate real-looking texts using
Markovian language models
The Hitchhiker Chaffer
• Content Chaff
– Random passages from the
Hitchhiker‟s Guide
– Footers from valid mail
“This must be Thursday,” said Arthur to himself, sinking low over his beer, “I never could get the hang of Thursdays.”
Express yourself with MSN Messenger 6.0…
Hitchhiker Chaffer‟s
Later Work
• There is nothing fancy
about this spam
– “A spam filter will catch
that in its sleep” –
anonymous
Hitchhiker Chaffer‟s
Later Work
• Hidden Text
• Content Chaff
• URL Spamming
Also included a number of unusual statements made by candidates during, „On display? I eventually had to go
down to the cellar to find them.‟
http://join.msn.com/?Pag e=features/es
More Tricks
• Encoded Text
• Distorted Text
Secret Decoder Ring Dude
• Another spam that looks
easy
Secret Decoder Ring Dude
• Character Encoding
• HTML word breaking
Pharmacy
Diploma Guy
• Word Obscuring
Dplmoia Pragorm
