Implementation and Evaluation

The structure of the SECURE-enhanced spam filtering application is illustrated in Figure 19. In this application, a principal represents an email user who has an email client that is configured to use the Simple Mail Transfer Protocol (SMTP) proxy, a protocol for sending email messages between mail servers and the Internet Message Access Protocol (IMAP) proxy, a protocol for retrieving email from a mail server and routing it to an email client. IMAP also allows for the creation of email folders on a server and the copying of messages between folders. SECURE is called by the proxy to make a decision as to whether or not to mark a message as spam.

Messages marked as spam are routed by IMAP to a spam folder, while legitimate non-spam messages are routed to the user’s email inbox. In the case of a false positive, i.e., a legitimate email is marked as spam, or a false negative, i.e., a spam email is let pass as legitimate, the user can move the message from or to the spam folder, and the move request is intercepted by the proxy and captured by SECURE in order to update trust evidence.

Mail Recipient, Jane Smith Mail Server IMAP & SMTP Proxies SECURE

The SECURE-enhanced spam filtering application has been evaluated in the SECURE Evaluation Framework (Bryce, Cahill et al. 2005). This experimental environment is illustrated in Figure 20.

SECURE KERNEL SECURE Evaluation Framework Engine

Jane Smith’s Profile

Community Profile

Mail File

Principal Jane Smith Community of other

principals

Forwards emails

Figure 20: The SECURE evaluation framework configuration

The environment models and evaluates mail messages sent to principal Jane Smith, and uses a fixed set of messages that are stored in a mail file. Each of these stored messages is tagged a priori as spam or legitimate. Messages are routed from the mail file to Jane Smith’s email client via the mail proxy, which calls SECURE to determine whether or not a message is spam. Each message is processed by the SECURE kernel’s trust and risk policies, and a classification is assigned, i.e., spam or valid. The results of the SECURE classification are compared to the pre-assigned spam tags which allows for the evaluation of SECURE’s accuracy with regard to correctly identifying both spam and legitimate email messages.

The mail file is compiled from the SpamAssassin benchmark using the easy_ham and spam files, composed respectively of valid messages and spam messages. There are 3051 messages in the benchmark, of which 2551 are legitimate and 500 are spam. The messages are sent from a community of 846 unique senders, of which 425 are spammers and 51 are repeat offenders, i.e., send more than one spam message. The number of valid mail senders is 421, of which 213 are once-off message senders and 208 send more than one message.

Two cases are evaluated. In the first case, Jane Smith uses no spam filter. All messages come into her inbox. She correctly identifies 2551 emails as valid and leaves them in her inbox. She also identifies 500 emails as spam and moves these messages to a spam folder, meaning that there were 500 false negatives in this case (spam emails that passed to the inbox as valid). No false positives occurred. The classification process in this case is 71.84% accurate.

In the second case, SECURE is used to make trust-based decisions about filtering spam based on Jane Smith’s observations about past mail sender behaviour. SECURE correctly classifies 2626 messages, with false negatives falling to 425 and false positives remaining at zero. The accuracy in this case rises to 86.07%, thus improving spam filtering by 14.23%.

With regard to recommended evidence, it is found that in the case where Jane Smith consults recommendations when she has no experience with a given email sender and when recommenders are fully trusted, i.e., are reliable to identify spammers, very high spam filtering accuracy can be achieved. In the case where some recommendations are false, if Jane Smith has some observations to rely on these observations act as a safety net against poor recommendations and valid classification can still be maintained, thus illustrating the importance of ‘buddy principals’ in whom Jane Smith can trust. Other important results of the evaluation are that SECURE can be implemented in a real autonomous system, and that SECURE provides a complementary approach to Bayesian spam filtering systems. Whereas Bayesian spam filters classify a mail message as spam based on a statistical analysis of message content, it was found that the SECURE spam filter can classify a message as spam based on a number of additional criteria, including trust in the message sender, acceptable risk levels for false positive or false negative errors, and confidence in the underlying system’s ability to recognise a mail sender.

However, we note that full SECURE capability is not utilised. For example, context is not used in this application, but it could be used, e.g., for providing information about the priority level associated with an incoming mail. Moreover, an alternative method for evaluating recommendations would have been to implement methods to assess recommendation integrity, especially given the evaluation result concerning the importance of recommendations when assessing email validity. Finally, it may have proven beneficial to provide the ability to evolve trust according to trust dynamic policies.