Spam Filtering Techniques

Feedback loops are schemes operated by ISP’s for email marketers, ESP’s and other organisations that send large amounts of email to optionally sign up for.

Typically to sign up the person or organisation applying for the feedback loop will need to prove that they are responsible for the mail being sent from the IP addresses or domains on which the feedback loop is being setup on so be prepared to fax or send documentation and to be able to receive email on your domains abuse address.

Once verified and the application is accepted by the ISP the ISP will then start emailing an email address defined within the application each time one of your emails is reported as spam by the subscriber by clicking the “this is spam”.

The email sent to this address will be an ARF message, abuse report format. This is an email identifying itself as an ARF message with an attachment of the original message that was sent to the complainant. The receiving mail server should pass this to either a human helpdesk if your volume is low to be unsubscribed from the list or alterntively be passed to a processing application if you send large volumes of email.

This processing application will read the ARF message and the email attached within it, identify either through the to email address or the headers in the message which subscriber is complaining and unsubscribe them from the lists automatically.

So how does this help me?

Although it may seem like a bit of a waste unsubscribing users who complain once, in the long term it will pay dividends. By unsubscribing users who complain once you are reducing the likelihood of them complaining again, this means your overall complaint rate per IP or domain is kept low which is the key metric that ISP’s use to choose whether or not to deliver your messages. By keeping your complaint rate low from your messages to ISP’s which have feedback loops they are much more likely to allow your messages straight through to the inbox ensuring you continue to get your emails to the subscribers who actually want to receive them.

In addition, some ISP’s, after you have established a good record on their feedback loop, will actually allow you to apply for a whitelist which allows you to skip some if not all of the spam filtering that they use. AOL for instance is one ISP which operates a whitelist scheme in addition to a feedback loop.

For a full list of feedback loops and information on how to sign up, click here.

If you use an ESP to deliver your campaigns on your behalf, they should already have these feedback loops in place. Ask them and if not see if these can be implemented on your behalf.

In our earlier post titled ‘What is SPF?‘ we established what sender policy framework was and how it works.

This time Email Manual will look at the reasons for implementing an SPF record on your domain and discussing the common objection.

Prevent potential damage to your brand.

By implementing an SPF record (sender policy framework) spammers and phishers are less likely to choose your domain as their ‘cover’ domain.

This is because spammers will seek domains which do not implement any form of email authentication as there is a higher chance they will get through spam filters at the receiving domain. Without SPF/email authentication when the remote mail server receives an email it will have no way to verify or check whether the email is likely to be genuine or not and will therefore have to rely on unreliable content filtering to establish whether the email is likely to be spam or not. Content filtering relies on patterns like words or phrases within the body or subject of the message and the receiving mail server will make a decision on whether your email is genuine purely on the content of your message. With SPF the spam filter at the remote mail server will be able to make a more informed decision as to whether the message is genuine or not.

If you do implement SPF however it is still possible that spammers will attempt to use your domain as their cover domain and attempt to deliver messages using it. When this happens the remote mail server will check against your SPF record and recognize that the source of the email it has just received is not authorised to send email for your domain and is likely to block or quarantine your message. This means that if your customers or potential customers are the target of the spam the vast majority of the emails will be blocked which will reduce the impact to any of your non internet savy client base. In addition to helping your own client base and protecting your own brand you are also allowing the wider internet community to cut down on the amount of spam on the internet by being able to pickup on more of it and block it.

SPF can also help you reduce the amount of users that blacklist your domain. Lets imagine a scenario when you have not implemented SPF, a spammer has successfully delivered email to many domains around the word seemingly coming from your domain/brand. In this instance there are likely to be many people around the world marking these emails as spam. Many spam filters use bayesian techniques for the spam filter to “auto learn”. This means that once a user has marked an email coming from your domain as spam, the spam filter will analyse  every part of the email, looking for phrases, html patterns, images and the from domain and email address. If the spammer is sending many different emails using your domain one of the few things that will remain consistent will be your domain name and the spam filter will pickup on this. If you then send the mail server that has auto learnt about your domain and the large amount of spam coming from it, a genuine email, the chances are it will be blocked. The chances of this happening will be much reduced if you had implemented SPF.

Spam filters don’t only negatively score emails which fail the SPF check. If an email from a domain passes an SPF check, it is quite feasible that the administrator of that spam filter has added a policy to positively score that email. This means that genuine email from your domain is MORE likely to be delivered to the inbox of the person you are sending an email to.

In Summary then the reasons for implementing SPF are :-

• You domain is less likely to be used by a spammer.
• If your domain is used by a spammer, the emails are far more likely to get blocked.
• You help reduce the amount of overall spam on the internet.
• Your domain is less likely to be blacklisted by bayesian spam filters.
• Genuine email is more likely to get through spam filters.

That being said however, the biggest objection to SPF is that many people think that it ‘breaks’ email forwarding. This is where the email is received by one mail server and then forwards it to another for a particular account or domain. Although this is true if the email server receiving the forwarded mail does inbound SPF checking there are several workarounds you can put in place.

One of most commonly used workarounds is to whitelist the mail server that forwards mail to your own. This means that it can fail the SPF check but you can still choose to allow the email through. The vast majority of mail servers allow whitelisting and this is normally very easy to implement. If you however can not support whitelisting there is another option. The other option is that instead of forwarding from the original domain that the email was sent from, you can instead “resend” the email, e.g take the contents of the original email, put it into a new email and send it from a domain which will not fail SPF checks.

I think you will agree that the benefits far outway the negatives, especially as only a small percentage of domains and users forward mail so no doubt you now want to know how you can go about implementing your SPF record on your domain(s). Email Manual will be posting a guide tomorrow on how to create your SPF record and within a couple of days we will have a guide on how you can implement the SPF record on both BIND and Microsoft DNS servers, why not subscribe via twitter or our RSS feed to be notified of when these guides are available.