Archive for the ‘email’ Category

TapCentral Spam Blocking

20 Oct 2006 12:08 by Rick

My hosting company are trialling a central spam blocking option on my domain. Actually it is a marketing exercise to get me to buy it but it was interesting enough to accept. It is a reputable system using Barracuda which incorporates a rule based algorithmic scoring system with user whitelisting and blacklisting plus, after a while, a Bayesian component. It is all web based and quite easy to use and seems to be effective.
The reason I won’t be continuing after the trial is that I now get as many “quarantine” messages from the system as there used to be spam and I have to read them. The reason is subtle and one that they didn’t think of.
One of the “features” of own-domain email is that you have the potential of a nearly infinite number of email addresses. The hosting people encouraged this by providing a “nobody” facility—a mechanism that sends any mail for which there isn’t a designated box or alias to a single address. I exploit this by giving every (organisation) that I deal with a unique address when I sign up; e.g. for mailing lists, payment accounts etc. That way I can tell if they forward it to “partners” or sell their address list. It has only happened to me once but was useful to prove the case.
A side effect of this is that if a spammer does a random name type of attack then I get it all in my inbox, not a huge amount but enough to notice. Unfortunately the new spam block system doesn’t recognise this so opens a quarantine management box for every new address it sees and sends me a summary and a password for it.
What it ought to do is activate the forwarding rules BEFORE processing the mail, then I would only get one box for the whole lot and it would start to get a significant amount of data for the Bayesian algorithm to kick in—but I can’t convince them of that so I will continue to process my own spam. Popfile is good enough and there is no bandwidth problem.

Tapemail delays

15 Aug 2006 09:24 by Rick

We have a problem that perhaps someone can help us with.
Emails from certain ISPs to west-penwith (hosted by DotEasy) take an extraordinary long time to arrive. The most noticable are mails from BlueYonder because we use it ourselves as do a lot of our friends locally. I have seen delays of anything from 20 to 25 hours. Studying the headers confirms that the delay is between the last Received: header in and the first at (after correctly compensating for timezone).
My question is “who is responsible?” Do I contact BlueYonder because they are hanging on to it in their queues or do I contact DotEasy for refusing to accept it in a timely manner?

TapHow email works (4) Viruses

27 Jul 2006 12:58 by Rick

Email viruses are another plague on the electronic utopia. To fulfil its definition a virus has to be able to replicate itself and pass on to another victim, preferably without manual intervention (else it is properly called a trojan, a subtle difference which can often be ignored). Because the email protocol is so simple, this is the easiest way for it to achieve its goal either via the native email client found on the machine or by a crude one built into the virus itself. As a result, once a machine has become infected it becomes the source for further infection. In the early days this involved attaching itself to legitimate outgoing mail and sending further infected mails to known addresses found on the source machine e.g. from the address book or a disk scan. Later this developed to the virus itself containing a database of potential targets which was shared out between victims in a cascade process. Ultimately it became more of a spam mechanism with the virus generating names semi-randomly using dictionary type techniques.

There are a number of places that viruses can be trapped and dealt with. All users are recommended to have a good anti-virus product installed. These are capable of scanning incoming and outgoing mail so attempting to deal with the symptoms as well as any potential infection. The better ones do this directly on the port drivers, between the applications and the outside world, so they can catch hidden email clients as well as the standard ones used by the owner. If your anti-virus tool reports outgoing infected mails then look to updating it and doing a major disk scan because you are probably infected.

Trapping infected emails on arrival is not really sufficient. Despite all the warnings, some people run without anti-virus software and can become infected increasing the problem. Also there is the cost involved with storing and distributing these emails. I said in part 3 that the mail servers only see the text stream and pass it though untouched. This is no longer true. Since the flood of malicious emails started there has grown, rather slowly in some places, a need to staunch it on the carriers rather than wait for the users. So now, most ISPs and company mail servers scan each email, decoding the MIME formats and checking each attachment before delivery. I believe that even some inter-network routers also do this.

This is a reasonably acceptable form of mail intercept—there are rarely false positive alerts with good mail being declared infected, and the better scanners just remove the attachment and pass the rest of the mail through with a note so that you know what has happened. The cruder ones strip off all attachments of certain types, such as .exe, which is less helpful and it becomes difficult to send legitimate files reliably. All sorts of subterfuge is used by good citizens to get their files though, the favourite being to change the file extension to something like .xex and give the recipient instructions how to restore it. The paranoid strip off all attachments reducing email to its basic text messaging form. This tends to only be short term at times of high risk.

TapHow email works (3) Attachments

14 Jul 2006 09:51 by Rick

The body of an email is a continuous stream of plain text lines, each no more than 80 characters long (remember punched cards?) and terminated by a line containing just a single dot “.” this doesn’t allow for much other than simple messages. To get round the restrictions and allow the sending of other types of data, uuencode (unix to unix encode) was borrowed from another system. This allows any data, including binary, to be transformed into plain printable characters, though you wouldn’t want to read them. A corresponding program uudecode converts it back to the original form without loss. The drawbacks to this are that the message increases in size and that the process is very manual. The first problem was addressed by compression but that made the second problem even worse as it is now a three stage process at each end i.e. compress, uuencode send—receive uudecode uncompress, and some data needed different handling, perhaps a different compress program.

The major development to resolve this was MIME (Multi-purpose Internet Mail Extension). Using this the body of the email is divided into a number of sections, still all in plain text and the whole lot terminated by the “.” The mail server knows nothing about it, it is all handled by the client at each end. Each section has its own sub-header which contains details about what sort of data the section contains and how it is encoded. Then when the mail arrives, it is all sorted out into the message (now perhaps formatted in HTML) and the attachments all with their correct type and name.

TapHow email works (2) Relays

12 Jul 2006 10:52 by Rick

What we saw in part 1 is the simple case with just a sending and receiving mail server, but quite often a number of intermediate hops are required to reach the ultimate destination. Using the “Received: ” headers in reverse order you can see what route a mail item took to get to you.

When email was invented, the networks weren’t as richly interconnected as they are now and consisted of small islands connected by narrow, unreliable pipes. So the technique to deliver a message was to forward it to another server which was a bit closer to the final destination. There was even a way you could suggest the route you wanted your mail to take. A system evolved where servers would discover the best routes by learned experience. This meant that servers became accustomed to accept any mail and forward it on as required.

In more recent times this custom was exploited as the servers didn’t care where the mail came from. Spammers arranged to send mail to random servers and specify obscure routes knowing that they would dutifully forward them on and their tracks would be hard to detect. This is called the “Open Relay Exploit.” To avoid this, servers began to be more selective and mostly they now only accept mail from their own clients or, alternatively, mail destined for places that they know about and are responsible for. They will no longer accept mail from anywhere to anywhere. In practice you will see from the headers that most mail gets moved around internally at the source end, makes a big leap to the destination, and then moves around a bit there before being stored.

An organisation called ORDB (Open Relay Database) tracks down the few remaining open relays and system managers can use this list to ban them from contact as their transmissions will be unreliable.

TapHow email works (1)

10 Jul 2006 16:18 by Rick

Consider an email that you send. It starts when you type it into a client program such as Thunderbird, Outlook Express or Hotmail. It then makes it way to your ISP. From there it goes to the recipients mail server where it is stored until they read it. Simple isn’t it. All except the last stage is controlled by SMTP; Simple Mail Transfer Protocol. Simple in this context is as in “Village Idiot” as we will see later.

An email is a plain text file consisting of some headers (consider this as the envelope) followed by the body which contains the letter. In theory no headers are required at all because you tell the courier who it is for but a destination line starting “To: ” followed by an email address is good. It would also be courteous if it had a “From: ” line and a “Subject: ” but they are not essential to the system. After the headers there is supposed to be a blank line before the body starts. At the end of the mail there will be a line containing a single “.” but you may not be able to see that when it arrives.

The email starts its journey going to your ISP mail server. It knows where that is because it is in your email client configuration. The conversation between the client and the server looks like this—the four letter codes are from the client, the server replies begin with a number.

220 Sendmail 8.6 ready at Mon, 10 Jul 2006 19:21:01
250 Hello goodclient [], pleased to meet you
        [yes, it really is this chatty—goodclient and the address are your client]
mail From:
250 ... Sender ok [note, it hasn't checked it really]
rcpt To:
250 ... Recipient ok
        [this gets minimum checking but it is important to get it right]
354 Enter mail, end with "." on a line by itself
Subject: greetings          [none of these are checked]

How are you today?
250 Jhub756hQ Message accepted for delivery [The random characters are an internal id]
        [now you can start again with a new "mail" or exit]
221 closing connection

For a bit of fun you don’t even need an email client, just connect to the mail server on port 25 using telnet and type the commands in by hand. It is rather tedious though and there is little room for error.

The first thing the mail server does is add a new header to the front “Received: from … by …” and perhaps a time stamp, recording where it came from; think of this as a postmark. It then uses the destination (from the “rcpt” command not the “To: ” header) to decide where the email should go next. It will look at the domain part of the address (after the “@”) and ask DNS (Domain Name Service) what the address of the mail server is—this is obtained from DNS “MX” (mail exchange) records. It then goes through the same chatter with the destination mail server.

When the email arrives, after adding its own “Received: ” header to the front, the destination mail server will check that it really does belong here and store it in the file, sometimes called the mail box corresponding to the user (the bit before the “@”) It will wait there for them to fetch it, usually using a different system called POP3 (Post Office Protocol v3). The POP3 server uses a similar set of 4 letter commands to list mail, retrieve it, delete it etc.

Apart from a bit of queuing to cope with delays, that is all a mail server does.

Note that during all this, almost all the headers are ignored. “From: ” is never used for instance and can say absolutely anything! “To: ” may sometimes be used if the message has to be queued but not otherwise. “Cc: ” (Carbon copy) headers are checked and processed as new destination addresses using the same process as above. “Bcc: ” (Blind copy) is stripped out very early before being processed like “Cc: ” so it remains invisible to everyone else.

TapEntertaining spam

10 Jun 2006 13:12 by Rick

I thought those two words were contradictory, except perhaps a good chortle at a particularly silly 419, but while checking my spam trap today I found this one. A spammer with a sense of humour? or a fortuitous harvesting of random text?

Vuja De: That feeling youve never been here before.

Woods Incomplete Maxims: Alls well that ends. A penny saved is a penny.

TapEmail dates

19 Mar 2006 17:26 by Rick

One of the oddities of Thunderbird, compared with other mail programs that I am used to, is that it time stamps the emails using the time they were (apparently) sent according to the clock on the sender’s system. This has some strange effects with mail from people with incorrect clocks, most noticable in spam. Whether it is because they are using hijacked systems to send the mail or it is some devious ploy I don’t know, but I have had mail in the last few weeks with dates from 1 Jan 1970 (just one) through 14 Mar 2000 up until 18 Mar 2007. The old ones go straight from the junk mail box to deleted, but the post-dated ones hang around at the front of the junk box for ages.

TapEmail Charging Revisited

11 Mar 2006 17:57 by Rick

Prompted by giafly’s comment on my earlier post, which pointed out that AOL had corrected/clarified/backtracked on their earlier statement, I have been taking another look at the issue.

There is a lot of comment around both for and against and, having sifted out the many trolls and flames, there is not much intelligent thinking around that I can see. What I have learned is that the principle of bulk email certification, for that is what this is, is not new. The major difference which has got people all heated is that the model is to charge per item rather than a flat fee. The remainder of the story only concerns AOL customers; that is that suitably certified mail, by a variety of agencies, will be presented as such in their inboxes. That is of no concern to the rest of us. The spectre that this heralds some sort of precedent for all mail, or that uncertified mail will become any less reliable than it is already, perhaps due to neglect of the spam filters, is just not plausible. AOL will continue to serve their customers in the best way that they can because their business depends on it.

TapThunderbird are go (just)

25 Feb 2006 16:51 by Rick

I have been meaning to change over from Outlook to Thunderbird email client for a while; mainly because I gave away my copy of Outlook to someone who needed it and I really ought to stop using it myself, but also because I have been inflicting Thunderbird on other people and ought to know a bit more about it.

My initial impressions are mixed; for a basic user it looks great, certainly a lot better than my last experience of Outlook Express (v5). It has all the things that you need and some more if you want to stretch out. For a user like me, who has been on Outlook for some years (even an old version) and has a complex and large email requirement, then it does seem to be lacking in some areas. Perhaps it is just because it is different and will take some getting used to, but I certainly don’t think that it quite lives up to some of the reviews that claim that it is fully featured. Perhaps if anyone knows a solution to any of these problems they could let me know, I wouldn’t be surprised if I just haven’t discovered how to configure it correctly. I have left out the weaknesses that I know to be fixed by extensions.

  • The structure of the mail files is good. Much better in a (standard) flat file format, one per mail folder (rather like OE) than a proprietary database like PST. However I am having trouble managing these folders: you get the choice, for each account, of creating a new top level structure, merging with an existing one or using the Global Inbox in “Local Folders.” There doesn’t seem to be a way of creating a new top level structure independent of the accounts. This may be possible by creating dummy accounts, but is a bit of a hack.
  • Also I can’t find any way to place these structures in a disk location of choice. I want to do this for two reasons—to get data for long term storage off my C: (software) drive and onto my data drive and also to segregate mail by project (which doesn’t match accounts) and to store that mail with the rest of the project data. This is said to be possible by setting the Local Directory for each account but I can’t find it.
  • I have imported all my live mail from Outlook which works ok but with some limitations. All the “follow-up flags” are lost even though both systems have the concept, and all mail comes in “Unread.” Having done this and marked Thunderbird as my default mail client, it doesn’t want to know any more. I can’t find any way to import my archives from detached Outlook mail files.
  • The spell check dictionary is not installed by default – I expected that downloading the British version of the client would have the British dictionary pre-installed. Although it installs like an extension, it has to be done from a login account with admin authority.

On the positive side, the real-time spell checker is great, the integrated spam filter, image and HTML controls and threaded views are good, though I understand they are in the latest version of Outlook too. I’ve not tried the RSS and News readers, multiple Address Books and Message Templates, but they look promising.

Some minor UI weaknesses which annoy more than break the application—

  • The Previous and Next button arrows go left and right but the mail is organised in a vertical list and so should be up and down.
  • There is no button to Resend a mail, you have to use the obscure “Edit as New” function which doesn’t have a button. Even after scouring all the extensions there are still some buttons I would like e.g. print preview.
  • Dragging mail folders around seems to create copies rather than move them. Sometimes using the standard “hold down the shift key” overrides this but not always (I think it fails when moving between top level structures.) Moved folders show as empty until selected.
  • The “Importance” flags don’t exist, but there is an Important Label which is not compatible. The “Follow-up” flags are limited to on or off – there is no “complete” option nor is any target date possible (perhaps because the calendar feature is not integrated as standard).
  • There are a number of different dates which you may want to use to manage mail—Date sent (I think this is the one that is displayed), Date downloaded and also Date received at server would be useful. A problem with using the Date sent is that it can be accidentally or deliberately wrong putting mail in unexpected positions in the list.
  • I don’t really need Drafts, Templates, Deleted and Sent folders cluttering up every top level folder but I don’t seem to be able to delete them.
  • If you switch on View Full Headers, there is no scroll bar and, if there are a lot, then the message itself drops right off the bottom to become inaccessible. This looks like a bug.
  • The mail item window has the subject too small and hidden away. I expect that it is changeable by an obscure option.
  • The address book is a bit limited. It has only provision for two email addresses per person. It could integrate with the main client more closely such as an optional search on the main toolbar and facilities to easily transfer addresses to new mail items made easier. I haven’t found the file yet so don’t know if it is possible to process it using other applications.
  • When reading a lot of mail at the start of the day, the junk mail occasionally gets flagged but not moved to the Junk folder. This has apparently been a known bug for some time. I think it is something to do with checking all accounts concurrently.
  • As it is using the same spell checker as Firefox (SpellBound extension) and it would be good if they used the same additional dictionary so I don’t have to teach all three separately about my unusual words.

^ Top