Order of the Bath

Now that it is available, upgrading from AVG 7.5 to AVG 8 is a logical step but there are some decision points to be made along the way so it is best to be prepared for them.

[Note that the Free edition has some quite rigid conditions about home use only.] First you have to find it. The link I gave before is still good but it is a few clicks of Grisoft determinedly trying to get you to buy the full suite. Some of the links on the way are a bit misleading. One says that AVG Anti-Spyware is being discontinued but others that it is now included with the Anti-Virus package. The eventual download location is either their own site or C|Net downloads.com.

When you come to install it there is no need to un-install the previous version. You will need to login to an admin account. Leaving a lot out, the sequence of events is:—

• Standard or Custom install—you will need custom if you don’t need the email scanner.
• For the Custom install, Un-tick the email scanner if you don’t want it.
• Un-tick the AVG Security Toolbar if you don’t want it. Everyone seems to want you to get one of those and if you loaded them all you wouldn’t have enough window left to browse in.
• Un-tick the “Enable Daily Scanning” box if you don’t want it. I find that it is a long process and very heavy on resources (though they have put in some sort of load-limiter now). I would rather do them when I want to—and certainly not daily.
• There is a tick box for informing AVG about potentially dangerous web sites that you come across. I haven’t checked the privacy statement for this yet so I would be cautious.
• Definitely SKIP the updates at the moment as the install is not really ready for them.
• Skip the registration for the time being.
• Now you will need to reboot (it prompts you).
• When it comes back the System Tray icon will probably be red. Right click to open the AVG User Interface.
• Click Update Now and it should go ahead and do it.

That is the install complete but you need to check one other thing. One of the features of AVG 8 is the AVG Search Shield, sometimes called the Link Scanner. This intercepts results from the search engines (Google etc.) and inspects them for malicious content—try it and see the little green icons after every hit. Quite how it does that I am not sure but it seemed to take a log time and have a lot of internet traffic. I would imagine that on a dial-up connection it would be impossible. The search engines themselves do some quality checking, if this is doing it real time then it would be better but at what cost. The other thing that bothers me about this is that it could be that you are automatically visiting sites that you wouldn’t otherwise touch with a barge pole (porn etc.) and it will leave the evidence of this in your cache even if it never displays it.

If you decide that you don’t want this facility there are two ways to switch it off. You can use the AVG interface, but if you switch it off there it will forever say that AVG is not fully functional. The other way is with the browser controls. It works using a browser plugin (both IE7 and Firefix, I don’t know about Opera or Safari) and these can be disabled. Go to Tools —>Manage Add-ons—>Enable or Disable Add-ons in IE7 or Tools —>Add-ons in Firefox. This will need to be done on EACH ACCOUNT on your computer.

Now you can register at leisure, if you can figure out how. I haven’t yet! It is worth remembering that, despite all my griping, this is still a free service for which we are grateful.

Update: 20 Jun. As far as I can tell, the Firefox plugin which drives LinkScanner is not Firefox 3 compatible. It will be interesting to see how they update it.

A new anti spam solution by Abaca aimed at ISPs and large corporations has what they claim is a unique and effective method of reducing errors, particularly false positives (that is marking perfectly good mail as spam in error). What they say is that, in addition to well known detection techniques, their ReceiverNet box looks at the reputation of the recipient to assess the likelihood that the mail is spam. The theory is that if you are promiscuous (or unlucky) with your address and you get a lot of span anyway, then there is a greater chance that this new one coming in is also spam. If, however, you normally receive very little rubbish then the chances are that this slightly suspicious one is ok.

I suspect that there is also a bit of psychology here as well. If you get a lot of spam then you are much less likely to complain if the odd mistake is made. Personally, I would be happier if these bulk filters had a coarse mesh, only trapping the obvious spam and viruses. This would be sufficient for their purposes of reducing network and server load. Leave it to me to fine tune it with a Bayesian algorithm which can learn the sort of mail that I receive and want.

As you may guess from the title, I have made the decision on how to switch away from Windows. It is not that Ubuntu had any problems but, having discovered that I would need to get new hardware anyway, the Mac PRO seemed to be the most reliable option at the best price. Yes, I spent more than I intended but they have jolly good salesmen who know their stuff down at Western Computer.

I have also scaled back the virtualisation plans to just a single XP guest running under VMware Fusion but I will talk more about that in a future post.

In general, installing software on the Mac is just as easy as on a PC; in many cases easier if you do the PC properly with a separate admin account—you don’t need to on OS X because of the Sudo based “run as admin” facility which really works. But some of the more obscure open source packages don’t have proper installers, probably due to low demand and/or lack of resources. One of these is the POPfile email classification (anti-spam) system. It works fine but is a pain to install with a dozen pre-req bits and pieces, some of which need to be compiled from scratch and even the documentation is a bit out of date. Having now done it, I will make my contribution to the project and publish an install script. Not a full blown installer, because I don’t know how to do that, but a scripted version of all the little bits that have to be done which will make it much simpler … [more]

This idea being promoted by MAAWG looks like it could be an effective way of limiting spam at source, and, as the members are high powered, it could actually get implemented.

The problem is that a large proportion of spam and associated phishing, viruses and other attacks are sent, not from huge malicious systems in a far off place, but many thousands of small home systems each adding their little bit to the flood and under common malicious control. They were infected by a previous attack and then join in themselves—these are called zombie systems and are collectively known as a bot-net.

The principle of this proposal is for ISP’s to identify customers on their own networks who are infected. Nothing new there except that they currently don’t do it because of the administrative overhead it would trigger. The difference is that once identified, the customer would have all their internet traffic automatically routed to a sanitised area called the Walled Garden within the local domain and that all browser requests result in a link to an internal site which provides education and disinfection tools. Until the customer systems are cleaned no traffic is permitted out onto the wider internet. Think of it as a quarantine with a pharmacy on hand for self treatment. The reasoning is that the majority of customers with infected systems are unaware of it and wouldn’t know what to do if they were told. This way they don’t have a choice.

There will still be some admin overhead—in calls to the help desk—and it would need to start easy to minimise false positive alarms, but it is probably the only way to force these infected zombie systems off the network.

As I said, there are some heavyweight people on this working group, AOL, AT&T, France Telecom (Orange) but not my ISP. But when(if?) the momentum gets under way, no ISP is going to be able to ignore it and stay in business.

There is currently a problem that Enigmail, the OpenPGP extension for Thunderbird doesn’t work with Gpg4Win. The latter is the GUI version of GnuPG for Windows. The versions tested were Enigmail 0.95.1 and Gpg4Win 1.1.0 but I understand other versions are affected.

There seems to be some dispute as to which program is at fault and the most seen recommendation is to un-install Gpg4Win and install the plain command line version of GnuPG. Although there is some overlap, both provide a key management GUI for instance, this would lose some of the useful disk management functions of Gpg4Win.

I have discovered that there is a much easier fix. In Thunderbird, go to the OpenPGP ==> Preferences menu item and in the “Files and Directories” window, tick Override and enter C:\Program Files\GNU\GnuPG\gpg.exe. Now stop and restart Thunderbird and every thing works just fine.

I had a bit of a fright this morning; AVG (free) kept saying that it had found an infected object but it wouldn’t put it in the Virus Vault where it should go. I was bothered because I don’t do viruses, I consider myself too smart for that (lookout, the sky is falling in). I see a few go past in email and I used to have trouble when my anti-spam system kept a copy of recent emails in plain text (it now keeps them in a database, so that is resolved). I have just installed a trial of Prevx so wondered if that may have triggered something but I don’t think so.

Some analysis and a few blunders later I discovered.

• The infected file was in C:\System Volume Information\_restore{DF9 …a lot of hex… F08}\RP108\A0024948.exe. If I remember rightly this is the System Restore area. I don’t recognise the file name, perhaps System Restore mangles them?
• This accounts for why my working (LUA) account could not vault it, because I don’t have access.
• It is reported as I-Worm/Stration.DJC. This is normally distributed by ICQ (which I don’t use) but has been seen recently in spam email—I am unlikely to have executed any attachments.

The blunder was that (in a panic) I deleted the system restore area before scanning the system; I seem to drop out of Security Analyst mode when I come home. Anyway I did a full system scan and a run of the Kaspersky Online Scanner for good measure. Nothing else was found.

What I don’t understand is

• How it got there. I thought System Restore was backing up things that changed during an install so that you could back them out later. If that is the case, it should have been live on my system before whatever install replaced it and there should be some other traces left.
• Why AVG should have been looking there in Resident Shield mode anyway. I thought it only checked files that you accessed, and that is not likely to be one of them.

It will, no doubt, remain a mystery.

One of the leading commercial anti-spam companies, Trend-Micro, who run the MAPS system compile a weekly rogues gallery of ISPs and the amount of spam generated from their networks.

Blueyonder (our ISP) is number 42 having generated 71.4M spam emails last week in the last 24 hours.

No wonder we keep getting blocked.

Mind you, that is nowhere near as bad as BT (number 24) or Orange (at number 8 with over 300M spam). These figures are not factored by the size of the customer base, but that is probably right because the more customers, the more collective pain they experience when they are blocked. Smaller ISPs don’t appear explicitly if they buy their network from larger suppliers.

I have commented before on the stupidity of blacklists, but at least this one is contactable and apparently accountable. Why should the misdemeanours of a few customers impact the whole community?

A while ago, when this web site was really getting going, I discovered the need for a mail form to help people communicate back. At the time I was having problems with random senders being blocked and this seemed the easiest way to do it. I found what looked to be a good system in Jack’s Formmail.php v5.0!. I went through it carefully, pulled out parts which uploaded files to the server which I thought were dangerous, and used it. This has been running for a couple of years—until today.

Today I got a heap of bounced emails into an account that is not normally used much and looking at a few, it was clear that the originals had been generated by my modified script; I had signed them.

The content of a form generated email from my script is as follows, much the same as the original in fact:

 1. To: [recipient]
 2. Subject: [subject]
 3. MIME-Version: 1.0
 4. From: “[realname]” <[email]>
 5. Reply-To: [email]
 6. X-Mailer: DT Formmail5.0_RJP_2
 7. Content-Type: multipart/mixed;
 8.         boundary=”—-=_OuterBoundary_000″
 9. This is a multi-part message in MIME format.
10.
11. ——=_OuterBoundary_000
12. Content-Type: multipart/alternative;
13.         boundary=”—-=_InnerBoundery_001″
14.
15.
16. ——=_InnerBoundery_001
17. Content-Type: text/plain;
18.         charset=”iso-8859-1″
19. Content-Transfer-Encoding: quoted-printable
20.
21. realname: [realname]
22. email: [email]
23. message: [message]
24.
25. Message sent by formail.php v5.0_RJP_2 from [HTTP_REFERER]
26.
27.
28. ——=_InnerBoundery_001–
29.
30. ——=_OuterBoundary_000–

Lines 1 & 2 are generated by the PHP mail() routine, lines 4 & 5 are generated by the script, lines 21-23 are obtained from the input form and the rest is pretty much fixed. I think my web host inserts Return-Path; [account email address] after line 8 and adds

Received: (from [account]@localhost)
by west-penwith.org.uk (8.12.11/8.12.11/Submit) id [id];
[datestamp]
Date: [datestamp]
Message-Id: [id]

to the front before the mail leaves home. [recipient] is fixed to my email address and coded in the script so that it can’t be harvested, [subject] is, I presume, sanitised by mail(). [email], the sender’s email address, is checked using a regexp and malformed ones rejected. [message] is not checked, but is protected by being inside a MIME type text/plain part.

Have you seen the flaw yet?

Looking at the bounced messages kindly provided by Yahoo! where they quoted the incoming message in full there were some strange additions. There was a big gap between lines 4 and 5 containing the spammy message. Also lines 21-23 were in a different order and there was a lot of additional text before line 25. This consisted of the same spammy message and a very long "bcc:" list. What they had done was inject

[bogus email address at west-penwith.org.uk]
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain
Subject: [spammy subject]
bcc: [lots of email addresses]

[Spammy content]

..

as the value for my variable [realname] and I hadn’t validated it. The insertion into the content of the message was benign, but in the header, it was taken and interpreted as written which is why all those bcc addresses were sent rubbish. As a last minute alteration, I thought it would be nice to have the mail come from a real person rather than just an email address and didn’t think of the consequences.

Verdict: Guilty as charged.

What I am curious about is a) why the mail stream wasn’t terminated by the “..” before line 5 as it is supposed to be and b) how they discovered the flaw. Unfortunately I haven’t been receiving mail from this form for a few weeks (this may be related) so I would have missed any test runs. Do they have a bot which picks up the variables used by a form and try injecting rubbish into them to see what happens, or has a human cracker been on the job?

I should point out that the flaw that was exploited was not in the original script, it was caused by an alteration I had made. The [subject] may also be vulnerable in a similar way, though it would depend on how mail() interprets it, the manual is vague on this. There are a lot of other things which even the original script doesn’t check so perhaps I should look around for a better one.

I know that the Add-ons for Firefox are created by volunteers but the organisation does itself no favours by allowing a fundamental extension go out of date. In most cases there is nothing wrong with the extension, just that the install package is out of date specifying a maximum version older than the current one. This has happened with the British English Dictionary so I have made available a hacked copy here.

British English Dictionary 1.19.99 (Click to install, right click to download.)

Sometimes you get a spam message that is so funny you can’t help laughing, mostly at their incompetence. Take this one which slipped through my filter; it looks like it is a multiple choice randomiser to generate basically the same message but each one with slightly different wording. At least, it would if it wasn’t for the fact that they have sent the source code not the generated text.

<”Hello”|”Hi”|”Hi there”|”Good day”>

I <”hope”|”sincerely hope”|”wish”> this message finds you in a great <”spirit”|”mood”>. <”For a start”|”First”|”First of all”> <”I would”|”I’d”> like to <”congratulate”|”welcome”> you on this <”offer”|”opportunity”> because our <”association”|”company”|”corporation”> just got your contact and your <”brief”|”short”> profile through an <”email”|”web”> listing affiliated with <”the UK Chamber of Commerce”|”Monster”|”Careerbuilder”|”Yahoo Jobs”|”Google Jobs”|”HotJobs”>
<”I would”|”I’d”> be <”very”|”extremely”|”highly”> interested in <”offering”|”giving”> you a <”work at home”|”great”|”flex-time”|”part-time”> <”job”|”career”> in which you <”could”|”can”|”would”> <”earn”|”get”> an extra income <”of about”|”nearly”|”up to”|”starting from”> J<”2000″|”3000″|”4000″> <”per month”|”monthly”>.

This work <”does not”|”doesn’t”|”will not”|”won’t”> affect your <”present”|”current”> <”job”|”career”> and this is a <”very “|”">limited offer in which I <”will”|”would”|”would really”> require your immediate response. I <”will be hoping”|”really hope”> to hear from you soon, since <”its”|”it’s”|”it is”|”this is”> a job that <”can”|”will”> enable you to <”work from home”|”work part-time”|”enjoy an easy work”|”work at home”>. You will also <”stand the chances”|”have a chance”|”be given the opportunity”> of being a part of our future and <”excellent”|”winning”|”our”> team in which <”you will”|”you’ll”> be highly <”appreciated”|”respected”>.
Please fill out our <”application”|”appointment”> form, <”no fees asked”|”no money upfront”>, just your <”name and a phone number”|”basic contact details”>:
[web address removed]

<”Your application”|”Your enquiry”> will be <”processed”|”answered”> <”as soon as possible”|”ASAP”>.

<”Thank you”|”Thanks”|”Have a nice day”|”Best wishes”|”Take care”|”Bye”>.

Soon afterwards I got the real thing!