Order of the Bath

There is growing concern among web site owners, their hosts and web marketing experts that AVG 8 is causing increased costs. The issue is LinkScanner and what it does to traffic. I have already commented that, for those users on limited bandwidth such as dial-up, it should be disabled and I have provided instructions on how to do this. But if it is also significantly affecting the other end of the internet—the web hosts—then AVG may be forced to modify it.

The way it works is that if you do a search using the major engines (at least Google, Yahoo and MSN Live) then you get a page of results, generally 10 at a time. AVG LinkScanner then steps in and visits every one of those results and checks the results for malware and sets a flag (Good, Doubtful or Bad) against each one to warn of potential problems.

The issues for users are:

• The increased bandwidth caused by the requests and results could have an impact on performance and possibly on any quotas you may have. This will be particularly true for dial-up users but could also affect capped broadband. On the other hand, users may judge that the benefit offsets the costs.
• Your logs and/or cache could show that you have visited sites that you had no intention of going to. This could have embarrassing or legal implications.
• This could also be reflected in any profiling that your ISP or the sites themselves are doing which could affect the advertising you receive (it could also be regarded as an asset as it may upset statistics gathered by Phorm type systems 🙂 ). A possible impact is that a site may think you have already seen a particular advert and not deliver it again—you never know, it may have been the offer you were waiting for.
• If the scanner itself were compromised then it is getting a lot of potential data to further infect your system.
• Because much malware is served via adverts, and adverts are rotated on every visit, the green tick may give you a false sense of security.

The issues for site owners and their friends are:

• They will see increased traffic, bandwidth which they have to pay for. Larger sites may need to deploy extra servers and connections to cope with the additional load.
• Sponsored results will also be visited and the agencies will charge the customer for each visit and it increases the apparent Click Through Rate with bogus visits. Update: Apparently AVG 8 goes direct to the raw URL and bypasses the Click Through detector so that the customer will not be charged. They will, however, still see the increased traffic.
• Ordinary pages that are funded by advertising appearing on them will see an apparent drop in Click Through Rate because the user never sees the ad to visit it.
• Web statistics become [even more] unreliable due to the increase in “bounces” i.e. visitors that come in from search and don’t go to any other pages.

At present the traffic is detectable for what it is, so concerned web owners can allow for it either in their analyses or even suppress responding to them. However, if that remains the case, then it will also be detectable by any malicious hosts or content to fool the scanner into returning a clean bill of health. It will be interesting watching the news in the next few weeks to see how this is resolved.

In the office today we had a spate of the Google “We’re Sorry” screen. We have no idea why but it was coming up on quite innocent single word searches. I had a search around and found a few pages of explanation but I think they have made a few mistakes with the error page. First of all, it doesn’t look like a Google page—the font for the logo is wrong—all the Google error pages are like this. Secondly, it is directing you to a less than perfect source of anti-virus and anti-spyware software. Although it is C|Net, there are some very dodgy downloads in there. But most of all, there is no clearly explained reason why the message appears. Admittedly it is better than earlier versions when it told you point-blank that you were infected but a list of possible reasons would be useful.

When it comes to the Captcha needed to continue working, I can’t read many of them.

These little things are a well kept secret of the browser world. You all know about Cookies? No, well I’ll first explain about them…Cookies are small pieces of data that are stored by your browser on behalf of the sites that you visit.

The problem discovered in the early days of web browsing is that it is a stateless process. Each request for data on a page is independent of every other. Although you know that your request for page 2 is related to your just having read page 1, the server at the other end sees it as an isolated call. If you are following a sequence, such as a process to purchase a book from Amazon, the server needs to know that the pages are all part of the same transaction. It does this by creating a cookie at your end which contains a unique identifier. This is sent along with each subsequent request so that the server can relate them all together without losing track.

There are two sorts of cookie—transient ones which are deleted as soon as the process is completed and longer term dated ones which carry forward information from one browser session to another. There is some security included which only allows a server to read the cookies that it created; this is done by domain name. A good example of a long term cookie is the one that holds your preferences for Google searches so it remembers which languages you prefer etc.

One use for cookies that have gained them a bad reputation is for advertising. The ad-server will store information about what ads it had sent you so it could ensure that you get different ones next time and perhaps also which ones you have clicked on so it can give you more of the same. These became known as tracking cookies, but it is not really as bad as it sounds; the security is still there and the only information that could be called personal is your network address. There is no suggestion that email addresses, personal names or other such things were disclosed, but by looking at the cookies on a user’s system you could get some idea of what sites they have been browsing. For more information see the Wikipedia article.

Due to their reputation, there is now a problem for companies that need to use them; up to 40% of people delete cookies on a regular basis. There is a built in feature in Firefox (and perhaps IE) to delete all cookies now or every time you shut down. As a result many advertising programs were not working properly.

Enter Macromedia (now Adobe) Flash. This system which operates on top of the standard web protocol is widely used by advertisers (and often disliked by users) because it allows animation and sound. It is also used by sites like YouTube to display short videos on demand and web designers to create really fancy (flashy!) sites. Flash has the capability to read and write cookies but it is cumbersome so they created their own (called Local Shared Objects). This was a good idea when they were used for the same purpose that cookes were designed for. But they are now being used as a backup to standard cookies because most people don’t know about them. If some sites spot that their standard cookie has been deleted, they will read the flash backup copy and immediately recreate the cookie, subverting the intention of the user.

Firefox extensions to the rescue—Objection. It is not very clever but does allow you to see the LSOs that have been created and delete them if needed. I am not suggesting that you get paranoid and delete everything in sight but you deserve to have control over your own browsing experience. Of course you could chose to block Flash altogether! I find animations distracting.

Update 14-Aug-2009: The new Firefox Private Browsing (sometimes know as porn mode) introduced in version 3.5 does nothing to stop the storage of or delete LSOs. Your private habits could easily be revealed by looking at what gets stored there. Also there is now a more comprehensive management. This is the Better Privacy plugin but be careful setting it up as it could affect sites that legitimately use LSOs (read the FAQ at the end). There is a management mechanism provided by Adobe which gives you some limited control over what is allowed. Not surprisingly, the options that you chose are themselves stored in an LSO for later retrieval by Flash.