TapThe Hidden Internet

A friend pointed out the recent article in the Guardian about the dark side of the internet. Whilst it is quite a good article it both only just skims the surface and rather heavily emphasizes the negative aspects as if it was all deliberate and malicious. Here I will provide some definitions and separate out the completely different aspects of the internet that cannot be seen. Much of this is in the original article but it is overshadowed by a little journalistic hyperbole. I have also added some reference links, something newspapers always leave out.

Dark Internet

This phrase is used to describe the hard to access parts of the internet using any protocol. This could be caused by router configuration or firewalls. MILNET is a notable deliberate example, kept separate for obvious reasons. The majority of corporate and domestic computer networks also fall into this category either by the use of deliberate managed firewalls or simple NAT based routers with limited port forwarding configured. Another proportion consists of machines that people have forgotten about and are still running but the connections are damaged in some way.

Deep Web

There are Web sites that are hard to find by conventional search. This could be because of dynamic content,
lack of incoming links, registration/login requirements, deliberate banning of search bots, scripted links and non-indexable media formats such as audio, video and images. This limits their presence in standard search engines so cannot be found by conventional means. It is considered to be many orders of magnitude larger than the surface web indexed by search engines. To a limited extent this has been addressed by human based searches such as StumbleUpon, manual submission to search engines like Open Directory and semi-automatic systems using things like Google Sitemap. That is assuming, of course, that the owners want them to be found. The majority of this information is proprietary information accessible only by companies and their customers and academic data used by researchers. Also there are old web sites, abandoned and unloved that simply don’t work any more and can’t be indexed. Think Geocities or Tripod. It may also contain a small amount of content that some may regard as illegal. The reason it is likely to be small is that it requires a more stable, permanent and sophisticated infrastructure than underground organisations can sustain and, in any case, the external port is publicly visible.

Darknet

This is where the author started. They are network connections using protocols other than HTTP, usually for private P2P file sharing but can also support forum and chat type connections. The best known are Waste (by Nullsoft) [now withdrawn] and Freenet and form a very small proportion of the internet. These networks are often hosted on what would normally be regarded as personal desktop machines rather than centralised servers and form a rapidly and dynamically changing mesh as machines come and go. In this respect it is similar to other P2P systems but no files exist in one place in their entirety, they are fragmented and scattered. The services are founded on principles of free speech which may be beyond what some people would regard as legal or moral. There is no central organisation and therefore no censorship of any kind. A proportion of the material on the darknet would be regarded as illegal by some governments. This ranges from dissident material placed and accessed by citizens of repressive regimes, through music and video file sharing in contravention of copyright up to illegal pornographic material and terrorist aids. It should remembered that free speech facilities are used to do good as well as evil (put those two in whichever order pleases you) and it should be noted that Google is one of the largesst sponsors of Freenet.

The latter systems should not be confused with archaic protocol networks from the early days of computing which are still running. These include Fidonet, Compuserve, IRC and Usenet (sometimes called Newsgroups). They require specialised software to access them but are only dark because they have dropped out of mainstream usage. One could say that modern but proprietary protocols such as AIM, MSN & Skype fit this category but they are well known and don’t generally hold permanent content.

Comments are closed.

^ Top