It’s Not Information Overload. Its Filter Failure. This is very much a discussion piece, starting with some puzzle pieces which I am playing with. I still do not have the whole picture, but it is becoming clearer.

In one of the pieces, Clay Shirky recently gave a talk titled ‘It’s Not Information Overload. Its Filter Failure’ (Web 2.0 Expo NY September 2008) which really struck a chord with me. There are a lot of other interesting ideas in the talk which I am sure I will return to, but for now I would like to focus on what I believe to be the core premise contained in the title – that of filter failure. Filter failure postulates that it is not a dramatic increase in information that leads to the feeling of information overload, but the failure of ones filters. The necessity to retool our filters really resonated with me.  His conclusion includes the comment “we are to information overload as fish are to water its just what we swim in”.

This leads me to another piece. At a recent Commencement Address (a US institution that is similar to a graduation ceremony) the author David Foster Wallace ( Transcription of the 2005 Kenyon Commencement Address – May 21, 2005) told the following story:

“There are these two young fish swimming along and they happen to meet an older fish swimming the other way, who nods at them and says “Morning, boys. How’s the water?” And the two young fish swim on for a bit, and then eventually one of them looks over at the other and goes “What the hell is water?” the point of the story “is merely that the most obvious, important realities are often the ones that are hardest to see and talk about” and correctly that “we have to keep reminding ourselves over and over:

“This is water.”

“This is water.”"

Closer to home (for me anyway) another piece is from Arthur Goldstuck writing on News 24 where he writes:

“One unforeseen (by most) consequence of all this bandwidth is that available content will mushroom even more dramatically than it has in the past decade. The information glut will become an information slum, and it will seem impossible for individuals to find their way to the best or most useful content, or at least to find their way through the useless content that will litter the internet, and possibly even TV. ” (Arthur Goldstuck 08/10/2008 09:16  - (SA) )

And then some pieces even closer to home, right here in the SWAT Blog: I Want Sandy (and others like her) (Chester do Nascimento 10/13/08) and
The web is a knowledge repository…not! (Jacques van Niekerk 10/10/08) and the response to that written by by G-J.

All of the above discuss the idea that there is too much information, an overload or a glut, and that this overload is too unwieldy or erratic. That all this information is not necessarily relevant to the context at hand. The premise being that we need tools or filters to deal with it and here is where it gets interesting for me. I am a self-confessed media addict and I do believe it forms a natural environment for me to the extent that I am not even aware of it, unless something impinges or changes  – and this is why Clay Shirky’s talk struck the chord it did with me. Something in the media environment has indeed changed and I/we need to retool. I had stopped seeing the water but something is muddying it.  Even while remaining a media addict one does not need to be the frog on slow boil!

For years now there has been talk of autonomous agents but nobody seems yet to have developed the killer app.  Micro-formats and the Semantic Web are technologies with similar promise which also have not delivered on their potential – yet.

So lets us consider a unified theory of filters.  These filters must apply across a wide variety of activities from business to pleasure, across a wide variety of applications from email to web feeds and across a wide variety of media from text to video to music.  Filters must be usable in or on a wide variety of media from the normal web to the mobile web, to the in-car GPS system to the Wifi connected camera.

In the same way people are discussing an internet of things, are we moving to an internet of filters?

Lets us consider some variables.  On one hand we want this filtering done for us and on the other we are the only people who can do it for ourselves.  The act of filtering must not be intrusive and it must not be complex.  The filters must be scalable and adaptable to new applications.  How can we enlist the help of others as Wikipedia has done? We cannot own it all because we cannot do it all.  Room for plugins is essential, an API mandatory.

What price privacy? Do we require filters in and filters out?

There are already some very big fish in the water but they do not seem to be providing quite the thing. The biggest fish at the moment is Google and they are certainly not resting on their laurels. There are others such as Cuil, new kids on the block Twine, DeepDyve and many other variations on the basic theme. How significant the management of filters can be is highlighted by Europeana, the EU response to European concerns over Google.

As Arthur Goldstuck goes on to propose,

“The solution to this will be information mapping at a level, on a scale and with built-in intelligence such as we have never seen before. An entire industry sector will emerge around the concept of mapping information, with the likes of Google and Wikipedia eventually being seen as the forerunners, but not the owners, of that space.”

We must not blind ourselves to the fact that we are talking about more than search -what we have here is a very exciting opportunity.