It’s interesting to compare this to another definition of information, from Claude Shannon’s information theory. This is a mathematical theory of information that states that the information content of a datum is inversely proportional to its probability of occurance. Shannon (and most communications engineers after him) used this definition in closed data sets, where we know beforehand what data is likely and what data is improbable. For example if we were to design a system that efficiently encodes (“compresses”) English texts, the letter “E” contains low information content, because it occurs 13% of the time. The occurence of the letter “Z” is rare (0.08%), and therefore its actual occurence carries substantial information. This is the basis on which most modern compression and data correction systems are built.

But the question is whether such a formal definition of information breaks down when we talk about knowledge as information. In a sense, yes, because the subjective probability of a datum differs from person to person. High-information data would be data that an individual finds suprising or new; low-information data would be obvious or common-sense. However, Shannon’s probability-based definition could still be a useful way of thinking about subjective information content. From this point of view, information content is inversely proportional to the probability that data is already accommodated in an individual’s knowledge framework. Data that changes this framework, that needs to be assimilated in some way, has high information content. Interestingly, data with too high information content, in other words, data that differs too much from an individual’s framework, may be rejected outright, and no useful information can be conveyed.

So does this give us any indication on how to allow users to gather more information from the web, with less knowledge-brokering by other humans? I think so, yes: but it points towards two difficult problems.

The first one stems from the subjectivity of information, and can only be solved if our information services can model a user’s knowledge framework in some or other way. In other words, user profiling must develop beyond simple clustering of favourite books and movies to a deeper cognitive model. Humans are good at this — good teachers and lecturers are those people with a knack for developing “thought models” of their students, and then offer data that maximises the transfer of information. Software systems are still pretty bad at this (but one of the better ones was designed by a psychologist, not an engineer, see http://snipurl.com/49jzu).

The second difficult problem is to extract semantic information, meaning, from the data cluttering the web. If you want to know whether data would carry high information for a user, you need a knowledge representation, and a way of comparing the semantic content of a page or a document or an idea to a user’s knowledge profile. Semantic analysis still has a long way to go before this is possible.

But although the general problem is difficult to solve, I think an information-theory approach can be feasible already in limited-domain applications. For example, if a user’s knowledge framework kan be estimated by cues such as observed interests and observed interaction with existing data within a specific domain (e.g. a news site, or an online encyclopedia), and if a knowledge ontology can be created for that domain, and available data clustered within that ontology, it might be possible to build some very useful applications. The difference from many current systems is that we shouldn’t continue to feed a user with data already within his framework (many recommendation systems seem to hammer me continuously with slight permutations of data or products that I already have), but with data this differs enough from the existing framework to be interesting.