Information Overload

November 23, 2005

George Kingsley Zipf stated the Principle of Least Effort: Each individual will adopt a course of action that will involve the expenditure of the least average of his work. Hence, when faced with too much information, we tend to acquire the easiest to access, rather than the best information for the task at hand. It is like looking for one’s lost keys wherever the light is good, not where they may be more likely to be found.

Therefore, too much information can easily be a bad thing.

Zipf is known for Zipf’s Law, which describes the frequency of words in text as following a power law. The frequency of the n-th ranked word in the English language is found to be proportional to , where ‘a’ is the power coefficient. ‘a’ is known to be close to 1. In the same vein, if ‘n’ is used to denote increasingly relevant sources of information, the frequency of good information drops off quickly. Hence, we need to be careful to spend more time on choosing good sources of information rather than acquiring any and all information.

In fact, collecting the easy-to-get information means that we may in fact collect too much of it, and thus end up spending more time and money on it, than if we were more selective.

This almost suggests a “toe-in-the-water” search model in two parts:

(1) Get only as much information as is needed to evaluate the quality of the source and no more, and then drop it if found wanting.

(2) Set a higher than average threshold for quality, so as to avoid waste.

I wrote about the emotional cost of information overload in a previous blog. That cost is distinct from the one being discussed here, which is one related to bad decision-making, not self-negativity. But the relevant point still is that bad information has a three-fold cost in terms of (a) emotion, (b) time, and (c) money.

Bad quality information breeds even more bad information. This is because too often information is simply served up in another form. The student term paper has become a special case of this form of abuse of information. There is an interesting negative dynamic to easy access to information. In the past, when researching a term paper, much effort was involved in gathering information. So before doing so, we thought a lot about what exactly we wanted to achieve, and then only did we begin information gathering. This process usually resulted in the “first access” of information being of reasonably high quality. Nowadays, because there is easy access to way too much information, the process of thought before retrieval is often skipped, resulting in low quality “first access”. In general, information gathering tends to follow a stopping rule which is just based on stopping some time after first access.

Mostly, once we have gathered a certain a certain quantity of information, we tend to stop. Hence, the quality of our data will almost always be a function of the quality level at initial access.

So, what can we do to avoid these pitfalls of information overload? There are many things I like to try.

(1) Avoid using the browser unless you absolutely have to. Even if it is work-related, determine if it can wait. The days I collect my web-based activity (including email) into one chunk of time are often the days I get quality work done.

(2) Do as much theorizing as possible before reaching out for data. I often find that I decide against pursuing an idea, and for good reason – after thinking a lot, it simply is not worth it. If I had instead rushed out and collected data, then it would be time spent badly, and, the much bigger problem is that once I have collected data, then I am much more committed to the project (sunk cost fallacy) and throw even more good time at it.

(3) If I am not sure about some information, I try not to waste time reading further to examine it. I simply save it away on my hard drive with a brief descriptor. (I have an excellent mnemonic system on my personal server for saving files, giving descriptive file names, syncing it with my other machines, and accessing the data from many places. This is in addition to the usual software for indexing my drive on my Mac and Linux machines). This way, if I need it, I can get it, but for now, I have saved time and effort in dealing with information “on the cusp”. I was reading the book on Fischer Black (by Perry Mehrling), and was struck by how close my system is to Black’s. He was clearly already grappling with the problem years ago, and had a detailed collection of manila folders where he stashed all his information and ideas. Were he alive today, I am sure he would have figured out something eons more sophisticated than what I have.

(4) Jeff Hawkins had this idea of the brain in his book “On Intelligence”. He likens the brain not to a fast CPU but to a vast memory bank. The brain solves problems and makes judgments exceedingly fast because all it does is fast retrieval, through pattern recognition of stored life scenarios. Every iota of learning is stored, to be dormant until recalled. This is similar to stashing information for probable future use.

(5) Pull at information, dont allow it to be pushed at you. Ignore Amazon’s recommendations. Given, that software is pretty smart, and often does very well in anticipating your latent needs, but then, it all comes at a cost.

(6) Never spend more time reading web sites in a day than you spend reading books or newspapers. You know why already.

Most of all, do not worry about being on top of all the information being thrown at you. If you want to live life like that in a reactionary fashion, then you are done for already.

Information is the food of the mind – eat well and do not overeat.


Information Angst

November 20, 2005

There is altogether too much information in our lives today. In the past, people broadly chose one of two levels of broad information, by deciding to read or ignore the newspaper. Or listen to the radio or not. Information was never “pushed” at you – except for the rare occassions when newspaper boys ran around screaming “Extra, Extra!”. How much times have changed.

The big difference is that information flow is now continuous. World events are continuous by definition, but reporting used to be very discrete. You got the newspaper but once a day, or heard the news a couple of times on the radio. Now, with everything in “streaming” mode, reporting major events as they unfold makes for an incessant barrage of information, mostly unavoidable if our lives have a semblance of engagement with the world. So it has become harder to “get away from it all”.

Continuous information creates an insidious need to compare ourselves with everyone and everything, simply because we can. We want to know if we are richer, happier, healthier or just plain better-looking than some benchmark. Most often, in an effort to compare and feel better, we usually end up feeling worse. Maybe we would be better off avoiding making comparisons at all. But that is not our nature. We are competitive to a fault, and information overload translates swiftly into competitive angst.

Its automatic. We dont start out a session with our computers with the goal of doing anything else but channeling our curiosity. Sooner or later the engagement turns into one where bits of information that are self comparative start becoming salient, and drive the emotional course of this human computer interaction. Its inevitable, its hardwired. Its a bias we cannot rationalize away. I find that I am somewhat calmer and happier on the days when I eschew or minimize the interaction with the computer. Is it the same for you?

We have been brainwashed into believing that more information is better than less. It all sounds pretty rational – take a large superset of data or information, simply distill out the relevant parts to get a concentrate of pure valuable knowledge. We believe that this approach is more optimal relative to one in which we receive a smaller subset of possibly less relevant information. But, there are many ifs: (a) How good are we at distilling knowledge from large sets of data? (b) Do we get better or worse doing this when data sets increase relative to knowledge sets? (c) What about the cost of all this information processing? (d) And finally, if as a byproduct, the interaction results in heavy competitive angst, is it all really worth it?

I think we need to carefully evaluate how much information our lives should have. Information is necessary for sustenance of our modern lives, just as food is necessary for our bodies. But, just as we know that we need to be careful about eating in a balanced way, we want to be measured in our intake of information. Mental and emotional well-being is as critically related to our information diet, as balanced meals are to our physiques.

The hard problem is that we do know what a balanced diet is when speaking of food. We also know better to avoid some foods in excess. But with information, it is often hard to know its bad for you till after you have consumed it! So maybe we should just limit information intake more generally. That said, maybe you should not have read this?

On Dabbling

November 4, 2005

The world usually rewards people of perseverance, those who stick to one thing, master it, or are just lucky to find what they are good at, and then proceed to excel in it. I am, on the other hand, driven in the opposite direction – far be it for me to spend all my time perfecting one thing. No, I am a self-confessed dabbler.

For me, life is too short to spend it all on one thing. In fact, I find a day is too short to spend it on one thing. I tend to read about ten books at a time, all of them in various stages. Quite understandably, I have a huge collection of bookmarks!

My eating habits are also those of a dabbler. I rarely eat the usual numbers of square meals a day. My consumption function appears to be more of a random walk, nibble here and there, and sometimes a binge. No steady diet for me.

Living a well-structured life is detrimental to dabbling. People who plan their next day by thinking through it and making a list the previous day will make poor dabblers. A true dabbler will, on the other hand, wake up in the morning, and ask – “So, what should I do today, if anything?” I don’t mean that I may not teach a class if I am to do so, but its more that, outside of the few hours of regimen, life seems wide open to possibility. The core of the dabbler mentality lies in placing trust in random-walking through life.

Being a dabbler is pretty hard-wired into one’s psyche. You cannot take someone with good discipline and turn them easily into dabblers, nor should you. There is a purpose to everyone born with organized habits and routines. Books are written about these people, such “The Organized Life” and so on. But no book has been written about the benefits of the dabbler ethic, presumably because we have exercised social value judgments and agreed that the dabbler is the root source of much of what ails us. Dabblers have been labelled as people with ADD/ADHD, when what it really was is a heightened sense of curiosity, one that does not need satiation but just one that needs constant feeding. Dabblers are not people who lack concentration – dabblers are simply people who need to act on every random thought that comes along.

What good is a dabbler then? Dabblers see connections between things, which does not always happen when we are too focused on something to the exclusion of much else. Dabblers seem to have deeply internalized the wisdom that the entire world is closely connected, and by foraging widely, these connections will come to light.

Dabblers are more likely to be open-minded, as they are naturally receptive to alternative ways of looking at things. They are hard-wired to tune into hundreds of frequencies, easily switching between them as often as necessary.

Dabblers are less likely to want to protect their turf. This is an easy virtue, for a dabbler does not have a turf. The question of protecting it does not really arise. There is a smug satisfaction in claiming all the world as one’s own stage.

If you know you are a dabbler, make time for your gift. I take a day off every week to make sure I give my habit the time it deserves. It pays off handsomely on the days I do not goof off. I am fresh with ideas, drummed up in dabbling model.

Every now and then, life conspires to sorely test even the die-hard, seasoned dabbler. Life demandingly asks for undivided focus on a single task. And it is in these moments that the true dabbler shows his colors, by refusing to cave in and dabble on regardless.

Dabblers of the world unite! Oops, I forgot, dabblers are defined out from doing that!