01 Mar 2001
Claude Shannon died on Tuesday, at 86. Shannon was the father of information theory, and a student of Vannevar Bush, a thinker whose work in the 1940s laid the foundations for what we know as the web. As part of his work in information theory, Shannon introduced the term ‘bit’. And by the end of his life, just a few decades later, we have a network transmitting multiple petabytes every day. From his 1948 paper A Mathematical Theory of Communication:
The ratio of the entropy of a source to the maximum value it could have while still restricted to the same
symbols will be called its relative entropy. This is the maximum compression possible when we encode into
the same alphabet. One minus the relative entropy is the redundancy. The redundancy of ordinary English,
not considering statistical structure over greater distances than about eight letters, is roughly 50%. This
means that when we write English half of what we write is determined by the structure of the language and
half is chosen freely.