Here is a term that you may not have heard very often and that it will be downplayed in the future: Big Data. What is Big Data? Glad you asked because you should know about one of the biggest tools that will be used to oppress you. Big Data is simply a handy term making reference to ultra-large repositories of data or databases in the hands of a "selected" few organizations.
A LITTLE BIT OF HAZY HISTORY
Governments have been collecting peoples' data since written language was invented. We can go back to Babylon or Ancient Egypt -3000 BC- or about 5000+ years back in time to find written records of… tax events… bureaucratic necessities and rulers' edicts among other things. The problem was always the same. In order to remain a ruler (with all the privileges it entails), people must be controlled. But a control system is only as good as its memory and human memory is fallible; hence the need to write things down. This is how the collection of personal information begun, as the necessity of a more efficient ruling (i.e. control) system.
However and over time those tireless bureaucrats discovered that they had a problem. There is such thing as too much information, particularly if you don't have a central repository and you don't have an efficient classification system. Hence top bureaucrats added lower-level bureaucrats who kept occupied by taking care of the records and delivered them to their bosses on demand.
Unfortunately (for bureaucrats) but fortunately for us, population growth took care of this problem. With few exceptions (such as Germany under Hitler) it was extremely difficult to keep records of masses of people. There simply wasn't enough bureaucratic manual labour to keep all the records up to date and available. This created the enormous benefit that for as long as you did not make yourself a royal pain in the butt to governments, they would leave you alone for lack of resources. For any intent and purpose you were invisible if you choose to be so.
A LITTLE BIT OF HISTORY
On or around the beginning of the industrial revolution machines began to be used to store and tabulate information on an ever larger scale. The real information revolution began with the birth of companies such as IBM in or around 1880's. This revolution continued unabated and has produced the one invention our civilization absolutely cannot do without: the computer.
The real computer revolution began around 1950's when mainframes' drop in price was sufficient as to make them accessible to corporations. The next big step occurred with the development of the personal computer around mid-60's and the rest is recent history. Computers have been improved one way or another with every step, but the two key elements that concern us are the CPU processing speed and data storage capacity.
RECENTLY
Up to recently or about 1980's there weren't too many practical (and cheap enough) technical ways of store information. This was so because hard disk drives (or HDD's) are expensive since mechanically, electrically and chemically they are difficulty to build in order to be reliable. Consider that even today corporations have the tendency to backup their systems on magnetic tape! In addition, up to about the 1980's the world was more or less analogue, not digital. However with the advent of advanced personal computers and the internet, everything could, was and is being digitized. Today the world in which we live is primarily digital. Old records may still be analogue, but they have been deprecated except for historians. And this had massive consequences.
DATA HARVESTING
The improvement of CPU processing, data storage and network speeds means that digital messages can now be intercepted and stored with ease (by the government) for a low-low price. It also means that few large corporations who are in the business of "collecting" data can do the same with "business" information. This process lead to our current reality. Massive amounts of our data are stored in massive vaults belonging to a few companies or few governments around the world.
How much is "massive"? We are talking about several zebibytes of information or roughly a number one followed by 21 zeros. If you consider that the world-wide wealth in play in derivatives is "only" 1.5 Quadrillion USD (or 1.5 followed by 15 zeroes), you get the idea.
THE USUAL SUSPECTS
Big Data is quite secretive. Nobody knows for sure who has what data, but few of the usual suspects always float to the top of the list. Some of they (in no particular order) are:
- IBM
- HP
- Teradata
- US Government
- UK Government
- Russian Government
- Oracle
- SAP
- EMC
- Amazon
- Microsoft
- VMWare
- Cloudera
THE PARADIGM SHIFT
For as long as people have been collating data, there were people willing to buy it. This is not new. Most of the companies listed above make very nice profits selling their data, primarily to governments. Up to recently this wasn't a problem because when you have databases of the size of zebibytes, there is no computer system capable of analyzing it and coming up with usable information. Very few (government) organizations -if any- approach such capability (e.g. NSA in US or the KGB/FSB in Russia). Basically, unless you were a "person of interest" nobody really had the capability to use your data in their possession on a timely manner against you. For any intent and purpose, you were invisible and this was good.
But things are changing and they are changing rapidly. We are at the onset of yet another informational revolution. The Big Data revolution. If you pay attention and you keep your digital ears to the ground, you will notice that - quite quietly- there is a steady upsurge of job offers asking for programmers specializing in Big Data. Furthermore, you will notice that this upsurge is driven to a large degree by government. Furthermore and as we mentioned before, government are by far the most assiduous purchasers of Big Data. Do you begin to see the problem now?
Governments are harvesting, collecting, purchasing, acquiring, and obtaining digital information at a breathtaking pace. Furthermore as all information has now the tendency to be digital, this means any and all information anywhere. Government thirst for data is insatiable (see for example They Want Everything). Basically, if it is digital information sooner or later it will become accessible to your government. And now, governments are bridging the last divide: the capacity to extract timely, targeted, useful and actionable information about you from Big Data. That's right. You are the target.
Note: please see the Glossary if you are unfamiliar with certain words.
