In light of the recent terrorist attacks I’ve felt more than a little underwhelmed with our government’s ability to deal with databases full of information. As it turns out Abdulmutallab was in the TIDE database but nobody bothered to check the database until after the plane was in the air. Hello people. This is not a difficult software problem:

SELECT COUNT(*) FROM TIDE..BadDude WHERE LastName = 'Abdulmutallab';

Unfortunately it’s not always quite so simple. The TIDE database feeds a number of other downstream databases which are used to create no fly lists, etc. I suspect despite spending billions of dollars on federal IT projects these types of data marts aren’t being used to their full potential. In addition, we are collecting data at an enormous rate and the government is not prepared to deal with it.

Just look at this story, apparently we’ve captured 24 years worth of video data from drones flying over Afghanistan in 2008. A bunch of smart people sit in a room and analyze it in real-time, and then it gets stored in a massive archive for some undetermined purpose. What if we could retroactively analyze the data and looked for patterns in Al-Qaeda’s movements? But then the fundamental problem is the government is not good at software projects. They end up hiring contractors to build these systems, frittering away billions of dollars, and the resulting systems still don’t work properly.

How about we let Google have a crack at the data? They’re by far the most capable group of engineers on the planet, and if protecting our national safety is the highest priority, wouldn’t you want the best people working on the problem? Let’s borrow a few of their brightest people, load all this data into BigTable and see if they can do better!