Thursday, September 23, 2010

Yahoo demonstrates the awesome power of Hadoop

The recent revelation by Yahoo that they had found the two quadrillionth bit of pi when expressed in binary notation must certainly rank among the most useless calculations ever. That they consumed 1,000 computers for 27 days to deliver the news that this bit is 0 is even more ridiculous. With exactly zero compute time (and zero power consumption), I could have had a 50% chance of getting it right. In fact, I'm not sure I have any greater confidence that the answer is 0 after Yahoo's work because one small bug in that code could easily cause the answer to flip. Finding ever more digits of pi has been a test of cleverness and computer speed for a long time and is almost equally useless but such a struggle to know the two quadrillionth bit without knowing any of the other bits boggles the mind.

[ Yes, I know this is just a demonstration of Hadoop but those of us who care didn't need to be beat over the head about the power of parallelization. ]

No comments:

Post a Comment