[colug-432] Hadoop interest ?

[Angelo McComis]

Tom / all:

I was speaking to a friend of mine who works at Google, and he was
intimating how wonderfully awesome the Map Reduce / Hadoop stuff is. His
example was a computational job that would not be able to complete in his
lifetime on a single server can be distributed out to multiple nodes and
crunched and completed in minutes or hours, depending on how much capacity I
had to throw at the work.

To consider the direction of the IT industry as a whole, this is certainly
an interesting discussion to have -

- Companies are trying to do more cloud-like things, and a Hadoop elastic
cloud makes a lot of sense there, but getting that much data from onsite to
the cloud is a challenge. But, if the data set is that big, would you not
spend more $ on bandwidth transfer putting it to and getting from the cloud
than the GDP of some smaller countries?

- Doing an internal Hadoop architecture - certainly the way to go, but what
is the value of redesigning your data and processes to take advantage of
Hadoop when the investment has already been made in the larger, vertically
scaling hardware?

- Doing an internal Hadoop architecture that's based on an internal elastic
cloud (e.g. use capacity when needed, give it back when finished) makes
sense, but to the point of making the investment of taking existing data and
processes and converting it to the style needed to be able to distribute the
rows out to Hadoop, then it becomes problematic.

I guess in short, I get it, but I don't see where it makes sense yet, unless
you are Google, Amazon, or one of the other "Top 10" biggies out there.

Maybe this is where more public forum and discussion comes into play.

Interested in others' comments on what cloud is, and where it makes sense.


Angelo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.colug.net/pipermail/colug-432/attachments/20101215/2caa9d9e/attachment.html