October 29, 2014

Why after the Manager comes the Director: Cloudera grows Big Dataecosystem

Two weeks ago Cloudera announced a new product called Director. This product is aimed at orchestrating different CDH environments in one single console. More over it defines a tipping point in the maturity of Hadoop as an enterprise technology, and Big Data as a whole. In the announcement, Cloudera Director is clearly positioned as a Cloud product, but we need to step back and think what exactly does that mean for enterprises in different levels of maturity of Hadoop adoption.

When Cloudera first started, the Hadoop environment was quite clumsy and so Cloudera Manager became a fantastic tool to manage all the complexities of an Hadoop cluster, together with its many services. The tool was said to be one of the most flexible, responsive and easy to work with managing tools in the Hadoop market, and a clear differentiator for Cloudera.

The more pieces of the jigsaw puzzle Cloudera started to bring into play, like Impala, the more important Cloudera Manager became as a de-facto standard tool for managing the so called Enterprise Data Hub suite of Cloudera products. This is not a free product, in fact it was the crown jewel in terms of payed software developed by a company that bangs the open-source drum. Many organisations that wanted just to get started with Cloudera's Hadoop (CDH), were not paying for support let alone for the Cloudera Manager tool. Then came the big technology companies like Dell and Oracle, that eased the way in which CDH could be steered into a data center with Hadoop based Appliances. All of the sudden data center operators found themselves with a sprawl of CDH instances to manage. Cloudera Manager was not enough to orchestrate this anymore.

Although the apparent simplicity of such tools can point to a world where these environments seem easy to manage, the fact is that they hide a huge complexity, not just in the variety of services they manage, but also in the dependencies between these intricate environments. So when Cloudera hit a wall in terms of making it more simple to manage complexity, companies start realising that starting easy is not synonym of starting free or very cheap. Investments needed to be done and the Cloud became a question popping up in every board room.

So we have here two forces combined:

1) Growth pains (Private cloud needs)

2) Need to ease the adoption of non-free, non-open-source software that helps manage free open source software (Public cloud)

Hence the timing for Cloudera Director. It's free, but you'll need to apply it in a payed CDH world. Could have been born sooner, but not later.

The console presents a way in which operators can not just manage different CDH clusters, but also built-in cloud mechanisms like charge back and cloud storage backup.

No comments:

Post a Comment