February 11, 2013 11:08 AM
Attention start-ups: Big Data does not end with Hadoop
For many people, the phrase Big Data has become almost synonymous with Apache Hadoop - the open source framework originally developed by Yahoo.
But as organisations become increasingly familiar with the nuances of Big Data, it is starting to look like there will be a much greater appreciation for different approaches to managing Big Data in the years ahead, presenting an exciting opportunity for start-up companies working in this space.
Last month, analyst firm Ovum produced a report which claimed that although Hadoop garners much of the spotlight as a Big Data platform, alternatives from the likes of 10gen, IBM and Teradata, as well as newer companies such as Splunk, are attracting plenty of attention.
A separate survey of business and IT managers, commissioned by EMC revealed that big data analytics is one of the most likely targets for investment over the next 12 months, with 47 percent of business leaders citing big data as a disruptive technology in their industry.
Hadoop has revolutionised data warehousing by moving away from the idea of having a centralised database system. Instead, an application can run on a number of different servers simultaneously, and the results can then be reduced down to a single set of data.
This means that data processing is no longer restricted by the server CPUs, or the fact that the data in question does not fit nicely into tables. The infrastructure can scale to accommodate the amount of data that needs to be processed - whether that data is structured or unstructured.
However, Hadoop was built on the paradigm of legacy data warehouses, which assumed that organisations could wait hours or days for the answer to a business question and its distributors are still working to overcome this problem. It also requires the integration of several components to create a solution, which can make it quite costly.
Some companies are now looking at alternative methods of handling and processing their data, so that the same insights can be gained in real-time, and with very little investment on the hardware side.
One company attempting to do this is British start-up Logscape. Rather than having to pull all of the data into dedicated servers to do the data processing, Logscape agents are installed on the various end points where data already resides.
These agents can pick up the file system, detect all the changes in
line, and build a little warehouse in the machine - plus there is no need to transport the data, so the whole process is a lot faster.
If a company has
200 machines doing this they create what is essentially a
large distributed database, which can then run the background and
enable the machines to carry on with their primary tasks. There is no need to spend money on dedicated infrastucture as it's all already there.
Logscape is specifically designed for systems monitoring - as in the case of its customer Sportingbet, which uses the technology to improve application performance by monitoring the end points in real time. The emphasis is on providing quick feedback at low cost, which is something that Hadoop simply cannot deliver.
However, systems monitoring is just one of a myriad of scenarios where the primary aim is to gain real-time insight into a company's existing data with minimal investment, meaning that there are many more opportunities for start-ups looking to carve out a niche in the Big Data landscape.
Hadoop will of course continue to play an important role too, allowing customers to get affordable storage for heavy-duty data sets and offering solutions to some of the really big and complex data processing problems.
However, many companies have not yet found a use for those kinds of Big Data applications. A lot of them are simply looking for ways to gain data insights into the day-to-day running of their businesses, and simple solutions that can carry out these Big Data functions at minimal cost may well win out.
With strong indications that venture capitalists are increasingly turning their attention to investment in enterprise technology start-ups, entrepreneurs could do worse than finding new and intelligent ways to solve companies' Big Data problems.