Follow Us

Incubator

Attention start-ups: Big Data does not end with Hadoop



For many people, the phrase Big Data has become almost synonymous with Apache Hadoop - the open source framework originally developed by Yahoo.


But as organisations become increasingly familiar with the nuances of Big Data, it is starting to look like there will be a much greater appreciation for different approaches to managing Big Data in the years ahead, presenting an exciting opportunity for start-up companies working in this space.


Last month, analyst firm Ovum produced a report which claimed that although Hadoop garners much of the spotlight as a Big Data platform, alternatives from the likes of 10gen, IBM and Teradata, as well as newer companies such as Splunk, are attracting plenty of attention.


A separate survey of business and IT managers, commissioned by EMC revealed that big data analytics is one of the most likely targets for investment over the next 12 months, with 47 percent of business leaders citing big data as a disruptive technology in their industry.


Hadoop has revolutionised data warehousing by moving away from the idea of having a centralised database system. Instead, an application can run on a number of different servers simultaneously, and the results can then be reduced down to a single set of data.


This means that data processing is no longer restricted by the server CPUs, or the fact that the data in question does not fit nicely into tables. The infrastructure can scale to accommodate the amount of data that needs to be processed - whether that data is structured or unstructured.


However, Hadoop was built on the paradigm of legacy data warehouses, which assumed that organisations could wait hours or days for the answer to a business question and its distributors are still working to overcome this problem. It also requires the integration of several components to create a solution, which can make it quite costly.


Some companies are now looking at alternative methods of handling and processing their data, so that the same insights can be gained in real-time, and with very little investment on the hardware side.


One company attempting to do this is British start-up Logscape. Rather than having to pull all of the data into dedicated servers to do the data processing, Logscape agents are installed on the various end points where data already resides.


These agents can pick up the file system, detect all the changes in line, and build a little warehouse in the machine - plus there is no need to transport the data, so the whole process is a lot faster.


If a company has 200 machines doing this they create what is essentially a large distributed database, which can then run the background and enable the machines to carry on with their primary tasks. There is no need to spend money on dedicated infrastucture as it's all already there.

Logscape is specifically designed for systems monitoring - as in the case of its customer Sportingbet, which uses the technology to improve application performance by monitoring the end points in real time. The emphasis is on providing quick feedback at low cost, which is something that Hadoop simply cannot deliver.


However, systems monitoring is just one of a myriad of scenarios where the primary aim is to gain real-time insight into a company's existing data with minimal investment, meaning that there are many more opportunities for start-ups looking to carve out a niche in the Big Data landscape.


Hadoop will of course continue to play an important role too, allowing customers to get affordable storage for heavy-duty data sets and offering solutions to some of the really big and complex data processing problems.


However, many companies have not yet found a use for those kinds of Big Data applications. A lot of them are simply looking for ways to gain data insights into the day-to-day running of their businesses, and simple solutions that can carry out these Big Data functions at minimal cost may well win out.


With strong indications that venture capitalists are increasingly turning their attention to investment in enterprise technology start-ups, entrepreneurs could do worse than finding new and intelligent ways to solve companies' Big Data problems.

Enhanced by Zemanta

Tags: apache hadoop, big data, hadoop, ibm, splunk, sportingbet, teradata, yahoo

RSSSubscribe to this blog

More from Techworld

More relevant IT news

Contact Us

For editorial queries:
Mike Simons Mike_Simons@idg.co.uk

For website issues:
Email webmaster@techworld.com

For commercial queries
Russell Kearney russell_kearney@idg.co.uk


For more contact details click here.


Email this to a friend

* indicates mandatory field





Techworld White Papers

Optimising data protection for virtual environments

VM environments require the same level of data protection as does the physical server environment. Companies may use data protection tools built for the physical environment in the virtual world, but this has serious disadvantages.

Download Whitepaper

PCI Compliance: Are UK businesses ready?

Exploring the results of a recent survey, including: ? Levels of understanding of the standard ? Current perceptions of actual compliance status ? Attitudes toward addressing compliance

Download Whitepaper

Mobility Management for Dummies

Your complete guide to managing and securing mobile devices such as laptops and smartphones.

Download Whitepaper

Magic Quadrant for midrange and high-end NAS solutions

It is difficult to find one midrange or high-end NAS product that can cater to all needs. File systems embedded in NAS are often designed to solve one major pain point, with additional features being added later to broaden use cases and benefits.

Download Whitepaper

Techworld UK - Technology - Business

Oracle Video

Enabling agile and intelligent businesses

 Changing markets, competitive pressures and evolving customer needs are placing increasing pressure on IT to deliver greater flexibility and speed. Explore truly flexible SOA foundations with this Oracle video.

Watch
COLT White Paper

IT Misuse Survey

Complete this survey and you could win a Nexus One

Techworld are running a short survey to discover how UK businesses are managing Internet and email misuse in the Enterprise.

Complete Survey

Complete our survey and you could win a Sony E-book Reader.
Techworld have teamed up with HP to compile a survey relating to server virtualisation. Complete the short survey and you could be the lucky winner of a Sony E-book reader.

Complete the survey here

Site Map

Test