Big Data Apps and Big Data PaaS


Enterprises no longer have a lack of data. Data can be obtained from everywhere. The hard part is to convert data into valuable information that can trigger positive actions. The problem is that you need currently four experts to get this process up and running:

1) Data ETL expert – is able to extract, transform and load data into a central system.

2) Data Mining expert – is able to suggest great statistical algorithms and able to interpret the results.

3) Big Data programmer – is an expert in Hadoop, Map-Reduce, Pig,  Hive, HBase, etc.

4) A business expert – that is able to guide all the experts into extracting the right information and taking the right actions based on the results.

A Big Data PaaS should focus on making sure that the first three are needed as little as possible. Ideally they are not needed at all.

How could a business expert be enabled in Big Data?

The answer is Big Data Apps and Big Data PaaS. What if a Big Data PaaS is available, ideally open source as well as hosted, that comes with a community marketplace for Big Data ETL connectors and Big Data Apps? You would have Big Data ETL connectors to all major databases, Excel, Access, Web server logs, Twitter, Facebook, Linkedin, etc. For a fee different data sources could be accessed in order to enhance the quality of data. Companies should be able to easily buy access to data of others on a Pay-as-you-use basis.

The next steps are Big Data Apps. Business experts often have very simple questions: “Which age group is buying my product?”, “Which products are also bought by my customers?”, etc. Small re-useable Big Data Apps could be built by experts and reused by business experts.

A Big Data App example

A medium sized company is selling household appliances. This company has a database with all the customers. Another database with all the product sales. What if a Big Data App could find which products tend to be sold together and if there are any specific customer features (age, gender, customer since, hobbies, income, number of children, etc.) and other features (e.g. time of the year) that are significant? Customer data in the company’s database could be enhanced with publicly available information (from Facebook, Twitter, Linkedin, etc.). Perhaps the Big Data App could find out that parents (number of children >0), whose children like football (Facebook), are 90% more likely to buy waffle makers, pancake makers, oil fryers, etc. three times a year. Local football clubs might organize events three times a year to gain extra funding. Sponsorship, direct mailing, special offers, etc. could all help to attract more parents, of football-loving-kids, to the shop.

The Big Data Apps would focus on solving a specific problem each: “Finding products that are sold together”, “Clustering customers based on social aspects”, etc. As long as a simple wizard can guide a non-technical expert in selecting the right data sources and understanding the results, it could be packaged up as a Big Data App. A marketplace could exist for the best Big Data Apps. External Big Data PaaS platforms could also allow data from different enterprises to be brought together and generate extra revenue as long as individual persons can not be identified.

About these ads
  1. March 21, 2012 at 9:44 pm

    Interesting viewpoint. But can’t we use traditional ETL/DWH/BI to implement the example in the article? The industry has spend 1.000+ Man years to try to enable the vision that end users (“business expert”) can utilize the data without the need for 1), 2) and 3). We can argue if they are successfull, but certainly the current Big Data hype does not provide any value in this area.

    IMHO BigData really requires huge data, and isn’t the problem with using Hadoop etc. that it is cheap and big, but provides only lower usability compared to traditional approach?

    It will be interesting to see if traditional vendors can embrace Big Data technologies or if they will be replaced by new generation of vendors.

    • March 21, 2012 at 10:41 pm

      Some very good points. NoSQL solutions have one big advantage compared to existing ETL/DWH/BI: unstructured data support. Most of the traditional solutions require experts to define fixed data structures. Here NoSQL has an advantage.

      My guess is that traditional vendors will be divided into two groups. Those that embrace progress and those that fight progress. EMC with Greenplum and HP with Vertica seem to embrace. Although lately Oracle and SAP are painting Cloud on their websites, they are not really embracing the new reality yet: e.g. RAC or SAP on Amazon with pay by the hour licensing are not available although their dotcom competition is offering it…

  2. September 3, 2012 at 9:39 am

    Nice read! IMO, the most important sentence is this one: “A Big Data PaaS should focus on making sure that the first three are needed as little as possible. Ideally they are not needed at all.”

    I agree. However, I do not see any relation to PaaS (i.e. cloud deployment model). A common non-cloud product would help.

    Best regards,
    Kai Wähner (Twitter: @KaiWaehner)

  3. Kai Wähner
    September 3, 2012 at 9:40 am

    Nice read! IMO, the most important sentence is this one: ”A Big Data PaaS should focus on making sure that the first three are needed as little as possible. Ideally they are not needed at all.”

    I agree. However, I do not see any relation to PaaS (i.e. cloud deployment model). A common non-cloud product would help.

    Best regards,
    Kai Wähner (Twitter: @KaiWaehner)

    • September 3, 2012 at 10:28 am

      PaaS is the big Cloud promise. Whereas IaaS is about deployment. SaaS about easy usage. PaaS is force.com and others. Instead of one SaaS app, it is a platform to create SaaS. This is where big data paas should be about enabling big data SaaS. Hope this clarifies it.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 328 other followers

%d bloggers like this: