BIG DATA SURVEY 2014 SURVEY

(1)

There has been a tremendous amount of hype around Big Data projects and applications in recent

years, but relatively little quantifiable evidence proving what, if any, business value these initiatives

are delivering to organizations. The database has risen to prominence as the single most important

determining factor behind whether a Big Data project succeeds or fails. While organizations have a

plentitude of offerings now available to them, ranging from traditional RDBMS vendors and cloud

databases to NoSQL and NewSQL, it’s a double-edged sword—making the right choice has never

been more difficult.

This survey was conducted with the goal of identifying opinions, attitudes and trends on the use

of database technology to create applications that leverage the massive amounts of data coming

into organizations today. What it uncovered is a great divide: the ability to successfully capture and

store huge amounts of data is not translating to improved bottom-line business results. Seventy-two

percent (72%) of the respondents surveyed admitted that their organizations cannot access, let alone

utilize the vast majority of the Big Data they collect. And the reason their data is going to waste—

database performance deficiencies.

Although most organizations are running multiple databases, the vast majority were not authorized

to select an optimized database for their new Big Data application. Instead, they were forced to apply

short-term fixes to improve system performance of existing databases (see figures 5.A-5.B). These

measures, including adding additional caching and stream processing, were still not enough for some,

as one in five respondents with the requisite project knowledge (i.e., excluding those answering

“don’t know”) indicated they had to abandon their Big Data projects altogether (see figure 6).

As evidenced by this survey, organizations that cannot quickly access, analyze and decision on the

majority of their data are missing golden opportunities to deliver a more personalized customer

experience, drive revenue growth and create competitive advantage.

SURVEY

(2)

SURVEY mEThoDoloGY AnD RESponDEnT pRofIlE

Answer Options Response Percent

programmer/Analyst/Developer 35.9%

other IT professional 16.0%

Development manager 11.7%

other IT manager 11.4%

Database Administrator 9.2%

Database Administration Manager 7.9%

non-IT professional 4.9%

non-IT manager 3.0%

What is your PRIMARY role?

Figure 1.B 35% 3% 16% 11.7% 11.4% 9.2% 7.9% 4.9% Programmer/Analyst/ Developer Other IT Professional Development Manager Other IT Manager Database Administrator Database Administration Manager Non-IT Professional Non-IT Manager Figure 1.A

Commissioned January, 2014, the VoltDB survey collected responses from 368 qualified IT professionals, including programmers, data analysts, database developers and administrators, and others. These respondents spanned a wide cross-section

(3)

Please indicate the industry your company operates in:

Information Technology 34.2%

finance & Insurance 13.9%

Retail Trade 6.5%

Professional, Scientific, Technical Services 6.0%

Transportation, Warehousing 5.7%

health Care, Social Assistance 4.6%

Arts, Entertainment & Recreation 3.8%

manufacturing 3.3%

Educational Services 3.3%

Utilities 2.7%

Wholesale Trade 1.9%

Real Estate, Rental & leasing 1.6%

Mining, Quarrying, Oil & Gas Extraction 1.4%

Public Administration 1.1%

Construction 0.8%

management of Companies & Enterprises 0.8%

Agriculture, Forestry, Fishing & Hunting 0.3%

Administrative and Support of Waste Management &

Remediation Services 0.3%

Accommodation, Food Services 0.3%

other 7.6%

Figure 2 What was your company’s revenue for 2013 (in USD)?

more than $5 Billion 12.2%

$1 - 4.9 Billion 11.4% $500 - 999 million 4.6% $250 - 499 million 6.5% $100 - 249 million 9.0% $50 - 99.9 million 6.0% $10 - 49.9 million 12.0% $1 - 9.9 million 9.0%

less than $1 million 16.8%

Don’t know 12.5%

Figure 3

24%

of respondents were employed by organizations earning

> $500 million

45%

of respondents were employed by organizations earning

(4)

Which type of database is your organization currently using? (select all that apply)

Traditional SQL-based RDBMS 70.9%

mySQl 48.4%

noSQl 37.8%

Data Warehouse 32.9%

Large data set processing in a distributed computing environment 27.2%

mainframe hierarchical Database 17.9%

newSQl 17.4% other 7.6% Figure 4.A Traditional SQL-based RDBmS Data Warehouse mainframe hierarchical Database newSQl noSQl mySQl

(5)

Which of the following have you had to implement due to your database-of-record’s performance shortcomings? (select all that apply)

Database caches 50.7%

Batch ETl 35.4%

Additional database licenses and supporting hardware 35.1%

Stream processing engine because the database

isn’t fast enough to process streaming data 30.9%

We have never encountered performance problems with

our database-of-record 15.6% 50.7% 35.4% 35.1% 30.9% 15.6% Database caches Batch ETL

Additional database licenses and supporting hardware Stream processing engine because the database isn’t fast enough to process streaming data

We have never encountered performance problems with our database-of-record

In the last 24 months, have you had to abandon any Big Data projects because your database hasn’t been able to support your application requirements?

(6)

50.0% 46.8% 37.1% 32.3% 32.3% 25.8% 6.5% Could not meet application

performance requirements Application could not meet minimum response time requirements

Database licensing fees were too expensive Inability to ensure transactional consistency Inability to extract business value from data in the required timeframe Could not ingest data at sufficient rate

Other

What are the primary reasons the project failed? (select all that apply)

Could not meet application performance requirements 50.0%

Application could not meet minimum response time requirements 46.8%

Database licensing fees were too expensive 37.1%

Inability to ensure transactional consistency 32.3%

Inability to extract business value from data in the required timeframe 32.3%

Could not ingest data at sufficient rate 25.8%

other 6.5%

Figure 7.A

(7)

What percentage of the data coming into your organization today are you able to access/utilize in an actionable way?

less than 50% 72.5%

more than 50% 27.5%

If you were able to access, analyze, and decision on the majority of data coming into your organization, what would you do with the insight? (select all that apply)

Deliver a more personalized customer experience 49.6%

Drive revenue growth 47.6%

Create competitive advantages (e.g., get to market faster, differentiate, adjust promotions

"on a dime" to achieve better results, etc.) 47.6%

Identify areas for improved productivity/operations 45.9%

Identify new market opportunities 39.8%

Help refine our go-to-market strategy 35.4%

Determine overall strategic business direction and and tactics 24.8%

Figure 8

Figure 9 The majority of respondents indicated that their

organizations can’t utilize most of the data they store for Big Data applications (see figure 8), despite

the fact that doing so could drive real bottom line business benefits (see figure 9).

UnUSED DATA hAS lITTlE oR no VAlUE.

most respondents recognized the advantages of in-memory database architecture (see figure 10) and predicted it will become mainstream in the near term (see figure 11). The drivers for adoption of in-memory

architecture include the ability to more quickly develop insights into the business as well as analyze real-time data and support real-time decision making (see figure 12).

(8)

Do you believe databases should move to in-memory to deliver better performance characteristics?

Yes 89.4%

no 10.6%

When do you believe in-memory databases will become mainstream?

0-5 years 61.6%

6-10 years 27.2%

more than 10 years 7.8%

never 3.4%

What do you believe to be the main value of in-memory analytics?

Allows for faster business insight 29.0%

Analyzes dynamic or real-time data 28.4%

Supports real-time decision making 20.2%

Allows us to simplify the stack as well as IT infrastructure and processes 6.8%

Don’t know 5.7%

Cost savings/decreased total cost of ownership 4.8%

no background in specialty databases required 4.0%

other 1.1%

Respondents also indicated they believe in-memory analytics will become mainstream in the near term in

order to close the “time to insight” gap in Big Data projects. [see figure 10].

AnALyTICS ARE MOVInG In-MEMORy WITH THE DATABASE

Figure 10

(9)

RECommEnDATIonS

• Organizations must accurately define project requirements and desired outcomes when developing a Big Data application. There are many new databases available today, but they solve different problems and need to be evaluated carefully to ensure they deliver the required performance characteristics. For example, graph databases are better suited for those situations where data is organized by relationships vs. by row or document, and specialized text search systems should be considered appropriate in situations requiring real-time search as users enter terms.

• The starting point to harnessing the power of Big Data is the ability to actually access and utilize data in an actionable way. Data is useless if it cannot be acted upon within a time frame that delivers business value. Extracting business value is most challenging with “fast” data – that is, data that provides maximum business value the moment it arrives, but loses value with the passage of time. In these situations, massive amounts of business value can be lost in a matter of milliseconds. It is critical to match database performance to the nature of the data that is being ingested, stored and analyzed. In the case of fast data, memory databases and in-memory analytics are the only practical option. Anything else is incapable of acting on the data fast enough to deliver optimal business value.

• In-memory database architecture combined with immediate, intelligent data processing is crucial to tapping Big Data value. Storing data entirely in main memory has the advantage of

being much faster than writing to and reading from a file system. Additionally, in-memory databases are designed to eliminate multi-threading and locking overhead, two of the main reasons for poor database performance. However, until and unless organizations employ a database architecture purpose-built for exceptional speed that also supports real-time analytics and instant decision making on the massive volumes of data streaming in – at the moment it arrives – Big Data’s value will go to waste.