Bring Dark Data Into the Light 12
Unused data can transform organizations’ analytics programs.
Architected for Success 14
Teradata Analytic Architecture Services maximize the business value from analytical systems.
See the Big Picture 17
Expand your business horizons with fast, easy enterprise access to Hadoop data.
A Better View PAGE 3
The Teradata ® Unified Data Architecture ™
T here’s no doubt that big data analytics is a big challenge for many organizations. Why? It comes down to two specific issues: integration and the last mile of delivery.
Every day, organizations generate massive volumes of multi-structured data from a wide range of sources—point-of-sale terminals, websites, social media, email, voicemail, you name it. It can be tough to determine which platforms are best for which data types, and how those plat- forms integrate so you can make use of the data throughout your organization.
The second challenge is what I call the last mile of delivery. It’s one thing to gather and integrate all of the data, but it’s quite another to extract value from it. Some newer technologies require very specialized talent to extract the value, and those people can be expensive to obtain—and retain.
Deploy and Deliver
The Unified Data Architecture
™—a collection of platforms, applications and tools for data management, process- ing and analytics—addresses those concerns. Not only does it give you the data integration that’s so important to creating business value, but it also provides for that last mile of delivery
using common tool sets and standard business intelligence (BI) tools.
With the Unified Data Architecture
™, you can effectively and quickly deploy big data analytics and deliver the resulting insights to the point in your business where decisions are made and value is gained. You can also leverage the BI and skill investments you’ve already made so you can focus future BI invest- ments on truly incremental analytics.
Remove the Confusion The world is moving from a single integrated analytics platform toward a logical analytics platform, and it needs a roadmap. Even if the IT organization hasn’t yet made that move, I would say the majority of users are already start- ing to build their own solutions to take advantage of newer tools and data that are being created. The Unified Data Architecture
™enables IT to talk with the business about how to integrate and deliver end-to-end solutions. A cornerstone to the logical analytics platform is still the Integrated Active
Warehouse that enables real-time delivery of integrated intelligence to front-line decision makers.
This is now critical because of the confusion in the market concerning new big data technology and vendors.
It’s even being labeled “disruptive tech- nology.” That hype, along with today’s very real business challenges, can make it difficult to see what the next move should be, which means it’s all too easy to get off on the wrong path.
With the Teradata
®Unified Data Architecture
™, you have a nice frame- work that removes the confusion and enables organizations to make proactive decisions about how they move forward with analytics. It integrates best-of- breed technologies to support the new kinds of data that are available and delivers tangible results.
TScott Gnau president of Teradata Labs
Best of the Best
The Teradata Unified Data Architecture is the key to capturing, analyzing and acting on all types of data.
VIEW FROM HERE
INDUSTRY EXPERTS TOUT THE PROBLEM-SOLVING CAPABILITIES OF THE UNIFIED DATA ARCHITECTURE.
by Colleen Marble
C
ompanies are being bombarded with more data and more types of data coming at them faster than ever before. The days of a controlled inflow of highly structured data are over, and a new world of multi-structured, high-volume data has begun.More data is better, right? Yes—provided organizations have the skill sets and platforms to capture, analyze and standardize that data to guide them to make the best business decisions possible. But many companies are learning that their data types and diverse analytics needs exceed the existing capabilities of standard data warehousing or business intelligence (BI) solutions. The answer to this challenge is the Teradata® Unified Data Architecture™ that brings together the processing power, storage capacity and analytic capabili- ties of multiple platforms into one cohesive environment. This enables users throughout the orga- nization to extract value from all available data, giving key stakeholders actionable intelligence on which to base their critical decisions. >>
A Better View
Look Forward, Not Back The market is evolving to a logical data warehouse. In this market, it’s crucial for enterprises to be ahead of the competition with their vision and technology. Although BI plat- forms have claimed for years to provide actionable intelligence, the Unified Data Architecture
™takes the concept to the next level. The solution now makes it possible to ask any question of any data and get near real-time answers.
In a traditional environment, it’s simply too difficult to conduct that kind of in-depth discovery analysis in a timely manner given the volume, variety and velocity of today’s data.
CEOs and other business leaders run the risk of not having the right data at the right time to make fully informed decisions about highly complex business problems. Users are restricted to asking simple questions of data, not asking questions at all, or getting partial answers too late to take advantage of opportunities.
They’re unable to see and understand unique associations in the data to predict outcomes or resolutions.
“In the good old days, most com- panies had BI, not analytics,” explains
Evan Quinn, senior principal analyst, data management and analytics, for Enterprise Strategy Group. “BI is typically ‘look backward.’ Most of the data you’re working with is structured, and there’s not a lot of iteration or discovery around BI. The questions are relatively straightforward: ‘How many people have met their sales quota this quarter?’ for example.”
Analytics, on the other hand, looks forward. “With analytics, we may or may not know what the structure of the data is. We may be dealing with semi-structured data, and we may have to go through many processes to build analytic models,” Quinn continues. “It’s no longer a question of ‘What have we done?’ Rather, it’s a question of ‘What should we do?’”
Trustworthy Results
In this brand new era of analytics, the integrated data warehouse (IDW) becomes one of several platforms used to capture and process data to create actionable intelligence.
“The IDW continues to play a very critical role in a hybrid ecosystem;
it’s just not the center of it,” explains Shawn Rogers, vice president of research at Enterprise Management
Associates. “There’s a need for flex- ibility and a need for companies to try to align their data and their workloads with the best possible platforms.”
Within this unified environment, data is stored and processed on the platform or platforms that best meet user requirements for response time, long-term storage and access. Not only does this environment return answers faster than previous architectures, it integrates the data so users will have faith in the results. They know the solution takes into account all available information to deliver a panoramic view of the business.
Consistency Across Platforms The integration enabled by the Unified Data Architecture
™allows for data governance and steward- ship practices to be consistent across all data platforms simultaneously.
“Previous architectures were very monolithic, which meant that you had a single platform that was predicated on having a single source of truth,” notes Tony Baer, principal analyst at Ovum. “A unified data environment acknowledges that there are many analytics and many paths to getting what you need, and
The Unified Data Architecture
™enables companies
to integrate and extract value from all types of data
throughout the organization to empower users to do
more across the enterprise.
you need to have the right platform for the right workload.”
“But it’s not just about availability,”
Baer adds. “It gets you closer to being able to manage something such as data qual- ity with a consistent policy even if you have different practices for carrying it out based on different data types and data requirements. You can more easily feder- ate data between different platforms.”
The same is true for other key practices. “If you have a UDA, you can also start applying backup, recovery, business continuity, secu- rity, compliance, data protection—all of the things you need to run an ERP class application,” Quinn points out.
“Those are going to be required for big data, and if you’ve got everything more or less in one place, that’s going to facilitate that process.”
Not only does this help across plat- forms, but it creates consistency. The Unified Data Architecture
™enables companies to integrate and extract value from all types of data through- out the organization to empower users to do more across the enterprise.
Now’s the Time
The Unified Data Architecture
™addresses a wide range of business
pain points that aren’t easily solved by traditional data warehouse or BI environments. By integrating a variety of platforms into one solu- tion, businesses can use the right platform for the right use case to deliver the best possible results in a timely and economical manner. The solution also makes it possible to apply governance, security, business continuity and other critical policies in a consistent manner across all data, regardless of platform.
The key benefit, however, is that the Unified Data Architecture
™facilitates more complex data discovery, which in turn provides more actionable insights. “It opens up new opportunities for flexibil- ity and savings to the company,”
Rogers says. “It allows you to do things you couldn’t do before. And at its highest or most sophisticated level, it allows you to do more complex work.”
That benefit comes just in time.
“Big data is going to require a better understanding, better management and better security of your data,”
notes Quinn. “The time to get on board with having a bigger picture is now.” T
Colleen Marble has been writing about business, marketing and infor- mation management since 1996.
BRIDGING THE IT/
BUSINESS GAP
Using the Unified Data Architecture™ as a solution for big data analytics is gaining momentum. However, there’s still a lot of work to be done to close the cultural divide between the business and IT sides of the house.
“It’s great to talk about the volume, velocity, variety, veracity—all the Vs—
but you’re really living in a vacuum if you only talk about those things. That’s the IT world,” explains Tony Cosentino, vice president and research director at Ventana Research. “When you really look at the business users, the guys who are tasked with insights for the organi- zation, they’re much more focused on what I call the Ws—‘what’ is the data,
‘so what’ are the inferences and implica- tions of the data, and ‘now what’ are the decisions that need to be made from that data.”
Cosentino suggests that the unified data environment is a marriage between the Vs (the IT side) and the Ws (the business side). The challenges are to overcome the cultural resistance to the integration of business and IT, and to meet executive expectations for what the value chain should be for the data.
“What good is an analysis if you don’t know what’s valuable at the end of the day?” Cosentino asks. “You can’t find the needle in the haystack if you don’t know what the needle looks like. The more we can build analytic centers of excellence that bring together business and IT to address this question, the better off we’ll be.”
ARCHITECTED FOR OPTIMAL VALUE
The Unified Data Architecture™ is a proven, safe and cost-effective framework for smarter data management, processing and analytics that enables organizations to exploit all their data, regardless of structure. This
collection of services, platforms, applications and tools helps organizations define and deploy an architecture that makes optimum use of available
technologies in a way that unleashes the full value of data.
Got to Have It!
THE MOST POWERFUL ANALYTICS PLATFORM ON THE PLANET DELIVERS ANALYTICS ON ANY DATA TO REVEAL NEW OPPORTUNITIES.
by Brett Martin
integrated solution and is also the most powerful and complete analytics archi- tecture available today. It integrates with and benefits from Teradata Database, Teradata Aster and open-source Apache
™Hadoop
®technologies. (See figure.) The result is a unified, high- performance architecture that aligns data warehousing, data discovery and data staging to unlock valuable insights
for increased productivity, lower costs and new business value. The key com- ponents of the solution include:
The Teradata Database
The database provides a single source of consistent, centralized and integrated data. The integrated approach supports the highest business value through cross-functional analysis. Users can ask
FIGURE THE TERADATA UNIFIED DATA ARCHITECTURE
H andling diverse data types is a challenge for most busi- nesses, requiring the use of multiple technologies. That challenge is compounded by a spike in data creation rates from an ever-increasing range of sources. Organizations need to capture, store and analyze all types of information, including multi- structured data from social media, texts, graphics, audio, machines and other sources. Making sense of this data requires powerful analytics and data warehousing tools to extract intelligence that gives businesses a competitive advantage.
Teradata has a solution: The Teradata
®Unified Data Architecture
™allows for the transparent movement of data in and out of complementary systems. This unified data and analyt- ics environment delivers a high-per- formance, innovative system that gives any user any analytics on any data.
The solution enables organizations to leverage all data for new insights and new business opportunities.
True Integration
The Teradata Unified Data
Architecture
™is the only truly
the most challenging questions—from determining complex trends and uncov- ering business anomalies to identifying customers for automated, custom Web offers—and get quick answers.
The Teradata Aster Database
This delivers the patented SQL-
MapReduce
®technology. Business users can run MapReduce functions using SQL, allowing data discovery through iterative analytics against structured and multi-structured data. Even users who don’t have a deep understanding of MapReduce can still leverage the robust discovery analytics. Bridging the gap between the business language of SQL and the analytics of MapReduce lets users gain insights that can curb customer churn, increase the success of marketing campaigns and ultimately improve the organization’s bottom line.
Hadoop
The Hortonworks Data Platform pro- vides an appliance for loading, storage and refinement of data. Unstructured data with multiple formats can be quickly loaded into open-source Hadoop. This could be the perfect solu- tion for storing semi-structured data, including weblogs, and performing data refinements such as sessionization and aggregation.
Benefit From Any and All Data
While these core technologies provide the heavy lifting, the value-add enabling software determines how well they’re integrated. The software also controls how much of a burden is left on an organization’s shoulders to get the technologies to work together to enable true business value. The Unified Data Architecture
™offers a rich set of software
and services that provide administra- tors and end users with transparent data access, seamless data movement throughout the environment and one operational management view that includes single-source support. This truly integrated environment allows organiza- tions to focus on creating business value, not on technology integration.
The Unified Data Architecture
™sim- plifies processing across massive data sets. Organizations can quickly perform iterative analytics against a broad, deep set of data using SQL, SQL-MapReduce or non-SQL languages and tools.
With its combined capabilities, the Unified Data Architecture
™allows businesses to explore vast stores of traditional and new data. They can capture, store and analyze the data and turn it into actionable intelligence.
Deriving meaning from all data allows companies to better understand and predict customer behavior to improve the customer experience. They also gain a panoramic view of their business and supply chain to improve forecasting and planning, answer questions and identify new revenue opportunities.
The solution is also supported by a team of technical experts with exten- sive industry experience. The experts deliver solutions that remove the common obstacles organizations face, such as deploying and managing new systems, and providing accessibility to enterprise data.
A True Business Advantage The Unified Data Architecture
™offers a breakthrough in analytics. It’s the only platform to bring together multiple technologies—the Teradata Database, Teradata Aster analytics platform and Hadoop—and integrate them with value-add software and support.
By combining the advantages of data warehousing, data discovery and data staging, the solution lets companies quickly and easily answer any business question, regardless of the type of data being analyzed. With the Unified Data Architecture
™, no data is too big or complex to benefit the business. T
Brett Martin is the senior editor for Teradata Magazine.
PROBLEM SOLVER
A large global bank was struggling to reduce churn in profitable customer segments. Part of the problem stemmed from the bank integrating customer interaction data across multiple channels from numerous siloed repositories.
Further complicating matters was the size of the data—billions of records per month.
The company turned to the Teradata® Unified Data Architecture™ to help detect and prevent churn. The solution enabled the bank to build an enterprise view of all customer interactions and identify the most frequent paths to account closure.
This solution allowed the bank to:
> Pinpoint the event causing churn
> Remove the event
> Reduce churn among profitable cus-
tomers by 5%
The results were achieved using:
> A Teradata Database-enabled enter-
prise data warehouse for historical customer transaction, profile and product information
> A Teradata Aster Database to analyze
and discover patterns to determine which actions likely caused churn
> Apache™ Hadoop® for loading,
storage and refinement of data, and optimizing storage costs
> Teradata Relationship Manager to
make real-time decisions and offers to improve customer satisfaction and prevent account closures
The Power
of Integrated
Analytıcs USE CASES DEMONSTRATE THE ABILITY OF THE UNIFIED DATA ARCHITECTURE TO DELIVER UNIQUE BUSINESS BENEFITS.
by John Edwards
Communications
Telcos can obtain a panoramic view of each customer to better lever- age transaction and interaction data and deliver next-generation customer experiences.
Carriers are able to monetize data and their existing infrastructure, increasing profitability. Highly sophisticated mar- keting segmentation can identify key groups such as prime customers who pay bills on time, loyally renew service
contracts and frequently purchase add- ons, such as extra minutes and more data capacity. Relevant and timely offers can then be crafted to retain profitable customers and to attract new customers possessing similar characteristics.
>
A Russian wireless provider deliv-ered a 10% uplift in retention and
$13 million in annual win-back of customers through social net- work analytics of SIM card data.
>
A European carrier uses “microtargeting” to focus on small cus- tomer segments with distinctive
profiles. By accurately applying event-based marketing to offers, the carrier increased close rates by up to 200%.
Financial Services
Banks, credit unions, brokerages and other financial service orga- nizations benefit from
an integrated picture of customer activity across multiple chan- nels (Web, branch, mobile and ATM), to
S queezing the maximum insight out of data collected from a rapidly growing number of sources should be a top priority for every organization. Yet finding an analytics framework capable of fully leveraging the value of both structured and unstructured data created by various applications and services can be a challenge.
The Teradata
®Unified Data Architecture
™is the answer. This tightly integrated set of platforms, tools and
services enables organizations to integrate and exploit all their data for competitive advantage. Here are just
a few examples:
customize online banking services, mobile apps and ATM features to match customer behavior patterns and preferences. They can also capture new business by recognizing in-stream activity in near real-time and prioritiz- ing the most relevant offers based on customer activity.
>
A financial institution increasedprofitability and reduced cus- tomer churn by 5% through identifying and then removing an event that was causing a high number of account closures.
>
A global bank achieved a 25%uplift in response rates to Web offers and a marketing return on investment (ROI) of 1,400% ($1 of marketing = $14 revenue).
>
A North American bank realized$7.5 million in incremental value by using a new “best customer”
profile to target new accounts.
>
A Southeast Asian bank achievedcampaigns that were 10 times more successful than before, gain- ing $961 million in new sales and realizing a 164% annualized ROI.
Healthcare
Integrating data from all patient sources (predictive behavior analytics, sensor data, pharmacy data, claim data, emergency room
visits, etc.) helps these organizations deliver next-generation care manage- ment and dramatically reduce plan costs.
>
One provider uses analytics to helpidentify false positives. This helped reduce a list of 35,000 potential diabetic patients by 25%.
>
A health insurer detects fraudbefore a claim is paid with constant
real-time monitoring. A payer’s model run time was reduced from days or weeks to minutes, enabling appropriate investigations.
>
A pharmacy enterprise saveda healthcare organization $40 million over six months due to accurate and quick generic conver- sion when a popular brand drug went off-patent.
Manufacturing
Manufacturers can pinpoint weak suppliers, partners who fail to meet deadlines or companies that deliver inferior materials. Conversely, they can also identify
suppliers that excel in speed, quality and value. The Unified Data Architecture
™also provides analytics for
immediate and predictive decision support by reducing the time to iden- tify, isolate and resolve manufacturing and operational issues. Enterprises are able to operationalize key findings and iterate quickly for continuous improvement, delivering reduced procurement costs and increased manufacturing quality.
>
A manufacturer improved yieldby 1%, saving millions of dollars.
Utilities
These companies can strengthen customer satisfaction by enabling customers to manage and reduce their own utility usage.
Utilities can also increase effi- ciency of theft investigations by analyzing interval usage data from smart meters, help- ing them detect suspicious usage
behavior and pinpoint fraud with a much higher degree of accuracy. In addition, utilities can also identify customers with unusually high levels of consumption at unusual times to target for energy efficiency programs.
>
At a U.S. utility, an integrated viewof data resulted in a $9 million cost savings and a reduction of 2,800 man-hours.
>
Another U.S. utility helps indi-vidual customers reduce their energy usage, with the goal to reduce total consumption by 1,000MW—roughly the output of one power plant.
>
An investor-owned utility in thewestern U.S. transformed how it identifies energy theft, moving from a manual system to a data- driven one. The new analytical system detects theft incidents with 70% accuracy, up from 30%
with the manual system.
Take the Lead
The Teradata Unified Data Architecture
™puts new analytics capabilities to work for organizations in any industry. Having the ability to rapidly and painlessly analyze large volumes of traditional and new data sources enables organizations to identify and capitalize on new oppor- tunities, learn more about every touch point in their supply chain and better meet customer expectations through
the value of integrated data. This intelligence enables organizations
to progress from competing in
their industry to leading it.
TJohn Edwards has covered the
technology industry for more than
two decades.
Benefit from
Any and All Data
What makes the Unified Data Architecture
™different from other analytics architectures?
CONNOLLY:
It’s really a question of how you evolve your existing archi- tecture to deal with the challenges of massive data volumes. How can you create a well-integrated system that derives value from data in ways that haven’t been possible until now? Also, is there a compelling economic model
that can be leveraged against existing skill and solution investments?
The Unified Data Architecture real- izes this vision by bringing together an enterprise data warehouse, a discovery platform and a big data platform.
They’re deeply integrated for not only efficient data access and sharing, but also for a robust operational experi- ence—which, in my opinion, is almost as important.
I n the age of big data analytics, organizations are looking for ways to capture, analyze and act upon the enormous volumes of multi-structured data generated every day. What they’re finding is that the best solution may not be a single solution at all.
Instead, the Teradata
®Unified Data Architecture
™embraces the brawn of an enterprise data ware- house, the brains of a discovery platform and the breadth of a big data platform. Teradata Magazine spoke with Shaun Connolly, vice president of corporate strategy for Hortonworks, to learn more about what the Unified Data Architecture
™can deliver.
HORTONWORKS VICE PRESIDENT SHAUN CONNOLLY DISCUSSES HOW AN INTEGRATED ENVIRONMENT DERIVES VALUE FROM DATA IN NEW WAYS.
by Colleen Marble
Does this offer benefits that aren’t available from other solutions?
CONNOLLY:
Yes. It enables companies to pick the right data system for the right use case at an optimal cost-per- formance benefit for their data process- ing needs. For instance, if you have 10 years’ worth of data but you’re not sure of its longer-term value, then a big data platform enables you to cost-effectively store that information while you decide what to do with it.
On the other hand, you also have traditional data warehousing, reporting and BI [business intelligence] solutions that come into play for business-crit- ical data that’s highly structured and operationalized.
The benefits extend into the area of multi-structured or unstructured data.
Before, you may have talked about analyzing that data, but you couldn’t access it. Now you can create a space to cost-effectively store, process and report on it in an appropriate way.
How does this make it pos- sible for organizations to obtain full value from data?
CONNOLLY:
Organizations need a good crawl-walk-run strategy, and the Unified Data Architecture spans all three phases. The crawl phase is about creating what we refer to as a data refinery. It captures a large data set and
puts it all into one place. It doesn’t mat- ter whether that data is structured or unstructured. That refinery outputs to a data warehouse, where you can blend data for reporting, visualization, etc.
The walk phase covers data explora- tion. People can “mash up” new data with existing data to identify emerging patterns. That might require you to bring together mobile data, Web data and transactional data so you can see patterns that influence, say, customer satisfaction and leverage that knowl- edge to improve the business.
The run phase enriches the online application with advanced analytics to create highly customized experiences. Yes, you’ll have batch processing, interactive querying and exploration, but you’ll also have the ability to get those results into your online applications very quickly.
Is it able to deliver insights faster than other solutions?
CONNOLLY:
Many companies share data between systems, and they do a lot of point-to-point system integra- tion in order to distribute insights in a timely manner. The well-integrated Unified Data Architecture lets you automate a lot of that data flow and get it where it needs to be a lot faster than traditional methods.
This is true regardless of format.
Rather than spending a lot of time figuring out how to transform the data
into a highly structured format, you can store it in its native format. Structure becomes somewhat irrelevant. It’s only applied when you decide what insight you want to operationalize.
Why should businesses consider the Unified Data Architecture
™if they already have an analytics solution in place?
CONNOLLY:
You want a logical archi- tecture that lets you store and process unstructured, semi-structured and highly structured data within the “system.” It might be in the data warehouse or in Apache Hadoop, but it doesn’t matter as long as it’s in a place where you can conduct more sophisticated analytics.
Many companies have already opera- tionalized analytics in highly structured relational databases. But when you consider the multi-structured data coming out of sensors, mobile devices or digital videos, traditional analytics don’t have a strong answer. The Unified Data Architecture is purpose-built to deal with any and all forms of data. That really resonates, particularly with large enterprises that are trying to integrate all their information and still use appropri- ate tools for particular data sets. T
Colleen Marble has been writing about business, marketing and infor- mation management since 1996.
“ The Unified Data Architecture is purpose-built to deal with any and all forms of data. That really resonates, particularly with large enterprises...”
— Shaun Connolly, vice president of corporate strategy for Hortonworks
BRING DARK DATA INTO THE LIGHT
UNUSED DATA HAS THE POTENTIAL TO TRANSFORM ORGANIZATIONS’ ANALYTICS PROGRAMS.
12 l Teradata Magazine l Special Edition l Unified Data Architecture
DEMAND FOR DATA
Companies are interested in myriad types of data for their analytics projects:
report they have too much data, not enough analysis
Document repositories/ECM 85
%Lack of maturity in big data tooling
Don’t have enough in-house expertise
Relational data from transaction systems
39
%52
%39
%38
%37
%34
%29
%70
%45
%37
%34
%25
%13
%say they can only access structured data sets say the data is there but their tools can’t make sense of it
Email 82
%Web behaviors, clickstreams
54
%External/public social media
52
%Lack of support for real-time data Poor data quality
Unsure of how to connect across all of their data sets Data is likely to be
too “dirty” to use
Unstructured data/documents such as PDF, Word, Excel docs Social media data
Semi-structured industry data
“Balancing Opportunity and Risk in Big Data:
A survey of enterprise priorities and strategies for harnessing big data,” Informatica
OPPORTUNITIES ABOUND
As organizations learn how to leverage
unstructured content and connect across multiple repositories, users have the followingconcerns about their analytics tools:
report they have too much data, not enough analysis
Document repositories/ECM 85
%Lack of maturity in big data tooling
Don’t have enough in-house expertise
Relational data from transaction systems
39
%52
%39
%38
%37
%34
%29
%70
%45
%37
%34
%25
%13
%say they can only access structured data sets say the data is there but their tools can’t make sense of it
Email 82
%Web behaviors, clickstreams
54
%External/public social media
52
%Lack of support for real-time data Poor data quality
Unsure of how to connect across all of their data sets Data is likely to be
too “dirty” to use
Unstructured data/documents such as PDF, Word, Excel docs Social media data
Semi-structured industry data
“Big Data: Extracting value from your digital landfills,” AIIM
UNSTRUCTURED CHALLENGE
Enterprises report large amounts of unstructured or semi-structured data repositories they already
analyze, monitor or query, or would like to analyze, monitor or query:
report they have too much data, not enough analysis
Document repositories/ECM 85
%Lack of maturity in big data tooling
Don’t have enough in-house expertise
Relational data from transaction systems
39
%52
%39
%38
%37
%34
%29
%70
%45
%37
%34
%25
%13
%say they can only access structured data sets say the data is there but their tools can’t make sense of it
Email 82
%Web behaviors, clickstreams
54
%External/public social media
52
%Lack of support for real-time data Poor data quality
Unsure of how to connect across all of their data sets Data is likely to be
too “dirty” to use
Unstructured data/documents such as PDF, Word, Excel docs Social media data
Semi-structured industry data
“Big Data: Extracting value from your digital landfills,” AIIM
MAKE SENSE OF ALL DATA
Many enterprises have plenty of data but they struggle to make use of it due
to the lack of capable tools:
report they have too much data, not enough analysis
Document repositories/ECM 85
%Lack of maturity in big data tooling
Don’t have enough in-house expertise
Relational data from transaction systems
39
%52
%39
%38
%37
%34
%29
%70
%45
%37
%25
%13
%say they can only access structured data sets say the data is there but their tools can’t make sense of it
Email 82
%Web behaviors, clickstreams
54
%External/public social media
52
%Lack of support for real-time data Poor data quality
Unsure of how to connect across all of their data sets Data is likely to be
too “dirty” to use
Unstructured data/documents such as PDF, Word, Excel docs Social media data
“Big Data: Extracting value from your digital landfills,”
AIIM
LINKED UP
60 %
Nearly 60% of companies would find it very useful
to link structured and
unstructured datasetsthey already have, but only 2% are able to
do so.
“Big Data: Extracting value from your digital landfills,”
AIIM
SOPHISTICATED ANALYTICS
56 %
56% of enterprises would consider the ability to do sophisticated analytics on unstructured content
and data streams very
valuable, including 18%
who say it would be hugely valuable.
“Big Data: Extracting value from your digital landfills,” AIIM
DARK MATTER WASTED
23 %
The vast majority of useful data is not used. 23% of the
data in the digital universe would be useful if it was tagged and analyzed, but
less than 1% is actually analyzed.
“The Digital Universe in 2020:
Big data, bigger digital shadows, biggest growth in the Far East,”
EMC
A growing amount of data is collected and stored by organizations in almost every industry, which is often driven by compliance and/or a mindset that “we should keep everything and sort it out later.” A report by Gartner, “Market Trends: Big Data Opportunities in Vertical Industries,” states that organizations see existing, underutilized dark data as one of the most immediate opportunities to transform their businesses. Here’s a big data opportunity heat map categorized by industry:
Very hot (compared with other industries) Hot
Moderate Low
Very low (compared with other industries) POTENTIAL BIG DATA OPPORTUNITY ON EACH DIMENSION IS:
Gartner (July 2012)
Volume of Data Velocity of Data Variety of Data Underutilized Dark Data Hardware Software Service
Banking and SecuritiesCommunica tions,
Media and Servic es
Educa tion
Government Healthcar
e
Providers Insur
anc e
Manuf acturing and Natural R
esour ces
Retail Transporta tion
Utilities Wholesale T rade
“Market Trends: Big Data Opportunities in Vertical Industries,” Gartner
SERVICES
Teradata Analytic Architecture Services maximize the business value gained from analytical systems.
by David R. Schiller, CCP, and Lance Miller
Architected for Success
B usinesses are faced with deci- sions regarding the waves of data coming their way. They have valid concerns about how to man- age the data and corresponding projects to maximize efficiency and gain opti- mum business value.
What’s needed is an architectural approach to effectively manage this data and turn it into valuable, insight- ful and actionable information. This approach helps align data to business needs, prioritizes projects and adjusts the project scope to address business priorities and pain points in the most
effective and leveraged way—regardless of the data size.
Exploit All Data
Teradata addresses the analytic chal- lenges of managing growing volumes and diverse types of data through the Teradata
®Unified Data Architecture
™. The systems architecture unifies mul- tiple forms of data and data-oriented technologies into an integrated, cohesive and transparent solution. This allows organizations to leverage the comple- mentary values of the industry-leading Teradata Database, patented Teradata
Aster SQL-MapReduce
®and open- source Apache
™Hadoop
®technologies.
Different data types can be housed in a manageable environment using the best technology to exploit the data to its fullest potential.
The Teradata Analytic Architecture
Services enable the building of the
architecture to deliver information to the
business. These services play a critical
role in designing and implementing ana-
lytic environments to support data and
its related components so business users
can get the answers they need, when they
need them.
The Unified Data Architecture
™makes all information available so organizations can explore new oppor- tunities, fully leverage existing ones, and address the myriad business needs and regulations they face. Teradata Analytic Architecture Services make this vision a reality by working with organizations to determine their business priorities, issues and other data-related needs, and how to handle them in the most efficient, effective manner.
Streamlined Architecture The Teradata Analytic Architecture Services handle information through the same architectural principles that apply to all types of data, including multi-struc- tured. This ensures a consistent method- ology and approach to help expand or improve the analytic environment.
A key benefit of having stream- lined, integrated analytic architecture services is the ability to deliver a uni- fied approach based on the business, information, applications and systems supporting the three major components of the Unified Data Architecture
™:
>
Data warehousing offers integratedand shared data environments to manage and deliver strategic and operational analytics to the business.
>
Data discovery provides theanalytics to unlock insights from data, which needs to be performed with a technology that has rapid exploration capabilities through a variety of analytic techniques and is accessible by mainstream business analysts.
>
Data staging enables loading, stor-ing and refining data in preparation for analytics.
Other benefits of the streamlined approach include leveraging global best practices, reduced risk, and consistent, efficient and cost-effective project implementations.
Supporting Services The Teradata Analytic Architecture Services help set up the systems archi- tecture to meet each organization’s unique needs. An infrastructure can be optimized to support each company’s analytic requirements based on answers to questions about the business strategy, the data being gathered and analyzed, who needs to consume information and other business functions.
The supporting and streamlined services include:
>
Analytic roadmap, a strategic con-sulting service, provides a structured framework to determine business priorities, the value of data and data-leveraging capabilities. It puts prioritized projects on a “roadmap”
that shows each project’s incremen- tal business value and increased business capabilities.
>
Opportunity workshop defines thescope of a specific project, creating the conceptual architecture docu- ment used to outline facts about the project.
>
Scoping service establishes the proj-ect parameters with an emphasis on ensuring the business requirements are realized in the delivered solution.
>
Design and delivery service pro-vides a complete, detailed solution architecture for a project.
Reach Data’s Full Potential Teradata addresses the challenge of growing volumes of diverse data types with the Unified Data Architecture
™. The architecture places data in a manage- able environment and utilizes the best technology to exploit information to its full potential. The Teradata Analytic Architecture Services are the enablers providing this flexible approach to derive maximum business value from all forms of data. T
David R. Schiller, CCP, has nearly 30 years of IT experience. He manages Teradata Professional Services marketing programs.
Lance Miller manages the Teradata Professional Services/Customer Services marketing group.
NEW CAPABILITIES VIA TERADATA ANALYTIC ARCHITECTURE SERVICES
Business analytical challenges that organizations face include:
> Inability to analyze data on a granular level, resulting in a lack of information on actuals, plans and forecasts
> Analytic capabilities that do not meet business needs due to duplication of efforts, inconsistent information and an inability to perform what-if scenarios
> Difficulty correlating customer satisfaction to labor and staffing
> Lack of rigorous data standards, causing confusion and difficulty when comparing historical performance
> Data access and security issues
New capabilities enabled by Teradata Analytic Architecture Services:
> Advanced analytics provides what-if capabilities using historical data and improved forecasts to predict, understand and plan business actions.
> Location attributes enable organizations to analyze data and support marketing strategies based on shared common attributes.
> Business intelligence dashboard delivers a snapshot of business performance.
The Power of Interaction
A discovery platform empowers organizations to capitalize on their analytic prowess.
by Randy Lea
O ver the years data ware- housing has moved from transactional analysis and report generation to being operational and mission-critical, actually driving a majority of businesses. With the introduction of big data, organiza- tions now want to know more about interactions for behavioral analysis around customers, products, machines and supply chains.
Facilitating this type of analysis calls for a discovery platform that can ana- lyze all data—non-relational, multi- structured and structured as well as transactional data—without requir- ing extensive data modeling, pre-prep of the data or stringent service level agreements (SLAs). It’s not trying to balance the books, or reconcile down to the penny. And as long as the qual- ity of the data is sound, its complete- ness can be “good enough.”
Furthermore, both the data analyst and the data scientist who know the business and the data can execute an iterative process using multiple analytic types, such as SQL, MapReduce, graph or statistical functions, in conjunction with each other to discover unique insights.
When the analysis encompasses a broader set of data including text, machine and sensor data, discovery
theory advances from transactions to interactions and empowers busi- nesses to execute behavior analysis.
Here’s just one example of the value derived from analyzing interactions.
U.S. healthcare providers receive a quality rating based on a five-point scale. Each point on the scale can be worth millions of dollars in business so negative consumer feedback can be very, very costly.
With a discovery platform, it’s possible to identify specific behavior that typically leads to a consumer complaint such as issues with office visits, billing mistakes or comments made during their
call center interaction. Through a series of steps, analysis can deter- mine when a patient may be headed down the path toward a complaint.
Armed with that information, the healthcare provider can put in place appropriate interactions to try to influence a more positive pathway.
In today’s ultra-competitive envi- ronment, knowing all of your orga- nization’s interactions can provide a winning edge—the discovery platform is your means to sharpen it. T
Randy Lea is vice president of the Teradata Aster Center of Innovation for the Americas.
USERS
Data scientist
Data analyst ANALYTICS
SQL MapReduce Statistical functions
Behavioral analytics
Customer Product Machine Supply chain ITERATIVE ANALYSIS
Graph ALL DATA
Structured data Multi- structured
data Non- relational
data
DISCOVERY
Discovery platform
OLTP DBMSs
Doesn’t require extensive modeling Doesn’t balance the books
Data completeness can be good enough
No stringent SLAs
•
•
•
•
•
•
•
•
Discovery Platform Requirements
A Teradata integrated data warehouse (IDW) gives organizations in every industry a fresh, big-picture view of their business. Now, with the Teradata
®Unified Data Architecture
™, that view just got even bigger.
The solution integrates open- source Apache
™Hadoop
®as part of its analytic data foundation, allowing organizations to store and access massive volumes of data with
ease. Companies can expand the scope of their application, business intelligence (BI) and data mining implementations by extracting hidden jewels from the data housed in Hadoop. Those hidden jewels can be incorporated with other data in the secure data warehouse to provide richer, more detailed insights into the business. Access to Hadoop data is enabled by new capabilities within Teradata Studio and Teradata Database 14.10.
Extraction Made Easy
Smart loader for Hadoop is a new fea- ture in Teradata Studio 14.02 that sim- plifies browsing Hadoop file systems within the new Hadoop View and provides bi-directional data transfer between Hadoop and Teradata sys- tems. The smart loader is composed of wizards within the Hadoop View to automate and simplify Hadoop con- nection and transfer tasks.
The smart loader consists of three elements. The first element, Teradata
See the Big Picture
Expand your business horizons with fast, easy enterprise access to Hadoop data.
by Arlene Zaima
APPLIED SOLUTIONS
Studio, runs on the user’s laptop or personal computer, which makes a Java database connectivity (JDBC) connec- tion to the second element, the Teradata Database. Once the connection is estab- lished, the data move is initiated on the third element, the Hadoop cluster.
Teradata Studio executes the Teradata Connector for Hadoop, which is a set of application program- ming interfaces (APIs) and tools that support high-performance, parallel bi-directional data movement. A drag-and-drop interface in Teradata Studio allows business analysts to extend their self-service capabili- ties. They no longer have to rely on Hadoop programmers to extract data.
Instead, analysts can easily experi- ment with and explore a combination of Hadoop and warehouse data in their secured and controlled Teradata Data Labs.
Pain-Free Data Movement The Hadoop View in Teradata Studio provides a connection management interface to allow users to create, edit and delete profilers describing their Hadoop system. Users enter HCatalog, port and system credentials to estab- lish a connection.
Teradata Studio connects to the HCatalog to determine the location of the associated files within the Hadoop Distributed File System (HDFS) and to read the metadata associated with the tables and files. The Hadoop View displays a tree of the Hadoop database, schemas and tables to simplify naviga- tion. Within this interface, users can move tables between the Teradata Database and Hadoop.
In addition to the Hadoop View, Teradata Studio also provides the Transfer Progress View and a Transfer History View to monitor and manage
current and past data transfers (as referenced in the tabs shown in the figure). Each entry consists of informa- tion about the transfer job, including:
> Job name
> Time stamp for start time
> Source and destination
> Job status
> Duration
> Number of rows transferred
> Notes
> Summary
The Wizards of Data Transfer An import wizard supports data transfers between Hadoop and the Teradata Database. The wizard prompts the user for the destination table name and source delimiters.
Default column names and types are provided, but can be overridden. As a default, no primary index (NOPI) tables are created to avoid skewing.
FIGURE HADOOP VIEW IN TERADATA STUDIO
Hadoop View in Teradata Studio provides a tree browser of the Hadoop tables with a drag-and-drop interface for table transfers, and provides both a Transfer Progress View and a Transfer History View.
APPLIED SOLUTIONS
To import data from Hadoop, Teradata Studio creates the target table in the Teradata Database by inter- preting the HCatalog metadata and generating the appropriate “
CREATE TABLE” SQL statement. Once the table is created, Teradata Studio executes the Teradata Connector for Hadoop within the Hadoop cluster. Data is imported directly from Hadoop to the Teradata Database across the Infiniband net- work for fast, seamless data movement.
An export wizard supports data transfers between the Teradata Database and Hadoop. This wizard navigates users through the process, allowing them to change the default Hadoop table name, column names and types, delimiter and job name.
Teradata Studio will invoke the
Teradata Connector for Hadoop within the Hadoop cluster to export the data.
Unlock the Value in Hadoop Data
Business users need a simple way to unlock and analyze data stored in Hadoop without employing complex Hadoop MapReduce programming and distributed processing skills. They need technologies that allow a stan- dard, easy-to-use business language like SQL to analyze data that has been captured or refined in the Hadoop environment. They also need the flexibility to use their standard BI and reporting tools against this data.
To that end, Teradata SQL-H
™allows business users to easily leverage the data in Hadoop.
Teradata SQL-H
™is a new query interface to analyze data from both Hadoop and the Teradata Database.
It provides standard ANSI SQL access to Hadoop data, allowing applications and analysts to continue using standard interfaces to access external data in Hadoop systems.
The solution can deliver unique benefits to businesses. For example, wireless communications service pro- viders can improve the accuracy of customer attrition scores by enrich- ing their data in the warehouse with call center records in unstructured formats, Web log data and other information. This integrated data can help analysts build more effective models with a higher level of preci- sion. The companies can benefit from reduced churn, improved customer satisfaction, and targeted up-sell and cross-sell opportunities.
Organizations can also use Teradata SQL-H
™to improve sales.
Analysts track sales of products for key customer segments, and a trend may show growing sales in a segment called “others,” previously considered outliers to their customer segments. In-database analytic tools allow analysts to drill down into the detailed transactions across multiple channels of the “other” category to
refine customer segmentation using the familiar query language SQL to reduce the learning curve, latency and staffing costs to exploit the trend.
Empower Analysts Smart loader for Hadoop and Teradata SQL-H
™provide business analysts the opportunity to work in a self-service environment within the security of the Teradata Database where sensitive data can be enriched with data from a less secure source.
These interfaces allow enterprise users to directly access and analyze vast amounts of Hadoop data with- out requiring complex programming or an understanding of the inter- working of the Hadoop system. These tools let analysts leverage the full set of analytic implementations available to Teradata Database users against Hadoop data. Analysts can maximize the power of the Teradata Database and the simplicity of standard SQL to benefit from a big data storage and staging environment.
TArlene Zaima is a program manager for Teradata Integrated Analytics solutions, including Geospatial and Agile Analytics Data Lab.
FREE DOWNLOAD
Download your complimentary copy of Teradata Studio at Teradata.com.
Analysts can maximize the power of the Teradata Database and the
simplicity of standard SQL to benefit from a big data storage and
staging environment.
10000 Innovation Drive Dayton, OH 45432 teradatamagazine.com
Unified Data Architecture and SQL-H are trademarks, and SQL MapReduce, Teradata and the Teradata logo are registered trademarks of Teradata Corporation and/or its affiliates in the U.S. and worldwide. Hadoop is a trademark of the Apache Software Foundation. Teradata continually improves
products as new technologies and components become available. Teradata, therefore, reserves the right to change specifications without prior notice. All features, functions and operations described herein may not be marketed in all parts of the world. Consult your Teradata representative
or Teradata.com for more information. Reproduction in whole or part of any material in this publication without written permission of Teradata Corporation is expressly prohibited.
EB-6790 0613 Copyright © 2013 by Teradata Corporation All Rights Reserved. Produced in USA.
