Cloud-based
predictive
analytics poised
for rapid growth
James Taylor
CEO, Decision Management Solutions
The results of research
into predictive analytics
in the cloud show that
early adopters are
breaking away and that
there is potential in
many kinds of
cloud-based predictive
analytic solutions.
©2012 Decision Management Solutions
Contents:
Introduction
1
Five scenarios for Predictive Analytics in the Cloud
5
Survey Results
15
Recommendations
27
Clario Analytics
31
Demographics
33
Conclusion
34
“Innovation happens at the intersection of
two or more different, yet similar, groups.
Where one technology meets another, one
discipline meets another, one department
meets another.”
Valdis Krebs, Founder & Chief Scientist, orgnet.com
Predictive Analytics in the
Cloud
Predictive Cloud
Analytics
1
Predictive Analytics in the Cloud
An Introduction
“Predictive analytics make sense of the tidal
wave of data available to companies today.
Cloud-based predictive analytic solutions make
predictive analytics more scalable, more
pervasive and easier to deploy.”
James Taylor, CEO Decision Management Solutions
Predictive analytics and cloud are hot topics in business today. Predictive analytics are increasingly the focus of many companies’ efforts to improve business
performance with analytics while cloud is fast becoming the default option for purchasing and deploying
software. Public, private and hybrid clouds are all evolving rapidly and are here to stay. But what’s
happening at the intersection of these two technologies? How can predictive analytics in the cloud add value and what are the critical risks and issues involved?
This paper explores the five key opportunities for organizations to use predictive analytics in the cloud: Using the cloud to deliver predictive
analytics-enabled “Decisions as a Service” solutions Embedding predictive analytics in Software as a
Service (SaaS) and other cloud-deployed applications Using the cloud to deliver predictive analytics to
non-cloud applications across the extended enterprise
Building predictive analytics against data in the cloud Using cloud computing to deliver elastic compute
power for building predictive analytic models
Before discussing the various options for predictive
analytics in the cloud it is worth clarifying exactly
what we mean by the various terms.
2
Introduction
Predictive Analytics
Predictive analytics is short hand for using historical data to build predictive analytic models. These models are mathematical formulae that use analysis of the past to calculate a value that represents a prediction about the future. It is sometimes said that “predictive analytics turn uncertainty about the future into a usable
probability.” Instead of simply being uncertain which offer is most attractive to our customers we can use a predictive analytic model to see which particular offer is most likely to be attractive to each specific customer; instead of being uncertain which claims are fraudulent we can use a predictive analytic model to predict how likely it is that a specific claim is fraudulent; instead of being uncertain how to treat our customers we can segment them into groups that are likely to respond similarly. Predictive analytics are often considered part of business intelligence or business analytics while data mining is often used as a synonym.
As we enter the era of “big data,” increasing amounts of data, both structured and unstructured, is available for analysis. Information that used to be “hidden” in emails or text fields is increasingly available for analysis. The growth of social media has added a whole new class of information that can be used to build a richer picture of customers and prospects. Gaining actionable insights from this data is rapidly becoming a business imperative. The use of predictive analytics to make sense of and exploit new and existing data sources is tracking to become pervasive, expanding from industries where it is already well established into every aspect of business and operations.
Predictive analytic models can be grouped into several categories:
Statistical—How likely is this fact to be statistically significant?
Association—If someone needs this item what else might they need?
Clustering—Group customers by the likelihood they will behave similarly.
Binary Predictions—will this shipment be late? Number in range predictions—how likely is this
claim to be fraud?
Selection from choices predictions—which route should be used for this delivery?
A wide range of possible techniques exist to develop these predictive analytic models. These will not be discussed here but a list of suitable references is provided in the bibliography.
Predictive analytic models can be built with an
organization’s own data, with data from external third-party sources and from data pooled from multiple organizations. In general more data sources, and more data, improve the effectiveness of predictive analytic models provided this data reflects the organization’s population of interest. Organizations that build
predictive analytic models typically invest in an analytic infrastructure that allows them to clean, integrate and analyze data.
Predictive analytics add value to an organization by improving the quality of decision-making. Knowing the prediction is not enough—the prediction must be acted on to add value. How a predictive analytic model is deployed, how it is used to drive automated decisions or to support manual decisions, is critical. Building predictive analytic models and deploying them are separate steps in the process but both must be mastered to get value from predictive analytics. Approaches for deploying predictive analytic models include writing or generating code that performs the same calculation as the model; deploying the model to a business rules management system or BRMS; and deploying the model “in-database” so that it is executed by a database or data warehouse server as required.
3
Introduction
Cloud
The key element in cloud computing is that computing resources are delivered as a service or a utility rather than as a set of products that must be installed and configured. A network is used to access the service, typically the internet. For example, instead of installing software to manage emails on a server at my location I use a cloud-based email service that can be accessed from anywhere using a browser. An analogy to the electrical grid is often used—companies don’t buy and install their own power generation capability, they buy power as and when they need it. Cloud computing, especially public cloud computing, makes computing capabilities available on a similar basis.
While there are those who consider cloud computing to be a new phenomenon, the basic principles of a cloud-based solution go back to the hosted application service providers of the 1990s and even to the shared environments provided by, for example, credit card processors since the 1980s. The recent explosion in interest has been driven by the replacement of custom interfaces and protocols with those based on Internet standards, by increasingly standardized hardware and software platforms suitable for developing cloud solutions, by the growth in virtualization software and by the widespread adoption of Service Oriented Architecture.
Three kinds of cloud computing are often discussed: Public cloud computing involves providing capabilities
as a service over the Internet using shared
infrastructure for all users—all users have access to the same compute capacity. Sometimes the
applications available on public clouds are available to everyone, sometimes only to those who have signed up for a particular service.
Private cloud computing involves using a private network so that only a particular company or organization can access the resources. This organization owns all the compute capacity being used.
Hybrid clouds use a mix of the two approaches, often running the same software in both public and private settings and using the two sets of
infrastructure for slightly different purposes. Public cloud infrastructure may be used to get started quickly before transitioning to private cloud functionality or some functions may be performed only on a private cloud—to keep data off the public networks for instance—while others are performed on both the private and public clouds.
Key elements of cloud solutions include:
Multi-tenancy, where multiple clients or projects are sharing the same hardware and software
infrastructure so that fluctuations in demand between them can be managed.
Transaction or usage pricing so that those using the capacity are paying for what they actually use not simply for the right to use it.
Location transparency and network access so that the solutions offer API-based or browser-based access available from any device that can access the network (public or private).
Service level agreements that define reliability and performance.
Reliability and availability through shared and sharable resources, failover from one piece of capacity to the next and high levels of overall redundancy.
Predictive Analytics in the Cloud
The characteristics of both cloud and predictive analytics solutions make the combination interesting. The combination of the power of predictive analytics to turn data into actionable insight and of cloud to deliver this insight broadly and cost effectively is potentially transformative. Five distinct opportunities for using predictive analytics and cloud together can be identified: Pre-packaged cloud-based solutions.Solution offerings, delivered as cloud-based or SaaS solutions, that provide decision-making based on predictive analytics as a core feature of the solution. For example cloud-based applications offering next
4
Introduction
best action, offer selection, fraud detection or instant credit decisions.
Predictive analytics for SaaS.
The use of predictive analytics solutions that are cloud-based to inject predictive analytics into other software products that are cloud-based or delivered as SaaS. For example embedding customer churn predictions in SaaS CRM solutions or delivering risk predictions into cloud-based dashboards.
Predictive analytics for on-premise.
The use of predictive analytics solutions that are cloud-based to inject predictive analytics into disparate internal systems and multi-channel
environments. E.g. using cloud-based deployment to link an internally developed cross-sell offer service (that uses propensity to buy models) to multiple customer-facing systems.
Modeling with the data cloud.
The use of cloud-based predictive analytic solutions to respond to the increasing amount of relevant data available in the cloud rather than on-premise. For instance building predictive analytic models in the cloud-based on customer purchase and behavior data stored in a SaaS CRM system as well as third party data available from a cloud-based web service. Elastic compute power for modeling.
The use of cloud technology to provide predictive analytics solutions that can scale elastically to meet demand. E.g. assigning extra resources dynamically when optimization or other demanding algorithms are being used to build sophisticated predictive analytic models against large datasets.
The characteristics of each of these opportunities, their value to organizations that adopt them and the way in which cloud and predictive analytic approaches
intersect in each case are the topic of the remainder of this paper.
5
“Clario's cloud technology has given us a
marketing tool that optimizes our contact
decisions and improves our planning process.
Additionally, their talented staff and deep
industry expertise has helped us improve our
sales forecasting.”
VP of Planning, Multi Channel Retailer
Three main characteristics define pre-packaged cloud-based solutions. First, and most important, these are solutions and not infrastructure. They make or enable specific decisions that can be described in business terms. The solution is not a predictive analytic solution so much as a decision-making solution. These solutions use predictive analytic models to make these decisions more accurate and effective. The predictive analytic models are embedded within a solution framework so that what the customer gets are better decisions not simply predictions.
For example a multi-channel cross-sell solution decides which product to offer as a cross sell to customers in different channels. This is based on predictive analytic models that predict how likely it is that the customer in question will buy each product and on rules and policies regarding how and when the products are sold.
The predictive analytic models involved are provided to the end customer—they are not required to build their own models. These models may be built automatically by software embedded in the solution or be built by the solution provider directly. Typically these models are built using the end customer’s own data. Some of these models may be built using data pooled from many organizations and multiple users of the solution may therefore use the same predictive analytic models. For instance applications for credit card fraud detection may use scores developed from credit card transactions
Pre-packaged cloud-based
solutions
6
Decisions as a Service
across all card issuers to predict how likely a particular transaction is to be fraudulent.
Value
Because offerings in this category are focused on solving a specific business problem they are straightforward to budget for and to justify. Business people can identify that the solution will help address a known business problem and can use this to justify purchase. In addition, because the offerings are packaged solutions, they offer a rapid time to value with little need for configuration, integration or modeling before value is realized. The value of predictive analytic models is also directly realized by providing a solution that acts on the predictions being made.
Because these are packaged solutions there is little or no need for the business buyers or users to understand either predictive analytics or the cloud. Typically, everything is packaged up into a solution with simple interfaces for both installation and usage. Users can get value from the solution without having specific skills in predictive analytics or even in the solution area itself. These solutions are often purchased because they address a business need and the fact that they are both cloud-based and embed predictive analytics matters only to the extent that these drive the desired behavior and cost profile for the end user. This simplicity and ease of use has a cost, of course, in that these solutions are typically hard to expand beyond their specific purpose.
The intersection
Because decisions have a simple interface—some information is provided and a decision is returned— they are easy to embed in services. It is straightforward to hide the complexity of analytic decision-making behind a simple API. This characteristic of decision-
making solutions makes them a good candidate for cloud-based delivery.
7
Benchmark Brands through it’s FootSmart brand, is a specialty retailer of comfort footwear and foot health products marketed through the FootSmart catalog and FootSmart.com. Its catalog mailings make an appearance in more 10 million households every year.
Benchmark Brands partnered with Clario Analytics to find the optimal contact stream for each individual customer across marketing media, channel, and time. Initially, a test group of customers received a contact stream dictated by Clario Stream, while a control group of customers received the contact treatment prescribed by the existing circulation methods. The test was evaluated on performance over several months.
A typical catalog mailing involves between 15 and 20 percent waste. With catalog costs at $0.50 or higher and mailing lists in the millions, that waste quickly multiplies Using Clario Stream allowed Benchmark Brands to focus catalogs on those most likely to show an acceptable return on the cost of mailing them the catalog. Reducing waste and targeting mailings increased profit and helped Benchmark Brands show double digit sales growth.
In addition, the use of the cloud allows data for the analytics to be pulled together by the solution provider so that it can be analyzed effectively. The cloud allows the solution provider to “push” new models to all users rapidly and gives them access to pooled data from many customers if this is part of the solution’s value.
Solutions that embed predictive analytics in cloud-based decision-making services are powerful but how can an organization deploy its own models to cloud-based applications it already uses?
8
“Clario offers us the opportunity to quickly
integrate best practices for marketing analytics
and strategic planning through their ‘Core’
cloud applications to support our internal
marketing resources without having to commit
to large investments in infrastructure projects or
tying up our internal IT department.”
Dave Mathews, President, CEO Healthy Directions
Companies are increasingly investing in SaaS offerings for their operational systems. SaaS solutions for
Customer Relationship Management, Human Resources, Marketing and more are well established. Business Intelligence, social and collaboration platforms as well as hosted versions of applications from the major
enterprise platform vendors are also available. Like many enterprise applications, many SaaS applications don’t apply predictive analytics. A cloud-based
predictive analytics solution may be the most effective way to embed more advanced analytics into these operational systems.
The defining characteristic of this category is that it uses the cloud to deliver predictions to SaaS or cloud-based applications. The SaaS applications already support or automate some decision-making and the predictive analytics are being delivered to improve the accuracy of these decisions. Predictive scores or characteristics are generated using predictive analytic models and
embedded in the SaaS application using the cloud. For example a credit risk score could be delivered to a SaaS CRM solution and then used by a customer routing script to route customers with low credit scores to an agent who specializes in helping those with poor credit. The predictive analytic models in question could be developed by the customer, the solution provider or a third party. They could also be based on pooled data as discussed above. The models could be built
Predictive Analytics for SaaS
9
Predictive Analytics for SaaS
automatically using software or be built using an existing analytic infrastructure. Regardless, the focus is on making those predictions available to SaaS or cloud-based environments.
Similarly the solutions could be specific to a particular kind of predictive analytic model or a particular domain, or they could offer a more general purpose capability. A general purpose capability for instance might allow any model to be deployed into a reporting environment using a standard such as the Predictive Model Markup Language (PMML) while a more focused solution might use the Force.com environment to integrate predictive analytic models with Salesforce.com.
Value
The value of this category derives from the lack of analytic sophistication in most SaaS and cloud-based applications, especially when it comes to predictive analytics. While many SaaS offerings are described as being “analytic” this usually mean only that they provide integrated reporting and dashboards. Some are not strong even at these most basic analytic tasks. The absence of support for data mining and predictive analytics in these solutions creates a need for offerings that can push predictive analytics into these applications to improve decision-making.
The predictions can improve decision-making in one of two ways. They could be pushed into an automated decision where they are combined with business rules or scripting in the target application. This allows the SaaS application to make more sophisticated decisions. They could also be added to a dashboard or reporting environment designed to support a person making a specific decision, helping them make a better one.
The intersection
In many situations, predictive analytics are a much more powerful way to use data than reporting or dashboards.
Being able to use historical data to make predictions about the future turns a largely passive data asset into one that can be used to actively improve business results. This is especially true because SaaS applications are often concerned with day to day operations. The power of predictive analytics to improve the way operational decisions are made is well established so embedding predictive analytics into these operational SaaS applications is very valuable.
SaaS applications need cloud-based delivery of predictive analytics if they are to use these kinds of analytic models. SaaS applications can’t easily access predictive analytic models deployed on premise (the traditional way organizations deploy their models). It is also not necessarily possible or desirable to deploy predictive analytic models as scores in the SaaS application database while newer in-database
deployment technologies may not work with a hosted database. As a result scores must be made available as a cloud-based service call for maximum utility.
Cloud-based deployment of predictive analytics can improve the accuracy of SaaS systems. It can also help make analytics more pervasive in on-premise
10
“Clario Stream applies intelligence to the
decision of who we to mail to, optimizing the
process.”
Alan Beychok, President and CEO of Benchmark Brands
Operational systems are where companies are seeing the greatest payback on their predictive analytic investments. The ability to package up predictive analytics in cloud-based services to enhance an organization’s existing systems can lower barriers to predictive analytics deployment. Many predictive analytic models are built by organizations and then not deployed, not put into production. These undeployed models represent lost opportunity for organizations. The pervasiveness of cloud-based solutions and the ease with which applications can be connected to the cloud mean that a cloud-based predictive analytic deployment approach may significantly increase the effective use of predictive analytic models.
The characteristics of cloud-based services to deliver predictive analytics to on-premise solutions are very similar to those embedding predictive analytics in SaaS systems. The difference is that the models are being deployed not to SaaS or cloud-based solutions but to internal legacy applications or to applications managed by an organization’s partners at their own locations. In this category the models are more likely to be the organization’s own predictive analytic models though they could also be built in the cloud (see “Elastic
compute power for modeling” below). For example, the predicted target price for a product might need to be distributed to multiple channel partners who each have their own systems.
Predictive Analytics for On Premise
11
Predictive Analytics for On Premise
Value
Many organizations have a disparate set of legacy applications into which they wish to deploy predictive analytic models. As organizations use an increasingly diverse set of partners and integrate these partners more tightly into their operations, the need to push predictive analytics to these other organizations also grows. Widespread deployment of predictive analytic models using a cloud-based approach allows consistency of decision-making across channels, partners and legacy applications.
A cloud-based approach is likely to be an effective solution in part because of the difficulty of embedding models in multiple applications and because a single database or a single environment may not be accessible to all the solutions that need the predictions in
question. Multiple systems built using multiple technologies might not all be able to execute code generated for a predictive analytic model where they will likely all be able to access a cloud-based service. Similarly all these systems are unlikely to share a singledatabase so in-database or in-warehouse
deployment of a predictive analytic model will not be an option.
The intersection
Predictive analytic models are particularly valuable in improving day to day operations. These front-line operations have both the most partners and require the most diverse set of enterprise software applications. The need to push predictive analytic models to every point of contact to drive consistently excellent decisions benefits tremendously from a cloud-based approach.
A fitness company that sells directly to
consumers through a catalog, other direct mail, the internet, TV and retail channel, this company has several brands. Clario’s cloud-based analytics solution was used to build buyer and inquiry models to target catalogs for one of the brands. This brand's catalog goes out three times per year and because of the high price point of a typical fitness equipment purchase, identifying the right customers to target is critical. Attributes were created from a variety of internal and cloud-based sources such as customer, order, item, inquiry activity and demographic data. These attributes drove predictive response models for current buyers and catalog inquirers based on historical mailing and purchase information. The final predictive analytic models rank order customers and inquirers according to expected sales. Model scores are updated prior to each campaign using Clario Core. For one campaign the buyer model had a 25% higher response rate and 44% higher ROI compared to prior campaigns. The inquirer model had an ROI increase of 174%.
The cloud is a great tool for driving analytic behavior into disparate systems. But as more of an organization’s data moves to the cloud, the potential for modeling in the cloud grows.
12
“Our catalog profits did increase as a result of
using Clario Stream. We are a disciplined
profit-driven company that does not spend our
advertising dollars on initiatives that do not
have an acceptable return. Clario Stream helps
us identify the bad spends in our catalog
mailings, making the whole effort more
profitable.”
Pete Bather, Vice President of CRM and Analytics, Benchmark Brands
As companies use more and more SaaS applications, a greater percentage of the data they have use and manage is already in the cloud. For example web
analytic data, credit bureau data and social media data as well as CRM and sales transaction data. This creates issues as well as opportunity.
An increasing number of the data sources that an organization needs to use to build predictive analytic models are thus available in the cloud. Where
previously organizations had on-premise solutions that contained all their customer, sales transaction, human resources, marketing and web data, now this data is often stored in SaaS and cloud-based solutions. In addition social media and other unstructured data are often available only through the cloud. The increasingly widespread adoption of “Big Data” technology is driven in part by a need to access and analyze the large volumes of new data available in the cloud.
This category of solution pulls all the data available in SaaS applications as well as third party web services into a cloud-based data management and modeling
environment. It pushes predictive analytic modeling to the data, given that the data is already in the cloud. For instance a company may bring all its internal data sources as well as the standard third party data it uses for enrichment into a cloud-based environment so that its whole analytic team can access it, and build models against it, from anywhere.
Modeling with the data cloud
13
Modeling with the data
cloud
Value
The value of this approach is that it saves on data transmission at the end points. Instead of data having to be pulled down into a modeling environment it remains in the cloud and is accessed and modeled there. This means that data is being moved cloud to cloud not cloud to end point, improving response and load times. By pushing the integration and cleaning of data into the cloud, it gets closer to cloud-based source data and becomes a more readily sharable asset. This allows analytic modeling teams to build models and do analysis on integrated data from multiple cloud sources without having to pull it all down on to a local machine.
It also makes it possible to build models against shared or pooled data without having to deal with downloading it to multiple locations. If multiple companies are using the same SaaS application then it is possible to create a pool of anonymous data that all can access to develop predictive analytic models. For instance, credit card issuers working with a cloud-based processor can develop fraud models using pooled data by analyzing the pooled data in a private cloud. This requires that modeling move to the data, in the cloud.
The intersection
Cloud-based solutions offer elastic storage and cloud to cloud data access. This makes it easier to pull together all the data needed for modeling and takes advantage of the flexibility of the cloud-based provisioning of new data space. In addition it allows access to a massively enriched set of data for modeling as more of the data used to enrich an organization’s own data is available in the cloud. As companies move increasingly to the use of SaaS solutions more of their core data will also be available only in the cloud, further increasing the value of this approach.
“Cloud applications are a great way to
augment internal company systems. For
example, a company that uses a customer
database of millions of customers may have
developed good internal systems for
operations and for a data warehouse. But
the company may be unsure of what
direction to go for an analytical platform to
support marketing and business intelligence.
It wants this capability, but is not sure about
the affordability of IT resources or the
resources needed to run the analytical
function in the marketing department. Cloud
computing can provide a solution.”
Doug Faherty, VP, Marketing Strategy Solutions, Clario Analytics
One final area of opportunity exists—taking advantage of the elastic compute power of the cloud to build predictive analytic models more efficiently.
14
“We have grown our sales by double digits
every year using Clario Stream. During that
time we invested in other marketing channels,
increased our prospecting and expanded our
product offering. The additional profits resulting
from Clario Stream helped fund that growth.”
Pete Bather, Vice President of CRM and Analytics, Benchmark Brands
When companies are building predictive analytic models the amount of compute power needed varies widely during the process so an elastic solution seems
inherently appealing. Building predictive analytic models in the cloud offers potentially infinite scaling.
Such a solution is designed to improve the creation and updating of predictive analytic models. The ability of clouds (private or public) to deliver elastic compute power is used for modeling activities that demand a lot of compute power. These solutions make it easy to add and provision new hardware as needed for modeling activities rather than requiring a pre-defined amount of hardware to be purchased, provisioned and configured. For instance when large datasets must be analyzed or when complex simulations are required to produce predictive analytic models, the team will need a lot more processing power then when they are analyzing results or investigating the data.
Elastic Compute Power for Modeling
15
Elastic Compute Modeling
Value
Building, updating and validating models can require a lot of compute power especially for things like Monte Carlo simulations and optimization. Taking advantage of new sources of data, especially when these sources involve very large amounts of data also requires a great deal of compute power or the data will have to be sampled to make it more manageable. This problem is especially acute if an organization is also taking advantage of new cloud data sources. Taken together these aspects of predictive analytic modeling can lead to big spikes in compute power and a cloud-based solution accommodates these intermittent spikes effectively. Modelers are also a key and constrained resource in most organizations. Using elastic compute resources can get them their results more quickly without having to invest in all the compute power they might need at a particular moment. The ability of elastic compute power to deliver rapid results without dedicated hardware keeps costs manageable while still supporting the kind of rapid iteration that is so important for good models. It is also true that in most organizations the existing operational hardware and data warehouse
infrastructure is being run at or near capacity. This can make it difficult for analytic modeling teams to try new techniques that require more compute power or to investigate new datasets—their existing infrastructure becomes a limiting factor on their ability to develop better models.
Of course budgets are not completely elastic so there needs to be a mechanism to manage the use of resources. In particular ways to predict future usage— predictive analytics about predictive analytics—would be useful.
The intersection
Elastic compute power for modeling takes advantage of the elasticity of cloud solutions to make predictive analytic activities more efficient. Not all the steps in the analysis, design, testing and deployment of predictive analytic models require the same amount of compute power. Using the elastic nature of cloud computing resources to ensure that those building predictive analytic models have the capacity they need when they need it makes for faster time to build predictive analytic models without a massive investment in hardware. Faster time to build models means more iterations and more effective analytic teams
16
“What’s most impressive? It’s the amount of
money (millions of dollars) that can be returned
to the company's bottom line using good
predictive analytics.”
Survey respondent
“Applying advanced analytic decisions to impact
my customer interactions without being limited
by where my data is and incorporating data
gathered to re-decision in real time.”
“Tools must be easy for my business teams to
use and understand the results, they aren't
sophisticated modelers!”
“Lower entry barrier for new application fields.
Less organisational overhead.”
Survey responses to “What other opportunities do you see from predictive analytics in the cloud?”
Rather than report survey results question by question the results, and their implications, have been grouped into a number of sections. Each section highlights significant results from the survey and discusses its implication.
Business solutions are what organizations need Predictive analytics are showing real strength Customers are the focus for predictive analytics
and cloud
Cloud-based predictive analytic scenarios are gaining momentum
Early adopters are gaining a competitive advantage
Decision Management matters to predictive analytic success
There are still some barriers and concerns with cloud-based predictive analytics
Industries vary in their adoption and concerns A mix of clouds is appropriate
Traditional data sources dominate predictive analytic models
After the survey results and implications are discussed we will make some recommendations and identify pros and cons of the various options. Demographics and vendor profiles complete the paper.
17
Survey Results and Implications
Figure 1 The impact of predictive analytic models on responding organizations
Business Solutions Needed
Most potential buyers of predictive analytics in the cloud are not really looking for “cloud” solutions. Years of successful industry adoptions of predictive analytics and growing awareness are resulting in more demand for analytically-based solutions. Yet many organizations are not looking for “predictive analytic” solutions either. What the vast majority of organizations seek is a solution to a specific business challenge. Predictive analytics can help them address the challenges they face. A cloud-based approach can make these solutions faster to deploy, more cost effective and more collaborative. Nevertheless the primary driver remains a need for a solution to a business problem.
One of the reasons for this is that companies realize that technology is not enough, they also need best practices and industry or solution-specific
implementation help. Few solutions are purely software-based; most involve configuration and specialization to work for a specific organization. This requires domain expertise as well as technical know-how. Similarly, successfully adopting solutions that embed predictive analytics often involves significant business change such as a willingness to establish control groups and
experimentation or to assign marketing priorities differently across the organization.
This solution focus goes back to some of the earliest “cloud-based” predictive analytic
solutions. Nearly 30 years ago shared, hosted applications for fraud detection and credit risk management were offered by credit card processors. A key driver for the adoption of these packages was a desire on the part of banks and credit card issues to get access to advanced analytic solutions as a packaged offering. This driver
remains front and center decades later.
Predictive Analytics Strong
This focus on business solutions cannot obscure the strength of demand for solutions based on predictive analytics. 82% of those surveyed work at companies that either have specific plans to adopt or are already using Predictive Analytics. Many, probably most, of these companies have gone through years of investment in increasingly sophisticated data infrastructure such as data warehouses and business intelligence. The value of this data infrastructure has always been in its ability to improve decision-making. Given that decisions affect the future and that the available data is all about the past, the potential value of this data can be increased if it can be used to make effective predictions that can shape decision-making.
The survey clearly showed that the use of predictive analytic models has started to make a real difference in organizations (Figure 1). Few (18%) have no plans to adopt predictive analytics while far more (43%) have already seen an impact from successful predictive analytic implementations. For an approach that has only recently become a mainstream topic in organizations this is an impressive showing. Perhaps even more impressive is that more than one in 10 of those responding (11%) say that this impact has been transformative.
18
Survey Results and Implications
Figure 2 Outcomes achieved or desired from predictive analytics
In terms of the number of models involved the vast majority of respondents had less than 10. Some were up in the 1,000s, however, with a core group of 20% having between 10 and 100 models in use.
Customers are the Focus
Survey respondents were given a number of different areas to identify their focus areas in both predictive analytics and cloud-based solutions. The top outcomes identified for predictive analytics (respondents could select more than one) were (Figure 2):
Improved targeting/allocation of resources (59%) Improved targeting/development of customers (58%) Improved targeting/acquisition of prospects (48%)
rounded out the top 3.
Managing non-credit risks and increased ROI from data scored over 40%. Interestingly credit risk scored 22% and fraud reduction only 28%. Given the long history of using predictive analytics for risk and fraud management these were somewhat surprising. They may reflect the rapid growth of predictive analytics in customer-facing activities or the widespread use of packaged predictive analytic models in fraud may mean it is just not top of mind for respondents.
Improved targeting and development of customers
scoring well was not a surprise but improved targeting/allocation of resources was equally widely cited and that was frankly unexpected. Some companies reported that although their initial interest is driven by demand-side or marketing operations, there is a significant interest in supply-side optimizations as well and that may account for the strong showing of resource allocation.
A similar question focused on the areas in which predictive analytic models were going to be adopted or were already in use. The top two areas were clearly marketing/customer acquisition and customer
retention—this was true whether “Likely to adopt” or just actual implementations plus specific plans were considered (Figure 4). Other areas that scored well included the broader category of customer
management, sales and cross-sell/up-sell. Fraud
detection scored quite highly on currently implemented but showed little growth with few plans, probably reflecting how well established predictive analytics is already in this area.
Among those experienced with predictive analytics and getting positive results it was much more likely that customer management and customer retention would be implemented and much less likely that planning and scheduling or operational efficiency would be the focus.
19
Survey Results and Implications
Figure 4 Areas where predictive analytics are being applied
Figure 3 Areas where cloud is being applied
A similar question focused on cloud adoption showed website, campaign management, and CRM scoring highest (Figure 3). Taken with the focus of predictive analytics above and it is clear that it is the effective acquisition, management and retention of customers that is the sweet spot for predictive analytics in the cloud. Given the demand from business people for solutions to improve customer acquisition and retention and the pressure to develop existing customers more effectively this is hardly surprising. The need for large numbers of often geographically dispersed staff to work on customer-facing
processes also contributes as
cloud-based options are very appealing in these kinds of tasks.
One other note on the focus areas for cloud. Platform as a Service or PaaS shows the biggest potential —it has far fewer using today but similar numbers for likely to use or more to the top categories thanks to a large group of “Likely to Use” shown in dark green in Figure 3. It is to be hoped that PaaS vendors will integrate predictive analytics more quickly than the developers of traditional application development platforms, most of which still lack this capability.
20
Survey Results and Implications
Figure 5 How important are the 5 scenarios
5 Scenarios Gaining Momentum
Five distinct opportunities for using predictive analytics and cloud were identified in the research and discussed in the survey:
Pre-packaged cloud-based solutions.
Solution offerings, delivered as cloud-based or SaaS solutions, that provide decision-making based on predictive analytics as a core feature of the solution. For example cloud-based applications offering next best action, offer selection, fraud detection or instant credit decisions.
Predictive analytics for SaaS.
The use of predictive analytics solutions that are cloud-based to inject predictive analytics into other software products that are cloud-based or delivered as SaaS. For example embedding customer churn predictions in SaaS CRM solutions or delivering risk predictions into cloud-based dashboards.
Predictive analytics for on-premise.
The use of predictive analytics solutions that are cloud-based to inject predictive analytics into disparate internal systems and multi-channel
environments. E.g. using cloud-based deployment to link an internally developed cross-sell offer service (that uses propensity to buy models) to multiple customer-facing systems.
Modeling with the data cloud.
The use of cloud-based predictive analytic solutions to respond to the increasing amount of relevant data available in the cloud rather than on-premise. For instance building predictive
analytic models in the cloud-based on customer purchase and behavior data stored in a SaaS CRM system as well as third party data available from a cloud-based web service. Elastic compute power for
modeling.
The use of cloud technology to provide predictive analytics solutions that can scale
elastically to meet demand. E.g. assigning extra resources dynamically when optimization
or other demanding algorithms are being used to build sophisticated predictive analytic models against large datasets.
All five of the scenarios were seen as potentially powerful solutions with over 2/3 of respondents
reporting that each of them has real potential (Figure 5). None of them are that widely adopted yet but pre-packaged analytic applications have the greatest penetration. Given the solution focus of many
customers, and the newness of predictive analytics for many organizations, this is not a surprise. Pre-packaged solutions are easier to link to business problems and generally involve less development and consulting work, making them easier to adopt.
The runner up was the use of cloud-based solutions to embed predictive analytics into on premise applications. This interesting result shows the importance of
deploying predictive analytics. It is not enough to build accurate predictive analytic models; organizations must also get these models into their (legacy) applications.
21
Survey Results and Implications
Figure 6 How widely adopted are the 5 scenarios
Figure 7 How likely are the 5 scenarios to be adopted
While none of the solution types are widely adopted among the respondents, pre-packaged solutions and the use of the cloud to embed predictive analytics into on premise systems as well as the use of the cloud to embed
predictive analytic models into SaaS systems all had some level of adoption in over 50% of respondents (Figure 6). A
deployment orientation seems to dominate predictive analytics in the cloud at least for now—
pervasiveness through the cloud seems more important than scalability and data availability. It is also true that the number of solutions for building predictive analytic models in the cloud is smaller than those for deploying and using the models once constructed and this must have slowed adoption.
Only modeling in the cloud (using data in the cloud) failed to get over 50% of respondents to say it was at least somewhat likely to be adopted (Figure 7). It got almost the same “Very Likely” score as the two options for embedding predictive analytics in existing systems but fell behind when those saying “Somewhat likely” were also included. Pre-packaged solutions that include predictive analytic models and the use of the cloud to
embed those models in existing systems did the best with elastic compute also showing well when those somewhat likely to adopt are considered. Given the lower scores for modeling in the cloud this implies that on-premise grids and private clouds are being
considered to provide a more elastic compute environment for model construction.
Early Adopters, Competitive Advantage
When a new technology is being adopted, organizations must choose whether to be leaders or laggards. They must decide if they will adopt the new technology before it is obvious whether it will be successful or notrather than wait for more data. There are clear risks in being an early adopter of any technology and these must be balanced against the potential competitive advantage. This balance is different for each wave of technology.
22
Survey Results and Implications
Predictive Analytics for SaaS
Survey Results and Implications
Figure 9 Percentage rating scenarios as very likely to deploy
Figure 8 Percentage rating scenarios as very important
In the case of cloud-based predictive analytic solutions, this balance seems to come down firmly on the side of being an early adopter. A number of factors suggest that the gap between these early adopters and their competitors is likely to widen as early adopters see more positive results, have fewer objections and are determined to adopt the technologies more aggressively. For instance:
Organizations with the most experience with predictive
analytics were more likely to have plans to adopt more cloud-based predictive analytic solutions. They were much less likely to have performance or
privacy concerns about the solutions.
They were more likely to embed predictive analytics in operational systems, a driver of positive results discussed earlier, and to recognize and value the deployment agility and cost reduction inherent in cloud-based predictive analytics solutions. They were more likely to take advantage of “big
data” from the cloud with 60% seeing themselves as very likely to use cloud data in models.
Comparing those respondents with no predictive analytics in the cloud solution widely deployed with those that have at least one widely deployed we see a
dramatic increase in the degree of estimated
importance every one of the scenarios (Figure 8). They were twice as likely to rate cloud-based predictive analytics as very important with pre-packaged solutions getting a 3x rating improvement.
It is also clear that those already deploying at least one cloud-based predictive analytics solution are much more likely to adopt solutions going forward (Figure 9). They say they are very likely to deploy the solutions at 4x the rate of those who have not yet started. Pre-packaged solutions were rated as very likely to be adopted at more than 5x the rate.
These early adopters contrast with those not using cloud-based predictive analytics yet who see themselves as much less likely to adopt any of the solutions of any kind. This was even true of packaged solutions, the most
preferred option for those who have not yet adopted any solution.
Taken together this implies that those already adopting cloud-based predictive analytic solutions are not only getting positive results but that these results make them more likely to accelerate and broaden their
23
Survey Results and Implications
Figure 10 How predictive analytic models are used
adoption of these solutions. Those hesitating about adopting them run the risk that they will be left behind, watching early adopters establish a lead that grows with time.
Decision Management Matters
Decision Management was clearly an important element for successful analytic adopters. Respondents were asked how they used predictive analytics in their decision making. Four options were presented—
predictive analytics provide occasional insight, predictive analytics are in regular use, predictive analytics are the primary driver for some decisions, and predictive analytics are embedded in operations. Overall respondents were most likely to say that predictive analytics were providing occasional insight or that predictive analytics were being tightly integrated in their operational systems—both at 28%. This last option is the basis of Decision Management Systems that embed predictive analytics in automated decision-making systems.
However the results become more interesting when considering those who have already seen significant positive results from predictive analytics. For these the percentage tightly integrating predictive analytics into operations rose while occasional use dropped (Figure 10). Among those transformed by predictive analytics a
whopping 2/3 (64%) said they tightly integrate their
predictive analytics with day to day operations—a clear illustration of the power of decision management to transform organizations with predictive analytics. As the graph below shows the initial impact often comes from occasional use of predictive analytics but as more impact is reported so the likelihood that predictive analytics are used in a more operational context grows. It should be noted that many of these operational integrations are still batch-oriented today. This is rapidly moving to more interactive systems, one analytic decision at a time in real-time, and this change will require even tighter operational integration in the future.
One other point should be made about early adopters. When asked about the need to understand predictive analytic models in a solution 58% of all respondents said it was essential to understand and control how the models are developed and tuned. This rose among those with more experience (rather than declining). It rose to 80% for those transformed by predictive analytics. The result could mean that experience does not make people any more comfortable with “black box” models. However it is possible that experience with predictive analytics makes clear the value of really understanding what is predictive and that this drives business innovation. As a result the increasing demand for transparency may not be an issue of trust so much as a perspective on the value of understanding.
24
Survey Results and Implications
Still Some Barriers and Concerns
While there is a clear enthusiasm for and optimism about these solutions there are still some barriers and issues as companies consider predictive analytics in the cloud. Data security and privacy was clearly the top issue with 65% listing it as very important. Regulatory issues and concerns with complexity and responsiveness were also significant. Interestingly those who had already had positive results worried a little less about data security and privacy, about complexity, and about latency and responsiveness. Only regulatory concerns remained as strong. Familiarity clearly results in at least a little less concern about these issues.
From conversations with companies, these concerns are particularly strong in those industries where the core data required for predictive analytics is regulated data. It is also clear that some of the variation in cloud choices (public, private or hybrid) is driven by these concerns with private clouds, for instance, being prioritized where the data involved is sensitive or where
responsiveness is critical. It should be noted that many organizations’ thinking about cloud solutions is still in its infancy as noted below.
Industries Vary
There was significant variation across industries in the results:
Retail banks and other financial services much more likely (2x) to have had significant or transformative impact. Given the long history of analytics in these industries this was to be expected.
These two industries were much more likely to focus on credit risk, other risks and fraud though customers are still #1. These are risk-centric industries so a stronger focus on risk and fraud is unsurprising. The financial services industry is trying to adopt a more customer-centric (rather than account or product-customer-centric)
view of their business also, however, so the focus on customers remains strong.
As you would expect given their results, these organizations were more likely to tightly integrate predictive analytics into operational processes. Decision Management is well established in these industries.
Telcos have aggressive plans with 67% having specific plans to adopt predictive analytics. Healthcare delivery organizations meanwhile
are clear laggards with over 40% with having no plans at all for predictive analytics.
Among the five scenarios, Financial Services and Banking are much more likely to adopt both elastic compute and modeling with cloud data. As noted in the recommendations section, this reflects their maturity level. In addition they show a generally greater sense that predictive analytics in the cloud is going to be important and a higher likelihood of adoption across the board.
Financial Services and Banking organizations are significantly more focused on deployment agility and cost reduction than is typical.
Financial Services and Telco are more concerned about privacy and security.
Retail banking respondents are big believers in pooled data, unsurprising given the strong showing for pooled data in helping build predictive analytic models for predicting credit risk and credit card fraud.
25
Survey Results and Implications
Figure 11 Outcomes achieved or desired from cloud solutions
A Mix of Clouds
Cloud-based solutions can be public, private or hybrid solutions. Respondents were currently weighted to private clouds (55% on average) with few using hybrid public/private clouds (<10%). Over the next few years the respondents expected to see a clear drop in private clouds, down by an average of 9% while hybrid clouds rose 13% and public clouds rose 6%. This likely reflects a growing comfort in public cloud solutions as well as a
broadening array of public cloud offerings as vendors move from offering only private cloud solutions to true public cloud, on-demand solutions. Of those using cloud-based solutions, about 32% are still using them tactically while 34% have well defined guidelines and another 34% are developing such guidelines. This focus on standardization implies that cloud is solidly on the IT department’s agenda even if impact has been limited so far.
In general the solution itself is the main driver with the use of cloud, and the type of cloud, being largely a secondary issue. Once it is understood that a solution is cloud-based, the value of the cloud in being able to “get around” IT barriers and get to value quickly is clearly appreciated, however. Conversations with organizations suggest that most private cloud users are trying to connect disparate elements of their business to improve collaboration and sharing. Public clouds are more popular once predictive analytic models are developed and when predictive scores and decisions need to be widely distributed.
Many vendors report being asked about hybrid clouds such as with an on-premise appliance being used to deliver a private cloud while using the public cloud to provide some cloud-based management and results distribution. The line between public and private clouds
is clearly still in flux and it is not clear where the line will ultimately be drawn. Within an organization their choice of public, private and hybrid is largely driven by the need for privacy and information security with public cloud being selected for lower risk scenarios. Overall responsiveness of the solution (private clouds can be more responsive with lower latency) and the time/cost to upload data also play a role in selection. Of note, almost half of those responding see cloud primarily as a way to reduce demand on internal IT (Figure 11). This is perhaps an indication that many organizations are not yet internalizing cloud as a source of business solutions not just IT infrastructure and that could be limiting the impact of cloud on organizations. Other results commonly discussed as benefits of cloud computing scored well also including replacing capital expenditure (on hardware and software) with operating expenditures, improving business user control of IT and reducing TCO.
Interestingly transaction based pricing did poorly which was somewhat unexpected—transaction pricing has a lot to offer, especially in a “decisions as a service” cloud-based solution but it’s not apparent that it is a big driver amongst those taking the survey.
26
Survey Results and Implications
Figure 12 Percentage rating different kinds of data as important
Traditional Data Sources Dominate
The analytics community is abuzz with discussions of “Big Data”, unstructured data or text analytics and new data sources beyond the traditional structured data managed in an organization’s data infrastructure. Despite this buzz, respondents were much more focused on the effective use of structured data when building predictive analytic models in the cloud.Structured data from other cloud sources such as SaaS was the most important for building predictive analytic models in the cloud followed by pooled data
(structured data from multiple companies pooled for analysis) and structured data uploaded from on-premise solutions to the cloud. Unstructured data was just not as highly rated.
This became more pronounced when you consider only organizations that already have some successful results from predictive analytics—those with some positive impact already. These more experienced organizations ranked structured cloud and pooled data as more important and unstructured data sources as less important (Figure 12). Success with predictive analytics seems to focus organizations on the effective use of structured data much more than on new, less structured data sources.
A case could be made that companies would be better investing in consortia to manage pooled data to improve their models rather than in trying to include
unstructured data in their analysis. While it is trendy to focus on unstructured sources the evidence from successful implementers of predictive analytics clearly ranked pooled data as the more useful resource. Pooling data across multiple organizations allows for predictive analytic models to be based on a larger sample and, generally, a sample that is more representative of the overall population.
Besides the kinds of data being included the survey asked about different styles of data access. How important, survey respondents were asked, is real-time or near real-time data relative to batch and more static data? This too showed an interesting difference
between those with more experience and those with less.
Overall near real-time and real-time data slightly out performed batch and static data in terms of their usefulness for predictive analytic model development, though all four categories scored highly. However when those without positive results from predictive analytics are compared with those who have had positive results there are some clear differences (Figure 13). Near real-time data was slightly more important, static data was somewhat more important and batch data was
significantly more important among those with more
experience and positive results. Real-time data actually dropped when experienced users of predictive analytics were considered. Real-time data, it seems, is widely expected to produce better results by those with limited experience. Those who have successfully built and deployed models seem to show that this is not necessarily true,
27
Survey Results and Implications
Figure 13 Importance of data timeliness to predictive analytic models
however, showing a stronger preference for batch data. Taken together this implies that organizations can get started with predictive analytics, and get positive results from using predictive analytic models, even if those models are built only from structured data in a batch environment. Organizations don’t need to wait until they can also manage unstructured data sources or manage their data in real-time before they can be effective users of predictive analytics.
In the next section we will explore recommendations for organizations at different levels of maturity as well as the pros and cons of cloud-based predictive analytics solutions.
28
“Predictive Analytics has not hit the mainstream
yet because most businesses see at as a
fairground voodoo magic trick to be marveled
at, and then go back home to regular business.
For those that see potential the entry barrier
can seem too high so that only the largest and
the bravest businesses venture forth. Using a
cloud-based model to address Predictive
Analytics may be the solution to this vicious
cycle,”
Gagan Saxena, CIO Apple Vacations, a survey participant
The basic value proposition of predictive analytics in the cloud is clear: organizations can make predictive
analytics more scalable, more pervasive and easier to deploy using cloud technologies. As more organizations seek competitive advantage through analytics, their ability to rapidly make analytics pervasive and to tightly integrate analytics into their business strategy and day to day operations is going to be a critical factor. For many of the challenges such organizations face in their journey to becoming analytic organizations, cloud-based solutions have much to offer.
The research and the survey results make it clear that organizations should make cloud-based Predictive Analytics part of both their development approach and their deployment architecture. Using cloud-based predictive analytic solutions to develop predictive analytic models makes it easier to adopt new data sources, especially cloud-based “Big Data”, while improving the effectiveness of scarce modeling experts by making the power they need available on demand. The pervasiveness of the cloud and the ability to define simple, standard interfaces make it a compelling deployment platform for predictive analytic models, helping to put predictive analytics to work throughout an organization’s operational systems and processes. The reality is that organizations are at different stages in their adoption of predictive analytics and they must pick the right approach for their level of sophistication. They
Recommendations
29
Recommendations
can begin their journey with pre-packaged solutions, use based deployment to expand and adopt cloud-based development to scale their predictive analytic operations. A blend of on-premise, private and public cloud solutions can be found to match their privacy, security, regulatory needs. Organizations also need to be aware of the pros and cons of cloud-based solutions so that they can maximize their advantages while minimizing their risks.
Three Stages of Adoption
The different kinds of solutions available under the umbrella of predictive analytics in the cloud allow organizations at every level of sophisticated to adopt cloud-based predictive analytics. Organizations can use predictive analytics in the cloud to jump-start their adoption of predictive analytics, speed and support expansion or refine an already sophisticated approach. The different solutions also allow different parts of an organization to share a focus on the cloud, and potentially cloud resources, while still moving forward in a way that makes sense—allowing more sophisticated or experienced departments to have a different
adoption strategy to those with no prior predictive analytics experience.
Before adopting cloud-based predictive analytic solutions organizations should understand where they fall on the maturity curve. A first group is those just getting started with predictive analytics, with little or no prior experience. A second group is those with some prior experience building and using predictive analytics that have not yet made predictive analytics widespread in their organizations but wish to do so. The final group is those widely using predictive analytics and looking for ways to make their approach more effective.
New to Predictive Analytics
Organizations that are new to predictive analytics can get a jump start on becoming a more analytic
organization using pre-packaged cloud-based solutions. These “Decision as a Service” solutions are focused squarely on being a complete solution to a business problem, making them easier to budget for and justify even in organizations with no prior history of success with predictive analytics. These solutions typically have very simple interfaces, minimizing or even reducing to zero the need to bring IT resources to bear on adoption. By returning a simple list of the right customers to contact, a fraud flag for a transaction or the best offer to make a customer, they make it easy to improve your day to day operations using analytics. Because the analytics are embedded in a solution, organizations don’t need any analytic expertise to adopt them.
Early Adopters of Predictive Analytics
For organizations that have already started to see some value from predictive analytics, but are not yet widely adopting analytic decision-making, cloud solutions offer the potential for rapid expansion. The only way to get value from predictive analytics is to act on it. Therefore the analytic insight you generate must be embedded into your organization in the form of improveddecision-making. The pervasiveness of the cloud makes it easy to connect any existing operational process or system to a cloud-based predictive analytic solution. Standard and simple interfaces are the norm in cloud solutions, making integration easier and reducing the need for extensive IT work. Cloud-based solutions also support the extended enterprise by allowing integration with systems managed by the organization as well as those being used by its channel and supply chain partners.
Both cloud-based predictive analytics for SaaS and cloud-based predictive analytics for on premise systems
