BIG DATA FOR YOUR DC
Nov 2014 AK Schultz
Page 2
It is data.
And it is BIG.
Page 3
Big Data Is:
These are data sets so
large and complex …
That it becomes
difficult to process…
Using
traditional data processing
applications
Page 4 Swisslog Confidential
90%
of ALL the
data
in the world
has been generated over the
last 2 years
!
But How Big?
Page 5 Swisslog Confidential
DATA TODAY
Page 6 Swisslog Confidential
Let’s Give You A Sense of Scale
Page 7 Swisslog Confidential
Then from 2003 to 2012…
Page 8 Swisslog Confidential
And now…
8 ZB = 8 Trillion GB
Page 9
Page 10
DATA-WISDOM CONTINUUM
DATA INFORMATION KNOWLEDGE WISDOM DECISIONS
§ Facts § Gathering § Research § Presentation § Organization § Gives Meaning § Application § Synthesized § Learning § Understanding § Interpretation § Actionable § Change § Movement § Optimization
THE PAST THE FUTURE WHY? Reveals Patterns WHAT? Reveals Relationships WHAT IS BEST? Reveals principles WHAT ACTION? Reveals direction
Page 11
In supply chain, what do we care about?
§
Enterprise Application
Data
§
Automation Sensor
Data, aka, the “Industrial
Internet”
Page 12
Page 13
§
A typical warehouse storing pallets will generate 1 to 2 GB
per mos in its WMS.
§
A typical retail distribution ASRS dealing primarily in case
handling will generate
– ~25 times the data for the same case volume (25 to 50 GB per
mos in its WMS).
– 5 to 10 GB per mos of industrial internet data. – 3 to 5 GB of data in a data warehouse.
§
This means we are likely not leveraging 300 to 660 GBs
annually per site.
§
This is not so much data.
Examples from Warehousing
Page 14
COST OF STORING BIG DATA
§
1 TB (Solid State) is <
$500
§
Less if you buy lots of
servers
§
What is there to
discuss?
Log first. Ask questions later.
1980 $190,000 1990 $9,000 2000 $15 2010 $0.07
Page 15
Valuable Data is Thrown Away
Waterfall versus AGILE
“Just tell me what you want to keep and give us
the business case to justify the server space.”
“I am not sure what I want to keep, but I will know
when I see it.”
“We can’t get help to build reports, but they we
send us a flat file. So we will just build
spreadsheets.”
“I spend more time building these spreadsheets
and charts that no one seems to care about.”
Page 16
Just Say No!
§ These documents are a dead end of knowledge
§ Expensive to create
§ You are not saving hard drive space but simply sprinkling to data around
§ Can be created by few. § Distributed in a clumsy way
§ Accessible by few, and not usually by those who can take action
§ Viewable to the who can take action, most likely on print outs in break rooms
Flat Files
Excel
Macros
Access
Page 17
Why Today is a New Day
Internet of Things
Low Cost
Bandwidth
Processing
Power &
Memory
Cloud
Computing
Unstructured
data storage
High leverage, data driven, actionable insights
Social
Media
Large
Scale
Enterpise
Systems
eCommerce
Industrial
Ethernet
Page 18 Swisslog Confidential
ENTER THE DATA LAKE
Data lake for structured and unstructured data
DATA LAKE
Traditional
Structured
Data
Raw Unstructured Data
• Allows for the distributed processing of large data sets across clusters of computers
• It is designed to scale up from single servers to thousands of machines
NoSQL!!!
Page 19 Swisslog Confidential
Data Lake vs Data Warehouse
Benefits
§ Generally less expensive per GB
§ Variety – better equipped to everything from RDBMS to Video Feeds
§ Volume – Traditional data warehouses can bog down. § Velocity – Distributed
processing enables faster storage and processing
§ Uses MapReduce to split data in chunks that can be process in an parallel manner
Page 20
Industrial
Ethernet
1.
Inputs
2.
Outputs
3.
Scanners
4.
….
DATA SOURCES
For a Warehouse
ERP
1.
Financial
2.
Production
3.
…
WMS
1.
Order Data
2.
SKU info
3.
…
LMS
1.
Labor
Standards
2.
Performance
3.
…
DATA LAKE
CMMS
1.
Maint Data
2.
Parts
3.
…
Imagine this data all living in completely separate data
warehouses.
Page 21 Swisslog Confidential
A Way to Help Find the Unk-Unks
Most of it does not matter. Some of it is crazy important.
Page 22 Swisslog Confidential
WE ARE SEARCHING FOR CAUSALITY
CAUSE
EFFECT
TIME
SPACE
RLTNSHP
Page 23 Swisslog Confidential
CAUSE
EFFECT
TIME
SPACE
RLTNSHP
BIATHALON EFFECT
THE DATA LAKE DOES NOT CARE
Time, Geography, Org Charts, Politics
Page 24 Swisslog Confidential
You Need to Do Something With the Data
But WHO is WHO?
DOMAIN
KNOWLEDGE
DATA
SCIENCE
TECHNOLOGY
Page 25 Swisslog Confidential
You Need to Do Something With the Data
But WHO is WHO?
DOMAIN
KNOWLEDGE
DATA
SCIENCE
TECHNOLOGY
Page 26
HOW SHOULD YOU DISPLAY DATA?
Customer
WMS / ERP / LMS
SmartLIFT
SmartLIFT
DASHBOARD
SmartLIFT
DRIVER GUI
Page 27
BIG DATA INTELLIGENCE
SmartLift: Big Data Meets Forklift
Cockpit:
§
A web-hosted business
intelligence tool
§
Interactive and user
configurable charts
§
Threshold based email
alerts
§
WIDGET BASED!
§
It is not possible to
create on GUI that meets
every employee’s needs
on a 15” monitor!
In order for Big Data to be actionable, you need to provide:
Right information, to the right people, at the right time.
Page 28 Swisslog Confidential
General Guidelines
Actionable
information
Right info, right
person, right
time.
Predict
Failure
Resolve before
Impact
Forecast
throughput
volume
Historical
Real-time
Predictive
Da
ta
A
gg
re
g
a
tion
Age of Data
Line Mgr
Executive Mgr
1
2
3
Page 29 Swisslog Confidential
Swisslog Data-Wisdom Solutions
DATA INFORMATION KNOWLEDGE WISDOM DECISIONS
§ Facts § Gathering § Research § Presentation § Organization § Gives Meaning § Application § Synthesized § Learning § Understanding § Interpretation § Actionable § Change § Movement § Optimization
Cockpit
ConditionMonitoring
CrystalBall
Page 30
§
What Big Data is NOT:
– Big data is NOT a panacea; it will not magically make all of your
business problems go away.
– Big data is NOT a replacement for relational database
management systems—at least, not today … maybe in a few years when query performance radically improves (or new methods arise, e.g. Stinger).
– Big data solutions are NOT simple; spinning up an HDInsight
cluster is a breeze, but then what?
– By its very definition, you are dealing with vast amounts of data,
most of which is unstructured.
– There is a lot to wade through to find the nuggets of relevant
data; there is no easy way to perform this operation.
Page 31
§
Start storing more of your data! Log first, ask questions
later!
§
Begin to turn your data warehouses into data lakes.
§
Build a Big Data Team (Domain, Data Science,
Technology). Likely you will need to outsource some of it.
§
Master Information and Knowledge before trying to get into
Wisdom.
§
Think AGILE not Waterfall. There are many Unk-Unks. A
waterfall approach will likely get bogged down.
§
Start small and grow.
Page 32
§
Technology convergence makes this an exciting time.
§
Things we wanted to do 10 years ago are now technically
and financially viable.
§
I am fully confident that if you seriously embrace Big Data,
the business upside is tremendous.
Page 33
