) 7 1 0 2 E II A ( g n ir e e n i g n E l a ir t s u d n I d n a e c n e g il l e t n I l a i c if it r A n o e c n e r e f n o C l a n o it a n r e t n I d r 3 7 1 0 2
8 7 9 : N B S
I -1-60595- 05 -9 2
f
o
t
e
n
r
e
t
n
I
n
i
m
a
e
r
t
S
a
t
a
D
e
v
i
s
s
a
M
f
o
m
e
t
s
y
S
g
n
i
s
s
e
c
o
r
P
e
m
i
T
l
a
e
R
m
r
o
t
S
e
h
c
a
p
A
n
o
d
e
s
a
B
s
g
n
i
h
T
G
N
A
W
e
i
J
,
S
h
u
a
i
Z
H
A
O
a
n
d
B
o
C
H
E
N
G
g n i h c ti w S d n a g n i k r o w t e N f o y r o t a r o b a L y e K e t a t
S Technology,BeiijngUniverstiyof
a n i h C , g n ij i e B , s n o it a c i n u m m o c e l e T d n a s t s o P
: s d r o w y e
K IoT,ApacheStorm ,Kafka,Massivedata ,Rea ltimeprocessing.
.t c a r t s b
A ThedataofInterne tofThingshasthecharacteristicsofrealtimea ndlargescale .Forthe l
a e r , a t a
d -timeprocessingi susuallynecessary ,especiallyi nsomesecurityscenarios,t hispapert ries l
a e r e h t t r o p p u s n a c h c i h w m e t s y s a n g i s e d o
t -timeprocessingoflargedataflow .Thispaperfirstly t
n i w o l f a t a d e h t f o s c i t s i r e t c a r a h c e h t s e z y l a n
a heInterne tofThings ,andbyusingt heresutlsoft he s m e l b o r p n i a m e h t d n a m e t s y s e h t n i d e s u e b n a c t a h t s e i g o l o n h c e t y e k e h t d r a w r o f s t u p I , s i s y l a n a
d f o s t c e p s a e e r h t m o r f l i a t e d n i d e n g i s e d I m e t s y s e h t f o e r u t c e ti h c r a e h t e b i r c s e d I , y ll a n i F . d e c a
f ata
. e g a r o t s a t a d d n a g n i s s e c o r p a t a d , s s e c c a
n o it c u d o r t n I
s g n i h T f o t e n r e t n
I (IoT) is anetwork with three layers of perception layer ,transpor tlayerand e c i v e d g n i s n e s n o i t a m r o f n i e h t h g u o r h t t e n r e t n I e h t h t i w g n i h t y n a t c e n n o c n a c t I . r e y a l n o it a c i l p p
a on
d n a e g n a h c x e n o it a m r o f n i r o f e l b i s n o p s e r y l n i a m s i r e y a l t r o p s n a r t e h T . r e y a l n o it p e c r e p e h t
, y l e t a m i tl U . n o i t a c i n u m m o
c w e can reailze intelilgen t identification , positioning , tracking , .
r e y a l n o it a c i l p p a e h t n o t n e m e g a n a m d n a g n i r o t i n o m
I m o r f a t a d e h
T nterne tofThingsmainlyhast wocharacteristics ,namelyl argescaleandrealt ime . e h t , d n o c e S . e c n e g il l e t n i n i e l o r a y a l p t c e j b o e h t e k a m n a c a t a d e h t , e l a c s e g r a l e h t h t i w y l n o , t s r i F
l a e r s i a t a
d -timeand need to beprocessed inrea ltime .Through theembedded orattached sensing m
r o f n i l a n r e t x e d n a s m e t i n o e c i v e
d ation acquisition technology ,IoTsystemcanreflec tthestateof , y t i d i m u h d n a e r u t a r e p m e t , s u o r e g n a d r o e f a s , g n i v o m r o y r a n o i t a t s g n i d u l c n i , e m i t t r o h s y r e v e s d o o g
s u y l e d i w e b n a c t I . c t
e edi nvariousscenesofrealtiy ,suchasenvironmenta lmontioring ,community .
c t e , y t i r u c e s
l a e
R -timecomputingi susuallycarriedou tformassivedata .I trequiresareal-timeprocessingand l
a e r e h t t e e m o t r e d r o n I . a t a d e h t o t e s n o p s e
r -timerequirementsoft hesystem,t heresponsei susually .
s d n o c e s n
i Rea ltime computing generally consistsof threestages :datageneration and collection .
e s a h p e c i v r e s e d i v o r p o t e g a r o t s , e s a h p g n i s s e c o r p s i s y l a n a d n a n o i s s i m s n a r t , e s a h p
R )
1 ea litmedataacquisiiton :Thispar tshouldguaranteet hecompletecollectionofalll ogdataand l
a e r e d i v o r
p -timedataforreal-timeapplications .Att hesamet ime ,weshouldensuret hatt hesystem e
h t ,t s a l t A . t n e m y o l p e d y s a e , n o i t a r u g i f n o c e l p m i s s w o l l a d n a e l b a i l e r , e l b a t s s
i responset imeshould
l a e r e
b -timeandl owl atency .A tpresen,tt hemainproductsareScribe ,Kafka ,FlumeandTimeTunnel , . d n o c e s r e p n o i s s i m s n a r t d n a n o i t i s i u q c a a t a d g o l B M f o s d e r d n u h f o s d e e n e h t t e e m l l a n a c y e h T . c t e
. s t c e j o r p e c r u o s n e p o l l a e r' y e h T
)
2 Rea ltimedata computation: In thechanging datastream ,real-timeanalysisiscarried ou tto .t
u o t l u s e r e h t d n e s o t d n a r e s u e h t o t l u f e s u e b y a m t a h t n o it a m r o f n i l u f e s u e r u t p a
c Thecurren tmain
b i r t s i d a s i m r o t S . a m u P d n a m r o t S , 4 S e d u l c n i s t c u d o r
p uted ,fault-toleran treal-time computing r o f t r o p p u s t i d n a , e s a b a t a d e h t e t a d p u d n a n o i t a m r o f n i g n i s s e c o r p r o f d e s u e b n a c t I . m e t s y s
w o l f f o m r o f e h t n i t n e i l c e h t o t s t l u s e r n r u t e r d n a , s m a e r t s a t a d r e v o s e i r e u q s u o u n i t n o
c [ 1].
)
s e it l u c if fi D
e g r a
L ScaleD aa t
t p a d a o t r e d r o n i ,t e n r e t n i d n a y g o l o n h c e t n o i t a m r o f n i f o t n e m p o l e v e d d i p a r e h t f o d n u o r g k c a b e h t n I
, r a e p p a s e p y t r o s n e s e r o m d n a e r o m , s o i r a n e c s t n e r e f f i d o
t andsensorperformancehasbeengreatly n a s a h m e t s y s n o i t i s i u q c a a t a d e h t y b d e t c e l l o c a t a d e h t f o e z i s e h t w o n o S . e r o f e b n a h t d e v o r p m i
l a e r e h t , e r o f e r e h T . h t w o r g e v i s o l p x
e -timecomputing system needsto dea lwtih themassive data r
e t n I e h t y b d e t c e l l o
c ne tofThingssystem .Howto mee tthereal-timeprocessingbasedonmassive .
h t i w t l a e d e b o t m e l b o r p t n a t r o p m i y r e v a s i a t a d
l a e
R -itmeProces isng
l a e r n i d e t c e l l o c s i a t a d e s u a c e
B -timei nInterne tofThings ,weneedt omaker eal-timeprocessingand o
p s e
r nse .With the enormous scale of data ,how to improve the speed of data processing is the .
r e d i s n o c o t d e e n e w t a h t m e l b o r p
y ti r u c e S
e h t n i , e l p m a x e r o F . n o i s s i m s n a r t a t a d f o y ti l i b a i l e r e h t e r u s n e o t d e e n e w , n o i t a u t i s y t i r u c e s h g i h n I
s y t i r u c e s y t i n u m m o
c ystem ,weno tonlyneedtoachievereal-timedataprocessing ,bu talsoneedto o S . a t a d f o e c e i p e l tt i l a n i r a e p p a y a m a t a d e v i t i s n e s e h t f o e s u a c e b , t s o l t o n s i a t a d e h t t a h t e r u s n e
t n a t r o p m i y r e v s i n o i s s i m s n a r t e h t a t a d f o y t il i b a i l e r e h t e r u s n e o t w o
h .Int hesameway ,weneedt he t i n e h w d e d r a c s i d n a h t r e h t a r d e h c a c e b n a c a t a d e h t t a h t e r u s n e o t m s i n a h c e m g n i h c a c e t a i r p o r p p a
g n i s s e c o r p n a h t r e t s a f s e m o
c .
TimingSequence fo D aa t
n i a t a d d e h c a c e h t t e g o t w o h r e d i s n o c o t d e e n o s l a e w , y t i r u c e s s e d i s e
B timesequencef orprocessing . p e e k o t d e e n e w d n a , n o i t a m r o f n i f o d n i k a s i t i e s u a c e b , t n a t r o p m i y r e v s i a t a d f o e c n e u q e s e m i t e h T
. e t a r u c c a e r o m s i s y l a n a a t a d e k a m o t n o i t a m r o f n i s i h t
l a e
R -itmeQuery
l a e r e h t t e e m o t w o H ? a t a d e v i s s a m e r o t s o t w o
H -timerequirementsofvariousqueries?Thesetwo e g r a l t r o p p u s o t m e t s y s e s a b a t a d e l b a n o s a e r a n g i s e d o t d e e n e W . t u o b a e r a c e w t a h w e r a s n o i t s e u
q
-l a e r e h t t e e m d n a y r e u q d n a e g a r o t s a t a d e l a c
s -timerequirementsoft hereal-timeprocessingsystem.
e r u t c e ti h c r
A De is gn
n g is e
D a Pnd rinciples
l a e r y h w t u o b a s n o s a e r t n a t r o p m i y r e v o w t e r a e r e h
T -timeprocessingt echniquesareusedt oprocess .
s g n i h t f o t e n r e t n I e h t n i a t a d
e g r a L . e v i s s a m s i s g n i h T f o t e n r e t n I e h t m o r f d e r i u q c a a t a d e h t s i e n
O -scaledatagenerallyhasfour t
s i r e t c a r a h
c ics :large ,fast ,miscellaneous and suspected .I thas thecharacteristics of theflow ,and l
a e r f o e c n a m r o f r e p h g i h r o f s e r i u q e
r -time .Aslongastheprocessingisfas tand correct ,thesedata S
. e l a c s r i e h t o t d e t a l e r e u l a v e g u h g n i r b n a
c ot heaccuracyandreal-timepropertyplayveryi mportan t l
a e r e h t d n a , m e t s y s e l o h w e h t r o f s e l o
r -timeprocessingtechnologyisj us tforthefas tandreal-time .
a t a d m a e r t s f o g n i s s e c o r p
e t c a r a h c d e t c e p s u s d n a s u o e n a ll e c s i m , e g r a l e h t s i n o s a e r d n o c e S e h
T risticsofdatadeterminet ha t l
a e r f o r e t s u l c e h t d n a , h t i w l a e d o t s e c r u o s e r g n i t u p m o c f o t n u o m a e g r a l a d e e n y e h
t -timeprocessing
h c i h w s e c r u o s e r g n i t u p m o c e g u h e d i v o r p n a c m e t s y
s cannotbeprovidedbysinglemachine.
l l a r e v
O Technica lArchtiecture
e h
T system is mainly divided into severa lparts ,data acquisition ,data access ,data cache ,data .
s m d f
@ 0
s m d f
@ 1 Bnkkdbsnq 'j`ej`oqnctbdq(
Zsnohb0\
` j e ` J Zsnohb1\
q d o d d j n n y
c d y h l n s r t B
s t n o R 0
'j`ej`bnmrtldq( s
k n
A 0
s k n
A 1
c d y h l n s r t B
s t n o R 1
s k n
A 2
s k n
A 3
d r ` A ` s ` C
l d s r x R
x f n k n o n
S 0
g i
F u 1 .re Systemarchitecture.
n i a
M Components a
t a
D Acqui isiton and D a Ata ccess .Data acquisition is accomplished by a variety of sensing : w o l e b 2 e r u g i F n i n w o h s s i e l p m a x e n a d n a , s g n i h T f o t e n r e t n I e h t f o r e y a l n o i t p e c r e p e h t n i s e c i v e d
y a W e t a
G GateWay
C
S C
V
e c i v e D 3
e c i v e D 1
e c i v e D 2
m n h s o d b q
d K`xdq Sq`mronqsK`xdq
. . .
e
m
u
l
F
g i
F u 2 .re Dataacquisition. n
e e h t f o e c n a r t n e e h t s i t r a p s s e c c a a t a d e h
T tire real-timeprocessing system ,which hides the a s e d i v o r p d n a k r o w t e n r o s n e s s u o e n e g o r e t e h e h t y b d e t c e l l o c a t a d g n i y l r e d n u e h t f o y t i x e l p m o c
r e p p u r o f e c a f r e t n i e c i v r e s s s e c c a a t a d d e i f i n
u -layerapplications. a
d g n i s s e c c a r o f e l b i s n o p s e r s i s s e c c a a t a
D tat ot heapplicationl ayer .Thispaperusesopen-source l
a e r l a r e v e s o t s s e c c a d i p a r e v e i h c a o t m e t s y s s d r o c e r e m u l F e h c a p
A -timesensing datasource[2] . h
g i h a s i e m u l
F -reliabliity and high-availability distributed massivelogcollection ,aggregationand e z i m o t s u c o s l a n a c e W . a t a d l a c i r o t s i h f o n o i t c e l l o c e h t r o f d e s u e b o s l a n a c t I . m e t s y s n o i s s i m s n a r t
. 3 e r u g i f n i n w o h s s i t r a h c w o l f e h T . a t a d f o g n i s s e c o r p e l p m i s a e v e i h c a o t e l u d o m k n i s e h t
O B S
d b q t n R
k d m m ` g B
n q u
@
j m h R
f m h r m d R
d b h u d
c r
n q u @
d b q t n R
k d m m ` g B
j m h R
` j e ` J
g i
F u 3 .re Dataaccess. n
o m e m u l
F itorst hemachineportst hroughi tsAgent .Thesourcecomponen tofAgen tenablesfor e
h t f o e d i s e n o s a , g n i r o ti n o m t r o p
c e l l o c e h t d n a , l e n n a h c h g u o r h t k n i S o t d e t t i m s n a r t s i t n e v E e h T . t n e v
E tormoduleconvergessevera l . a k f a K e h t o t a t a d e h t t i m s n a r t o t s t n e v E a t a
D Cache .Takingi ntoaccountt hedatafromInterne tofThingssystemi ssensitiveandofgrea t e w , e u l a
v cannot simply discard the data when large data flow in the system cannot be timely m e t s y s e h c a c e h t f o n o it c u d o r t n i e h T . r e t a l s s e c o r p e r d n a d e h c a c e b d l u o h s a t a d e h t t u b , g n i s s e c o r p e l u d o m e h c a c a t a d a s a d e s u s i a k f a K e h t , e r o f e r e h T . w o l f a t a d f o k a e p e h t n o e m i t n w o d d i o v a n a c o s l a h g i h a s i m e t s y s a k f a K e h t . m e t s y s e h t o t n
i throughput ,distributed ,pubilsh-subscribe messaging t u p h g u o r h t s t i d n a , e r u t c u r t s a t a d k s i d ) 1 ( O a y b y t i li b a t s d n a e c n e t s i s r e p e g a s s e m e r u s n e t i , m e t s y s r o f n o i t c n u f e g a r o t s s e d i v o r p a k f a K , r e p a p s i h t n I . ] 3 [ d n o c e s r e p s e g a s s e m f o s n o i l l i m t r o p p u s n a c
databefore processing .I tstores data tha thas no tbeen processed by storm in order ,wtih a topic a t a d e h t o t p m a t s e m it a d d a o t d e e n e w t a h t e t o n o t t n a t r o p m i s i t I . a t a d f o e p y t e n o o t g n i d n o p s e r r o c o r p a e e t n a r a u g n a c e w t a h t o s , a k f a K d e r e t n e t i e r o f e
b pert ime-orderfort hedatawhen t akingi tou t e
h t f
o Kafkasystem.
a t a
D Proces isng .The system designed in this paperuses Storm to process data .The Spou tof l a e r m r o f r e p e w n e h t d n a , m e t s y s a k f a K e h t m o r f a t a d d a e r y l s u o u n it n o c n a c m r o t
S -timeoperation
e h t t a h t h c u s , e l d n a h o t y g o l o p o T a e v a h l l i w a t a d f o s e p y t e s e h t f o h c a E . r e t s u l c m r o t S e h t h g u o r h t n e e w t e b e c n e u l f n i d n a , r e h t o h c a e f o t n e d n e p e d n i e b n a c s e p y t a t a d t n e r e f f i d s s e c o r p o t s e l u d o m b m u n t n e r e f f i d a n g i s s a n a c e W . l l a m s y l e v i t a l e r s i s e l u d o
m er of resources for differen tmodules f o a t a d e h t s s e c o r p o t s e c r u o s e r e r o m e t a c o l l a n a c e w , e l p m a x e r o f , n o i t a u ti s l a u t c a e h t o t g n i d r o c c a . e l a c s e g r a l r o y t i r o i r p h g i h p m o c o t d e e n o s l a t u o p S , a t a d t u o g n i k a t m o r f t r a p a , t a h t t u o t n i o p o t y r a s s e c e n s i t
I letethework
e h t t r o s o t e e r t r e s o l e h t d n a m h t i r o g l a t r o s e g r e m e h t e s u e w , r e p a p s i h t n I . e m i t y b a t a d e c n e u q e s o t e w e m i t y r e v e , e c n e u q e s e m i t n i s i a t a d r i e h t t a h t e e t n a r a u g n a c n o i ti t r a p e l g n i s e h t s a e r e h W . a t a d a e d e l d n a h n u e h t t u o e k a t o t d e e n y l n
o rlies tdatafromal lpartitionofonetopic ,thenwecanuset he g n i s s e c o r p e h T . s s e c o r p o t e n o t s e i l r a e e h t t c e l e s d n a m e h t t r o s o t a t a d f o p m a t s e m i t e h t d n a e e r t r e s o l n n a c t u o p S t a h t s i d o h t e m s i h t f o e g a t n a v d a s i d e h T . s t l o B t x e n n i d e t e l p m o c s i a t a d f
o o tuse
a t a d t a h t r e d i s n o c n a c e w , e m it s s e c c a n a h t r e t s a f h c u m s i g n i s s e c o r p a t a d s a e r e h W . g n i d a e r h t i t l u m e l g n i s a f o e s u e h t o s , m e t s y s e h t f o k c e n e l t t o b e h t t o n s i s s e c c
a -threadedspou tdoesnotl eadt oslower . g n i s s e c o r p a t a d p s m r o t S , r e m u s n o c a k f a K s
A ou treadt hedatafromKafkai norder ,andt hedatai sdeal twiththe e d o n t s a l e h T . g n i s s e c o r p a t a d f o e p y t a o t g n i d n o p s e r r o c s i y g o l o p o T A . s s e c o r p y g o l o p o T d e n i f e d n a c e W . e s a b a t a d e h t o t n i a t a d d e s s e c o r p e h t e r o t s o t d e s u t l o b e h t s i y g o l o p o T h c a e f
o readdatafrom
. s n o i t a c i l p p a r e h t o n i ti e s u r o e s a b a t a d e g a r o t
S .Thestoragemodule ,composed ofRedis+MySQL+HDFS ,isresponsibleforproviding y r e u q d n a e g a r o t s a t a
d functions[4] .Theprocessed datain Storm is stored directly to distributed h c i h w s i d e
R usesamaster-slaveform .Distributed Rediscan improvethequeryefficiencyofdata . d
e h
T atasynchronizationprocessi s: a b a t a d r e t s a m e h t o t a t a d s e t i r w m r o t S )
1 s e;
; B D R e v a l s o t s n o i t a r e p o e h t s e r o t s r e t s a M ) 2 . y r o m e m o t n i B D R s d a o l e v a l S ) 3 l u o h s e
W dsett het imingt askt opersistt hedataofRedist oMySQLashistorica ldata ,andt henwe e h t r o f e m i t y b e l b a t n o it i t r a p n a c e W . a t a d f o t n u o m a d e t i m il a e r u s n e o t s i d e R m o r f m e h t r a e l c n a c S y M . y r e u q f o y c n e i c i f f e e h t e r u s n e o t L Q S y M n i a t a d l a c i r o t s i
h QLcanprovidedatafort heprogram o t m e t s y s e l i f S F D H g n i s u y b s s o l a t a d t n e v e r p n a c L Q S y M , n o i t i d d a n I . s i s y l a n a a t a d l a c i r o t s i h f o . e l i f a t a d s ti r o f s p u k c a b e k a m n a m r o f r e p e h t e v o r p m i y l e v it c e f f e n a c L Q S y M r o f l o o p n o i t c e n n o c e s a b a t a d d i u r D f o e s u e h
T ce
9Qd`c V9Vqhsd
c d s t a h q s r h C
r h c d Q
K P R x
L GCER
,vqhsd d b ` e q d s m h
Q.V ` s ` C
k ` d q ,shld
` s ` c
k ` b h q n s r h g
` s `
c a`bjto
d b ` e q d s m
h C`s`
Q.V ` s ` C
s q n o l h
g i
F u 4 .re DataStorage.
tl u s e R d n a n o it a u l a v E
l a t n e m i r e p x
E Environment
s t n e m i r e p x e f o p u o r g o w t e r a e r e h
T conducted .The firs tgroup of experiments isto use the Java a t a d g n i v i e c e r d n a g n i d n e s e h t d n a , e n i h c a m e l g n i s a n i a t a d d n e s o t r o s n e s e h t e t a l u m i s o t m a r g o r p
r g d n o c e s e h T . d a e r h t e n o p u s e k a t h c a e g n i s s e c o r p d n a g n i v i e c e r , g n i d n e S . P C T y b d e t e l p m o c e r
a oup
e h t n i s e n i h c a m o w t e r a e r e h T . r e t s u l c m r o t s a n i k r o w e m a s e h t h s i l p m o c c a o t s i s t n e m i r e p x e f o
y g o l o p o T . r o s i v r e p u s a s a s k r o w r e h t o e h t d n a , r o s i v r e p u s a d n a s u b m i n e h t s a s k r o w e n O . r e t s u l c
e m i r e p x e e h t f o s t l u s e r e h T . 1 e r u g i f n i n w o h s s i e r u t c u r t
s n tareobtainedbyrecordingt henumberof e
u e u q e h t n i g n i n i a m e r a t a d f o r e b m u n e h t d n a a t a d d e t t i m s n a r
t [ 5].
tl u s e R
s t n e m i r e p x e f o p u o r g t s r i f e h T ) 1
d n a 2 e l b a t ; n o i ti d n o c e n i h c a m e l g n i s r e d n u d n o c e s e n o n i a t a d 0 1 g n i d n e s f o t l u s e r e h t s i 1 e l b a T
e l b a
t 3 are resutls of sending data through a single thread and two threads a tthe maximum rate .
e n i h c a m e h t y b d e t t i m r e p
e l b a
T 1 .10data/1secondinsinglemachine.
s
0 1 0s 2 0s 3 0s 4 0s )
% ( U P
C 6 8 8 9 8
) % ( y r o m e
M 5 2 5 4 5 4 5 4 5 4
d e s s e c o r
P 0 1 00 2 00 3 00 4 00
Not process 0 0 0 0 0
d e d n e
S 0 1 00 2 00 3 00 4 00
e l b a
T 2. M axspeedwithonethreadinsinglemachine. s
0 1 0s 2 0s 3 0s 4 0s )
% ( U P
C 6 8 4 8 6 8 3 8 4
) % ( y r o m e
M 5 2 7 8 7 8 7 8 7 8
d e s s e c o r
P 0 383242 335534 106268 24652 s
s e c o r p t o
N 0 11283 25138 6272 10203 d
e d n e
S 0 394525 743914 831316 859899 e
l b a
T 3. Maxspeedwitht wothreadinsinglemachine. s
0 1 0s 2 0s 3 0s 4 0s )
% ( U P
C 6 9 6 9 5 9 5 9 5
) % ( y r o m e
M 5 2 7 7 7 7 7 7 7 7
d e s s e c o r
P 0 374671 281907 180780 13535 Not process 0 23532 2617 5360 10050
d e d n e
S 0 398203 659195 842718 860943 s
a n o g n i n n u r y g o l o p o T a d n a d a e r h t e l g n i s a y b a t a d g n i d n e s f o e s a c e h t r e d n u t l u s e r e h t s i 4 e l b a T
s d a e r h t o w t h t i w r o s i v r e p u s o w t n o y g o l o p o T o w t g n i n n u r f o t l u s e r e h t s i 5 s e l b a t ; r o s i v r e p u s e l g n i s
e s o
t nddataa tmaximumratepermittedbyt hemachine. e
l b a
T 4 .10data/1secondi nstormcluster.
0s 1 0s 2 0s 3 0s 4 0s )
% ( U P
C 6 9 9 9 8
) % ( y r o m e
M 5 2 5 6 5 6 5 6 5 6
d e s s e c o r
P 0 1 00 2 00 3 00 4 00
Not process 0 0 0 0 0
d e d n e
S 0 1 00 2 00 3 00 4 00 e
l b a
T 5. Maxspeedwitht wothreadinstormcluster.
0s 1 0s 2 0s 3 0s 4 0s )
% ( U P
C -1 6 9 3 9 5 9 7 9 6
) % ( y r o m e
M -1 5 2 8 0 8 0 8 0 8 0 d
e s s e c o r
P -1 0 381355 290571 80967 72230 Not process-1 0 4967 15759 20149 9 66
) % ( U P
C -2 5 8 7 9 5 9 5 9 5
) % ( y r o m e
M -2 4 5 7 5 7 5 7 5 7 5 d
e s s e c o r
P -2 0 361076 370981 82113 32074 Not process-2 0 22129 21573 6757 9066
d e d n e
S 0 769527 1441315 1593969 1681399
si s y l a n A
y r o m e m d n a U P C e h t , y l t a e r g s w o r g a t a d f o t n u o m a e h t n e h w t a h t w o h s 2 d n a 1 e l b a t n i s t l u s e R
k r o w t e n a s i e r e h t e e s n a c e w 3 d n a 2 e l b a t m o r F . y l t n a c i f i n g i s e s a e r c n i l l i w e n i h c a m f o n o i t p m u s n o c
, d e h c a e r s i k c e n e l t t o b e h t n e h W . e n i h c a m e h t n i k c e n e l t t o b O
I thespeed of sending and receiving a
t a
d cannotcontinuet oi mprove .Thet asksexecutedi nstormaresimilart ot heonesdirectlyexecuted e h t n O . 4 d n a 1 e l b a t n i t n e i c i f f u s e r a s e c r u o s e r k r o w t e n d n a y r o m e m , U P C n e h w e n i h c a m e h t n i
n i e r a e n i h c a m e h t f o s e c r u o s e r k r o w t e n d n a y r o m e m , U P C n e h w , y r a r t n o
c sufficien,tt heperformance g n i d r o c c A . e n i h c a m e l g n i s n i g n i n n u r t a h t n a h t r e t t e b y l s u o i v b o s i r e t s u l c m r o t s n i g n i n n u r s k s a t f o
e h t t a h t w o n k n a c e w , e u e u q e h t n i g n i n i a m e r a t a d f o t n u o m a e h t d n a d e e p s g n i s s e c o r p a t a d e h t o t
s s e c c a a t a d m o r f d e m u s n o c e m i
t toprocessingi si nseconds ,whichcanmeett heneedsforrealt ime.
y r a m m u S
, r e p a p s i h t n
I wefocusont hedesignoft hesystemarchitecturefromt hreeaspectsofdataaccess ,data .
e g a r o t s a t a d d n a g n i s s e c o r
p Andexperimentsshowtha tthesystemcansatisfytheneedofrealtime .
a t a d e v i s s a m g n i s s e c o r p n i e c n a m r o f r e p r e t t e b g n i v a h e li h w e c n a m r o f r e
p Parameters in every
.t l u s e r d e r i s e d e h t g n i r b d n a r o r r e e h t e c u d e r o t s e c n a t s m u c r i c c i f i c e p s n i d e t s u j d a e b o t e v a h e l u d o m
t n e m e g d e l w o n k c A
r o p p u s s i k r o w s i h
T tedbyNationa lNatura lScienceFoundationofChina(Gran tNo.61501048) ;The l a r u t a N l a n o i t a N ; ) 2 1 C R 7 1 0 2 . o N t n a r G ( s e i t i s r e v i n U l a r t n e C e h t r o f s d n u F h c r a e s e R l a t n e m a d n u F
. o N t n a r G ( a n i h C f o n o i t a d n u o F e c n e i c
S 61372115) .
s e c n e r e f e R
, u Q g n i q o u G ] 1
[ The Design and Implementation of Log Analysis Based on Storm [D] .Nanjing .
6 1 0 2 , y t i s r e v i n U
] 2
[ Mingyu Zeng .A Design and Implementation of Distributed Real-time Log Data Storage and b
d o g n o M d n a m r o t S n o d e s a B m e t s y S g n i s s e c o r
B ] 3
[ o Chen, Research on Design Method of Massive Data Statistic System based on STORM g
n i t u p m o C d e t u b i r t s i
D [J] .ChinaComputer & Communication ,2017 ,(04): 21 -2 125 . ]
4
[ LinZhao, Research andImplementationofDataProcessingFrameworkofIoTBased onStorm .
] D
[ Northwes tUniversity ,2015. ]
5
[ JinsongL , i ResearchandImplementationofaDistributedReal-timeandIncrementa lCalculaiton m
r o t S n o d e s a B k r o w e m a r