• No results found

Real Time Processing System of Massive Data Stream in Internet of Things Based on Apache Storm

N/A
N/A
Protected

Academic year: 2020

Share "Real Time Processing System of Massive Data Stream in Internet of Things Based on Apache Storm"

Copied!
7
0
0

Loading.... (view fulltext now)

Full text

(1)

) 7 1 0 2 E II A ( g n ir e e n i g n E l a ir t s u d n I d n a e c n e g il l e t n I l a i c if it r A n o e c n e r e f n o C l a n o it a n r e t n I d r 3 7 1 0 2

8 7 9 : N B S

I -1-60595- 05 -9 2

f

o

t

e

n

r

e

t

n

I

n

i

m

a

e

r

t

S

a

t

a

D

e

v

i

s

s

a

M

f

o

m

e

t

s

y

S

g

n

i

s

s

e

c

o

r

P

e

m

i

T

l

a

e

R

m

r

o

t

S

e

h

c

a

p

A

n

o

d

e

s

a

B

s

g

n

i

h

T

G

N

A

W

e

i

J

,

S

h

u

a

i

Z

H

A

O

a

n

d

B

o

C

H

E

N

G

g n i h c ti w S d n a g n i k r o w t e N f o y r o t a r o b a L y e K e t a t

S Technology,BeiijngUniverstiyof

a n i h C , g n ij i e B , s n o it a c i n u m m o c e l e T d n a s t s o P

: s d r o w y e

K IoT,ApacheStorm ,Kafka,Massivedata ,Rea ltimeprocessing.

.t c a r t s b

A ThedataofInterne tofThingshasthecharacteristicsofrealtimea ndlargescale .Forthe l

a e r , a t a

d -timeprocessingi susuallynecessary ,especiallyi nsomesecurityscenarios,t hispapert ries l

a e r e h t t r o p p u s n a c h c i h w m e t s y s a n g i s e d o

t -timeprocessingoflargedataflow .Thispaperfirstly t

n i w o l f a t a d e h t f o s c i t s i r e t c a r a h c e h t s e z y l a n

a heInterne tofThings ,andbyusingt heresutlsoft he s m e l b o r p n i a m e h t d n a m e t s y s e h t n i d e s u e b n a c t a h t s e i g o l o n h c e t y e k e h t d r a w r o f s t u p I , s i s y l a n a

d f o s t c e p s a e e r h t m o r f l i a t e d n i d e n g i s e d I m e t s y s e h t f o e r u t c e ti h c r a e h t e b i r c s e d I , y ll a n i F . d e c a

f ata

. e g a r o t s a t a d d n a g n i s s e c o r p a t a d , s s e c c a

n o it c u d o r t n I

s g n i h T f o t e n r e t n

I (IoT) is anetwork with three layers of perception layer ,transpor tlayerand e c i v e d g n i s n e s n o i t a m r o f n i e h t h g u o r h t t e n r e t n I e h t h t i w g n i h t y n a t c e n n o c n a c t I . r e y a l n o it a c i l p p

a on

d n a e g n a h c x e n o it a m r o f n i r o f e l b i s n o p s e r y l n i a m s i r e y a l t r o p s n a r t e h T . r e y a l n o it p e c r e p e h t

, y l e t a m i tl U . n o i t a c i n u m m o

c w e can reailze intelilgen t identification , positioning , tracking , .

r e y a l n o it a c i l p p a e h t n o t n e m e g a n a m d n a g n i r o t i n o m

I m o r f a t a d e h

T nterne tofThingsmainlyhast wocharacteristics ,namelyl argescaleandrealt ime . e h t , d n o c e S . e c n e g il l e t n i n i e l o r a y a l p t c e j b o e h t e k a m n a c a t a d e h t , e l a c s e g r a l e h t h t i w y l n o , t s r i F

l a e r s i a t a

d -timeand need to beprocessed inrea ltime .Through theembedded orattached sensing m

r o f n i l a n r e t x e d n a s m e t i n o e c i v e

d ation acquisition technology ,IoTsystemcanreflec tthestateof , y t i d i m u h d n a e r u t a r e p m e t , s u o r e g n a d r o e f a s , g n i v o m r o y r a n o i t a t s g n i d u l c n i , e m i t t r o h s y r e v e s d o o g

s u y l e d i w e b n a c t I . c t

e edi nvariousscenesofrealtiy ,suchasenvironmenta lmontioring ,community .

c t e , y t i r u c e s

l a e

R -timecomputingi susuallycarriedou tformassivedata .I trequiresareal-timeprocessingand l

a e r e h t t e e m o t r e d r o n I . a t a d e h t o t e s n o p s e

r -timerequirementsoft hesystem,t heresponsei susually .

s d n o c e s n

i Rea ltime computing generally consistsof threestages :datageneration and collection .

e s a h p e c i v r e s e d i v o r p o t e g a r o t s , e s a h p g n i s s e c o r p s i s y l a n a d n a n o i s s i m s n a r t , e s a h p

R )

1 ea litmedataacquisiiton :Thispar tshouldguaranteet hecompletecollectionofalll ogdataand l

a e r e d i v o r

p -timedataforreal-timeapplications .Att hesamet ime ,weshouldensuret hatt hesystem e

h t ,t s a l t A . t n e m y o l p e d y s a e , n o i t a r u g i f n o c e l p m i s s w o l l a d n a e l b a i l e r , e l b a t s s

i responset imeshould

l a e r e

b -timeandl owl atency .A tpresen,tt hemainproductsareScribe ,Kafka ,FlumeandTimeTunnel , . d n o c e s r e p n o i s s i m s n a r t d n a n o i t i s i u q c a a t a d g o l B M f o s d e r d n u h f o s d e e n e h t t e e m l l a n a c y e h T . c t e

. s t c e j o r p e c r u o s n e p o l l a e r' y e h T

)

2 Rea ltimedata computation: In thechanging datastream ,real-timeanalysisiscarried ou tto .t

u o t l u s e r e h t d n e s o t d n a r e s u e h t o t l u f e s u e b y a m t a h t n o it a m r o f n i l u f e s u e r u t p a

c Thecurren tmain

b i r t s i d a s i m r o t S . a m u P d n a m r o t S , 4 S e d u l c n i s t c u d o r

p uted ,fault-toleran treal-time computing r o f t r o p p u s t i d n a , e s a b a t a d e h t e t a d p u d n a n o i t a m r o f n i g n i s s e c o r p r o f d e s u e b n a c t I . m e t s y s

w o l f f o m r o f e h t n i t n e i l c e h t o t s t l u s e r n r u t e r d n a , s m a e r t s a t a d r e v o s e i r e u q s u o u n i t n o

c [ 1].

)

(2)

s e it l u c if fi D

e g r a

L ScaleD aa t

t p a d a o t r e d r o n i ,t e n r e t n i d n a y g o l o n h c e t n o i t a m r o f n i f o t n e m p o l e v e d d i p a r e h t f o d n u o r g k c a b e h t n I

, r a e p p a s e p y t r o s n e s e r o m d n a e r o m , s o i r a n e c s t n e r e f f i d o

t andsensorperformancehasbeengreatly n a s a h m e t s y s n o i t i s i u q c a a t a d e h t y b d e t c e l l o c a t a d e h t f o e z i s e h t w o n o S . e r o f e b n a h t d e v o r p m i

l a e r e h t , e r o f e r e h T . h t w o r g e v i s o l p x

e -timecomputing system needsto dea lwtih themassive data r

e t n I e h t y b d e t c e l l o

c ne tofThingssystem .Howto mee tthereal-timeprocessingbasedonmassive .

h t i w t l a e d e b o t m e l b o r p t n a t r o p m i y r e v a s i a t a d

l a e

R -itmeProces isng

l a e r n i d e t c e l l o c s i a t a d e s u a c e

B -timei nInterne tofThings ,weneedt omaker eal-timeprocessingand o

p s e

r nse .With the enormous scale of data ,how to improve the speed of data processing is the .

r e d i s n o c o t d e e n e w t a h t m e l b o r p

y ti r u c e S

e h t n i , e l p m a x e r o F . n o i s s i m s n a r t a t a d f o y ti l i b a i l e r e h t e r u s n e o t d e e n e w , n o i t a u t i s y t i r u c e s h g i h n I

s y t i r u c e s y t i n u m m o

c ystem ,weno tonlyneedtoachievereal-timedataprocessing ,bu talsoneedto o S . a t a d f o e c e i p e l tt i l a n i r a e p p a y a m a t a d e v i t i s n e s e h t f o e s u a c e b , t s o l t o n s i a t a d e h t t a h t e r u s n e

t n a t r o p m i y r e v s i n o i s s i m s n a r t e h t a t a d f o y t il i b a i l e r e h t e r u s n e o t w o

h .Int hesameway ,weneedt he t i n e h w d e d r a c s i d n a h t r e h t a r d e h c a c e b n a c a t a d e h t t a h t e r u s n e o t m s i n a h c e m g n i h c a c e t a i r p o r p p a

g n i s s e c o r p n a h t r e t s a f s e m o

c .

TimingSequence fo D aa t

n i a t a d d e h c a c e h t t e g o t w o h r e d i s n o c o t d e e n o s l a e w , y t i r u c e s s e d i s e

B timesequencef orprocessing . p e e k o t d e e n e w d n a , n o i t a m r o f n i f o d n i k a s i t i e s u a c e b , t n a t r o p m i y r e v s i a t a d f o e c n e u q e s e m i t e h T

. e t a r u c c a e r o m s i s y l a n a a t a d e k a m o t n o i t a m r o f n i s i h t

l a e

R -itmeQuery

l a e r e h t t e e m o t w o H ? a t a d e v i s s a m e r o t s o t w o

H -timerequirementsofvariousqueries?Thesetwo e g r a l t r o p p u s o t m e t s y s e s a b a t a d e l b a n o s a e r a n g i s e d o t d e e n e W . t u o b a e r a c e w t a h w e r a s n o i t s e u

q

-l a e r e h t t e e m d n a y r e u q d n a e g a r o t s a t a d e l a c

s -timerequirementsoft hereal-timeprocessingsystem.

e r u t c e ti h c r

A De is gn

n g is e

D a Pnd rinciples

l a e r y h w t u o b a s n o s a e r t n a t r o p m i y r e v o w t e r a e r e h

T -timeprocessingt echniquesareusedt oprocess .

s g n i h t f o t e n r e t n I e h t n i a t a d

e g r a L . e v i s s a m s i s g n i h T f o t e n r e t n I e h t m o r f d e r i u q c a a t a d e h t s i e n

O -scaledatagenerallyhasfour t

s i r e t c a r a h

c ics :large ,fast ,miscellaneous and suspected .I thas thecharacteristics of theflow ,and l

a e r f o e c n a m r o f r e p h g i h r o f s e r i u q e

r -time .Aslongastheprocessingisfas tand correct ,thesedata S

. e l a c s r i e h t o t d e t a l e r e u l a v e g u h g n i r b n a

c ot heaccuracyandreal-timepropertyplayveryi mportan t l

a e r e h t d n a , m e t s y s e l o h w e h t r o f s e l o

r -timeprocessingtechnologyisj us tforthefas tandreal-time .

a t a d m a e r t s f o g n i s s e c o r p

e t c a r a h c d e t c e p s u s d n a s u o e n a ll e c s i m , e g r a l e h t s i n o s a e r d n o c e S e h

T risticsofdatadeterminet ha t l

a e r f o r e t s u l c e h t d n a , h t i w l a e d o t s e c r u o s e r g n i t u p m o c f o t n u o m a e g r a l a d e e n y e h

t -timeprocessing

h c i h w s e c r u o s e r g n i t u p m o c e g u h e d i v o r p n a c m e t s y

s cannotbeprovidedbysinglemachine.

l l a r e v

O Technica lArchtiecture

e h

T system is mainly divided into severa lparts ,data acquisition ,data access ,data cache ,data .

(3)

s m d f

@ 0

s m d f

@ 1 Bnkkdbsnq 'j`ej`oqnctbdq(

Zsnohb0\

` j e ` J Zsnohb1\

q d o d d j n n y

 c d y h l n s r t B

s t n o R 0

'j`ej`bnmrtldq( s

k n

A 0

s k n

A 1

 c d y h l n s r t B

s t n o R 1

s k n

A 2

s k n

A 3

 d r ` A ` s ` C

l d s r x R

x f n k n o n

S 0

g i

F u 1 .re Systemarchitecture.

n i a

M Components a

t a

D Acqui isiton and D a Ata ccess .Data acquisition is accomplished by a variety of sensing : w o l e b 2 e r u g i F n i n w o h s s i e l p m a x e n a d n a , s g n i h T f o t e n r e t n I e h t f o r e y a l n o i t p e c r e p e h t n i s e c i v e d

y a W e t a

G GateWay

C

S C

V

e c i v e D 3

e c i v e D 1

e c i v e D 2

 m n h s o d b q

d K`xdq Sq`mronqsK`xdq

. . .

e

m

u

l

F

g i

F u 2 .re Dataacquisition. n

e e h t f o e c n a r t n e e h t s i t r a p s s e c c a a t a d e h

T tire real-timeprocessing system ,which hides the a s e d i v o r p d n a k r o w t e n r o s n e s s u o e n e g o r e t e h e h t y b d e t c e l l o c a t a d g n i y l r e d n u e h t f o y t i x e l p m o c

r e p p u r o f e c a f r e t n i e c i v r e s s s e c c a a t a d d e i f i n

u -layerapplications. a

d g n i s s e c c a r o f e l b i s n o p s e r s i s s e c c a a t a

D tat ot heapplicationl ayer .Thispaperusesopen-source l

a e r l a r e v e s o t s s e c c a d i p a r e v e i h c a o t m e t s y s s d r o c e r e m u l F e h c a p

A -timesensing datasource[2] . h

g i h a s i e m u l

F -reliabliity and high-availability distributed massivelogcollection ,aggregationand e z i m o t s u c o s l a n a c e W . a t a d l a c i r o t s i h f o n o i t c e l l o c e h t r o f d e s u e b o s l a n a c t I . m e t s y s n o i s s i m s n a r t

. 3 e r u g i f n i n w o h s s i t r a h c w o l f e h T . a t a d f o g n i s s e c o r p e l p m i s a e v e i h c a o t e l u d o m k n i s e h t

 O B S

d b q t n R

k d m m ` g B

n q u

@ 

j m h R

 f m h r m d R

d b h u d

c r

 n q u @

d b q t n R

k d m m ` g B

j m h R

` j e ` J

g i

F u 3 .re Dataaccess. n

o m e m u l

F itorst hemachineportst hroughi tsAgent .Thesourcecomponen tofAgen tenablesfor e

h t f o e d i s e n o s a , g n i r o ti n o m t r o p

(4)

c e l l o c e h t d n a , l e n n a h c h g u o r h t k n i S o t d e t t i m s n a r t s i t n e v E e h T . t n e v

E tormoduleconvergessevera l . a k f a K e h t o t a t a d e h t t i m s n a r t o t s t n e v E a t a

D Cache .Takingi ntoaccountt hedatafromInterne tofThingssystemi ssensitiveandofgrea t e w , e u l a

v cannot simply discard the data when large data flow in the system cannot be timely m e t s y s e h c a c e h t f o n o it c u d o r t n i e h T . r e t a l s s e c o r p e r d n a d e h c a c e b d l u o h s a t a d e h t t u b , g n i s s e c o r p e l u d o m e h c a c a t a d a s a d e s u s i a k f a K e h t , e r o f e r e h T . w o l f a t a d f o k a e p e h t n o e m i t n w o d d i o v a n a c o s l a h g i h a s i m e t s y s a k f a K e h t . m e t s y s e h t o t n

i throughput ,distributed ,pubilsh-subscribe messaging t u p h g u o r h t s t i d n a , e r u t c u r t s a t a d k s i d ) 1 ( O a y b y t i li b a t s d n a e c n e t s i s r e p e g a s s e m e r u s n e t i , m e t s y s r o f n o i t c n u f e g a r o t s s e d i v o r p a k f a K , r e p a p s i h t n I . ] 3 [ d n o c e s r e p s e g a s s e m f o s n o i l l i m t r o p p u s n a c

databefore processing .I tstores data tha thas no tbeen processed by storm in order ,wtih a topic a t a d e h t o t p m a t s e m it a d d a o t d e e n e w t a h t e t o n o t t n a t r o p m i s i t I . a t a d f o e p y t e n o o t g n i d n o p s e r r o c o r p a e e t n a r a u g n a c e w t a h t o s , a k f a K d e r e t n e t i e r o f e

b pert ime-orderfort hedatawhen t akingi tou t e

h t f

o Kafkasystem.

a t a

D Proces isng .The system designed in this paperuses Storm to process data .The Spou tof l a e r m r o f r e p e w n e h t d n a , m e t s y s a k f a K e h t m o r f a t a d d a e r y l s u o u n it n o c n a c m r o t

S -timeoperation

e h t t a h t h c u s , e l d n a h o t y g o l o p o T a e v a h l l i w a t a d f o s e p y t e s e h t f o h c a E . r e t s u l c m r o t S e h t h g u o r h t n e e w t e b e c n e u l f n i d n a , r e h t o h c a e f o t n e d n e p e d n i e b n a c s e p y t a t a d t n e r e f f i d s s e c o r p o t s e l u d o m b m u n t n e r e f f i d a n g i s s a n a c e W . l l a m s y l e v i t a l e r s i s e l u d o

m er of resources for differen tmodules f o a t a d e h t s s e c o r p o t s e c r u o s e r e r o m e t a c o l l a n a c e w , e l p m a x e r o f , n o i t a u ti s l a u t c a e h t o t g n i d r o c c a . e l a c s e g r a l r o y t i r o i r p h g i h p m o c o t d e e n o s l a t u o p S , a t a d t u o g n i k a t m o r f t r a p a , t a h t t u o t n i o p o t y r a s s e c e n s i t

I letethework

e h t t r o s o t e e r t r e s o l e h t d n a m h t i r o g l a t r o s e g r e m e h t e s u e w , r e p a p s i h t n I . e m i t y b a t a d e c n e u q e s o t e w e m i t y r e v e , e c n e u q e s e m i t n i s i a t a d r i e h t t a h t e e t n a r a u g n a c n o i ti t r a p e l g n i s e h t s a e r e h W . a t a d a e d e l d n a h n u e h t t u o e k a t o t d e e n y l n

o rlies tdatafromal lpartitionofonetopic ,thenwecanuset he g n i s s e c o r p e h T . s s e c o r p o t e n o t s e i l r a e e h t t c e l e s d n a m e h t t r o s o t a t a d f o p m a t s e m i t e h t d n a e e r t r e s o l n n a c t u o p S t a h t s i d o h t e m s i h t f o e g a t n a v d a s i d e h T . s t l o B t x e n n i d e t e l p m o c s i a t a d f

o o tuse

a t a d t a h t r e d i s n o c n a c e w , e m it s s e c c a n a h t r e t s a f h c u m s i g n i s s e c o r p a t a d s a e r e h W . g n i d a e r h t i t l u m e l g n i s a f o e s u e h t o s , m e t s y s e h t f o k c e n e l t t o b e h t t o n s i s s e c c

a -threadedspou tdoesnotl eadt oslower . g n i s s e c o r p a t a d p s m r o t S , r e m u s n o c a k f a K s

A ou treadt hedatafromKafkai norder ,andt hedatai sdeal twiththe e d o n t s a l e h T . g n i s s e c o r p a t a d f o e p y t a o t g n i d n o p s e r r o c s i y g o l o p o T A . s s e c o r p y g o l o p o T d e n i f e d n a c e W . e s a b a t a d e h t o t n i a t a d d e s s e c o r p e h t e r o t s o t d e s u t l o b e h t s i y g o l o p o T h c a e f

o readdatafrom

. s n o i t a c i l p p a r e h t o n i ti e s u r o e s a b a t a d e g a r o t

S .Thestoragemodule ,composed ofRedis+MySQL+HDFS ,isresponsibleforproviding y r e u q d n a e g a r o t s a t a

d functions[4] .Theprocessed datain Storm is stored directly to distributed h c i h w s i d e

R usesamaster-slaveform .Distributed Rediscan improvethequeryefficiencyofdata . d

e h

T atasynchronizationprocessi s: a b a t a d r e t s a m e h t o t a t a d s e t i r w m r o t S )

1 s e;

; B D R e v a l s o t s n o i t a r e p o e h t s e r o t s r e t s a M ) 2 . y r o m e m o t n i B D R s d a o l e v a l S ) 3 l u o h s e

W dsett het imingt askt opersistt hedataofRedist oMySQLashistorica ldata ,andt henwe e h t r o f e m i t y b e l b a t n o it i t r a p n a c e W . a t a d f o t n u o m a d e t i m il a e r u s n e o t s i d e R m o r f m e h t r a e l c n a c S y M . y r e u q f o y c n e i c i f f e e h t e r u s n e o t L Q S y M n i a t a d l a c i r o t s i

h QLcanprovidedatafort heprogram o t m e t s y s e l i f S F D H g n i s u y b s s o l a t a d t n e v e r p n a c L Q S y M , n o i t i d d a n I . s i s y l a n a a t a d l a c i r o t s i h f o . e l i f a t a d s ti r o f s p u k c a b e k a m n a m r o f r e p e h t e v o r p m i y l e v it c e f f e n a c L Q S y M r o f l o o p n o i t c e n n o c e s a b a t a d d i u r D f o e s u e h

T ce

(5)

9Qd`c V9Vqhsd 

c d s t a h q s r h C

r h c d Q

K P R x

L GCER

,vqhsd d b ` e q d s m h

Q.V ` s ` C

k ` d q ,shld

` s ` c

 k ` b h q n s r h g

` s `

c a`bjto

d b ` e q d s m

h C`s`

Q.V ` s ` C

s q n o l h

g i

F u 4 .re DataStorage.

tl u s e R d n a n o it a u l a v E

l a t n e m i r e p x

E Environment

s t n e m i r e p x e f o p u o r g o w t e r a e r e h

T conducted .The firs tgroup of experiments isto use the Java a t a d g n i v i e c e r d n a g n i d n e s e h t d n a , e n i h c a m e l g n i s a n i a t a d d n e s o t r o s n e s e h t e t a l u m i s o t m a r g o r p

r g d n o c e s e h T . d a e r h t e n o p u s e k a t h c a e g n i s s e c o r p d n a g n i v i e c e r , g n i d n e S . P C T y b d e t e l p m o c e r

a oup

e h t n i s e n i h c a m o w t e r a e r e h T . r e t s u l c m r o t s a n i k r o w e m a s e h t h s i l p m o c c a o t s i s t n e m i r e p x e f o

y g o l o p o T . r o s i v r e p u s a s a s k r o w r e h t o e h t d n a , r o s i v r e p u s a d n a s u b m i n e h t s a s k r o w e n O . r e t s u l c

e m i r e p x e e h t f o s t l u s e r e h T . 1 e r u g i f n i n w o h s s i e r u t c u r t

s n tareobtainedbyrecordingt henumberof e

u e u q e h t n i g n i n i a m e r a t a d f o r e b m u n e h t d n a a t a d d e t t i m s n a r

t [ 5].

tl u s e R

s t n e m i r e p x e f o p u o r g t s r i f e h T ) 1

d n a 2 e l b a t ; n o i ti d n o c e n i h c a m e l g n i s r e d n u d n o c e s e n o n i a t a d 0 1 g n i d n e s f o t l u s e r e h t s i 1 e l b a T

e l b a

t 3 are resutls of sending data through a single thread and two threads a tthe maximum rate .

e n i h c a m e h t y b d e t t i m r e p

e l b a

T 1 .10data/1secondinsinglemachine.

s

0 1 0s 2 0s 3 0s 4 0s )

% ( U P

C 6 8 8 9 8

) % ( y r o m e

M 5 2 5 4 5 4 5 4 5 4

d e s s e c o r

P 0 1 00 2 00 3 00 4 00

Not process 0 0 0 0 0

d e d n e

S 0 1 00 2 00 3 00 4 00

e l b a

T 2. M axspeedwithonethreadinsinglemachine. s

0 1 0s 2 0s 3 0s 4 0s )

% ( U P

C 6 8 4 8 6 8 3 8 4

) % ( y r o m e

M 5 2 7 8 7 8 7 8 7 8

d e s s e c o r

P 0 383242 335534 106268 24652 s

s e c o r p t o

N 0 11283 25138 6272 10203 d

e d n e

S 0 394525 743914 831316 859899 e

l b a

T 3. Maxspeedwitht wothreadinsinglemachine. s

0 1 0s 2 0s 3 0s 4 0s )

% ( U P

C 6 9 6 9 5 9 5 9 5

) % ( y r o m e

M 5 2 7 7 7 7 7 7 7 7

d e s s e c o r

P 0 374671 281907 180780 13535 Not process 0 23532 2617 5360 10050

d e d n e

S 0 398203 659195 842718 860943 s

(6)

a n o g n i n n u r y g o l o p o T a d n a d a e r h t e l g n i s a y b a t a d g n i d n e s f o e s a c e h t r e d n u t l u s e r e h t s i 4 e l b a T

s d a e r h t o w t h t i w r o s i v r e p u s o w t n o y g o l o p o T o w t g n i n n u r f o t l u s e r e h t s i 5 s e l b a t ; r o s i v r e p u s e l g n i s

e s o

t nddataa tmaximumratepermittedbyt hemachine. e

l b a

T 4 .10data/1secondi nstormcluster.

0s 1 0s 2 0s 3 0s 4 0s )

% ( U P

C 6 9 9 9 8

) % ( y r o m e

M 5 2 5 6 5 6 5 6 5 6

d e s s e c o r

P 0 1 00 2 00 3 00 4 00

Not process 0 0 0 0 0

d e d n e

S 0 1 00 2 00 3 00 4 00 e

l b a

T 5. Maxspeedwitht wothreadinstormcluster.

0s 1 0s 2 0s 3 0s 4 0s )

% ( U P

C -1 6 9 3 9 5 9 7 9 6

) % ( y r o m e

M -1 5 2 8 0 8 0 8 0 8 0 d

e s s e c o r

P -1 0 381355 290571 80967 72230 Not process-1 0 4967 15759 20149 9 66

) % ( U P

C -2 5 8 7 9 5 9 5 9 5

) % ( y r o m e

M -2 4 5 7 5 7 5 7 5 7 5 d

e s s e c o r

P -2 0 361076 370981 82113 32074 Not process-2 0 22129 21573 6757 9066

d e d n e

S 0 769527 1441315 1593969 1681399

si s y l a n A

y r o m e m d n a U P C e h t , y l t a e r g s w o r g a t a d f o t n u o m a e h t n e h w t a h t w o h s 2 d n a 1 e l b a t n i s t l u s e R

k r o w t e n a s i e r e h t e e s n a c e w 3 d n a 2 e l b a t m o r F . y l t n a c i f i n g i s e s a e r c n i l l i w e n i h c a m f o n o i t p m u s n o c

, d e h c a e r s i k c e n e l t t o b e h t n e h W . e n i h c a m e h t n i k c e n e l t t o b O

I thespeed of sending and receiving a

t a

d cannotcontinuet oi mprove .Thet asksexecutedi nstormaresimilart ot heonesdirectlyexecuted e h t n O . 4 d n a 1 e l b a t n i t n e i c i f f u s e r a s e c r u o s e r k r o w t e n d n a y r o m e m , U P C n e h w e n i h c a m e h t n i

n i e r a e n i h c a m e h t f o s e c r u o s e r k r o w t e n d n a y r o m e m , U P C n e h w , y r a r t n o

c sufficien,tt heperformance g n i d r o c c A . e n i h c a m e l g n i s n i g n i n n u r t a h t n a h t r e t t e b y l s u o i v b o s i r e t s u l c m r o t s n i g n i n n u r s k s a t f o

e h t t a h t w o n k n a c e w , e u e u q e h t n i g n i n i a m e r a t a d f o t n u o m a e h t d n a d e e p s g n i s s e c o r p a t a d e h t o t

s s e c c a a t a d m o r f d e m u s n o c e m i

t toprocessingi si nseconds ,whichcanmeett heneedsforrealt ime.

y r a m m u S

, r e p a p s i h t n

I wefocusont hedesignoft hesystemarchitecturefromt hreeaspectsofdataaccess ,data .

e g a r o t s a t a d d n a g n i s s e c o r

p Andexperimentsshowtha tthesystemcansatisfytheneedofrealtime .

a t a d e v i s s a m g n i s s e c o r p n i e c n a m r o f r e p r e t t e b g n i v a h e li h w e c n a m r o f r e

p Parameters in every

.t l u s e r d e r i s e d e h t g n i r b d n a r o r r e e h t e c u d e r o t s e c n a t s m u c r i c c i f i c e p s n i d e t s u j d a e b o t e v a h e l u d o m

t n e m e g d e l w o n k c A

r o p p u s s i k r o w s i h

T tedbyNationa lNatura lScienceFoundationofChina(Gran tNo.61501048) ;The l a r u t a N l a n o i t a N ; ) 2 1 C R 7 1 0 2 . o N t n a r G ( s e i t i s r e v i n U l a r t n e C e h t r o f s d n u F h c r a e s e R l a t n e m a d n u F

. o N t n a r G ( a n i h C f o n o i t a d n u o F e c n e i c

S 61372115) .

s e c n e r e f e R

, u Q g n i q o u G ] 1

[ The Design and Implementation of Log Analysis Based on Storm [D] .Nanjing .

6 1 0 2 , y t i s r e v i n U

] 2

[ Mingyu Zeng .A Design and Implementation of Distributed Real-time Log Data Storage and b

d o g n o M d n a m r o t S n o d e s a B m e t s y S g n i s s e c o r

(7)

B ] 3

[ o Chen, Research on Design Method of Massive Data Statistic System based on STORM g

n i t u p m o C d e t u b i r t s i

D [J] .ChinaComputer & Communication ,2017 ,(04): 21 -2 125 . ]

4

[ LinZhao, Research andImplementationofDataProcessingFrameworkofIoTBased onStorm .

] D

[ Northwes tUniversity ,2015. ]

5

[ JinsongL , i ResearchandImplementationofaDistributedReal-timeandIncrementa lCalculaiton m

r o t S n o d e s a B k r o w e m a r

References

Related documents

Once they collect their logs, respondents say the most useful feature of log management systems is “real-time alerts,” with 68 percent indicating they are very useful and 25

In face of this problem’s impact, we need to ascertain what TCP implementation artifacts cause such high am- plification rates. In general, we would expect to see ei- ther a single

Peran pelaksana program sangat menentukan keberhasilan pencapaian tujuan program karena kurangnya dukungan bidan terhadap program dapat mempengaruhi implementasinya terhadap

The Entity team will also work with Constabulary resources to up-skill them in order to be able to use the tool past their involvement with the Atlas project and the

Extrapolating from the 2012 revision time line, the 2017 revi- sion process is likely to commence as early as 2013 (Figure 1), giving the industry ap- proximately two years

Following successful completion of the course and assessment, a Royal Society of Biology certificate is awarded, which is used to support your Home Office Licence application..

There is no one route to getting a job with animals but an interest in science and a science based qualification teamed with work experience is a good place to start.. Some zoo

After training volunteers help with feeding, cleaning, all round care for wildlife and now is the best time for training before we get busy. Volunteer trades people