We contend th a t in order to effectively utilize m odern, shared com puting environm ents, applications should m onitor a n d a d a p t to changing system conditions. T h is research has pu rsued two d istin ct b u t com plem entary m echanism s to enable some classes of scientific applications to a d a p t to variations in C P U an d m em ory availability.
8.1
F le x ib le -p h a s e lo a d b a la n c in g
T h e first m echanism we stu d ied is a stra te g y for dynam ic load balancing of a class of ite ra tive algorithm s th a t possess w hat we te rm a flexible phase, du rin g w hich th e am o u n t of local work com pleted can vary w ith o u t d etrim en t to th e end result. T h e technique can d y n am i cally correct load im balance from b o th in tern al and ex ternal sources an d requires m inim al overhead. A lthough lim ited to a narrow class of algorithm s, this class includes some p o p u lar num erical algorithm s th a t form th e co m p u tatio n al kernels of m any scientific applications. We have experim entally d e m o n stra ted th e u tility of our m ethod in two Krylov-like m ethods: a m ulti-grain form ulation of a block Jacobi-D avidson eigensolver an d an FG M R E S linear solver em ploying additive-Schw arz preconditioning. O u r m ethod has proven able to cope w ith severe load im balance in th e Jacobi-D avidson eigensolver, an d it makes p ractical th e full-m ultigrain parallel form ulation th a t can hide netw ork latencies b u t m ay accen tu ate load im balance. In th e additive-Schw arz preconditioned FG M R ES solver, perform ance gains are significant b u t m ore m odest due to some intrinsic p roperties of th e num erical m ethod. We
also note th a t th e m eth o d can sm ooth load im balances due to differences in conditioning of th e subdom ains, which cannot be fixed by rep artitio n in g .
O ur load balancing stra te g y is able to rectify large load im balances in th e m ulti-grain, block eigensolver we tested , b u t is m ore lim ited in th e dom ain-decom position linear system solver. In fu tu re work, it m ay be w orthw hile to see if we can achieve b e tte r resu lts w ith a linear solver by working w ith a m ulti-grain block K rylov-type solver, such as block FG M RES. A dditionally, our stra te g y m ay show b e tte r perform ance in a dom ain-decom position context involving coarser subproblem s, such as an inexact N ew ton m ethod em ploying non linear additive Schwarz preconditioning [17, 16]. In such a m ethod, a nonlinear problem is solved on each subdom ain, involving m uch m ore co m p u tatio n th a n th e sim ple, linear sub d o m ain solves of th e additive-Schw arz preconditioner for FG M R ES.
8.2
A d y n a m ic m e m o r y -a d a p ta tio n fra m ew o rk
T h e second idea we have explored is a general fram ew ork th a t allows scientific a n d oth er data-intensive applications to dynam ically a d a p t th eir m em ory requirem ents, facilitatin g graceful deg rad atio n of perform ance as m em ory pressure arises. It does so w ith m inim al reliance on o p e ra tin g system provided inform ation, a n d can be em bedded in m any scientific kernels w ith little coding effort using th e su p p o rtin g library, M Ml i b, we have developed. T h e key technical hurdle in realizing our fram ew ork is th e lack of inform ation a b o u t m em ory availability provided by o p e ra tin g system s. We have developed a relatively sim ple alg o rith m th a t judges m em ory availability effectively using a single m etric, th e resident set size of th e application. T h e m em o ry -ad ap tatio n fram ew ork is especially su ited to non-centrally adm inistered, open system s, and su p p o rts th e following functionality:
• m anagem ent of m ultiple m em ory objects w ith m ultiple active panels a t th e sam e tim e
• m em ory objects th a t can be m odified or vary in size
C H A P T E R 8. CO NCLUSIO NS A N D F U T U R E W O R K 118
• a u to m a tic estim atio n of th e size of m em ory not m anaged w ith in th e fram ew ork • user-defined panel replacem ent policies, which can optionally be co o rdinated w ith the
system policy th ro u g h a “m ost-m issing” alternative.
We have used our M Ml ib lib ra ry to em bed m em ory a d a p tiv ity into th ree scientific a p
plications: a conjugate gradient linear system solver, a m odified G ram -Schm idt orthogonal- ization routine, and a M onte-C arlo Ising m odel sim ulation. T hese applications or kernels, though sim ple, are representative of elem ents of a wide range of scientific a n d engineer ing codes. E xp erim en tal evaluation of th e applications in Linux an d Solaris environm ents d em o n stra te th a t th e m em o ry -ad ap tatio n fram ew ork yields considerable perform ance a d vantages over reliance on th e v irtu a l m em ory system to handle m em ory pressure. T hese advantages stem not only from enabling th e use of application-specific replacem ent policies, b u t from avoidance of v irtu a l m em ory system overheads an d antagonism . Im p o rtan tly , m ul tip le jobs em ploying th e m em o ry -ad ap tatio n fram ew ork can coexist on a system w ith o u t th rash in g , m onopolizing, or otherw ise d isru p tin g th e system .
We believe th a t our resu lts have d e m o n stra ted th e u tility of our m em o ry -a d a p ta tio n fram ew ork, b u t m any topics rem ain for fu tu re work. We have briefly investigated th e use of a “m ost-m issing” replacem ent policy th a t is coordinated w ith th e page reclam ation of th e o p eratin g system . O ur resu lts using th is policy to evict panels th a t have been m ostly evicted by th e o p e ra tin g system show prom ise, b u t th ere are some questions th a t m ust be answered: We suspect th a t we should always circum vent th e applicatio n specific policy du rin g m em ory shortage, b u t we m ust verify th a t th is is th e case or w hether it is som etim es a p p ro p riate to strike a balance betw een th e two policies. If a balance should be struck, we m ust d eterm ine th e thresholds th a t d icta te when th e M RU policy should be followed for a page th a t is targ e te d by th e m ost-m issing policy.
We have designed M Ml ib to require as little o p e ra tin g system provided in form ation as
possible, in p a rt to facilitate portability. We have teste d M Ml ib extensively u n d er L inux
tested it cursorily un d er M acOS X 10.1. It is im p o rta n t to evaluate M Ml ib un d er oth er
o p e ra tin g system s, whose m em ory-m anagem ent strategies m ay differ. For instance, we have not evaluated M Ml ib un d er system s th a t em ploy a page-fault frequency (P F F ) stra te g y
for th ra sh in g prevention.
We have d e m o n stra ted w ith M Ml ib th a t it is possible to hide m em o ry -ad ap tatio n deci
sions an d bookkeeping b ehind a sim ple lib ra ry interface. T he c u rren t version of th e lib rary is still som ew hat rudim entary, however, an d work should be done to fu rth e r refine it. For exam ple, m ore functions im plem enting com m on panel replacem ent policies should be added, an d interfaces to facilitate m apping m ore com plex d a ta stru c tu re s to panels should be pro vided. D eveloping a m ore “p ro d u ctio n read y ” version of M Ml ib will m ake easier w hat is
perh ap s th e m ost im p o rta n t work to be pursued: using M Ml ib in m ore types of an d m ore
com plex applications. O ur eventual goal is to use M Ml ib to im plem ent m em ory a d a p tiv ity
in a widely used scientific toolkit such as P E T S c. By providing features such as a m em ory- adaptive sparse m a trix class th a t su p p o rts th e s ta n d a rd P E T S c m a trix m ethods, we can enable scientists to em bed, w ith m inim al effort, m em ory-adaptivity into th e ir existing ap p li cation codes. Such ease of use is vital if we w ish to see ad o p tio n of o ur m em ory-adaptivity techniques in real-w orld applications.