Amdahls and Gustafson-Barsis laws revisited

Amdahl's and Gustafson-Barsis la ws revisited Andrzej Karb o wski Institute of Con trol and Computation Engineering, W arsa w Univ ersit y of T e hnology , ul. No w o wiejsk a 15/19, 00-665 W arsa w, P oland, NASK (Resear h and A ademi Computer Net w ork), ul. W¡ w ozo w a 18, 02-796 W arsa w, P oland E-mail: A.Karb o wskiia.p w.edu.pl Otob er 27, 2018 Abstrat The pap er presen ts a simple deriv ation of the Gustafson-Barsis la w from the Amdahl's la w. In the omputer literature these t w o la ws desribing the sp eedup limits of parallel appliations are deriv ed separately . It is sho wn, that treating the time of the exeution of the sequen tial part of the appliation as a onstan t, in few lines the Gustafson-Barsis la w an b e obtained from the Amdahl's la w and that the p opular laim, that Gustafson-Barsis la w o v erthro ws Amdahl's la w is a mistak e. Keyw ords: parallel omputing, distributed omputing, sp eedup 1 In tro dution The Amdahl's la w form ulated ab out four deades ago [ 1℄ is onsidered to b e one of the most inuen tial onepts in parallel and distributed pro essing [7℄. It desrib es the limits on the sp eedup obtained o wing to the exeution of the appliation on the parallel ma hine with relation to the single-pro essor, sequen tial ma hine. More preisely , Amdahl's la w sa ys, that the sp eedup of an appliation obtained o wing to the exeution on the parallel ma hine an- not b e greater that the reipro al of the sequen tial fration of the program. Sp eedup restritions resulting from Amdahl's la w prev en ted designers from exploiting parallelism for man y y ears, b eing a n uisane to v endors of parallel omputers [4℄. The resue ame from Sandia Labs. On the basis of some exp erimen ts, Gustafson [2 ℄ laimed that "the assumptions underlying Am- dahl's 1967 argumen t are inappropriate for the urren t approa h to massiv e ensem ble parallelism". F urthermore, Gustafson form ulated "an alternativ e to Amdahl's la w suggested b y E. Barsis at Sandia". The so-alled Gustafson- Barsis la w is said to vindiate the use of massiv ely parallel pro essing [5℄, [6 ℄. Ho w ev er, in the author's opinion, when w e analyze deep er b oth la ws, w e will see, that Gustafson's results do not refute the Amdahl's la w, and the Gustafson-Barsis la w an b e diretly deriv ed from the Amdahl's la w. 2 Amdahl's and Gustafson-Barsis la ws in the original form Although in the original Amdahl's pap er [1℄ there w ere no equations, basing on the v erbal desription one ma y presen t his onept formally . The w a y of our presen tation is similar to that of [3℄, [4℄, with only one dierene, whi h will b e explained later on. It is assumed in the mo del, that the program onsists of t w o parts: sequen tial and parallel. While the time of the exeu- tion of the sequen tial part for a giv en size n is the same on all ma hines, indep enden tly of the n um b er of pro essors p , the parallel part is p erfetly salable, that is, the time of its exeution on a ma hine with p pro essors is one p -th of the time of the exeution on the ma hine with one pro essor. Let us denote b y β ( n, p ) the sequen tial fration of the total real-time T ( n, p ) of the exeution of the program on a ma hine with p pro essors (the men- tioned dierene in tro dued here is treating b oth the fration β and time T as funtions of n and p ; it will pro v e to b e v ery useful afterw ards). With this notation w e ma y alulate the sequen tial part time T s for the giv en problem size n from the expression T s ( n ) = β ( n, 1) · T ( n, 1) (1) and the parallel part time T p , whi h is dep enden t on the problem size n and 2 the n um b er of pro essors p , from the expression T p ( n, p ) = (1 − β ( n, 1 )) · T ( n, 1) p (2) If w e ignore omm uniation osts and o v erhead osts asso iated with op er- ating system funtions, su h as pro ess reation, memory managemen t, et. [4℄, the total time T ( n, p ) will b e the sum of sequen tial and parallel part time, that is T ( n, p ) = T s ( n ) + T p ( n, p ) = β ( n, 1) · T ( n, 1) + (1 − β ( n, 1 )) · T ( n, 1) p = =  β ( n, 1) + 1 − β ( n, 1 ) p  · T ( n, 1) (3) F rom (3) w e get diretly the form ula for the sp eedup S ( n, p ) obtained due to the parallelization of the appliation: S ( n, p ) = T ( n, 1) T ( n, p ) = 1 β ( n, 1) + 1 − β ( n, 1) p (4) The form ula (4) is alled Amdahl's la w. It is seen, that in the limit S ( n, p ) → p → ∞ 1 β ( n, 1) (5) It means, that ev en when w e use innitely man y parallel pro essors, w e annot aelerate the alulations more than the reipro al of the sequen tial fration of the exeution time of the program on a sequen tial ma hine. That is, for example, when this fator equals 1 2 , the program an b e aelerated at most t wie, when 1 10  ten times! Sp eedup restritions resulting from Amdahl's la w prev en ted designers from exploiting parallelism for man y y ears, b eing a problem to v endors of parallel omputers [4℄. The help ame from Sandia Labs. In some exp erimen ts desrib ed b y Gustafson [2℄ it w as tak en, that the run time w as onstan t, while the problem size saled with the n um b er of pro essors. More preisely , the time of the sequen tial part w as indep enden t, while the w ork to b e done in parallel v aried linearly with the n um b er of pro essors. Sine the time of the exeution in Gustafson's pap er [2 ℄ w as normalized to 1, that is T s ( n ) + T p ( n, p ) = 1 (6) 3 w e had atually the equiv alene β ( n, p ) ≡ T s ( n ) (7) and T p ( n, p ) = 1 − β ( n, p ) (8) F ollo wing Gustafson, a serial pro essor w ould require time T s ( n ) + T p ( n, p ) · p to p erform the task, so the saled sp eedup on the parallel system w as equal: S ( n, p ) = T s ( n ) + T p ( n, p ) · p T s ( n ) + T p ( n, p ) = T s ( n ) + T p ( n, p ) · p = p + (1 − p ) · T s ( n ) (9) Using the equiv alene (7 ) w e ma y write (9) in the follo wing form: S ( n, p ) = p + (1 − p ) · T s ( n ) = p + (1 − p ) · β ( n, p ) = p − ( p − 1) · β ( n, p ) (10) The last equation is alled Gustafson-Barsis la w. 3 The main results In the Gustafson's pap er [2℄, three things raise some doubts: 1. Mixing the problem size and the n um b er of pro essors, treating b oth as tigh tly onneted ("the problem size sales with the n um b er of pro- essors") 2. Normalizing the time of alulations on the sequen tial ma hine to 1 (eq. (6)) for all problem sizes and n um b ers of pro essors 3. T reating assessmen t (10 ) as a b etter alternativ e to Amdahl's la w, de- riv ed indep enden tly , basing on dieren t assumptions The truth is, that Gustafson-Barsis la w is nothing but a dieren t form of Amdahl's la w, and that b etter v alues of the sp eedup in the Gustafson's ex- p erimen ts with the gro wing size of the problem ould b e obtained diretly from the Amdahl's la w. T o sho w this it is suien t to notie, that for a giv en problem size n there is a onstan t in all exeutions of the program, on ma hines with dieren t n um b er of pro essors. This onstan t is the time of the exeution of 4 the sequen tial part T s ( n ) for the giv en problem size n . It is indep enden t of the n um b er of pro essors p , that is: T s ( n ) = β ( n, p ) · T ( n, p ) = const., p = 1 , 2 , 3 , . . . (11) So, it will b e for an y p = 1 , 2 , 3 , . . . β ( n, 1) · T ( n, 1) = β ( n, p ) · T ( n, p ) (12) F rom the equation (12 ) w e get: β ( n, 1) = β ( n, p ) · T ( n, p ) T ( n, 1) (13) Replaing β ( n, 1) in equation (3 ) b y (13) w e will get: T ( n, p ) = β ( n, p ) · T ( n, p ) + T ( n, 1) p − β ( n, p ) · T ( n, p ) p (14) No w, m ultiplying b oth sides b y p and mo ving all omp onen ts with T ( n, p ) to the left hand side w e reeiv e: p · T ( n, p ) − p · β ( n, p ) · T ( n, p ) + β ( n, p ) · T ( n, p ) = T ( n, 1) (15) W e will get the v alue of sp eedup S ( n, p ) obtained o wing to the parallelization dividing b oth sides of the equation (15 ) b y T ( n, p ) . So, it will b e equal: S ( n, p ) = T ( n, 1) T ( n, p ) = p − ( p − 1) · β ( n, p ) (16) In this w a y w e reeiv ed nothing but Gustafson-Barsis la w ( 10). What onerns the b etter sp eedup in Gustafson's exp erimen ts with the gro wing size of the problem (and the n um b er of pro essors whi h w as link ed there) w e ma y explain it in the follo wing w a y . Gustafson assumed, that the time sp en t in the serial part ("for v etor startup, program loading, serial b ottlene ks, I/O op erations") do not dep end on the problem size, that is T s ( n ) = const. = T s = β ( n, 1) · T ( n, 1) = β s · T (1 , 1 ) , ∀ n (17) while the total time of the exeution of the parallel part on the sequan tial ma hine w as prop ortional to the problem size n . In this w a y the serial fator on the sequen tial ma hine β ( n, 1) w as equal β ( n, 1) = β s · T (1 , 1 ) β s · T (1 , 1 ) + n · (1 − β s ) · T (1 , 1) = β s β s + n · (1 − β s ) = 1 1 + n · ( 1 β s − 1) (18) 5 A similar situation w ould b e when the time T s ( n ) is prop ortional to the prob- lem size n (e.g. n · β s ), but the time sp en t in the parallel part is prop ortional to n 2 (e.g. n 2 · (1 − β s ) ). In su h ases β ( n, 1) → n → ∞ 0 (19) what means, taking in to aoun t (4), that S ( n, p ) → n → ∞ p (20) In other w ords, also from Amdahl's la w w e ma y onlude, that the bigger the size of the problem, the loser the sp eedup to the n um b er of pro essors. 4 Conlusions In the pap er it is sho wn, that the Gustafson-Barsis la w an b e diretly deriv ed from the Amdahl's la w, without strange assumptions as normalizng to one the time of exeution of the program on the sequen tial ma hine. Moreo v er, the sp eedups approa hing the n um b er of pro essors observ ed in the exp erimen ts desrib ed in the Gustafson's pap er an b e onluded from the Amdahl's la w, when w e tak e in to aoun t as the argumen ts of the serial fator the size of the problem and the n um b er of pro essors. A  kno wledgmen ts This resear h w as supp orted b y the P olish Ministry of Siene and Higher Eduation under gran t N N514 416934 (y ears 2008-2010). Referenes [1℄ G. M. Amdahl, V alidit y of the Single-Pro essor Approa h to A  hieving Large Sale Computing Capabilities, In AFIPS Conferene Pro eedings v ol. 30 (A tlan ti Cit y , N.J., Apr. 18-20), AFIPS Press, Reston, V a., pp. 483485, April 1967. [2℄ J. Gustafson, Reev aluating Amdahls La w, Comm uniations of the A CM 31(5), pp. 532533, 1988. 6 [3℄ T. Lewis, The next 10 , 000 2 y ears: P art I, IEEE Computer, 29(4), pp. 64  70, 1996. [4℄ T.G. Lewis, H. El-Rewini, In tro dution to P arallel Computing, Pren tie- Hall, Englew o o d Clis, N.J., 1992. [5℄ E.D. Reilly , Milestones in Computer Siene and Information T e hnol- ogy , Green w o o d Press, 2003. [6℄ A. Ralston, E.D. Reilly , D. Hemmendinger (eds.), Enylop edia of Com- puter Siene, F ourth Edition, Wiley , 2003. [7℄ M.D. Theys, S. Ali, H.J. Siegel, M. Chandy , K. Hw ang , K. Kennedy , L. Sha, K.G. Shin, M. Snir, L. Sn yder, T. Sterling, What are the top ten most inuen tial parallel and distributed pro essing onepts of the past millenium?, Journal of P arallel and Distributed Computing, 61 (12), pp. 1827-1841, 2001. 7