AnalyticalModelingofParallel

Programs
S.Oliveira

08/25/05

S.Oliveira,22C177/22M178
Fall2005

ScalabilityofParallelSystems

Efficiencyofaparallelprogram
E=S/P=Ts/PTp

Usingtheparalleloverheadexpression
E=1/(1+To/Ts)

ThetotaloverheadfunctionToisanincreasing
functionofP.
Foragivenproblemsize,T remainsconstant,
s
asweincreasethenumberofP,Toincreases.
Efficiencydecreaseswithincreasingnumberof
processingelements~characteristiccommonto
S.Oliveira,22C177/22M178
allparallelprograms.
08/25/05
Fall2005
2

ScalabilityofParallelSystemsContd

SpeedupandefficiencyasfunctionstoP:

08/25/05

Tp=n/p+2logp

S=n/[(n/p)+2logp]
E=1/1+(2plogp/n)

BothEfficiencyandSpeedupincreasewithn,forthe
samenumberofP.
BothEfficiencyandSpeedupdropwithincreasingP.i
S.Oliveira,22C177/22M178
Fall2005

ScalabilityofParallelSystemsContd

Itispossibletokeeptheefficiencyinmany
parallelsystemsfixedbyincreasingboththe
sizeoftheproblemandthenumberof
processingelementssimultaneouslyscalable
parallelsystem.
Scalability:measureofaparallelsystems
capacitytoincreasespeedupinproportiontothe
numberofprocessingelements.
Recallthatacostoptimalparallelsystemhas
efficiencyof(1).So,scalabilityandcostoptimalityof

parallelsystemsarerelated.Wecanhaveacostoptimal
parallelsystemifthe#ofPandsizeofcomputationare
S.Oliveira,22C177/22M178
08/25/05
4
chosenappropriately. Fall2005

IsoefficiencyMetricofScalability

Wesaidthatforagivenproblemsize,asweincrease#of
P,theoverallefficiencyofparallelsystemdecreases.
ScalableParallelSystemisoneinwhichefficiencycanbe
keptconstantasthenumberofprocessingelementsis
increased,providedproblemsizeisalsoincreased.
TherateofchangeofbothPandproblemsizedetermine
thedegreeofscalabilityofaparallelsystem.Alowerrateis
moredesirablethanahighergrowthrateinproblemsize.
ProblemSizeisdefinedasthenumberofbasic
computationstepsinthebestsequentialalgorithmtosolve
theproblemonasingleP.
Weassumethatittakesunittimetoperformonebasic
computationofthealgorithm.ThusProblemsize=W=Ts
(timeoffastestknownsequentialalgorithm).
08/25/05

S.Oliveira,22C177/22M178
Fall2005

IsoefficiencyFunction

ParallelExecutionTimecanbeexpressedasafunctionof:

ProblemSize
Overheadfunction
NumberofProcessingelements

Wecanrewriteparallelruntime(Tp)as:

08/25/05

S.Oliveira,22C177/22M178
Fall2005

IsoefficiencyFunctioncontd

Conclusions:

Iftheproblemsize(W)iskeptconstantandpisincreased,
efficiency(E)decreasesbecausethetotaloverhead(To)
increaseswithp.
Fordifferentparallelsystems,Wmustbeincreasedatdifferent
rateswithrespecttopinordertomaintainafixedefficiency.
ForscalableparallelsystemsEcanbemaintainedatafixed
value(0<E<1)iftheratioTo/Wismaintainedconstant.

*
08/25/05

K=E/(1E),weget:
S.Oliveira,22C177/22M178
Fall2005

IsoefficiencyFunctioncontd
IsoefficiencyFunction:W=KTo(W,P).Thisfunction
determinestheeasewithwhichaparallelsystemcan
maintainaconstantefficiencyandachievespeedups
increasinginproportiontonumberofprocessingelements.
Asmallisoefficiencyfunctionmeansthatsmallincrementsin
problemsizearesufficientforefficientutilizationofincreasing
#ofP,indicatingthatthesystemishighlyscalable.
Howeveralargeisoefficiencyfunctionindicatesapoorly
scalableparallelsystem.
Theisoefficiencyfunctiondoesnotexistforunscalable
parallelsystemsbecausetheEcantbekeptatanyconstant
valueaspincreases,nomatterhowfasttheproblemsizeis
increased.
Finally,theisoefficiencyfunctioncapturesthecharacteristics
S.Oliveira,22C177/22M178
ofaparallelalgorithmaswellastheparallelarchitectureon
08/25/05
Fall2005
8
whichitisimplemented.

CostOptimality&Isoefficiency
Function

Aparallelsystemiscostoptimaliftheproductofthe
numberofprocessingelementsandtheparallel
executiontimeisproportionaltotheexecutiontimeof
thefastestknownsequentialalgorithmonasingle
processingelement.
AparallelsystemiscostoptimalIFF:

Theabove2equationssuggestthataparallelsystemis
costoptimalIFFitsoverheadfunctiondoesnot
asymptoticallyexceedtheproblemsize.
~mustbesatisfiedtoensurethecost
optimalityofaparallelsystemasitisscaledup.
S.Oliveira,22C177/22M178

08/25/05

Fall2005

LowerBoundonIsoefficiency
Function

Anideallyscalableparallelsystemmusthave
thelowestpossibleisoefficiencyfunction.
ForaproblemconsistingofWunitsofwork,no
morethanWprocessingelementscanbeused
costoptimally,asadditionalpwillbeidle.
Theproblemsizemustincreaseatleastasfast
as(p)tomaintainfixedefficiency;hence(p)
istheasymptoticlowerboundonthe
isoefficiencyfunction.

08/25/05

S.Oliveira,22C177/22M178
Fall2005

10

DegreeofConcurrency&
IsoefficiencyFunction

Themaximumnumberoftasksthatcanbeexecuted
simultaneouslyatanytimeinaparallelalgorithmis
calleditsdegreeofconcurrency.
DegreeofConcurrency:isameasureofthenumberof
operationsthatanalgorithmcanperforminparallelfora
problemofsizeW.
IfC(W)isthedegreeofconcurrencyofaparallel
algorithm,thenforaproblemofsizeW,nomorethanC
(W)processingelementscanbeemployedeffectively.
Theisoefficiencyfunctionduetoconcurrencyisoptimal,
i.e.(p),onlyifthedegreeofconcurrencyofthe
parallelalgorithmis
(W)
S.Oliveira,22C177/22M178

08/25/05

Fall2005

11

MinimumExecutionTime&Minimum
CostOptimalExecutionTime

Ifweincreasethenumberofprocessingelementsfora
givenproblemsize,eithertheparallelruntimecontinues
todecreaseandasymptoticallyapproachesaminimum
valueoritstartsrisingafterattainingaminimumvalue.
AproblemofsizeW,acostoptimalsolutionrequires
thatp=O(f1(W)).Sincetheparallelruntimeis(W/P)
foracostoptimalparallelsystem.

08/25/05

S.Oliveira,22C177/22M178
Fall2005

12

AsymptoticAnalysisofParallelPrograms.
Comparisonoffourdifferentalgorithmsforsorting
agivenlistofnumbers.Thetableshowsnumber
ofprocessingelements,parallelruntime,speedup,
efficiencyandthepTpproduct.

08/25/05

S.Oliveira,22C177/22M178
Fall2005

13