Escolar Documentos
Profissional Documentos
Cultura Documentos
Anintroductiontothe
MessagePassingInterface(MPI)
usingC
ThisisashortintroductiontotheMessagePassingInterface(MPI)designedtoconveythefundamental
operationanduseoftheinterface.Thisintroductionisdesignedforreaderswithsomebackground
programmingC,andshoulddeliverenoughinformationtoallowreaderstowriteandruntheirown(very
simple)parallelCprogramsusingMPI.
ThereexistsaversionofthistutorialforFortranprogramerscalledIntroductionthetheMessagePassing
Interface(MPI)usingFortran.
WhatisMPI?
MPIisalibraryofroutinesthatcanbeusedtocreateparallelprogramsinCorFortran77.StandardC
andFortranincludenoconstructssupportingparallelismsovendorshavedevelopedavarietyof
extensionstoallowusersofthoselanguagestobuildparallelapplications.Theresulthasbeenaspateof
nonportableapplications,andaneedtoretrainprogrammersforeachplatformuponwhichtheywork.
TheMPIstandardwasdevelopedtoamelioratetheseproblems.ItisalibrarythatrunswithstandardCor
Fortranprograms,usingcommonlyavailableoperatingsystemservicestocreateparallelprocessesand
exchangeinformationamongtheseprocesses.
MPIisdesignedtoallowuserstocreateprogramsthatcanrunefficientlyonmostparallelarchitectures.
Thedesignprocessincludedvendors(suchasIBM,Intel,TMC,Cray,Convex,etc.),parallellibrary
authors(involvedinthedevelopmentofPVM,Linda,etc.),andapplicationsspecialists.Thefinalversion
forthedraftstandardbecameavailableinMayof1994.
MPIcanalsosupportdistributedprogramexecutiononheterogenoushardware.Thatis,youmayruna
programthatstartsprocessesonmultiplecomputersystemstoworkonthesameproblem.Thisisuseful
withaworkstationfarm.
Helloworld
HereisthebasicHelloworldprograminCusingMPI:
#include<stdio.h>
#include<mpi.h>
main(intargc,char**argv)
{
intierr;
ierr=MPI_Init(&argc,&argv);
printf("Helloworld\n");
ierr=MPI_Finalize();
}
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 1/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
Ifyoucompilehello.cwithacommandlike
mpicchello.cohello
youwillcreateanexecutablefilecalledhello,whichyoucanexecutebyusingthempiruncommandasin
thefollowingsessionsegment:
$mpirunnp4hello
Helloworld
Helloworld
Helloworld
Helloworld
$
Whentheprogramstarts,itconsistsofonlyoneprocess,sometimescalledthe"parent","root",or
"master"process.WhentheroutineMPI_Initexecuteswithintherootprocess,itcausesthecreationof3
additionalprocesses(toreachthenumberofprocesses(np)specifiedonthempiruncommandline),
sometimescalled"child"processes.
Eachoftheprocessesthencontinuesexecutingseparateversionsofthehelloworldprogram.Thenext
statementineveryprogramistheprintfstatement,andeachprocessprints"Helloworld"asdirected.
Sinceterminaloutputfromeveryprogramwillbedirectedtothesameterminal,weseefourlinessaying
"Helloworld".
Identifyingtheseparateprocesses
Aswritten,wecannottellwhich"Helloworld"linewasprintedbywhichprocess.Toidentifyaprocess
weneedsomesortofprocessIDandaroutinethatletsaprocessfinditsownprocessID.MPIassignsan
integertoeachprocessbeginningwith0fortheparentprocessandincrementingeachtimeanewprocess
iscreated.AprocessIDisalsocalledits"rank".
MPIalsoprovidesroutinesthatlettheprocessdetermineitsprocessID,aswellasthenumberof
processesthatarehavebeencreated.
HereisanenhancedversionoftheHelloworldprogramthatidentifiestheprocessthatwriteseachline
ofoutput:
#include<stdio.h>
#include<mpi.h>
main(intargc,char**argv)
{
intierr,num_procs,my_id;
ierr=MPI_Init(&argc,&argv);
/*findoutMYprocessID,andhowmanyprocesseswerestarted.*/
ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);
printf("Helloworld!I'mprocess%ioutof%iprocesses\n",
my_id,num_procs);
ierr=MPI_Finalize();
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 2/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
Whenwerunthisprogram,eachprocessidentifiesitself:
$mpicchello2.cohello2
$mpirunnp4hello2
Helloworld!I'mprocess0outof4processes.
Helloworld!I'mprocess2outof4processes.
Helloworld!I'mprocess1outof4processes.
Helloworld!I'mprocess3outof4processes.
$
Notethattheprocessnumbersarenotprintedinascendingorder.Thatisbecausetheprocessesexecute
independentlyandexecutionorderwasnotcontrolledinanyway.Theprogramsmayprinttheirresultsin
differentorderseachtimetheyarerun.
(TofindoutwhichOriginprocessorsandmemoriesareusedtorunaprogramyoucanturnonthe
MPI_DSM_VERBOSEenvironmentvariablewith"exportMPI_DSM_VERBOSE=ON",orequivalent.)
Toleteachprocessperformadifferenttask,youcanuseaprogramstructurelike:
#include<mpi.h>
main(intargc,char**argv)
{
intmy_id,root_process,ierr,num_procs;
MPI_Statusstatus;
/*Createchildprocesses,eachofwhichhasitsownvariables.
*Fromthispointon,everyprocessexecutesaseparatecopy
*ofthisprogram.EachprocesshasadifferentprocessID,
*rangingfrom0tonum_procsminus1,andCOPIESofall
*variablesdefinedintheprogram.Novariablesareshared.
**/
ierr=MPI_Init(&argc,&argv);
/*findoutMYprocessID,andhowmanyprocesseswerestarted.*/
ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);
if(my_id==0){
/*dosomeworkasprocess0*/
}
elseif(my_id==1){
/*dosomeworkasprocess1*/
}
elseif(my_id==2){
/*dosomeworkasprocess2*/
}
else{
/*dothisworkinanyremainingprocesses*/
}
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 3/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
/*Stopthisprocess*/
ierr=MPI_Finalize();
}
BasicMPIcommunicationroutines
Itisimportanttorealizethatseparateprocessessharenomemoryvariables.Theyappeartobeusingthe
samevariables,buttheyarereallyusingCOPIESofanyvariabledefinedintheprogram.
Asaresult,theseprogramscannotcommunicatewitheachotherbyexchanginginformationinmemory
variables.InsteadtheymayuseanyofalargenumberofMPIcommunicationroutines.Thetwobasic
routinesare:
MPI_Send,tosendamessagetoanotherprocess,and
MPI_Recv,toreceiveamessagefromanotherprocess.
ThesyntaxofMPI_Sendis:
intMPI_Send(void*data_to_send,intsend_count,MPI_Datatypesend_type,
intdestination_ID,inttag,MPI_Commcomm);
data_to_send:variableofaCtypethatcorrespondstothesend_typesuppliedbelow
send_count:numberofdataelementstobesent(nonnegativeint)
send_type:datatypeofthedatatobesent(oneoftheMPIdatatypehandles)
destination_ID:processIDofdestination(int)
tag:messagetag(int)
comm:communicator(handle)
OnceaprogramcallsMPI_Send,itblocksuntilthedatatransferhastakenplaceandthedata_to_send
variablecanbesafelyreused.Asaresult,theseroutinesprovideasimplesynchronizationservicealong
withdataexchange.
ThesyntaxofMPI_Recvis:
intMPI_Recv(void*received_data,intreceive_count,MPI_Datatypereceive_type,
intsender_ID,inttag,MPI_Commcomm,MPI_Status*status);
received_data:variableofaCtypethatcorrespondstothereceive_typesuppliedbelow
receive_count:numberofdataelementsexpected(int)
receive_type:datatypeofthedatatobereceived(oneoftheMPIdatatypehandles)
sender_ID:processIDofthesendingprocess(int)
tag:messagetag(int)
comm:communicator(handle)
status:statusstruct(MPI_Status)
Thereceive_count,sender_ID,andtagvaluesmaybespecifiedsoastoallowmessagesofunknown
length,fromseveralsources(MPI_ANY_SOURCE),orwithvarioustagvalues(MPI_ANY_TAG).
Theamountofinformationactuallyreceivedcanthenberetrievedfromthestatusvariable,aswith:
countMPI_Get_count(&status,MPI_FLOAT,&true_received_count);
received_source=status.MPI_SOURCE;
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 4/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
received_tag=status.MPI_TAG;
MPI_Recvblocksuntilthedatatransferiscompleteandthereceived_datavariableisavailableforuse.
ThebasicdatatypesrecognizedbyMPIare:
MPIdatatypehandle Cdatatype
MPI_INT int
MPI_SHORT short
MPI_LONG long
MPI_FLOAT float
MPI_DOUBLE double
MPI_CHAR char
MPI_BYTE unsignedchar
MPI_PACKED
Therealsoexistothertypeslike:MPI_UNSIGNED,MPI_UNSIGNED_LONG,and
MPI_LONG_DOUBLE.
Acommonpatternofprocessinteraction
Acommonpatternofinteractionamongparallelprocessesisforone,themaster,toallocateworktoaset
ofslaveprocessesandcollectresultsfromtheslavestosynthesizeafinalresult.
Themasterprocesswillexecuteprogramstatementslike:
/*distributeportionsofarray1toslaves.*/
for(an_id=1;an_id<num_procs;an_id++){
start_row=an_id*num_rows_per_process;
ierr=MPI_Send(&num_rows_to_send,1,MPI_INT,
an_id,send_data_tag,MPI_COMM_WORLD);
ierr=MPI_Send(&array1[start_row],num_rows_per_process,
MPI_FLOAT,an_id,send_data_tag,MPI_COMM_WORLD);
}
/*and,thencollecttheresultsfromtheslaveprocesses,
*hereinavariablecalledarray2,anddosomethingwiththem.*/
for(an_id=1an_id<num_procs;an_id++){
ierr=MPI_Recv(&array2,num_rows_returned,MPI_FLOAT,
MPI_ANY_SOURCE,MPI_ANY_TAG,MPI_COMM_WORLD,&status);
/*dosomethingwitharray2here*/
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 5/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
/*andthenprintoutsomefinalresultusingthe
*informationcollectedfromtheslaves.*/
Inthisfragment,themasterprogramsendsacontiguousportionofarray1toeachslaveusingMPI_Send
andthenreceivesaresponsefromeachslaveviaMPI_Recv.Inpractice,themasterdoesnothavetosend
anarrayitcouldsendascalarorsomeotherMPIdatatype,anditcouldconstructarray1fromany
componentstowhichithasaccess.
Herethereturnedinformationisputinarray2,whichwillbewrittenovereverytimeadifferentmessage
isreceived.Therefore,itwillprobablybecopiedtosomeothervariablewithinthereceivingloop.
NotetheuseoftheMPIconstantMPI_ANY_SOURCEtoallowthisMPI_Recvcalltoreceivemessages
fromanyprocess.Insomecases,aprogramwouldneedtodetermineexactlywhichprocesssenta
messagereceivedusingMPI_ANY_SOURCE.status.MPI_SOURCEwillholdthatinformation,
immediatelyfollowingthecalltoMPI_Recv.
Theslaveprogramtoworkwiththismasterwouldresemble:
/*Receiveanarraysegment,herecalledarray2*/.
ierr=MPI_Recv(&num_rows_to_receive,1,MPI_INT,
root_process,MPI_ANY_TAG,MPI_COMM_WORLD,&status);
ierr=MPI_Recv(&array2,num_rows_to_receive,MPI_FLOAT,
root_process,MPI_ANY_TAG,MPI_COMM_WORLD,&status);
/*Dosomethingwitharray2here,placingtheresultinarray3,
*andsendarray3totherootprocess.*/
ierr=MPI_Send(&array3,num_rows_to_return,MPI_FLOAT,
root_process,return_data_tag,MPI_COMM_WORLD);
Therecouldbemanyslaveprogramsrunningatthesametime.Eachonewouldreceivedatainarray2
fromthemasterviaMPI_Recvandworkonitsowncopyofthatdata.Eachslavewouldconstructits
owncopyofarray3,whichitwouldthensendtothemasterusingMPI_Send.
Anonparallelprogramthatsumsthevaluesinanarray
Thefollowingprogramcalculatesthesumoftheelementsofaarray.Itwillbefollowedbyaparallel
versionofthesameprogramusingMPIcalls.
#include<stdio.h>
#definemax_rows10000000
intarray[max_rows];
main(intargc,char**argv)
{
inti,num_rows;
longintsum;
printf("pleaseenterthenumberofnumberstosum:");
scanf("%i",&num_rows);
if(num_rows>max_rows){
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 6/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
printf("Toomanynumbers.\n");
exit(1);
}
/*initializeanarray*/
for(i=0;i<num_rows;i++){
array[i]=i;
}
/*computesum*/
sum=0;
for(i=0;i<num_rows;i++){
sum+=array[i];
}
printf("Thegrandtotalis:%i\n",sum);
}
Designforaparallelprogramtosumanarray
ThecodebelowshowsacommonFortranstructureforincludingbothmasterandslavesegmentsinthe
parallelversionoftheexampleprogramjustpresented.Itiscomposedofashortsetupsectionfollowed
byasingleif...elseloopwherethemasterprocessexecutesthestatmentsbetweenthebracketsafter
theifstatement,andtheslaveprocessesexecutethestatementsbetweenthebracketsaftertheelse
statement.
/*ThisprogramsumsallrowsinanarrayusingMPIparallelism.
*Therootprocessactsasamasterandsendsaportionofthe
*arraytoeachchildprocess.Masterandchildprocessesthen
*allcalculateapartialsumoftheportionofthearrayassigned
*tothem,andthechildprocessessendtheirpartialsumsto
*themaster,whocalculatesagrandtotal.
**/
#include<stdio.h>
#include<mpi.h>
intmy_id,root_process,ierr,num_procs,an_id;
MPI_Statusstatus;
root_process=0;
/*Nowreplicatethisprocesstocreateparallelprocesses.
ierr=MPI_Init(&argc,&argv);
/*findoutMYprocessID,andhowmanyprocesseswerestarted*/
ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);
if(my_id==root_process){
/*Imustbetherootprocess,soIwillquerytheuser
*todeterminehowmanynumberstosum.
*initializeanarray,
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 7/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
*distributeaportionofthearraytoeachchildprocess,
*andcalculatethesumofthevaluesinthesegmentassigned
*totherootprocess,
*and,finally,Icollectthepartialsumsfromslaveprocesses,
*printthem,andaddthemtothegrandsum,andprintit*/
}
else{
/*Imustbeslaveprocess,soImustreceivemyarraysegment,
*calculatethesumofmyportionofthearray,
*and,finally,sendmyportionofthesumtotherootprocess.*/
/*Stopthisprocess*/
ierr=MPI_Finalize();
}
Thecompleteparallelprogramtosumaarray
HereistheexpandedparallelversionofthesameprogramusingMPIcalls.
/*ThisprogramsumsallrowsinanarrayusingMPIparallelism.
*Therootprocessactsasamasterandsendsaportionofthe
*arraytoeachchildprocess.Masterandchildprocessesthen
*allcalculateapartialsumoftheportionofthearrayassigned
*tothem,andthechildprocessessendtheirpartialsumsto
*themaster,whocalculatesagrandtotal.
**/
#include<stdio.h>
#include<mpi.h>
#definemax_rows100000
#definesend_data_tag2001
#definereturn_data_tag2002
intarray[max_rows];
intarray2[max_rows];
main(intargc,char**argv)
{
longintsum,partial_sum;
MPI_Statusstatus;
intmy_id,root_process,ierr,i,num_rows,num_procs,
an_id,num_rows_to_receive,avg_rows_per_process,
sender,num_rows_received,start_row,end_row,num_rows_to_send;
/*Nowreplictethisprocesstocreateparallelprocesses.
*Fromthispointon,everyprocessexecutesaseperatecopy
*ofthisprogram*/
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 8/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
ierr=MPI_Init(&argc,&argv);
root_process=0;
/*findoutMYprocessID,andhowmanyprocesseswerestarted.*/
ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);
if(my_id==root_process){
/*Imustbetherootprocess,soIwillquerytheuser
*todeterminehowmanynumberstosum.*/
printf("pleaseenterthenumberofnumberstosum:");
scanf("%i",&num_rows);
if(num_rows>max_rows){
printf("Toomanynumbers.\n");
exit(1);
}
avg_rows_per_process=num_rows/num_procs;
/*initializeanarray*/
for(i=0;i<num_rows;i++){
array[i]=i+1;
}
/*distributeaportionofthebectortoeachchildprocess*/
for(an_id=1;an_id<num_procs;an_id++){
start_row=an_id*avg_rows_per_process+1;
end_row=(an_id+1)*avg_rows_per_process;
if((num_rowsend_row)<avg_rows_per_process)
end_row=num_rows1;
num_rows_to_send=end_rowstart_row+1;
ierr=MPI_Send(&num_rows_to_send,1,MPI_INT,
an_id,send_data_tag,MPI_COMM_WORLD);
ierr=MPI_Send(&array[start_row],num_rows_to_send,MPI_INT,
an_id,send_data_tag,MPI_COMM_WORLD);
}
/*andcalculatethesumofthevaluesinthesegmentassigned
*totherootprocess*/
sum=0;
for(i=0;i<avg_rows_per_process+1;i++){
sum+=array[i];
}
printf("sum%icalculatedbyrootprocess\n",sum);
/*and,finally,Icolletthepartialsumsfromtheslaveprocesses,
*printthem,andaddthemtothegrandsum,andprintit*/
for(an_id=1;an_id<num_procs;an_id++){
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 9/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
for(an_id=1;an_id<num_procs;an_id++){
ierr=MPI_Recv(&partial_sum,1,MPI_LONG,MPI_ANY_SOURCE,
return_data_tag,MPI_COMM_WORLD,&status);
sender=status.MPI_SOURCE;
printf("Partialsum%ireturnedfromprocess%i\n",partial_sum,sender);
sum+=partial_sum;
}
printf("Thegrandtotalis:%i\n",sum);
}
else{
/*Imustbeaslaveprocess,soImustreceivemyarraysegment,
*storingitina"local"array,array1.*/
ierr=MPI_Recv(&num_rows_to_receive,1,MPI_INT,
root_process,send_data_tag,MPI_COMM_WORLD,&status);
ierr=MPI_Recv(&array2,num_rows_to_receive,MPI_INT,
root_process,send_data_tag,MPI_COMM_WORLD,&status);
num_rows_received=num_rows_to_receive;
/*Calculatethesumofmyportionofthearray*/
partial_sum=0;
for(i=0;i<num_rows_received;i++){
partial_sum+=array2[i];
}
/*andfinally,sendmypartialsumtohterootprocess*/
ierr=MPI_Send(&partial_sum,1,MPI_LONG,root_process,
return_data_tag,MPI_COMM_WORLD);
}
ierr=MPI_Finalize();
}
Thefollowingtableshowsthevaluesofseveralvariablesduringtheexecutionofsumarray_mpi.The
informationcomesfromatwoprocessorparallelrun,andthevaluesofprogramvariablesareshownin
bothprocessormemoryspaces.NotethatthereisonlyoneprocessactivepriortothecalltoMPI_Init.
Valuehistoriesofselectedvariables
withinthemasterandslaveprocesses
duringa2processexecutionofprogramsumarray_mpi
Before After After
Program Before After
MPI_Send MPI_Recv MPI_Recv
location MPI_Init MPI_Init
toslave byslave bymaster
Variable
Proc0 Proc0 Proc1 Proc0 Proc1 Proc0 Proc1 Proc0 Proc1
Name
root_process 0 0 0 0 0 0 0 0 0
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 10/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
my_id . 0 1 0 1 0 1 0 1
num_procs . 2 2 2 2 2 2 2 2
num_rows . . . 6 . 6 . 6 .
avg_rows_
. . . 3 . 3 . 3 .
per_process
num_rows_
. . . . . . 3 . 3
received
array[0] . . . 1.0 . 1.0 . 1.0 .
array[1] . . . 2.0 . 2.0 . 2.0 .
array[2] . . . 3.0 . 3.0 . 3.0 .
array[3] . . . 4.0 . 4.0 . 4.0 .
array[4] . . . 5.0 . 5.0 . 5.0 .
array[5] . . . 6.0 . 6.0 . 6.0 .
array2[0] . . . . . . 4.0 . 4.0
array2[1] . . . . . . 5.0 . 5.0
array2[2] . . . . . . 6.0 . 6.0
array2[3] . . . . . . . . .
array2[4] . . . . . . . . .
array2[5] . . . . . . . . .
partial_sum . . . . . . . 6.0 15.0
sum . . . . . . . 21.0 .
LoggingandtracingMPIactivity
ItispossibletousempiruntorecordMPIactivity,byusingtheoptionsmpilogandmpitrace.Formore
informationaboutthisfacilityseemanmpirun.
Collectiveoperations
MPI_SendandMPI_Recvare"pointtopoint"communicationsfunctions.Thatis,theyinvolveone
senderandonereceiver.MPIincludesalargenumberofsubroutinesforperforming"collective"
operations.CollectiveoperationareperformedbyMPIroutinesthatarecalledbyeachmemberofa
groupofprocessesthatwantsomeoperationtobeperformedforthemasagroup.Acollectivefunction
mayspecifyonetomany,manytoone,ormanytomanymessagetransmission.
MPIsupportsthreeclassesofcollectiveoperations:
synchronization,
datamovement,and
collectivecomputation
Theseclassesarenotmutuallyexclusive,ofcourse,sinceblockingdatamovementfunctionsalsoserve
tosynchronizeprocessactivity,andsomeMPIroutinesperformbothdatamovementandcomputation.
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 11/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
Synchronization
TheMPI_Barrierfunctioncanbeusedtosynchronizeagroupofprocesses.Tosynchronizeagroupof
processes,eachonemustcallMPI_Barrierwhenithasreachedapointwhereitcangonofurtheruntilit
knowsthatallitscohortshavereachedthesamepoint.OnceaprocesshascalledMPI_Barrier,itwillbe
blockeduntilallprocessesinthegrouphavealsocalledMPI_Barrier.
Collectivedatamovement
Thereareseveralroutinesforperformingcollectivedatadistributiontasks:
MPI_Bcast
Broadcastdatatootherprocesses
MPI_Gather,MPI_Gatherv
Gatherdatafromparticipatingprocessesintoasinglestructure
MPI_Scatter,MPI_Scatter
Breakastructureintoportionsanddistributethoseportionstootherprocesses
MPI_Allgather,MPI_Allgatherv
Gatherdatafromdifferentprocessesintoasinglestructurethatisthensenttoallparticipants
(Gathertoall)
MPI_Alltoall,MPI_Alltoallv
Gatherdataandthenscatterittoallparticipants(Alltoallscatter/gather)
Theroutineswith"V"suffixesmovevariablesizedblocksofdata.
ThesubroutineMPI_Bcastsendsamessagefromoneprocesstoallprocessesinacommunicator.
intMPI_Bcast(void*data_to_be_sent,intsend_count,MPI_Datatypesend_type,
intbroadcasting_process_ID,MPI_Commcomm);
Whenprocessesarereadytoshareinformationwithotherprocessesaspartofabroadcast,ALLofthem
mustexecuteacalltoMPI_BCAST.ThereisnoseparateMPIcalltoreceiveabroadcast.
MPI_Bcastcouldhavebeenusedintheprogramsumarray_mpipresentedearlier,inplaceofthe
MPI_Sendloopthatdistributeddatatoeachprocess.Doingsowouldhaveresultedinexcessivedata
movement,ofcourse.AbettersolutionwouldbeMPI_ScatterorMPI_Scatterv.
ThesubroutinesMPI_ScatterandMPI_Scattervtakeaninputarray,breaktheinputdataintoseparate
portionsandsendaportiontoeachoneoftheprocessesinacommunicatinggroup.
intMPI_Scatter(void*send_data,intsend_count,MPI_Datatypesend_type,
void*receive_data,intreceive_count,MPI_Datatypereceive_type,
intsending_process_ID,MPI_Commcomm);
or
intMPI_Scatterv(void*send_data,int*send_count_array,int*send_start_array,
MPI_Datatypesend_type,void*receive_data,intreceive_count,
MPI_Datatypereceive_type,intsender_process_ID,MPI_Commcomm);
data_to_send:variableofaCtypethatcorrespondstotheMPIsend_typesuppliedbelow
send_count:numberofdataelementstosend(int)
send_count_array:arraywithanentryforeachparticipatingprocesscontainingthenumberofdata
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 12/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
elementstosendtothatprocess(int)
send_start_array:arraywithanentryforeachparticipatingprocesscontainingthedisplacement
relativetothestartofdata_to_sendforeachdatasegmenttosend(int)
send_type:datatypeofelementstosend(oneoftheMPIdatatypehandles)
receive_data:variableofaCtypethatcorrespondstotheMPIreceive_typesuppliedbelow
receive_count:numberofdataelementstoreceive(int)
receive_type:datatypeofelementstoreceive(oneoftheMPIdatatypehandles)
sender_ID:processIDofthesender(int)
receive_tag:receivetag(int)
comm:communicator(handle)
status:statusobject(MPI_Status)
TheroutineMPI_Scattervcouldhavebeenusedintheprogramsumarray_mpipresentedearlier,inplace
oftheMPI_Sendloopthatdistributeddatatoeachprocess.
MPI_Bcast,MPI_Scatter,andothercollectiveroutinesbuildacommunicationtreeamongthe
participatingprocessestominimizemessagetraffic.IfthereareNprocessesinvolved,therewould
normallybeN1transmissionsduringabroadcastoperation,butifatreeisbuiltsothatthebroadcasting
processsendsthebroadcastto2processes,andtheyeachsenditonto2otherprocesses,thetotalnumber
ofmessagestransferredisonlyO(lnN).
Collectivecomputationroutines
Collectivecomputationissimilartocollectivedatamovementwiththeadditionalfeaturethatdatamay
bemodifiedasitismoved.Thefollowingroutinescanbeusedforcollectivecomputation.
MPI_Reduce
Performareductionoperation.Thatis,applysomeoperationtosomeoperandinevery
participatingprocess.Forexample,addanintegerresidingineveryprocesstogetherandputthe
resultinaprocessspecifiedintheMPI_Reduceargumentlist.
MPI_Allreduce
Performareductionleavingtheresultinallparticipatingprocesses
MPI_Reduce_scatter
Performareductionandthenscattertheresult
MPI_Scan
Performareductionleavingpartialresults(computeduptothepointofaprocess'sinvolvementin
thereductiontreetraversal)ineachparticipatingprocess.(parallelprefix)
ThesubroutineMPI_Reducecombinesdatafromallprocessesinacommunicatorusingoneofseveral
reductionoperationstoproduceasingleresultthatappearsinaspecifiedtargetprocess.
intMPI_Reduce(void*data_to_be_sent,void*result_to_be_received_by_target,
intsend_count,MPI_Datatypesend_type,MPI_Opoperation,
inttarget_process_ID,MPI_Commcomm);
Whenprocessesarereadytoshareinformationwithotherprocessesaspartofadatareduction,allofthe
participatingprocessesexecuteacalltoMPI_Reduce,whichuseslocaldatatocalculateeachprocess's
portionofthereductionoperationandcommunicatesthelocalresulttootherprocessesasnecessary.
Onlythetarget_process_IDreceivesthefinalresult.
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 13/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
MPI_Reducecouldhavebeenusedintheprogramsumarray_mpipresentedearlier,inplaceofthe
MPI_Recvloopthatcollectedpartialsumsfromeachprocess.
Collectivecomputationbuiltinoperations
ManyoftheMPIcollectivecomputationroutinestakebothbuiltinanduserdefinedcombination
functions.Thebuiltinfunctionsare:
Operationhandle Operation
MPI_MAX Maximum
MPI_MIN Minimum
MPI_PROD Product
MPI_SUM Sum
MPI_LAND LogicalAND
MPI_LOR LogicalOR
MPI_LXOR LogicalExclusiveOR
MPI_BAND BitwiseAND
MPI_BOR BitwiseOR
MPI_BXOR BitwiseExclusiveOR
MPI_MAXLOC Maximumvalueandlocation
MPI_MINLOC Minimumvalueandlocation
Acollectiveoperationexample
Thefollowingprogramintegratesthefunctionsin(X)overtherange0to2pi.Itwillbefollowedbya
parallelversionofthesameprogramthatusestheMPIlibrary.
/*programtointegratesin(x)between0andpibycomputing
*theareaofanumberofrectangleschosensoastoapproximate
*theshapeunderthecurveofthefunction.
*
*1)asktheusertochoosethenumberofintervals,
*2)computetheintervalwidth(rect_width),
*3)foreachinterval:
*
*a)findthemiddleoftheinterval(x_middle),
*b)computetheheightoftherectangle,sin(x_middle),
*c)findtheareaoftherectangleastheproductof
*theintervalwidthanditsheightsin(x_middle),and
*d)incrementarunningtotal.
**/
#include<stdio.h>
#include<math.h>
#definePI3.1415926535
main(intargc,char**argv)
{
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 14/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
inti,num_intervals;
doublerect_width,area,sum,x_middle;
printf("Pleaseenterthenumberofintervalstointerpolate:");
scanf("%i",&num_intervals);
rect_width=PI/num_intervals;
sum=0;
for(i=1;i<num_intervals+1;i++){
/*findthemiddleoftheintervalontheXaxis.*/
x_middle=(i0.5)*rect_width;
area=sin(x_middle)*rect_width;
sum=sum+area;
}
printf("Thetotalareais:%f\n",(float)sum);
}
ThenextprogramisanMPIversionoftheprogramabove.ItusesMPI_Bcasttosendinformationto
eachparticipatingprocessandMPI_Reducetogetagrandtotaloftheareascomputedbyeach
participatingprocess.
/*Thisprogramintegratessin(x)between0andpibycomputing
*theareaofanumberofrectangleschosensoastoapproximate
*theshapeunderthecurveofthefunctionusingMPI.
*
*Therootprocessactsasamastertoagroupofchildprocess
*thatactasslaves.Themasterpromptsforthenumberof
*interpolationsandbroadcaststhatvaluetoeachslave.
*
*Therearenum_procsprocessesalltogether,andaprocess
*computestheareadefinedbyeverynum_procsthinterval,
*collectsapartialsumofthoseareas,andsendsitspartial
*sumtotherootprocess.
**/
#include<stdio.h>
#include<math.h>
#include<mpi.h>
#definePI3.1415926535
main(intargc,char**argv)
{
intmy_id,root_process,num_procs,ierr,num_intervals,i;
doublerect_width,area,sum,x_middle,partial_sum;
MPI_Statusstatus;
/*Letprocess0betherootprocess.*/
root_process=0;
/*Nowreplicatethisprocesstocreateparallelprocesses.*/
ierr=MPI_Init(&argc,&argv);
/*FindoutMYprocessID,andhowmanyprocesseswerestarted.*/
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 15/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);
if(my_id==root_process){
/*Imustbetherootprocess,soIwillquerytheuser
todeterminehowmanyinterpolationintervalstouse.*/
printf("Pleaseenterthenumberofintervalstointerpolate:");
scanf("%i",&num_intervals);
}
/*Then...nomatterwhichprocessIam:
*
*Iengageinabroadcastsothatthenumberofintervalsis
*sentfromtherootprocesstotheotherprocesses,and...
**/
ierr=MPI_Bcast(&num_intervals,1,MPI_INT,root_process,
MPI_COMM_WORLD);
/*calculatethewidthofarectangle,and*/
rect_width=PI/num_intervals;
/*thencalculatethesumoftheareasoftherectanglesfor
*whichIamresponsible.Startwiththe(my_id+1)th
*intervalandprocesseverynum_procsthintervalthereafter.
**/
partial_sum=0;
for(i=my_id+1;i<num_intervals+1;i+=num_procs){
/*FindthemiddleoftheintervalontheXaxis.*/
x_middle=(i0.5)*rect_width;
area=sin(x_middle)*rect_width;
partial_sum=partial_sum+area;
}
printf("proc%icomputes:%f\n",my_id,(float)partial_sum);
/*andfinally,engageinareductioninwhichallpartialsums
*arecombined,andthegrandsumappearsinvariable"sum"in
*therootprocess,
**/
ierr=MPI_Reduce(&partial_sum,&sum,1,MPI_DOUBLE,
MPI_SUM,root_process,MPI_COMM_WORLD);
/*and,ifIamtherootprocess,printtheresult.*/
if(my_id==root_process){
printf("Theintegralis%f\n",(float)sum);
/*(yes,wecouldhavesummedjusttheheights,and
*postponedthemultiplicationbyrect_widthtilnow.)*/
}
/*Closedownthisprocesses.*/
ierr=MPI_Finalize();
}
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 16/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
Simultaneoussendandreceive
ThesubroutineMPI_Sendrecvexchangesmessageswithanotherprocess.Asendreceiveoperationis
usefulforavoidingsomekindsofunsafeinteractionpatternsandforimplementingremoteprocedure
calls.
AmessagesentbyasendreceiveoperationcanbereceivedbyMPI_Recvandasendreceiveoperation
canreceiveamessagesentbyanMPI_Send.
MPI_Sendrecv(&data_to_send,send_count,send_type,destination_ID,send_tag,
&received_data,receive_count,receive_type,sender_ID,receive_tag,
comm,&status)
data_to_send:variableofaCtypethatcorrespondstotheMPIsend_typesuppliedbelow
send_count:numberofdataelementstosend(int)
send_type:datatypeofelementstosend(oneoftheMPIdatatypehandles)
destination_ID:processIDofthedestination(int)
send_tag:sendtag(int)
received_data:variableofaCtypethatcorrespondstotheMPIreceive_typesuppliedbelow
receive_count:numberofdataelementstoreceive(int)
receive_type:datatypeofelementstoreceive(oneoftheMPIdatatypehandles)
sender_ID:processIDofthesender(int)
receive_tag:receivetag(int)
comm:communicator(handle)
status:statusobject(MPI_Status)
MPItags
MPI_SendandMPI_Recv,aswellasotherMPIroutines,allowtheusertospecifyatagvaluewitheach
transmission.Thesetagvaluesmaybeusedtospecifythemessagetype,or"context,"inasituation
whereaprogrammayreceivemessagesofseveraltypesduringthesameprogram.Thereceiversimply
checksthetagvaluetodecidewhatkindofmessageithasreceived.
MPIcommunicators
EveryMPIcommunicationoperationinvolvesa"communicator."Communicatorsidentifythegroupof
processesinvolvedinacommunicationoperationand/orthecontextinwhichitoccurs.Thesourceand
destinationprocessesspecifiedinpointtopointroutineslikeMPI_SendandMPI_Recvmustbe
membersofthespecifiedcommunicatorandthetwocallsmustreferencethesamecommunicator.
Collectiveoperationsincludejustthoseprocessesidentifiedbythecommunicatorspecifiedinthecalls.
ThecommunicatorMPI_COMM_WORLDisdefinedbydefaultforallMPIruns,andincludesall
processesdefinedbyMPI_Initduringthatrun.Additionalcommunicatorscanbedefinedthatincludeall
orpartofthoseprocesses.Forexample,supposeagroupofprocessesneedstoengageintwodifferent
reductionsinvolvingdisjointsetsofprocesses.Acommunicatorcanbedefinedforeachsubsetof
MPI_COMM_WORLDandspecifiedinthetworeductioncallstomanagemessagetransmission.
MPI_Comm_splitcanbeusedtocreateanewcommunicatorcomposedofasubsetofanother
communicator.MPI_Comm_dupcanbeusedtocreateanewcommunicatorcomposedofallofthe
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 17/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC
membersofanothercommunicator.Thismaybeusefulformanaginginteractionswithinasetof
processesinplaceofmessagetags.
Moreinformation
ThisshortintroductionomitsmanyMPItopicsandroutines,andcoversmosttopicsonlylightly.In
particular,itomitsdiscussionsoftopologies,unsafecommunicationandnonblockingcommunication.
Foradditionalinformationconcerningtheseandothertopicspleaseconsult:
themajorMPIWebsite,whereyouwillfindversionsofthestandards:
http://www.mcs.anl.gov/mpi
thebooks:
Gropp,Lusk,andSkjellum,UsingMPI:PortableParallelProgrammingwiththeMessage
PassingInterface,MITPress,1994
Foster,Ian,DesigningandBuildingParallelPrograms,availableinbothhardcopy
(AddisonWesleyPublishingCo.,1994)andonlineversions,
Pacheco,Peter,AUser'sGuidetoMPI,whichgivesatutorialintroductionextendedtocover
derivedtypes,communicatorsandtopologies,or
thenewsgroupcomp.parallel.mpi
Exercises
HerearesomeexercisesforcontinuingyourinvestigationofMPI:
Convertthehelloworldprogramtoprintitsmessagesinrankorder.
Converttheexampleprogramsumarray_mpitouseMPI_Scatterand/orMPI_Reduce.
Writeaprogramtofindallpositiveprimesuptosomemaximumvalue,usingMPI_Recvto
receiverequestsforintegerstotest.Themasterwillloopfrom2tothemaximumvalueon
1.issueMPI_Recvandwaitforamessagefromanyslave(MPI_ANY_SOURCE),
2.ifthemessageiszero,theprocessisjuststarting,
ifthemessageisnegative,itisanonprime,
ifthemessageispositive,itisaprime.
3.useMPI_Sendtosendanumbertotest.
andeachslavewillsendarequestforanumbertothemaster,receiveanintegertotest,testit,and
returnthatintegerifitisprime,butitsnegativevalueifitisnotprime.
Writeaprogramtosendatokenfromprocessortoprocessorinaloop.
Documentpreparedby:
DanielThomassetand
MichaelGrobe
AcademicComputingServices
TheUniversityofKansas
withassistanceandoverheadsprovidedby
TheNationalComputationalScienceAlliance(NCSA)at
TheUniversityofIllinois
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 18/18