A Data-Parallel Version of Aleph

MPI should not be confused with libraries implementing parallel numerical methods, or with parallelising compilers. MPI provides the message-passing infrastructure necessary for the communication between the nodes of a parallel computation, and does not automate in any way the actual parallelisation of the code as, for example, a parallelising compiler would.

It should also be pointed out that MPI is not a library, but an API specification. The advantage of conforming to the MPI specification is that programmes can link to any MPI library without modifications, allowing for greater portability between all kinds of varied and diverse parallel architectures. In the remainder of this section, the MPI functions that are pertinent to the implementation of the MPI version of Aleph will be introduced. For a more complete description of MPI, the reader is referred to the MPI standard defined and maintained by the MPI Forum [1,2] or user’s guides to either MPI in general or some particular MPI library [3,6].

It was mentioned above that MPI facilitates the communication between the nodes of a parallel architecture, but it would be more accurate to say that it facilitates the communication between the processes involved in a parallel computation. In order to start a program that is using an MPI library on a parallel machine, a separate, architecture-specific mechanism -a job-queueing system, for example -has to load identical copies of the program onto each and every node. The MPI library will then provide the methods for passing messages between these processes abstracting away from the architecture of the machine, so that they can be running on the nodes of a parallel computer, the workstations of a network, or even be multiple processes running on the same processor. It is of obvious benefit to spread the processes as evenly among the available resources as possible, but that is not part of the MPI protocol: it is the queueing system’s task to start the processes and enforce its queueing policy. For the remainder of this paper, process and node will be used interchangeably to refer to a node of the computation at the abstract, MPI level, regardless of how that maps to the actual processor nodes of the hardware.

The processes involved in a parallel computation are identified by communicator and rank. A communicator is a collection of nodes involved in a subtask of the computation, and a computation might keep all the available nodes within one communicator or split them among several. The goal of MPI is to facilitate fast and efficient intra-communicator communication, whereas intercommunicator communication is meant to be sparser and not as performancecritical.

Within each communicator, processes are identified by their rank, which is an enumeration of the processes, starting from 0. The node with rank 0 will be called the head node. MPI defines MPI_COMM_WORLD to be the global communicator that groups together all the nodes, useful for applications that do not need to group their nodes into separate communicators. Since this is case here, for the remainder of this paper all references to rank will refer to the node by that rank in the global communicator, and the communicator argument to MPI functions will not be shown, since it would invariably be MPI_COMM_WORLD.

The most basic operation that MPI facilitates is the point-to-point sending and receiving of a message. A message consists of an array of data, a type, and a tag. The type is one of several predefined data-types supported by MPI. All the the usual C data-types, like characters, integers, and floating-point numericals, are supported. It should, however, be noted that although MPI types are stored locally according to the native binary representation for that type, all the appropriate conversions1 are carried out when typed data is been transmitted, thus making it possible to spread a computation over heterogeneous workstation clusters.

Lastly, a message carries a numerical tag which can be interpreted as a message type. Messages with different tags ’live’ in a different space, and a receive action must specify (by tag) which sorts of messages it should be allowed to receive; the messages carrying any other tag are to be ignored. This mechanism allows for a certain degree of asynchroneity, as communications of different ‘sorts’ can be kept apart without having to rely on synchronising the sender and the receiver.

With the above concepts in mind, the elementary MPI operators will have the following C declarations: where MPI_Send() would dispatch count bytes from memory location message to the node of rank dest. To receive the message, the recipient must issue an MPI_Recv() specifying: the maxi

…(Full text truncated)…