Distributed parallelism

Learning outcomes

  • .
For teachers

Teaching goals are:

  • .

Prior:

  • .

Feedback:

  • .

Arnold (at the left): a robot that was controlled by MPI Cora, the robotic platform for Arnold

Why distributed parallelism is important

Type of parallelism Number of cores Number of nodes Memory Library
Single-threaded 1 1 As given by operating system None
Threaded/shared memory Multiple 1 Shared by all cores OpenMP
Distributed Multiple Multiple Distributed OpenMPI

Notes

  • Distributed programming. Uses a Message Passing Interface. For a job that use many different nodes, for example, a weather prediction.

Output

Using 2 OpenMP threads 

               Core t (s)   Wall t (s)        (%)
       Time:       86.902       43.452      200.0
                 (ns/day)    (hour/ns)
Performance:        1.740       13.794

               Core t (s)   Wall t (s)        (%)
       Time:      100.447       50.224      200.0
                 (ns/day)    (hour/ns)
Performance:        1.591       15.082

               Core t (s)   Wall t (s)        (%)
       Time:      150.753       37.689      400.0
                 (ns/day)    (hour/ns)
Performance:        3.783        6.345

               Core t (s)   Wall t (s)        (%)
       Time:      292.200       36.526      800.0
                 (ns/day)    (hour/ns)
Performance:        6.446        3.723

Remember

  • Use --ntasks=N
  • Use srun
  • Use an MPI version of your software: a ‘regular’ non-MPI version will never work!

Julia stuff here

MATLAB stuff here

R stuff here