SlideShare a Scribd company logo

May2010 hex-core-opt

•
Jeff Larkin, profile picture
Jeff Larkin

This document discusses porting, scaling, and optimizing applications on Cray XT systems. It covers topics such as choosing compilers, profiling and debugging applications at scale, understanding CPU affinity, and improvements in the Cray Message Passing Toolkit (MPT). The document provides guidance on leveraging different compilers, collecting performance data using hardware counters and CrayPAT, understanding MPI process binding, and enhancements in MPT 4.0 related to MPI standards support and communication optimizations.

Technology
1 of 101
Downloaded 27 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
Porting, Scaling, and Optimization on Cray XT Systems Jeff Larkin larkin@cray.com These slides are available at http://www.slideshare.net/jefflarkin/may2010-hex-core-opt
COMPILING May 10 Slide 2
Choosing a Compiler Try every compiler available to you, there’s no “best” PGI, Cray, Pathscale, Intel, GCC are all available, but not necessarily on all machines Each compiler favors certain optimizations, which may benefit applications differently Test your answers carefully Order of operation may not be the same between compilers or even compiler versions You may have to decide whether speed or precision is more important to you May 10 Slide 3
Choosing Compiler Flags PGI -fast –Mipa=fast man pgf90; man pgcc; man pgCC Cray <none, turned on by default> man crayftn; man craycc ; man crayCC Pathscale -Ofast man eko (“Every Known Optimization”) GNU -O2 / -O3 man gfortran; man gcc; man g++ Intel -fast man ifort; man icc; man iCC May 10 Slide 4
PROFILING AND DEBUGGING May 10 Slide 5
Profile and Debug “at Scale” Codes don’t run the same at thousands of cores as hundreds, 10s of thousands as thousands, or 100s of thousands as 10s Determine how many nodes you wish to run on and test at that size, don’t test for throughput Scaling (Time) 2000 1500 Time (s) 1000 v1 (time) 500 v2 (time) 0 May 10 Slide 6
Profile and Debug with Real Science Choose a real problem, not a toy What do you want to achieve with the machine? How will you really run your code? Will you really run without I/O? May 10 Slide 7
What Data Should We Collect? Compiler Feedback Most compilers can tell you a lot about what they do to your code Did your important loop vectorize? Was a routine inlined? Was this loop unrolled? It’s just as important to know what the compiler didn’t do Some C/C++ loops won’t vectorize without some massaging Some loop counts aren’t known at runtime, so optimization is limited Flags to know: PGI: -Minfo=all –Mneginfo=all Cray: -rm (Fortran) (or also –O[neg]msgs), -hlist=m (C) (or also –h [neg]msgs) Pathscale: -LNO:simd_verbose=ON Intel: -vec-report1 May 10 Slide 8
Compiler Feedback Examples: PGI ! Matrix Multiply 24, Loop interchange do k = 1, N produces reordered loop nest: 25,24,26 do j = 1, N 26, Generated an alternate do i = 1, N loop for the loop c(i,j) = c(i,j) + & Generated vector a(i,k) * b(k,j) sse code for the loop end do Generated 2 end do prefetch instructions end do for the loop May 10 Slide 9
Compiler Feedback Examples: Cray 23. ! Matrix Multiply 24. ib------------< do k = 1, N 25. ib ibr4-------< do j = 1, N 26. ib ibr4 Vbr4--< do i = 1, N 27. ib ibr4 Vbr4 c(i,j) = c(i,j) + a(i,k) * b(k,j) 28. ib ibr4 Vbr4--> end do 29. ib ibr4-------> end do 30. ib------------> end do i – interchanged b – blocked r – unrolled V - Vectorized May 10 Slide 10
Compiler Feedback Examples: Intel mm.F90(14): (col. 3) remark: PERMUTED LOOP WAS VECTORIZED. mm.F90(25): (col. 7) remark: LOOP WAS VECTORIZED. May 10 Slide 11
What Data Should We Collect? Hardware Performance Counters Using tools like CrayPAT it’s possible to know what the processor is doing We can get counts for FLOPS Cache Hits/Misses TLB Hits/Misses Stalls … We can derive FLOP Rate Cache Hit/Miss Ratio Computational Intensity … May 10 Slide 12
Gathering Performance Data Use Craypat’s APA First gather sampling for line number profile Light-weight Guides what should and should not be instrumented load module Second gather instrumentation (-g make mpi,io,blas,lapack,math …) pat_build -O apa a.out Hardware counters Execute MPI message passing pat_report *.xf information Examine *.apa I/O information pat_build –O *.apa Execute Math Libraries …
Sample APA File (Significantly Trimmed) # You can edit this file, if desired, and use it # ---------------------------------------------------------------------- # HWPC group to collect by default. -Drtenv=PAT_RT_HWPC=1 # Summary with TLB metrics. # ---------------------------------------------------------------------- # Libraries to trace. -g mpi,math # ---------------------------------------------------------------------- -w # Enable tracing of user-defined functions. # Note: -u should NOT be specified as an additional option. # 9.08% 188347 bytes -T ratt_i_ # 6.71% 177904 bytes -T rhsf_ # 5.61% 205682 bytes -T ratx_i_ # 5.59% 22005 bytes -T transport_m_computespeciesdiffflux_
CrayPAT Groups biolib Cray Bioinformatics library routines omp OpenMP API (not supported on blacs Basic Linear Algebra Catamount) communication subprograms omp-rtl OpenMP runtime library (not blas Basic Linear Algebra subprograms supported on Catamount) caf Co-Array Fortran (Cray X2 systems portals Lightweight message passing API only) pthreads POSIX threads (not supported on fftw Fast Fourier Transform library (64- Catamount) bit only) scalapack Scalable LAPACK hdf5 manages extremely large and shmem SHMEM complex data collections stdio all library functions that accept or heap dynamic heap return the FILE* construct io includes stdio and sysio groups sysio I/O system calls lapack Linear Algebra Package system system calls lustre Lustre File System upc Unified Parallel C (Cray X2 systems math ANSI math only) mpi MPI netcdf network common data form (manages array-oriented scientific data)
Useful API Calls, for the control-freaks Regions, useful to break up long routines int PAT_region_begin (int id, const char *label) int PAT_region_end (int id) Disable/Enable Profiling, useful for excluding initialization int PAT_record (int state) Flush buffer, useful when program isn’t exiting cleanly int PAT_flush_buffer (void)
Sample CrayPAT HWPC Data USER / rhsf_ ------------------------------------------------------------------------ Time% 74.6% Time 556.742885 secs Imb.Time 14.817686 secs Imb.Time% 2.6% Calls 2.3 /sec 1200.0 calls PAPI_L1_DCM 14.406M/sec 7569486532 misses PAPI_TLB_DM 0.225M/sec 117992047 misses PAPI_L1_DCA 921.729M/sec 484310815400 refs PAPI_FP_OPS 871.740M/sec 458044890200 ops User time (approx) 525.438 secs 1103418882813 cycles 94.4%Time Average Time per Call 0.463952 sec CrayPat Overhead : Time 0.0% HW FP Ops / User time 871.740M/sec 458044890200 ops 10.4%peak(DP) HW FP Ops / WCT 822.722M/sec Computational intensity 0.42 ops/cycle 0.95 ops/ref MFLOPS (aggregate) 1785323.32M/sec TLB utilization 4104.61 refs/miss 8.017 avg uses D1 cache hit,miss ratios 98.4% hits 1.6% misses D1 cache utilization (M) 63.98 refs/miss 7.998 avg uses May 10 Slide 17
CrayPAT HWPC Groups 0 Summary with instruction 11 Floating point operations mix metrics (2) 1 Summary with TLB metrics 12 Floating point operations mix (vectorization) 2 L1 and L2 metrics 13 Floating point operations mix 3 Bandwidth information (SP) 4 Hypertransport information 14 Floating point operations mix 5 Floating point mix (DP) 6 Cycles stalled, resources idle 15 L3 (socket-level) 7 Cycles stalled, resources full 16 L3 (core-level reads) 8 Instructions and branches 17 L3 (core-level misses) 9 Instruction cache 18 L3 (core-level fills caused by 10 Cache hierarchy L2 evictions) 19 Prefetches May 10 Slide 18
MPI Statistics How many times are MPI routines called and with how much data? Do I have a load imbalance? Are my processors waiting for data? Could I perform better by adjusting MPI environment variables? May 10 Slide 19
Sample CrayPAT MPI Statistics MPI Msg Bytes | MPI Msg | MsgSz | 16B<= | 256B<= | 4KB<= |Experiment=1 | Count | <16B | MsgSz | MsgSz | MsgSz |Function | | Count | <256B | <4KB | <64KB | Caller | | | Count | Count | Count | PE[mmm] 3062457144.0 | 144952.0 | 15022.0 | 39.0 | 64522.0 | 65369.0 |Total |--------------------------------------------------------------------------- | 3059984152.0 | 129926.0 | -- | 36.0 | 64522.0 | 65368.0 |mpi_isend_ ||-------------------------------------------------------------------------- || 1727628971.0 | 63645.1 | -- | 4.0 | 31817.1 | 31824.0 |MPP_DO_UPDATE_R8_3DV.in.MPP_DOMAINS_MOD 3| | | | | | | MPP_UPDATE_DOMAIN2D_R8_3DV.in.MPP_DOMAINS_MOD ||||------------------------------------------------------------------------ 4||| 1680716892.0 | 61909.4 | -- | -- | 30949.4 | 30960.0 |DYN_CORE.in.DYN_CORE_MOD 5||| | | | | | | FV_DYNAMICS.in.FV_DYNAMICS_MOD 6||| | | | | | | ATMOSPHERE.in.ATMOSPHERE_MOD 7||| | | | | | | MAIN__ 8||| | | | | | | main |||||||||------------------------------------------------------------------- 9|||||||| 1680756480.0 | 61920.0 | -- | -- | 30960.0 | 30960.0 |pe.13666 9|||||||| 1680756480.0 | 61920.0 | -- | -- | 30960.0 | 30960.0 |pe.8949 9|||||||| 1651777920.0 | 54180.0 | -- | -- | 23220.0 | 30960.0 |pe.12549 |||||||||=================================================================== May 10 Slide 20
I/O Statistics How much time is spent in I/O? How much data am I writing or reading? How many processes are performing I/O? May 10 Slide 21
Understanding Cray Message Passing Toolkit (mpt) COMMUNICATION May 10 Slide 22
Understanding CPU Affinity MPICH_CPUMASK_DISPLAY env variable Displays MPI process CPU affinity mask export MPICH_CPUMASK_DISPLAY=1 aprun –n 8 –N 8 –cc cpu ./mpi_exe [PE_0]: cpumask set to 1 cpu on nid00036, cpumask = 00000001 [PE_1]: cpumask set to 1 cpu on nid00036, cpumask = 00000010 [PE_2]: cpumask set to 1 cpu on nid00036, cpumask = 00000100 [PE_3]: cpumask set to 1 cpu on nid00036, cpumask = 00001000 [PE_4]: cpumask set to 1 cpu on nid00036, cpumask = 00010000 [PE_5]: cpumask set to 1 cpu on nid00036, cpumask = 00100000 [PE_6]: cpumask set to 1 cpu on nid00036, cpumask = 01000000 [PE_7]: cpumask set to 1 cpu on nid00036, cpumask = 10000000 aprun –n 8 –N 8 –cc numa_node ./mpi_exe [PE_0]: cpumask set to 4 cpus on nid00036, cpumask = 00001111 [PE_1]: cpumask set to 4 cpus on nid00036, cpumask = 00001111 [PE_2]: cpumask set to 4 cpus on nid00036, cpumask = 00001111 [PE_3]: cpumask set to 4 cpus on nid00036, cpumask = 00001111 [PE_4]: cpumask set to 4 cpus on nid00036, cpumask = 11110000 [PE_5]: cpumask set to 4 cpus on nid00036, cpumask = 11110000 [PE_6]: cpumask set to 4 cpus on nid00036, cpumask = 11110000 [PE_7]: cpumask set to 4 cpus on nid00036, cpumask = 11110000 July 2009 Slide 23
MPT 4.0 (Released Q4 2009) Merge to ANL MPICH2 1.1 Support for the MPI 2.1 Standard (except dynamic processes) Binary-compatible with Intel MPI Additional MPI-IO Optimizations Improvements in Allgatherv at > 2K processors 10X – 2000x improvement Improvements in Scatterv at > 2KB message size 20-80% improvement Initial Gemini Support July 2009 Slide 24
Auto-Scaling MPI Environment Variables Key MPI variables that change their default values dependent on job size MPICH_MAX_SHORT_MSG_SIZE MPICH_PTL_UNEX_EVENTS MPICH_UNEX_BUFFER_SIZE MPICH_PTL_OTHER_EVENTS Higher scaling of MPT jobs with fewer tweaks to env variables “Default” values are based on total number of ranks in job See MPI man page for specific formulas used We don’t always get it right Adjusted defaults aren't perfect for all applications Assumes a somewhat communication-balanced application Users can always override the new defaults Understanding and fine-tuning these variables may help performance July 2009 Slide 25
Cray MPI XT Portals Communications Short Message Eager Protocol The sending rank “pushes” the message to the receiving rank Used for messages MPICH_MAX_SHORT_MSG_SIZE bytes or less Sender assumes that receiver can handle the message Matching receive is posted - or - Has available event queue entries (MPICH_PTL_UNEX_EVENTS) and buffer space (MPICH_UNEX_BUFFER_SIZE) to store the message Long Message Rendezvous Protocol Messages are “pulled” by the receiving rank Used for messages greater than MPICH_MAX_SHORT_MSG_SIZE bytes Sender sends small header packet with information for the receiver to pull over the data Data is sent only after matching receive is posted by receiving rank July 2009 Slide 26
Match Entries Posted by MPI MPT Eager Protocol Incoming Msg S E to handle Unexpected Msgs Data “pushed” to the receiver A S App ME Eager Rendezvous (MPICH_MAX_SHORT_MSG_SIZE bytes or less) Short Msg ME Long Msg ME T A R STEP 1 STEP 2 MPI_RECV call Sender MPI_SEND call Receiver Post ME to Portals MPI RANK 0 RANK 1 Unexpected Buffers (MPICH_UNEX_BUFFER_SIZE) STEP 3 Portals DMA PUT Unexpected Msg Queue Other Event Queue (MPICH_PTL_OTHER_EVENTS) Unexpected Event Queue MPI_RECV is posted prior to MPI_SEND call (MPICH_PTL_UNEX_EVENTS) July 2009 Slide 27
Match Entries Posted by MPI MPT Eager Protocol Incoming Msg S E to handle Unexpected Msgs Data “pushed” to the receiver A S Eager Short Msg ME Rendezvous Long Msg ME (MPICH_MAX_SHORT_MSG_SIZE bytes or less) T A R STEP 3 STEP 1 MPI_RECV call Sender MPI_SEND call Receiver No Portals ME MPI RANK 0 RANK 1 Unexpected Buffers STEP 2 STEP 4 (MPICH_UNEX_BUFFER_SIZE) Portals DMA PUT Memcpy of data Unexpected Msg Queue Unexpected Event Queue (MPICH_PTL_UNEX_EVENTS) MPI_RECV is not posted prior to MPI_SEND call July 2009 Slide 28
MPT Rendezvous Protocol S Match Entries Posted by MPI to handle Unexpected Msgs Data “pulled” by the receiver Incoming Msg E ( > MPICH_MAX_SHORT_MSG_SIZE bytes ) A Eager Rendezvous S T Short Msg ME Long Msg ME A App ME R STEP 1 STEP 3 MPI_SEND call MPI_RECV call Portals ME created Triggers GET request Sender Receiver MPI RANK 0 RANK 1 Unexpected STEP 2 Buffers Portals DMA PUT of Header STEP 4 Receiver issues GET request to match Sender ME Unexpected Msg Queue STEP 5 Portals DMA of Data Unexpected Event Queue Data is not sent until MPI_RECV is issued July 2009 Slide 29
Auto-Scaling MPI Environment Variables Default values for various MPI jobs sizes MPI Environment Variable Name 1,000 PEs 10,000 PEs 50,000 PEs 100,000 PEs MPICH_MAX_SHORT_MSG_SIZE (This size determines whether 128,000 20,480 4096 2048 the message uses the Eager or bytes Rendezvous protocol) MPICH_UNEX_BUFFER_SIZE 60 MB 60 MB 150 MB 260 MB (The buffer allocated to hold the unexpected Eager data) MPICH_PTL_UNEX_EVENTS (Portals generates two events for 20,480 22,000 110,000 220,000 each unexpected message events received) MPICH_PTL_OTHER_EVENTS 2048 (Portals send-side and expected 2500 12,500 25,000 events) events July 2009 Slide 30
Cray MPI Collectives Our Collectives Strategy Improve performance over standard ANL MPICH2 algorithms Tune for our interconnect(s) Work for any intra-communicator (not just MPI_COMM_WORLD) Enabled by default Can be selectively disabled via MPICH_COLL_OPT_OFF export MPICH_COLL_OPT_OFF=mpi_bcast,mpi_allreduce Many have user-adjustable cross-over points (see man page) Cray Optimized Collectives MPI_Allgather ( for small messages ) MPI_Alltoall ( changes to the order of exchanges ) MPI_Alltoallv / MPI_Alltoallw ( windowing algorithm ) Cray Optimized SMP-aware Collectives MPI_Allreduce MPI_Barrier MPI_Bcast ( new in MPT 3.1.1 ) MPI_Reduce ( new in MPT 3.1.2 ) July 2009 Slide 31
SMP-aware Collectives – Allreduce Example STEP 1 STEP 2 STEP 3 Identify Node-Captain rank. Perform an Allreduce with node- Perform a local on-node Perform a local on-node captains only. This reduces the bcast. NO network traffic. reduction to node-captain. process count by a factor of 8 on NO network traffic. XT5. July 2009 Slide 32
Performance Comparison of MPI_Allreduce Default vs MPICH_COLL_OPT_OFF=MPI_Allreduce Percent Improvement of SMP-aware MPI_Allreduce (compared to MPICH2 algorithm) 1024 PEs on an Istanbul System 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% Allreduce 30.0% 20.0% 10.0% 0.0% Message Size July 2009 Slide 33
Performance Comparison of MPI_Bcast Default vs MPICH_COLL_OPT_OFF=MPI_Bcast Percent Improvement of SMP-aware MPI_Bcast Percent Improvement of SMP-aware MPI_Bcast (compared to MPICH2 algorithm) 1024 PEs on an Istanbul System 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% Bcast 30.0% 20.0% 10.0% 0.0% Message Size July 2009 Slide 34
Performance Comparison of MPI_Reduce Default vs MPICH_COLL_OPT_OFF=MPI_Reduce Percent Improvement of SMP-aware MPI_Reduce (compared to MPICH2 algorithm) 1024 PEs on an Istanbul System 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% Reduce 30.0% 20.0% 10.0% 0.0% Message Size July 2009 Slide 35
Performance Comparison of MPI_Allgather Default vs MPICH_COLL_OPT_OFF=MPI_Allgather Percent Improvement of Optimized MPI_Allgather (compared to MPICH2 algorithm) 1024 PEs on an Istanbul System 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% Allgather 40.0% 30.0% 20.0% 10.0% 0.0% 1 2 4 8 16 32 64 128 256 512 1024 2048 Message Size July 2009 Slide 36
MPI-IO Improvements MPI-IO collective buffering MPICH_MPIIO_CB_ALIGN=0 Divides the I/O workload equally among all aggregators Inefficient if multiple aggregators reference the same physical I/O block Default setting in MPT 3.2 and prior versions MPICH_MPIIO_CB_ALIGN=1 Divides the I/O workload up among the aggregators based on physical I/O boundaries and the size of the I/O request Allows only one aggregator access to any stripe on a single I/O call Available in MPT 3.1 MPICH_MPIIO_CB_ALIGN=2 Divides the I/O workload into Lustre stripe-sized groups and assigns them to aggregators Persistent across multiple I/O calls, so each aggregator always accesses the same set of stripes and no other aggregator accesses those stripes Minimizes Lustre file system lock contention Default setting in MPT 3.3 July 2009 Slide 37
IOR benchmark 1,000,000 bytes MPI-IO API , non-power-of-2 blocks and transfers, in this case blocks and transfers both of 1M bytes and a strided access pattern. Tested on an XT5 with 32 PEs, 8 cores/node, 16 stripes, 16 aggregators, 3220 segments, 96 GB file 1800 1600 1400 MB/Sec 1200 1000 800 600 400 200 0 July 2009
HYCOM MPI-2 I/O On 5107 PEs, and by application design, a subset of the PEs(88), do the writes. With collective buffering, this is further reduced to 22 aggregators (cb_nodes) writing to 22 stripes. Tested on an XT5 with 5107 PEs, 8 cores/node 4000 3500 3000 2500 MB/Sec 2000 1500 1000 500 0 July 2009
OPTIMIZING YOUR CODE May 10 Slide 40
Step by Step 1. Fix any load imbalance 2. Fix your hotspots 1. Communication • Pre-post receives • Overlap computation and communication • Reduce collectives • Adjust MPI environment variables • Use rank reordering 2. Computation • Examine the hardware counters and compiler feedback • Adjust the compiler flags, directives, or code structure to improve performance 3. I/O • Stripe files/directories appropriately • Use methods that scale • MPI-IO or Subsetting At each step, check your answers and performance. Between each step, gather your data again. May 10 Slide 41
Detecting a load imbalance COMMUNICATION July 2009 Slide 42
Craypat load-imbalance data Table 1: Profile by Function Group and Function Time % | Time | Imb. Time | Imb. | Calls |Experiment=1 | | | Time % | |Group | | | | | Function | | | | | PE='HIDE' 100.0% | 1061.141647 | -- | -- | 3454195.8 |Total |-------------------------------------------------------------------- | 70.7% | 750.564025 | -- | -- | 280169.0 |MPI_SYNC ||------------------------------------------------------------------- || 45.3% | 480.828018 | 163.575446 | 25.4% | 14653.0 |mpi_barrier_(sync) || 18.4% | 195.548030 | 33.071062 | 14.5% | 257546.0 |mpi_allreduce_(sync) || 7.0% | 74.187977 | 5.261545 | 6.6% | 7970.0 |mpi_bcast_(sync) ||=================================================================== | 15.2% | 161.166842 | -- | -- | 3174022.8 |MPI ||------------------------------------------------------------------- || 10.1% | 106.808182 | 8.237162 | 7.2% | 257546.0 |mpi_allreduce_ || 3.2% | 33.841961 | 342.085777 | 91.0% | 755495.8 |mpi_waitall_ ||=================================================================== | 14.1% | 149.410781 | -- | -- | 4.0 |USER ||------------------------------------------------------------------- || 14.0% | 148.048597 | 446.124165 | 75.1% | 1.0 |main |====================================================================
Fixing a Load Imbalance What is causing the load imbalance Computation Is decomposition appropriate? Would RANK_REORDER help? Communication Is decomposition appropriate? Would RANK_REORDER help? Are receives pre-posted? OpenMP may help Able to spread workload with less overhead Large amount of work to go from all-MPI to Hybrid • Must accept challenge to OpenMP-ize large amount of code May 10 Slide 44
Pre-Post your Recvs COMMUNICATION July 2009 Slide 45
5/10/2010 46
5/10/2010 47
5/10/2010 48
5/10/2010 49
5/10/2010 50
5/10/2010 51
5/10/2010 52
5/10/2010 53
5/10/2010 54
5/10/2010 55
Tweak the Library COMMUNICATION July 2009 Slide 56
MPICH Performance Variable MPICH_MAX_SHORT_MSG_SIZE Controls message sending protocol (Default:128000 byte) Message sizes <= MSG_SIZE: Use EAGER Message sizes > MSG_SIZE: Use RENDEZVOUS Increasing this variable may require that MPICH_UNEX_BUFFER_SIZE be increased Increase MPICH_MAX_SHORT_MSG_SIZE if App sends large msgs(>128K) and receives are pre-posted Can reduce messaging overhead via EAGER protocol Can reduce network contention Decrease MPICH_MAX_SHORT_MSG_SIZE if: App sends msgs in 32k-128k range and receives not pre-posted Cray Inc. Proprietary 57
MPICH Performance Variable: MPICH_PTL_MATCH_OFF If set => Disables Portals matching Matching happens on the Opteron Requires extra copy for EAGER protocol Reduces MPI_Recv Overhead Helpful for latency-sensitive application Large # of small messages Small message collectives (<1024 bytes) When can this be slower? Pre-posted Receives can slow it down When extra copy time longer than post-to-Portals time For medium to larger messages (16k-128k range) 58
Custom Rank Ordering CrayPAT can provide suggestions based on metrics from a profiled run. If you understand your decomposition well enough, you may be able to map it to the network We can help you find a better order based on your grid Is your “nearest neighbor” really your “nearest neighbor?” May 10 Slide 59
COMPUTATION May 10 Slide 60
Vectorization Stride one memory accesses Loop on the “fast” indices No IF tests No subroutine calls Inline Module Functions Statement Functions What is size of loop? Vectorizing might not help with very small loops Unroll small loops Loop nest Stride one on inside Longest on the inside Increase computational intensity CI = (vector flops/number of memory accesses)
Bad Striding (64) ( 5) COMMON A(8,8,IIDIM,8),B(8,8,iidim,8) ( 59) DO 41090 K = KA, KE, -1 ( 60) DO 41090 J = JA, JE ( 61) DO 41090 I = IA, IE ( 62) A(K,L,I,J) = A(K,L,I,J) - B(J,1,i,k)*A(K+1,L,I,1) ( 63) * - B(J,2,i,k)*A(K+1,L,I,2) - B(J,3,i,k)*A(K+1,L,I,3) ( 64) * - B(J,4,i,k)*A(K+1,L,I,4) - B(J,5,i,k)*A(K+1,L,I,5) ( 65) 41090 CONTINUE ( 66) PGI 59, Loop not vectorized: loop count too small 60, Interchange produces reordered loop nest: 61, 60 Loop unrolled 5 times (completely unrolled) 61, Generated vector sse code for inner loop Pathscale (lp41090.f:62) Non-contiguous array "A(_BLNK__.0.0)" reference exists. Loop was not vectorized. (lp41090.f:62) Non-contiguous array "A(_BLNK__.0.0)" reference exists. Loop was not vectorized. (lp41090.f:62) Non-contiguous array "A(_BLNK__.0.0)" reference exists. Loop was not vectorized. (lp41090.f:62) Non-contiguous array "A(_BLNK__.0.0)" reference exists. Loop was not vectorized. April 2008 ORNL/NICS Users' Meeting 62
Rewrite ( 6) COMMON AA(IIDIM,8,8,8),BB(IIDIM,8,8,8) ( 95) DO 41091 K = KA, KE, -1 ( 96) DO 41091 J = JA, JE ( 97) DO 41091 I = IA, IE ( 98) AA(I,K,L,J) = AA(I,K,L,J) - BB(I,J,1,K)*AA(I,K+1,L,1) ( 99) * - BB(I,J,2,K)*AA(I,K+1,L,2) - BB(I,J,3,K)*AA(I,K+1,L,3) ( 100) * - BB(I,J,4,K)*AA(I,K+1,L,4) - BB(I,J,5,K)*AA(I,K+1,L,5) ( 101) 41091 CONTINUE PGI 95, Loop not vectorized: loop count too small 96, Outer loop unrolled 5 times (completely unrolled) 97, Generated 3 alternate loops for the inner loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Pathscale (lp41090.f:99) LOOP WAS VECTORIZED. April 2008 ORNL/NICS Users' Meeting 63
April 2008 ORNL/NICS Users' Meeting 64
C pointers ( 53) void mat_mul_daxpy(double *a, double *b, double *c, int rowa, int cola, int colb) ( 54) { ( 55) int i, j, k; /* loop counters */ ( 56) int rowc, colc, rowb; /* sizes not passed as arguments */ ( 57) double con; /* constant value */ ( 58) ( 59) rowb = cola; ( 60) rowc = rowa; ( 61) colc = colb; ( 62) ( 63) for(i=0;i<rowc;i++) { ( 64) for(k=0;k<cola;k++) { ( 65) con = *(a + i*cola +k); ( 66) for(j=0;j<colc;j++) { ( 67) *(c + i*colc + j) += con * *(b + k*colb + j); ( 68) } ( 69) } ( 70) } ( 71) } mat_mul_daxpy: 66, Loop not vectorized: data dependency Loop not vectorized: data dependency Loop unrolled 4 times Slide 65
C pointers, rewrite ( 53) void mat_mul_daxpy(double* restrict a, double* restrict b, double* restrict c, int rowa, int cola, int colb) ( 54) { ( 55) int i, j, k; /* loop counters */ ( 56) int rowc, colc, rowb; /* sizes not passed as arguments */ ( 57) double con; /* constant value */ ( 58) ( 59) rowb = cola; ( 60) rowc = rowa; ( 61) colc = colb; ( 62) ( 63) for(i=0;i<rowc;i++) { ( 64) for(k=0;k<cola;k++) { ( 65) con = *(a + i*cola +k); ( 66) for(j=0;j<colc;j++) { ( 67) *(c + i*colc + j) += con * *(b + k*colb + j); ( 68) } ( 69) } ( 70) } ( 71) } Slide 66
C pointers, rewrite 66, Generated alternate loop with no peeling - executed if loop count <= 24 Generated vector sse code for inner loop Generated 2 prefetch instructions for this loop Generated vector sse code for inner loop Generated 2 prefetch instructions for this loop Generated alternate loop with no peeling and more aligned moves - executed if loop count <= 24 and alignment test is passed Generated vector sse code for inner loop Generated 2 prefetch instructions for this loop Generated alternate loop with more aligned moves - executed if loop count >= 25 and alignment test is passed Generated vector sse code for inner loop Generated 2 prefetch instructions for this loop • This can also be achieved with the PGI safe pragma and –Msafeptr compiler option or Pathscale –OPT:alias option Slide 67
Nested Loops ( 47) DO 45020 I = 1, N ( 48) F(I) = A(I) + .5 ( 49) DO 45020 J = 1, 10 ( 50) D(I,J) = B(J) * F(I) ( 51) DO 45020 K = 1, 5 ( 52) C(K,I,J) = D(I,J) * E(K) ( 53) 45020 CONTINUE PGI 49, Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Loop unrolled 2 times (completely unrolled) Pathscale (lp45020.f:48) LOOP WAS VECTORIZED. (lp45020.f:48) Non-contiguous array "C(_BLNK__.0.0)" reference exists. Loop was not vectorized. 68
Rewrite ( 71) DO 45021 I = 1,N ( 72) F(I) = A(I) + .5 ( 73) 45021 CONTINUE ( 74) ( 75) DO 45022 J = 1, 10 ( 76) DO 45022 I = 1, N ( 77) D(I,J) = B(J) * F(I) ( 78) 45022 CONTINUE ( 79) ( 80) DO 45023 K = 1, 5 ( 81) DO 45023 J = 1, 10 ( 82) DO 45023 I = 1, N ( 83) C(K,I,J) = D(I,J) * E(K) ( 84) 45023 CONTINUE 69
PGI 73, Generated an alternate loop for the inner loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop 78, Generated 2 alternate loops for the inner loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop 82, Interchange produces reordered loop nest: 83, 84, 82 Loop unrolled 5 times (completely unrolled) 84, Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Pathscale (lp45020.f:73) LOOP WAS VECTORIZED. (lp45020.f:78) LOOP WAS VECTORIZED. (lp45020.f:78) LOOP WAS VECTORIZED. (lp45020.f:84) Non-contiguous array "C(_BLNK__.0.0)" reference exists. Loop was not vectorized. (lp45020.f:84) Non-contiguous array "C(_BLNK__.0.0)" reference exists. Loop was not vectorized. 70
71
Big Loop ( 52) C THE ORIGINAL ( 53) ( 54) DO 47020 J = 1, JMAX ( 55) DO 47020 K = 1, KMAX ( 56) DO 47020 I = 1, IMAX ( 57) JP = J + 1 ( 58) JR = J - 1 ( 59) KP = K + 1 ( 60) KR = K - 1 ( 61) IP = I + 1 ( 62) IR = I - 1 ( 63) IF (J .EQ. 1) GO TO 50 ( 64) IF( J .EQ. JMAX) GO TO 51 ( 65) XJ = ( A(I,JP,K) - A(I,JR,K) ) * DA2 ( 66) YJ = ( B(I,JP,K) - B(I,JR,K) ) * DA2 ( 67) ZJ = ( C(I,JP,K) - C(I,JR,K) ) * DA2 ( 68) GO TO 70 ( 69) 50 J1 = J + 1 ( 70) J2 = J + 2 ( 71) XJ = (-3. * A(I,J,K) + 4. * A(I,J1,K) - A(I,J2,K) ) * DA2 ( 72) YJ = (-3. * B(I,J,K) + 4. * B(I,J1,K) - B(I,J2,K) ) * DA2 ( 73) ZJ = (-3. * C(I,J,K) + 4. * C(I,J1,K) - C(I,J2,K) ) * DA2 ( 74) GO TO 70 ( 75) 51 J1 = J - 1 ( 76) J2 = J - 2 ( 77) XJ = ( 3. * A(I,J,K) - 4. * A(I,J1,K) + A(I,J2,K) ) * DA2 ( 78) YJ = ( 3. * B(I,J,K) - 4. * B(I,J1,K) + B(I,J2,K) ) * DA2 ( 79) ZJ = ( 3. * C(I,J,K) - 4. * C(I,J1,K) + C(I,J2,K) ) * DA2 ( 80) 70 CONTINUE ( 81) IF (K .EQ. 1) GO TO 52 ( 82) IF (K .EQ. KMAX) GO TO 53 ( 83) XK = ( A(I,J,KP) - A(I,J,KR) ) * DB2 ( 84) YK = ( B(I,J,KP) - B(I,J,KR) ) * DB2 ( 85) ZK = ( C(I,J,KP) - C(I,J,KR) ) * DB2 ( 86) GO TO 71 5/10/2010 72
Big Loop ( 87) 52 K1 = K + 1 ( 88) K2 = K + 2 ( 89) XK = (-3. * A(I,J,K) + 4. * A(I,J,K1) - A(I,J,K2) ) * DB2 ( 90) YK = (-3. * B(I,J,K) + 4. * B(I,J,K1) - B(I,J,K2) ) * DB2 ( 91) ZK = (-3. * C(I,J,K) + 4. * C(I,J,K1) - C(I,J,K2) ) * DB2 ( 92) GO TO 71 ( 93) 53 K1 = K - 1 ( 94) K2 = K - 2 ( 95) XK = ( 3. * A(I,J,K) - 4. * A(I,J,K1) + A(I,J,K2) ) * DB2 ( 96) YK = ( 3. * B(I,J,K) - 4. * B(I,J,K1) + B(I,J,K2) ) * DB2 ( 97) ZK = ( 3. * C(I,J,K) - 4. * C(I,J,K1) + C(I,J,K2) ) * DB2 ( 98) 71 CONTINUE ( 99) IF (I .EQ. 1) GO TO 54 ( 100) IF (I .EQ. IMAX) GO TO 55 ( 101) XI = ( A(IP,J,K) - A(IR,J,K) ) * DC2 ( 102) YI = ( B(IP,J,K) - B(IR,J,K) ) * DC2 ( 103) ZI = ( C(IP,J,K) - C(IR,J,K) ) * DC2 ( 104) GO TO 60 ( 105) 54 I1 = I + 1 ( 106) I2 = I + 2 ( 107) XI = (-3. * A(I,J,K) + 4. * A(I1,J,K) - A(I2,J,K) ) * DC2 ( 108) YI = (-3. * B(I,J,K) + 4. * B(I1,J,K) - B(I2,J,K) ) * DC2 ( 109) ZI = (-3. * C(I,J,K) + 4. * C(I1,J,K) - C(I2,J,K) ) * DC2 ( 110) GO TO 60 ( 111) 55 I1 = I - 1 ( 112) I2 = I - 2 ( 113) XI = ( 3. * A(I,J,K) - 4. * A(I1,J,K) + A(I2,J,K) ) * DC2 ( 114) YI = ( 3. * B(I,J,K) - 4. * B(I1,J,K) + B(I2,J,K) ) * DC2 ( 115) ZI = ( 3. * C(I,J,K) - 4. * C(I1,J,K) + C(I2,J,K) ) * DC2 ( 116) 60 CONTINUE ( 117) DINV = XJ * YK * ZI + YJ * ZK * XI + ZJ * XK * YI ( 118) * - XJ * ZK * YI - YJ * XK * ZI - ZJ * YK * XI ( 119) D(I,J,K) = 1. / (DINV + 1.E-20) ( 120) 47020 CONTINUE ( 121) 5/10/2010 73
PGI 55, Invariant if transformation Loop not vectorized: loop count too small 56, Invariant if transformation Pathscale Nothing 5/10/2010 74
Re-Write ( 141) C THE RESTRUCTURED ( 142) ( 143) DO 47029 J = 1, JMAX ( 144) DO 47029 K = 1, KMAX ( 145) ( 146) IF(J.EQ.1)THEN ( 147) ( 148) J1 = 2 ( 149) J2 = 3 ( 150) DO 47021 I = 1, IMAX ( 151) VAJ(I) = (-3. * A(I,J,K) + 4. * A(I,J1,K) - A(I,J2,K) ) * DA2 ( 152) VBJ(I) = (-3. * B(I,J,K) + 4. * B(I,J1,K) - B(I,J2,K) ) * DA2 ( 153) VCJ(I) = (-3. * C(I,J,K) + 4. * C(I,J1,K) - C(I,J2,K) ) * DA2 ( 154) 47021 CONTINUE ( 155) ( 156) ELSE IF(J.NE.JMAX) THEN ( 157) ( 158) JP = J+1 ( 159) JR = J-1 ( 160) DO 47022 I = 1, IMAX ( 161) VAJ(I) = ( A(I,JP,K) - A(I,JR,K) ) * DA2 ( 162) VBJ(I) = ( B(I,JP,K) - B(I,JR,K) ) * DA2 ( 163) VCJ(I) = ( C(I,JP,K) - C(I,JR,K) ) * DA2 ( 164) 47022 CONTINUE ( 165) ( 166) ELSE ( 167) ( 168) J1 = JMAX-1 ( 169) J2 = JMAX-2 ( 170) DO 47023 I = 1, IMAX ( 171) VAJ(I) = ( 3. * A(I,J,K) - 4. * A(I,J1,K) + A(I,J2,K) ) * DA2 ( 172) VBJ(I) = ( 3. * B(I,J,K) - 4. * B(I,J1,K) + B(I,J2,K) ) * DA2 ( 173) VCJ(I) = ( 3. * C(I,J,K) - 4. * C(I,J1,K) + C(I,J2,K) ) * DA2 ( 174) 47023 CONTINUE ( 175) ( 176) ENDIF 5/10/2010 75
Re-Write ( 178) IF(K.EQ.1) THEN ( 179) ( 180) K1 = 2 ( 181) K2 = 3 ( 182) DO 47024 I = 1, IMAX ( 183) VAK(I) = (-3. * A(I,J,K) + 4. * A(I,J,K1) - A(I,J,K2) ) * DB2 ( 184) VBK(I) = (-3. * B(I,J,K) + 4. * B(I,J,K1) - B(I,J,K2) ) * DB2 ( 185) VCK(I) = (-3. * C(I,J,K) + 4. * C(I,J,K1) - C(I,J,K2) ) * DB2 ( 186) 47024 CONTINUE ( 187) ( 188) ELSE IF(K.NE.KMAX)THEN ( 189) ( 190) KP = K + 1 ( 191) KR = K - 1 ( 192) DO 47025 I = 1, IMAX ( 193) VAK(I) = ( A(I,J,KP) - A(I,J,KR) ) * DB2 ( 194) VBK(I) = ( B(I,J,KP) - B(I,J,KR) ) * DB2 ( 195) VCK(I) = ( C(I,J,KP) - C(I,J,KR) ) * DB2 ( 196) 47025 CONTINUE ( 197) ( 198) ELSE ( 199) ( 200) K1 = KMAX - 1 ( 201) K2 = KMAX - 2 ( 202) DO 47026 I = 1, IMAX ( 203) VAK(I) = ( 3. * A(I,J,K) - 4. * A(I,J,K1) + A(I,J,K2) ) * DB2 ( 204) VBK(I) = ( 3. * B(I,J,K) - 4. * B(I,J,K1) + B(I,J,K2) ) * DB2 ( 205) VCK(I) = ( 3. * C(I,J,K) - 4. * C(I,J,K1) + C(I,J,K2) ) * DB2 ( 206) 47026 CONTINUE ( 207) ENDIF ( 208) 5/10/2010 76
Re-Write ( 209) I = 1 ( 210) I1 = 2 ( 211) I2 = 3 ( 212) VAI(I) = (-3. * A(I,J,K) + 4. * A(I1,J,K) - A(I2,J,K) ) * DC2 ( 213) VBI(I) = (-3. * B(I,J,K) + 4. * B(I1,J,K) - B(I2,J,K) ) * DC2 ( 214) VCI(I) = (-3. * C(I,J,K) + 4. * C(I1,J,K) - C(I2,J,K) ) * DC2 ( 215) ( 216) DO 47027 I = 2, IMAX-1 ( 217) IP = I + 1 ( 218) IR = I – 1 ( 219) VAI(I) = ( A(IP,J,K) - A(IR,J,K) ) * DC2 ( 220) VBI(I) = ( B(IP,J,K) - B(IR,J,K) ) * DC2 ( 221) VCI(I) = ( C(IP,J,K) - C(IR,J,K) ) * DC2 ( 222) 47027 CONTINUE ( 223) ( 224) I = IMAX ( 225) I1 = IMAX - 1 ( 226) I2 = IMAX - 2 ( 227) VAI(I) = ( 3. * A(I,J,K) - 4. * A(I1,J,K) + A(I2,J,K) ) * DC2 ( 228) VBI(I) = ( 3. * B(I,J,K) - 4. * B(I1,J,K) + B(I2,J,K) ) * DC2 ( 229) VCI(I) = ( 3. * C(I,J,K) - 4. * C(I1,J,K) + C(I2,J,K) ) * DC2 ( 230) ( 231) DO 47028 I = 1, IMAX ( 232) DINV = VAJ(I) * VBK(I) * VCI(I) + VBJ(I) * VCK(I) * VAI(I) ( 233) 1 + VCJ(I) * VAK(I) * VBI(I) - VAJ(I) * VCK(I) * VBI(I) ( 234) 2 - VBJ(I) * VAK(I) * VCI(I) - VCJ(I) * VBK(I) * VAI(I) ( 235) D(I,J,K) = 1. / (DINV + 1.E-20) ( 236) 47028 CONTINUE ( 237) 47029 CONTINUE ( 238) 5/10/2010 77
PGI 144, Invariant if transformation Loop not vectorized: loop count too small 150, Generated 3 alternate loops for the inner loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop 160, Generated 4 alternate loops for the inner loop Generated vector sse code for inner loop Generated 6 prefetch instructions for this loop Generated vector sse code for inner loop ooo 5/10/2010 78
Pathscale (lp47020.f:132) LOOP WAS VECTORIZED. (lp47020.f:150) LOOP WAS VECTORIZED. (lp47020.f:160) LOOP WAS VECTORIZED. (lp47020.f:170) LOOP WAS VECTORIZED. (lp47020.f:182) LOOP WAS VECTORIZED. (lp47020.f:192) LOOP WAS VECTORIZED. (lp47020.f:202) LOOP WAS VECTORIZED. (lp47020.f:216) LOOP WAS VECTORIZED. (lp47020.f:231) LOOP WAS VECTORIZED. (lp47020.f:248) LOOP WAS VECTORIZED. 5/10/2010 79
LP47020 2500 2000 Original PS-Quad 1500 Restructured PS-Quad Original PS-Dual MFLOPS Restructured PS-Dual Original PGI-Dual Restructured PGI-Dual 1000 Original PGI-Quad Restructured PGI-Quad 500 0 0 50 100 150 200 250 Vector Length 5/10/2010 80
Original ( 42) C THE ORIGINAL ( 43) ( 44) DO 48070 I = 1, N ( 45) A(I) = (B(I)**2 + C(I)**2) ( 46) CT = PI * A(I) + (A(I))**2 ( 47) CALL SSUB (A(I), CT, D(I), E(I)) ( 48) F(I) = (ABS (E(I))) ( 49) 48070 CONTINUE ( 50) PGI 44, Loop not vectorized: contains call Pathscale Nothing 5/10/2010 81
Restructured ( 69) C THE RESTRUCTURED ( 70) ( 71) DO 48071 I = 1, N ( 72) A(I) = (B(I)**2 + C(I)**2) ( 73) CT = PI * A(I) + (A(I))**2 ( 74) E(I) = A(I)**2 + (ABS (A(I) + CT)) * (CT * ABS (A(I) - CT)) ( 75) D(I) = A(I) + CT ( 76) F(I) = (ABS (E(I))) ( 77) 48071 CONTINUE ( 78) PGI 71, Generated an alternate loop for the inner loop Unrolled inner loop 4 times Used combined stores for 2 stores Generated 2 prefetch instructions for this loop Unrolled inner loop 4 times Used combined stores for 2 stores Generated 2 prefetch instructions for this loop Pathscale (lp48070.f:71) LOOP WAS VECTORIZED. 5/10/2010 82
LP48070 3500 3000 2500 Original PS-Quad Restructured PS-Quad 2000 Original PS-Dual MFLOPS Restructured PS-Dual Original PGI-Dual 1500 Restructured PGI-Dual Original PGI-Quad Restructured PGI-Quad 1000 500 0 0 50 100 150 200 250 300 350 400 450 500 Vector Length 5/10/2010 83
Cache-Blocking OPTIMIZING YOUR CODE July 2009 Slide 84
What is Cache Blocking? Cache blocking is a combination of strip mining and loop interchange, designed to increase data reuse. Takes advantage of temporal reuse: re-reference array elements already referenced Good blocking will take advantage of spatial reuse: work with the cache lines! Many ways to block any given loop nest Which loops get blocked? What block size(s) to use? Analysis can reveal which ways are beneficial But trial-and-error is probably faster May 10 Slide 85
Cache Use in Stencil Computations j=1 j=8 2D Laplacian i=1 do j = 1, 8 do i = 1, 16 a = u(i-1,j) + u(i+1,j) & - 4*u(i,j) & + u(i,j-1) + u(i,j+1) end do end do Cache structure for this example: Each line holds 4 array elements Cache can hold 12 lines of u data No cache reuse between outer loop i=16 iterations 120 30 18 15 13 12 10 9 7 6 4 3 May 10 Slide 86
Blocking to Increase Reuse j=1 j=8 Unblocked loop: 120 cache misses i=1 Block the inner loop do IBLOCK = 1, 16, 4 do j = 1, 8 i=5 do i = IBLOCK, IBLOCK + 3 a(i,j) = u(i-1,j) + u(i+1,j) & - 2*u(i,j) & + u(i,j-1) + u(i,j+1) i=9 end do end do end do i=13 Now we have reuse of the “j+1” data 80 20 12 10 11 9 8 7 6 4 3 May 10 Slide 87
Blocking to Increase Reuse j=1 j=5 One-dimensional blocking reduced i=1 misses from 120 to 80 Iterate over 4 4 blocks do JBLOCK = 1, 8, 4 i=5 do IBLOCK = 1, 16, 4 do j = JBLOCK, JBLOCK + 3 do i = IBLOCK, IBLOCK + 3 a(i,j) = u(i-1,j) + u(i+1,j) & i=9 - 2*u(i,j) & + u(i,j-1) + u(i,j+1) end do end do i=13 end do end do Better use of spatial locality (cache 15 13 12 10 60 30 18 17 16 11 9 8 7 6 4 3 lines) May 10 Slide 88
Obligatory GEMM discussion Matrix-matrix multiply (GEMM) is the canonical cache-blocking example Operations can be arranged to create multiple levels of blocking Block for register Block for cache (L1, L2, L3) Block for TLB No further discussion here. Interested readers can see Any book on code optimization Sun’s Techniques for Optimizing Applications: High Performance Computing contains a decent introductory discussion in Chapter 8 Insert your favorite book here Gunnels, Henry, and van de Geijn. June 2001. High-performance matrix multiplication algorithms for architectures with hierarchical memories. FLAME Working Note #4 TR- 2001-22, The University of Texas at Austin, Department of Computer Sciences Develops algorithms and cost models for GEMM in hierarchical memories Goto and van de Geijn. 2008. Anatomy of high-performance matrix multiplication. ACM Transactions on Mathematical Software 34, 3 (May), 1-25 Description of GotoBLAS DGEMM
What Could Go Wrong? “I tried cache-blocking my code, but it didn’t help” You’re doing it wrong. Your block size is too small (too much loop overhead). Your block size is too big (data is falling out of cache). You’re targeting the wrong cache level (?) You haven’t selected the correct subset of loops to block. The compiler is already blocking that loop. Prefetching is acting to minimize cache misses. Computational intensity within the loop nest is very large, making blocking less important.
A Real-Life Example: NPB MG Multigrid PDE solver Class D, 64 MPI ranks do i3 = 2, 257 Global grid is 1024 1024 do i2 = 2, 257 1024 do i1 = 2, 257 Local grid is 258 258 258 ! update u(i1,i2,i3) Two similar loop nests account for ! using 27-point stencil end do >50% of run time end do 27-point 3D stencil end do There is good data reuse along leading dimension, even without blocking i2+1 i2 cache lines i2-1 i3+1 i3 i3-1 i1-1 i1 i1+1
I’m Doing It Wrong Block the inner two loops Mop/s/proces Block size Creates blocks extending along i3 direction s unblocked 531.50 do I2BLOCK = 2, 257, BS2 do I1BLOCK = 2, 257, BS1 16 16 279.89 do i3 = 2, 257 do i2 = I2BLOCK, & 22 22 321.26 min(I2BLOCK+BS2-1, 257) do i1 = I1BLOCK, & 28 28 358.96 min(I1BLOCK+BS1-1, 257) ! update u(i1,i2,i3) 34 34 385.33 ! using 27-point stencil end do 40 40 408.53 end do end do 46 46 443.94 end do end do 52 52 468.58 58 58 470.32 64 64 512.03 70 70 506.92
That’s Better Block the outer two loops Mop/s/proces Block size Preserves spatial locality along i1 direction s unblocked 531.50 do I3BLOCK = 2, 257, BS3 do I2BLOCK = 2, 257, BS2 16 16 674.76 do i3 = I3BLOCK, & min(I3BLOCK+BS3-1, 257) 22 22 680.16 do i2 = I2BLOCK, & min(I2BLOCK+BS2-1, 257) 28 28 688.64 do i1 = 2, 257 ! update u(i1,i2,i3) 34 34 683.84 ! using 27-point stencil end do 40 40 698.47 end do end do 46 46 689.14 end do end do 52 52 706.62 58 58 692.57 64 64 703.40 70 70 693.87
Tuning Malloc OPTIMIZING YOUR CODE July 2009 Slide 94
GNU Malloc GNU malloc library malloc, calloc, realloc, free calls Fortran dynamic variables Malloc library system calls Mmap, munmap =>for larger allocations Brk, sbrk => increase/decrease heap Malloc library optimized for low system memory use Can result in system calls/minor page faults 95
Improving GNU Malloc Detecting “bad” malloc behavior Profile data => “excessive system time” Correcting “bad” malloc behavior Eliminate mmap use by malloc Increase threshold to release heap memory Use environment variables to alter malloc MALLOC_MMAP_MAX_ = 0 MALLOC_TRIM_THRESHOLD_ = 536870912 Possible downsides Heap fragmentation User process may call mmap directly User process may launch other processes PGI’s –Msmartalloc does something similar for you at compile time 96
Google TCMalloc Google created a replacement “malloc” library “Minimal” TCMalloc replaces GNU malloc Limited testing indicates TCMalloc as good or better than GNU malloc Environment variables not required TCMalloc almost certainly better for allocations in OpenMP parallel regions There’s currently no pre-built tcmalloc for Cray XT, but some users have successfully built it. 97
Memory Allocation: Make it local Linux has a “first touch policy” for memory allocation *alloc functions don’t actually allocate your memory Memory gets allocated when “touched” Problem: A code can allocate more memory than available Linux assumed “swap space,” we don’t have any Applications won’t fail from over-allocation until the memory is finally touched Problem: Memory will be put on the core of the “touching” thread Only a problem if thread 0 allocates all memory for a node Solution: Always initialize your memory immediately after allocating it If you over-allocate, it will fail immediately, rather than a strange place in your code If every thread touches its own memory, it will be allocated on the proper socket July 2009 Slide 98
Using Libraries OPTIMIZING YOUR CODE July 2009 Slide 99
Using Libraries Cray’s Scientific Libraries team has worked to optimize common library calls for each architecture There are more library routines than developers, so tell us what’s important Let the wrappers choose the right library for you As long as you have xtpe-<arch> loaded, the wrappers will pick the best library for you Linking against the wrong library can dramatically reduce performance Library calls are tuned for general cases, if you have a particular size, they may be able to do better GEMMs are tuned for square matrices, if yours aren’t square, they may be able to help you do better. July 2009 Slide 100
NCCS: help@nccs.gov NICS: help@teragrid.org Contact Your Liaison Jeff Larkin: larkin@cray.com The Best Optimization Technique: SEEK HELP May 10 Slide 101

More Related Content

What's hot (20)

PDF
Pragmatic Optimization in Modern Programming - Demystifying the Compiler
Marina Kolpakova
 
PDF
SFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMU
Linaro
 
PDF
eBPF/XDP
Netronome
 
PDF
Intel Nervana Graph とは？
Mr. Vengineer
 
PDF
한컴MDS_Virtual Target Debugging with TRACE32
HANCOM MDS
 
PPTX
Onnc intro
Luba Tang
 
PPTX
QEMU - Binary Translation
Jiann-Fuh Liaw
 
PDF
JIT compilation for CPython
delimitry
 
PDF
Translation Cache Policies for Dynamic Binary Translation
Saber Ferjani
 
PDF
Code GPU with CUDA - Optimizing memory and control flow
Marina Kolpakova
 
PDF
An evaluation of LLVM compiler for SVE with fairly complicated loops
Linaro
 
PDF
RISC-V : Berkeley Boot Loader & Proxy Kernelのソースコード解析
Mr. Vengineer
 
PDF
Performance evaluation with Arm HPC tools for SVE
Linaro
 
PDF
Programming at Compile Time
emBO_Conference
 
PDF
Machine Trace Metrics
Wang Hsiangkai
 
PDF
Arm tools and roadmap for SVE compiler support
Linaro
 
PDF
Code GPU with CUDA - Device code optimization principle
Marina Kolpakova
 
PDF
PyCon TW 2017 - PyPy's approach to construct domain-specific language runtime...
Tsundere Chen
 
PDF
1 Vampir Overview
PTIHPA
 
PDF
Profiling deep learning network using NVIDIA nsight systems
Jack (Jaegeun) Han
 
Pragmatic Optimization in Modern Programming - Demystifying the Compiler
Marina Kolpakova
 
SFO15-202: Towards Multi-Threaded Tiny Code Generator (TCG) in QEMU
Linaro
 
eBPF/XDP
Netronome
 
Intel Nervana Graph とは？
Mr. Vengineer
 
한컴MDS_Virtual Target Debugging with TRACE32
HANCOM MDS
 
Onnc intro
Luba Tang
 
QEMU - Binary Translation
Jiann-Fuh Liaw
 
JIT compilation for CPython
delimitry
 
Translation Cache Policies for Dynamic Binary Translation
Saber Ferjani
 
Code GPU with CUDA - Optimizing memory and control flow
Marina Kolpakova
 
An evaluation of LLVM compiler for SVE with fairly complicated loops
Linaro
 
RISC-V : Berkeley Boot Loader & Proxy Kernelのソースコード解析
Mr. Vengineer
 
Performance evaluation with Arm HPC tools for SVE
Linaro
 
Programming at Compile Time
emBO_Conference
 
Machine Trace Metrics
Wang Hsiangkai
 
Arm tools and roadmap for SVE compiler support
Linaro
 
Code GPU with CUDA - Device code optimization principle
Marina Kolpakova
 
PyCon TW 2017 - PyPy's approach to construct domain-specific language runtime...
Tsundere Chen
 
1 Vampir Overview
PTIHPA
 
Profiling deep learning network using NVIDIA nsight systems
Jack (Jaegeun) Han
 

Similar to May2010 hex-core-opt (20)

PDF
XT Best Practices
Jeff Larkin
 
PPT
f37-book-intarch-pres-pt2.ppt
ssuserf06014
 
PPT
f37-book-intarch-pres-pt2.ppt
Vhhvf
 
PDF
Parallelism in a NumPy-based program
Ralf Gommers
 
PDF
Library Operating System for Linux #netdev01
Hajime Tazaki
 
PDF
MLOps Case Studies: Building fast, scalable, and high-accuracy ML systems at ...
Masashi Shibata
 
PDF
Understand and Harness the Capabilities of Intel® Xeon Phi™ Processors
Intel® Software
 
PDF
Unmanaged Parallelization via P/Invoke
Dmitri Nesteruk
 
PPT
20081114 Friday Food iLabt Bart Joris
imec.archive
 
PDF
Adding a BOLT pass
Amir42407
 
PDF
SOFA Tutorial
NTU CSIE, Taiwan
 
PDF
Matopt
Afaf Soumia Medjden
 
PPTX
Dpdk applications
Vipin Varghese
 
PDF
不深不淺，帶你認識 LLVM (Found LLVM in your life)
Douglas Chen
 
PDF
Preparing to program Aurora at Exascale - Early experiences and future direct...
inside-BigData.com
 
PPT
Virtual platform
sean chen
 
PDF
LibOS as a regression test framework for Linux networking #netdev1.1
Hajime Tazaki
 
PDF
Trying and evaluating the new features of GlusterFS 3.5
Keisuke Takahashi
 
PDF
Computação Paralela: Benefícios e Desafios - Intel Software Conference 2013
Intel Software Brasil
 
PDF
Performance tweaks and tools for Linux (Joe Damato)
Ontico
 
XT Best Practices
Jeff Larkin
 
f37-book-intarch-pres-pt2.ppt
ssuserf06014
 
f37-book-intarch-pres-pt2.ppt
Vhhvf
 
Parallelism in a NumPy-based program
Ralf Gommers
 
Library Operating System for Linux #netdev01
Hajime Tazaki
 
MLOps Case Studies: Building fast, scalable, and high-accuracy ML systems at ...
Masashi Shibata
 
Understand and Harness the Capabilities of Intel® Xeon Phi™ Processors
Intel® Software
 
Unmanaged Parallelization via P/Invoke
Dmitri Nesteruk
 
20081114 Friday Food iLabt Bart Joris
imec.archive
 
Adding a BOLT pass
Amir42407
 
SOFA Tutorial
NTU CSIE, Taiwan
 
Matopt
Afaf Soumia Medjden
 
Dpdk applications
Vipin Varghese
 
不深不淺，帶你認識 LLVM (Found LLVM in your life)
Douglas Chen
 
Preparing to program Aurora at Exascale - Early experiences and future direct...
inside-BigData.com
 
Virtual platform
sean chen
 
LibOS as a regression test framework for Linux networking #netdev1.1
Hajime Tazaki
 
Trying and evaluating the new features of GlusterFS 3.5
Keisuke Takahashi
 
Computação Paralela: Benefícios e Desafios - Intel Software Conference 2013
Intel Software Brasil
 
Performance tweaks and tools for Linux (Joe Damato)
Ontico
 
Ad

More from Jeff Larkin (15)

PDF
Best Practices for OpenMP on GPUs - OpenMP UK Users Group
Jeff Larkin
 
PDF
FortranCon2020: Highly Parallel Fortran and OpenACC Directives
Jeff Larkin
 
PDF
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Jeff Larkin
 
PDF
Performance Portability Through Descriptive Parallelism
Jeff Larkin
 
PDF
GTC16 - S6510 - Targeting GPUs with OpenMP 4.5
Jeff Larkin
 
PDF
GTC16 - S6410 - Comparing OpenACC 2.5 and OpenMP 4.5
Jeff Larkin
 
PDF
SC13: OpenMP and NVIDIA
Jeff Larkin
 
PDF
Refactoring Applications for the XK7 and Future Hybrid Architectures
Jeff Larkin
 
PDF
Optimizing GPU to GPU Communication on Cray XK7
Jeff Larkin
 
PPTX
Progress Toward Accelerating CAM-SE
Jeff Larkin
 
PDF
HPCMPUG2011 cray tutorial
Jeff Larkin
 
PDF
CUG2011 Introduction to GPU Computing
Jeff Larkin
 
PDF
Maximizing Application Performance on Cray XT6 and XE6 Supercomputers DOD-MOD...
Jeff Larkin
 
PDF
A Comparison of Accelerator Programming Models
Jeff Larkin
 
PDF
Practical Examples for Efficient I/O on Cray XT Systems (CUG 2009)
Jeff Larkin
 
Best Practices for OpenMP on GPUs - OpenMP UK Users Group
Jeff Larkin
 
FortranCon2020: Highly Parallel Fortran and OpenACC Directives
Jeff Larkin
 
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Jeff Larkin
 
Performance Portability Through Descriptive Parallelism
Jeff Larkin
 
GTC16 - S6510 - Targeting GPUs with OpenMP 4.5
Jeff Larkin
 
GTC16 - S6410 - Comparing OpenACC 2.5 and OpenMP 4.5
Jeff Larkin
 
SC13: OpenMP and NVIDIA
Jeff Larkin
 
Refactoring Applications for the XK7 and Future Hybrid Architectures
Jeff Larkin
 
Optimizing GPU to GPU Communication on Cray XK7
Jeff Larkin
 
Progress Toward Accelerating CAM-SE
Jeff Larkin
 
HPCMPUG2011 cray tutorial
Jeff Larkin
 
CUG2011 Introduction to GPU Computing
Jeff Larkin
 
Maximizing Application Performance on Cray XT6 and XE6 Supercomputers DOD-MOD...
Jeff Larkin
 
A Comparison of Accelerator Programming Models
Jeff Larkin
 
Practical Examples for Efficient I/O on Cray XT Systems (CUG 2009)
Jeff Larkin
 
Ad

Recently uploaded (20)

PDF
Build with AI and GDG Cloud Bydgoszcz- ADK .pdf
jaroslawgajewski1
 
PDF
The Future of Artificial Intelligence (AI)
Mukul
 
PPTX
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
PDF
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PDF
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
PPTX
Farrell_Programming Logic and Design slides_10e_ch02_PowerPoint.pptx
bashnahara11
 
PDF
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PPTX
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
PPTX
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
PDF
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 
PDF
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
 
PDF
How Open Source Changed My Career by abdelrahman ismail
a0m0rajab1
 
PDF
introduction to computer hardware and sofeware
chauhanshraddha2007
 
PPTX
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
PDF
Per Axbom: The spectacular lies of maps
Nexer Digital
 
PDF
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
PDF
State-Dependent Conformal Perception Bounds for Neuro-Symbolic Verification
Ivan Ruchkin
 
PPTX
AVL ( audio, visuals or led ), technology.
Rajeshwri Panchal
 
PDF
Make GenAI investments go further with the Dell AI Factory
Principled Technologies
 
Build with AI and GDG Cloud Bydgoszcz- ADK .pdf
jaroslawgajewski1
 
The Future of Artificial Intelligence (AI)
Mukul
 
IT Runs Better with ThousandEyes AI-driven Assurance
ThousandEyes
 
A Strategic Analysis of the MVNO Wave in Emerging Markets.pdf
IPLOOK Networks
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
Farrell_Programming Logic and Design slides_10e_ch02_PowerPoint.pptx
bashnahara11
 
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
AI and Robotics for Human Well-being.pptx
JAYMIN SUTHAR
 
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
Research-Fundamentals-and-Topic-Development.pdf
ayesha butalia
 
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
 
How Open Source Changed My Career by abdelrahman ismail
a0m0rajab1
 
introduction to computer hardware and sofeware
chauhanshraddha2007
 
Applied-Statistics-Mastering-Data-Driven-Decisions.pptx
parmaryashparmaryash
 
Per Axbom: The spectacular lies of maps
Nexer Digital
 
OFFOFFBOX™ – A New Era for African Film | Startup Presentation
ambaicciwalkerbrian
 
State-Dependent Conformal Perception Bounds for Neuro-Symbolic Verification
Ivan Ruchkin
 
AVL ( audio, visuals or led ), technology.
Rajeshwri Panchal
 
Make GenAI investments go further with the Dell AI Factory
Principled Technologies
 

May2010 hex-core-opt

  • 1. Porting, Scaling, and Optimization on Cray XT Systems Jeff Larkin [email protected] These slides are available at http://www.slideshare.net/jefflarkin/may2010-hex-core-opt
  • 3. Choosing a Compiler Try every compiler available to you, there’s no “best” PGI, Cray, Pathscale, Intel, GCC are all available, but not necessarily on all machines Each compiler favors certain optimizations, which may benefit applications differently Test your answers carefully Order of operation may not be the same between compilers or even compiler versions You may have to decide whether speed or precision is more important to you May 10 Slide 3
  • 4. Choosing Compiler Flags PGI -fast –Mipa=fast man pgf90; man pgcc; man pgCC Cray <none, turned on by default> man crayftn; man craycc ; man crayCC Pathscale -Ofast man eko (“Every Known Optimization”) GNU -O2 / -O3 man gfortran; man gcc; man g++ Intel -fast man ifort; man icc; man iCC May 10 Slide 4
  • 6. Profile and Debug “at Scale” Codes don’t run the same at thousands of cores as hundreds, 10s of thousands as thousands, or 100s of thousands as 10s Determine how many nodes you wish to run on and test at that size, don’t test for throughput Scaling (Time) 2000 1500 Time (s) 1000 v1 (time) 500 v2 (time) 0 May 10 Slide 6
  • 7. Profile and Debug with Real Science Choose a real problem, not a toy What do you want to achieve with the machine? How will you really run your code? Will you really run without I/O? May 10 Slide 7
  • 8. What Data Should We Collect? Compiler Feedback Most compilers can tell you a lot about what they do to your code Did your important loop vectorize? Was a routine inlined? Was this loop unrolled? It’s just as important to know what the compiler didn’t do Some C/C++ loops won’t vectorize without some massaging Some loop counts aren’t known at runtime, so optimization is limited Flags to know: PGI: -Minfo=all –Mneginfo=all Cray: -rm (Fortran) (or also –O[neg]msgs), -hlist=m (C) (or also –h [neg]msgs) Pathscale: -LNO:simd_verbose=ON Intel: -vec-report1 May 10 Slide 8
  • 9. Compiler Feedback Examples: PGI ! Matrix Multiply 24, Loop interchange do k = 1, N produces reordered loop nest: 25,24,26 do j = 1, N 26, Generated an alternate do i = 1, N loop for the loop c(i,j) = c(i,j) + & Generated vector a(i,k) * b(k,j) sse code for the loop end do Generated 2 end do prefetch instructions end do for the loop May 10 Slide 9
  • 10. Compiler Feedback Examples: Cray 23. ! Matrix Multiply 24. ib------------< do k = 1, N 25. ib ibr4-------< do j = 1, N 26. ib ibr4 Vbr4--< do i = 1, N 27. ib ibr4 Vbr4 c(i,j) = c(i,j) + a(i,k) * b(k,j) 28. ib ibr4 Vbr4--> end do 29. ib ibr4-------> end do 30. ib------------> end do i – interchanged b – blocked r – unrolled V - Vectorized May 10 Slide 10
  • 11. Compiler Feedback Examples: Intel mm.F90(14): (col. 3) remark: PERMUTED LOOP WAS VECTORIZED. mm.F90(25): (col. 7) remark: LOOP WAS VECTORIZED. May 10 Slide 11
  • 12. What Data Should We Collect? Hardware Performance Counters Using tools like CrayPAT it’s possible to know what the processor is doing We can get counts for FLOPS Cache Hits/Misses TLB Hits/Misses Stalls … We can derive FLOP Rate Cache Hit/Miss Ratio Computational Intensity … May 10 Slide 12
  • 13. Gathering Performance Data Use Craypat’s APA First gather sampling for line number profile Light-weight Guides what should and should not be instrumented load module Second gather instrumentation (-g make mpi,io,blas,lapack,math …) pat_build -O apa a.out Hardware counters Execute MPI message passing pat_report *.xf information Examine *.apa I/O information pat_build –O *.apa Execute Math Libraries …
  • 14. Sample APA File (Significantly Trimmed) # You can edit this file, if desired, and use it # ---------------------------------------------------------------------- # HWPC group to collect by default. -Drtenv=PAT_RT_HWPC=1 # Summary with TLB metrics. # ---------------------------------------------------------------------- # Libraries to trace. -g mpi,math # ---------------------------------------------------------------------- -w # Enable tracing of user-defined functions. # Note: -u should NOT be specified as an additional option. # 9.08% 188347 bytes -T ratt_i_ # 6.71% 177904 bytes -T rhsf_ # 5.61% 205682 bytes -T ratx_i_ # 5.59% 22005 bytes -T transport_m_computespeciesdiffflux_
  • 15. CrayPAT Groups biolib Cray Bioinformatics library routines omp OpenMP API (not supported on blacs Basic Linear Algebra Catamount) communication subprograms omp-rtl OpenMP runtime library (not blas Basic Linear Algebra subprograms supported on Catamount) caf Co-Array Fortran (Cray X2 systems portals Lightweight message passing API only) pthreads POSIX threads (not supported on fftw Fast Fourier Transform library (64- Catamount) bit only) scalapack Scalable LAPACK hdf5 manages extremely large and shmem SHMEM complex data collections stdio all library functions that accept or heap dynamic heap return the FILE* construct io includes stdio and sysio groups sysio I/O system calls lapack Linear Algebra Package system system calls lustre Lustre File System upc Unified Parallel C (Cray X2 systems math ANSI math only) mpi MPI netcdf network common data form (manages array-oriented scientific data)
  • 16. Useful API Calls, for the control-freaks Regions, useful to break up long routines int PAT_region_begin (int id, const char *label) int PAT_region_end (int id) Disable/Enable Profiling, useful for excluding initialization int PAT_record (int state) Flush buffer, useful when program isn’t exiting cleanly int PAT_flush_buffer (void)
  • 17. Sample CrayPAT HWPC Data USER / rhsf_ ------------------------------------------------------------------------ Time% 74.6% Time 556.742885 secs Imb.Time 14.817686 secs Imb.Time% 2.6% Calls 2.3 /sec 1200.0 calls PAPI_L1_DCM 14.406M/sec 7569486532 misses PAPI_TLB_DM 0.225M/sec 117992047 misses PAPI_L1_DCA 921.729M/sec 484310815400 refs PAPI_FP_OPS 871.740M/sec 458044890200 ops User time (approx) 525.438 secs 1103418882813 cycles 94.4%Time Average Time per Call 0.463952 sec CrayPat Overhead : Time 0.0% HW FP Ops / User time 871.740M/sec 458044890200 ops 10.4%peak(DP) HW FP Ops / WCT 822.722M/sec Computational intensity 0.42 ops/cycle 0.95 ops/ref MFLOPS (aggregate) 1785323.32M/sec TLB utilization 4104.61 refs/miss 8.017 avg uses D1 cache hit,miss ratios 98.4% hits 1.6% misses D1 cache utilization (M) 63.98 refs/miss 7.998 avg uses May 10 Slide 17
  • 18. CrayPAT HWPC Groups 0 Summary with instruction 11 Floating point operations mix metrics (2) 1 Summary with TLB metrics 12 Floating point operations mix (vectorization) 2 L1 and L2 metrics 13 Floating point operations mix 3 Bandwidth information (SP) 4 Hypertransport information 14 Floating point operations mix 5 Floating point mix (DP) 6 Cycles stalled, resources idle 15 L3 (socket-level) 7 Cycles stalled, resources full 16 L3 (core-level reads) 8 Instructions and branches 17 L3 (core-level misses) 9 Instruction cache 18 L3 (core-level fills caused by 10 Cache hierarchy L2 evictions) 19 Prefetches May 10 Slide 18
  • 19. MPI Statistics How many times are MPI routines called and with how much data? Do I have a load imbalance? Are my processors waiting for data? Could I perform better by adjusting MPI environment variables? May 10 Slide 19
  • 20. Sample CrayPAT MPI Statistics MPI Msg Bytes | MPI Msg | MsgSz | 16B<= | 256B<= | 4KB<= |Experiment=1 | Count | <16B | MsgSz | MsgSz | MsgSz |Function | | Count | <256B | <4KB | <64KB | Caller | | | Count | Count | Count | PE[mmm] 3062457144.0 | 144952.0 | 15022.0 | 39.0 | 64522.0 | 65369.0 |Total |--------------------------------------------------------------------------- | 3059984152.0 | 129926.0 | -- | 36.0 | 64522.0 | 65368.0 |mpi_isend_ ||-------------------------------------------------------------------------- || 1727628971.0 | 63645.1 | -- | 4.0 | 31817.1 | 31824.0 |MPP_DO_UPDATE_R8_3DV.in.MPP_DOMAINS_MOD 3| | | | | | | MPP_UPDATE_DOMAIN2D_R8_3DV.in.MPP_DOMAINS_MOD ||||------------------------------------------------------------------------ 4||| 1680716892.0 | 61909.4 | -- | -- | 30949.4 | 30960.0 |DYN_CORE.in.DYN_CORE_MOD 5||| | | | | | | FV_DYNAMICS.in.FV_DYNAMICS_MOD 6||| | | | | | | ATMOSPHERE.in.ATMOSPHERE_MOD 7||| | | | | | | MAIN__ 8||| | | | | | | main |||||||||------------------------------------------------------------------- 9|||||||| 1680756480.0 | 61920.0 | -- | -- | 30960.0 | 30960.0 |pe.13666 9|||||||| 1680756480.0 | 61920.0 | -- | -- | 30960.0 | 30960.0 |pe.8949 9|||||||| 1651777920.0 | 54180.0 | -- | -- | 23220.0 | 30960.0 |pe.12549 |||||||||=================================================================== May 10 Slide 20
  • 21. I/O Statistics How much time is spent in I/O? How much data am I writing or reading? How many processes are performing I/O? May 10 Slide 21
  • 22. Understanding Cray Message Passing Toolkit (mpt) COMMUNICATION May 10 Slide 22
  • 23. Understanding CPU Affinity MPICH_CPUMASK_DISPLAY env variable Displays MPI process CPU affinity mask export MPICH_CPUMASK_DISPLAY=1 aprun –n 8 –N 8 –cc cpu ./mpi_exe [PE_0]: cpumask set to 1 cpu on nid00036, cpumask = 00000001 [PE_1]: cpumask set to 1 cpu on nid00036, cpumask = 00000010 [PE_2]: cpumask set to 1 cpu on nid00036, cpumask = 00000100 [PE_3]: cpumask set to 1 cpu on nid00036, cpumask = 00001000 [PE_4]: cpumask set to 1 cpu on nid00036, cpumask = 00010000 [PE_5]: cpumask set to 1 cpu on nid00036, cpumask = 00100000 [PE_6]: cpumask set to 1 cpu on nid00036, cpumask = 01000000 [PE_7]: cpumask set to 1 cpu on nid00036, cpumask = 10000000 aprun –n 8 –N 8 –cc numa_node ./mpi_exe [PE_0]: cpumask set to 4 cpus on nid00036, cpumask = 00001111 [PE_1]: cpumask set to 4 cpus on nid00036, cpumask = 00001111 [PE_2]: cpumask set to 4 cpus on nid00036, cpumask = 00001111 [PE_3]: cpumask set to 4 cpus on nid00036, cpumask = 00001111 [PE_4]: cpumask set to 4 cpus on nid00036, cpumask = 11110000 [PE_5]: cpumask set to 4 cpus on nid00036, cpumask = 11110000 [PE_6]: cpumask set to 4 cpus on nid00036, cpumask = 11110000 [PE_7]: cpumask set to 4 cpus on nid00036, cpumask = 11110000 July 2009 Slide 23
  • 24. MPT 4.0 (Released Q4 2009) Merge to ANL MPICH2 1.1 Support for the MPI 2.1 Standard (except dynamic processes) Binary-compatible with Intel MPI Additional MPI-IO Optimizations Improvements in Allgatherv at > 2K processors 10X – 2000x improvement Improvements in Scatterv at > 2KB message size 20-80% improvement Initial Gemini Support July 2009 Slide 24
  • 25. Auto-Scaling MPI Environment Variables Key MPI variables that change their default values dependent on job size MPICH_MAX_SHORT_MSG_SIZE MPICH_PTL_UNEX_EVENTS MPICH_UNEX_BUFFER_SIZE MPICH_PTL_OTHER_EVENTS Higher scaling of MPT jobs with fewer tweaks to env variables “Default” values are based on total number of ranks in job See MPI man page for specific formulas used We don’t always get it right Adjusted defaults aren't perfect for all applications Assumes a somewhat communication-balanced application Users can always override the new defaults Understanding and fine-tuning these variables may help performance July 2009 Slide 25
  • 26. Cray MPI XT Portals Communications Short Message Eager Protocol The sending rank “pushes” the message to the receiving rank Used for messages MPICH_MAX_SHORT_MSG_SIZE bytes or less Sender assumes that receiver can handle the message Matching receive is posted - or - Has available event queue entries (MPICH_PTL_UNEX_EVENTS) and buffer space (MPICH_UNEX_BUFFER_SIZE) to store the message Long Message Rendezvous Protocol Messages are “pulled” by the receiving rank Used for messages greater than MPICH_MAX_SHORT_MSG_SIZE bytes Sender sends small header packet with information for the receiver to pull over the data Data is sent only after matching receive is posted by receiving rank July 2009 Slide 26
  • 27. Match Entries Posted by MPI MPT Eager Protocol Incoming Msg S E to handle Unexpected Msgs Data “pushed” to the receiver A S App ME Eager Rendezvous (MPICH_MAX_SHORT_MSG_SIZE bytes or less) Short Msg ME Long Msg ME T A R STEP 1 STEP 2 MPI_RECV call Sender MPI_SEND call Receiver Post ME to Portals MPI RANK 0 RANK 1 Unexpected Buffers (MPICH_UNEX_BUFFER_SIZE) STEP 3 Portals DMA PUT Unexpected Msg Queue Other Event Queue (MPICH_PTL_OTHER_EVENTS) Unexpected Event Queue MPI_RECV is posted prior to MPI_SEND call (MPICH_PTL_UNEX_EVENTS) July 2009 Slide 27
  • 28. Match Entries Posted by MPI MPT Eager Protocol Incoming Msg S E to handle Unexpected Msgs Data “pushed” to the receiver A S Eager Short Msg ME Rendezvous Long Msg ME (MPICH_MAX_SHORT_MSG_SIZE bytes or less) T A R STEP 3 STEP 1 MPI_RECV call Sender MPI_SEND call Receiver No Portals ME MPI RANK 0 RANK 1 Unexpected Buffers STEP 2 STEP 4 (MPICH_UNEX_BUFFER_SIZE) Portals DMA PUT Memcpy of data Unexpected Msg Queue Unexpected Event Queue (MPICH_PTL_UNEX_EVENTS) MPI_RECV is not posted prior to MPI_SEND call July 2009 Slide 28
  • 29. MPT Rendezvous Protocol S Match Entries Posted by MPI to handle Unexpected Msgs Data “pulled” by the receiver Incoming Msg E ( > MPICH_MAX_SHORT_MSG_SIZE bytes ) A Eager Rendezvous S T Short Msg ME Long Msg ME A App ME R STEP 1 STEP 3 MPI_SEND call MPI_RECV call Portals ME created Triggers GET request Sender Receiver MPI RANK 0 RANK 1 Unexpected STEP 2 Buffers Portals DMA PUT of Header STEP 4 Receiver issues GET request to match Sender ME Unexpected Msg Queue STEP 5 Portals DMA of Data Unexpected Event Queue Data is not sent until MPI_RECV is issued July 2009 Slide 29
  • 30. Auto-Scaling MPI Environment Variables Default values for various MPI jobs sizes MPI Environment Variable Name 1,000 PEs 10,000 PEs 50,000 PEs 100,000 PEs MPICH_MAX_SHORT_MSG_SIZE (This size determines whether 128,000 20,480 4096 2048 the message uses the Eager or bytes Rendezvous protocol) MPICH_UNEX_BUFFER_SIZE 60 MB 60 MB 150 MB 260 MB (The buffer allocated to hold the unexpected Eager data) MPICH_PTL_UNEX_EVENTS (Portals generates two events for 20,480 22,000 110,000 220,000 each unexpected message events received) MPICH_PTL_OTHER_EVENTS 2048 (Portals send-side and expected 2500 12,500 25,000 events) events July 2009 Slide 30
  • 31. Cray MPI Collectives Our Collectives Strategy Improve performance over standard ANL MPICH2 algorithms Tune for our interconnect(s) Work for any intra-communicator (not just MPI_COMM_WORLD) Enabled by default Can be selectively disabled via MPICH_COLL_OPT_OFF export MPICH_COLL_OPT_OFF=mpi_bcast,mpi_allreduce Many have user-adjustable cross-over points (see man page) Cray Optimized Collectives MPI_Allgather ( for small messages ) MPI_Alltoall ( changes to the order of exchanges ) MPI_Alltoallv / MPI_Alltoallw ( windowing algorithm ) Cray Optimized SMP-aware Collectives MPI_Allreduce MPI_Barrier MPI_Bcast ( new in MPT 3.1.1 ) MPI_Reduce ( new in MPT 3.1.2 ) July 2009 Slide 31
  • 32. SMP-aware Collectives – Allreduce Example STEP 1 STEP 2 STEP 3 Identify Node-Captain rank. Perform an Allreduce with node- Perform a local on-node Perform a local on-node captains only. This reduces the bcast. NO network traffic. reduction to node-captain. process count by a factor of 8 on NO network traffic. XT5. July 2009 Slide 32
  • 33. Performance Comparison of MPI_Allreduce Default vs MPICH_COLL_OPT_OFF=MPI_Allreduce Percent Improvement of SMP-aware MPI_Allreduce (compared to MPICH2 algorithm) 1024 PEs on an Istanbul System 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% Allreduce 30.0% 20.0% 10.0% 0.0% Message Size July 2009 Slide 33
  • 34. Performance Comparison of MPI_Bcast Default vs MPICH_COLL_OPT_OFF=MPI_Bcast Percent Improvement of SMP-aware MPI_Bcast Percent Improvement of SMP-aware MPI_Bcast (compared to MPICH2 algorithm) 1024 PEs on an Istanbul System 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% Bcast 30.0% 20.0% 10.0% 0.0% Message Size July 2009 Slide 34
  • 35. Performance Comparison of MPI_Reduce Default vs MPICH_COLL_OPT_OFF=MPI_Reduce Percent Improvement of SMP-aware MPI_Reduce (compared to MPICH2 algorithm) 1024 PEs on an Istanbul System 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% Reduce 30.0% 20.0% 10.0% 0.0% Message Size July 2009 Slide 35
  • 36. Performance Comparison of MPI_Allgather Default vs MPICH_COLL_OPT_OFF=MPI_Allgather Percent Improvement of Optimized MPI_Allgather (compared to MPICH2 algorithm) 1024 PEs on an Istanbul System 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% Allgather 40.0% 30.0% 20.0% 10.0% 0.0% 1 2 4 8 16 32 64 128 256 512 1024 2048 Message Size July 2009 Slide 36
  • 37. MPI-IO Improvements MPI-IO collective buffering MPICH_MPIIO_CB_ALIGN=0 Divides the I/O workload equally among all aggregators Inefficient if multiple aggregators reference the same physical I/O block Default setting in MPT 3.2 and prior versions MPICH_MPIIO_CB_ALIGN=1 Divides the I/O workload up among the aggregators based on physical I/O boundaries and the size of the I/O request Allows only one aggregator access to any stripe on a single I/O call Available in MPT 3.1 MPICH_MPIIO_CB_ALIGN=2 Divides the I/O workload into Lustre stripe-sized groups and assigns them to aggregators Persistent across multiple I/O calls, so each aggregator always accesses the same set of stripes and no other aggregator accesses those stripes Minimizes Lustre file system lock contention Default setting in MPT 3.3 July 2009 Slide 37
  • 38. IOR benchmark 1,000,000 bytes MPI-IO API , non-power-of-2 blocks and transfers, in this case blocks and transfers both of 1M bytes and a strided access pattern. Tested on an XT5 with 32 PEs, 8 cores/node, 16 stripes, 16 aggregators, 3220 segments, 96 GB file 1800 1600 1400 MB/Sec 1200 1000 800 600 400 200 0 July 2009
  • 39. HYCOM MPI-2 I/O On 5107 PEs, and by application design, a subset of the PEs(88), do the writes. With collective buffering, this is further reduced to 22 aggregators (cb_nodes) writing to 22 stripes. Tested on an XT5 with 5107 PEs, 8 cores/node 4000 3500 3000 2500 MB/Sec 2000 1500 1000 500 0 July 2009
  • 41. Step by Step 1. Fix any load imbalance 2. Fix your hotspots 1. Communication • Pre-post receives • Overlap computation and communication • Reduce collectives • Adjust MPI environment variables • Use rank reordering 2. Computation • Examine the hardware counters and compiler feedback • Adjust the compiler flags, directives, or code structure to improve performance 3. I/O • Stripe files/directories appropriately • Use methods that scale • MPI-IO or Subsetting At each step, check your answers and performance. Between each step, gather your data again. May 10 Slide 41
  • 42. Detecting a load imbalance COMMUNICATION July 2009 Slide 42
  • 43. Craypat load-imbalance data Table 1: Profile by Function Group and Function Time % | Time | Imb. Time | Imb. | Calls |Experiment=1 | | | Time % | |Group | | | | | Function | | | | | PE='HIDE' 100.0% | 1061.141647 | -- | -- | 3454195.8 |Total |-------------------------------------------------------------------- | 70.7% | 750.564025 | -- | -- | 280169.0 |MPI_SYNC ||------------------------------------------------------------------- || 45.3% | 480.828018 | 163.575446 | 25.4% | 14653.0 |mpi_barrier_(sync) || 18.4% | 195.548030 | 33.071062 | 14.5% | 257546.0 |mpi_allreduce_(sync) || 7.0% | 74.187977 | 5.261545 | 6.6% | 7970.0 |mpi_bcast_(sync) ||=================================================================== | 15.2% | 161.166842 | -- | -- | 3174022.8 |MPI ||------------------------------------------------------------------- || 10.1% | 106.808182 | 8.237162 | 7.2% | 257546.0 |mpi_allreduce_ || 3.2% | 33.841961 | 342.085777 | 91.0% | 755495.8 |mpi_waitall_ ||=================================================================== | 14.1% | 149.410781 | -- | -- | 4.0 |USER ||------------------------------------------------------------------- || 14.0% | 148.048597 | 446.124165 | 75.1% | 1.0 |main |====================================================================
  • 44. Fixing a Load Imbalance What is causing the load imbalance Computation Is decomposition appropriate? Would RANK_REORDER help? Communication Is decomposition appropriate? Would RANK_REORDER help? Are receives pre-posted? OpenMP may help Able to spread workload with less overhead Large amount of work to go from all-MPI to Hybrid • Must accept challenge to OpenMP-ize large amount of code May 10 Slide 44
  • 45. Pre-Post your Recvs COMMUNICATION July 2009 Slide 45
  • 46. 5/10/2010 46
  • 47. 5/10/2010 47
  • 48. 5/10/2010 48
  • 49. 5/10/2010 49
  • 50. 5/10/2010 50
  • 51. 5/10/2010 51
  • 52. 5/10/2010 52
  • 53. 5/10/2010 53
  • 54. 5/10/2010 54
  • 55. 5/10/2010 55
  • 56. Tweak the Library COMMUNICATION July 2009 Slide 56
  • 57. MPICH Performance Variable MPICH_MAX_SHORT_MSG_SIZE Controls message sending protocol (Default:128000 byte) Message sizes <= MSG_SIZE: Use EAGER Message sizes > MSG_SIZE: Use RENDEZVOUS Increasing this variable may require that MPICH_UNEX_BUFFER_SIZE be increased Increase MPICH_MAX_SHORT_MSG_SIZE if App sends large msgs(>128K) and receives are pre-posted Can reduce messaging overhead via EAGER protocol Can reduce network contention Decrease MPICH_MAX_SHORT_MSG_SIZE if: App sends msgs in 32k-128k range and receives not pre-posted Cray Inc. Proprietary 57
  • 58. MPICH Performance Variable: MPICH_PTL_MATCH_OFF If set => Disables Portals matching Matching happens on the Opteron Requires extra copy for EAGER protocol Reduces MPI_Recv Overhead Helpful for latency-sensitive application Large # of small messages Small message collectives (<1024 bytes) When can this be slower? Pre-posted Receives can slow it down When extra copy time longer than post-to-Portals time For medium to larger messages (16k-128k range) 58
  • 59. Custom Rank Ordering CrayPAT can provide suggestions based on metrics from a profiled run. If you understand your decomposition well enough, you may be able to map it to the network We can help you find a better order based on your grid Is your “nearest neighbor” really your “nearest neighbor?” May 10 Slide 59
  • 60. COMPUTATION May 10 Slide 60
  • 61. Vectorization Stride one memory accesses Loop on the “fast” indices No IF tests No subroutine calls Inline Module Functions Statement Functions What is size of loop? Vectorizing might not help with very small loops Unroll small loops Loop nest Stride one on inside Longest on the inside Increase computational intensity CI = (vector flops/number of memory accesses)
  • 62. Bad Striding (64) ( 5) COMMON A(8,8,IIDIM,8),B(8,8,iidim,8) ( 59) DO 41090 K = KA, KE, -1 ( 60) DO 41090 J = JA, JE ( 61) DO 41090 I = IA, IE ( 62) A(K,L,I,J) = A(K,L,I,J) - B(J,1,i,k)*A(K+1,L,I,1) ( 63) * - B(J,2,i,k)*A(K+1,L,I,2) - B(J,3,i,k)*A(K+1,L,I,3) ( 64) * - B(J,4,i,k)*A(K+1,L,I,4) - B(J,5,i,k)*A(K+1,L,I,5) ( 65) 41090 CONTINUE ( 66) PGI 59, Loop not vectorized: loop count too small 60, Interchange produces reordered loop nest: 61, 60 Loop unrolled 5 times (completely unrolled) 61, Generated vector sse code for inner loop Pathscale (lp41090.f:62) Non-contiguous array "A(_BLNK__.0.0)" reference exists. Loop was not vectorized. (lp41090.f:62) Non-contiguous array "A(_BLNK__.0.0)" reference exists. Loop was not vectorized. (lp41090.f:62) Non-contiguous array "A(_BLNK__.0.0)" reference exists. Loop was not vectorized. (lp41090.f:62) Non-contiguous array "A(_BLNK__.0.0)" reference exists. Loop was not vectorized. April 2008 ORNL/NICS Users' Meeting 62
  • 63. Rewrite ( 6) COMMON AA(IIDIM,8,8,8),BB(IIDIM,8,8,8) ( 95) DO 41091 K = KA, KE, -1 ( 96) DO 41091 J = JA, JE ( 97) DO 41091 I = IA, IE ( 98) AA(I,K,L,J) = AA(I,K,L,J) - BB(I,J,1,K)*AA(I,K+1,L,1) ( 99) * - BB(I,J,2,K)*AA(I,K+1,L,2) - BB(I,J,3,K)*AA(I,K+1,L,3) ( 100) * - BB(I,J,4,K)*AA(I,K+1,L,4) - BB(I,J,5,K)*AA(I,K+1,L,5) ( 101) 41091 CONTINUE PGI 95, Loop not vectorized: loop count too small 96, Outer loop unrolled 5 times (completely unrolled) 97, Generated 3 alternate loops for the inner loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Pathscale (lp41090.f:99) LOOP WAS VECTORIZED. April 2008 ORNL/NICS Users' Meeting 63
  • 64. April 2008 ORNL/NICS Users' Meeting 64
  • 65. C pointers ( 53) void mat_mul_daxpy(double *a, double *b, double *c, int rowa, int cola, int colb) ( 54) { ( 55) int i, j, k; /* loop counters */ ( 56) int rowc, colc, rowb; /* sizes not passed as arguments */ ( 57) double con; /* constant value */ ( 58) ( 59) rowb = cola; ( 60) rowc = rowa; ( 61) colc = colb; ( 62) ( 63) for(i=0;i<rowc;i++) { ( 64) for(k=0;k<cola;k++) { ( 65) con = *(a + i*cola +k); ( 66) for(j=0;j<colc;j++) { ( 67) *(c + i*colc + j) += con * *(b + k*colb + j); ( 68) } ( 69) } ( 70) } ( 71) } mat_mul_daxpy: 66, Loop not vectorized: data dependency Loop not vectorized: data dependency Loop unrolled 4 times Slide 65
  • 66. C pointers, rewrite ( 53) void mat_mul_daxpy(double* restrict a, double* restrict b, double* restrict c, int rowa, int cola, int colb) ( 54) { ( 55) int i, j, k; /* loop counters */ ( 56) int rowc, colc, rowb; /* sizes not passed as arguments */ ( 57) double con; /* constant value */ ( 58) ( 59) rowb = cola; ( 60) rowc = rowa; ( 61) colc = colb; ( 62) ( 63) for(i=0;i<rowc;i++) { ( 64) for(k=0;k<cola;k++) { ( 65) con = *(a + i*cola +k); ( 66) for(j=0;j<colc;j++) { ( 67) *(c + i*colc + j) += con * *(b + k*colb + j); ( 68) } ( 69) } ( 70) } ( 71) } Slide 66
  • 67. C pointers, rewrite 66, Generated alternate loop with no peeling - executed if loop count <= 24 Generated vector sse code for inner loop Generated 2 prefetch instructions for this loop Generated vector sse code for inner loop Generated 2 prefetch instructions for this loop Generated alternate loop with no peeling and more aligned moves - executed if loop count <= 24 and alignment test is passed Generated vector sse code for inner loop Generated 2 prefetch instructions for this loop Generated alternate loop with more aligned moves - executed if loop count >= 25 and alignment test is passed Generated vector sse code for inner loop Generated 2 prefetch instructions for this loop • This can also be achieved with the PGI safe pragma and –Msafeptr compiler option or Pathscale –OPT:alias option Slide 67
  • 68. Nested Loops ( 47) DO 45020 I = 1, N ( 48) F(I) = A(I) + .5 ( 49) DO 45020 J = 1, 10 ( 50) D(I,J) = B(J) * F(I) ( 51) DO 45020 K = 1, 5 ( 52) C(K,I,J) = D(I,J) * E(K) ( 53) 45020 CONTINUE PGI 49, Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Loop unrolled 2 times (completely unrolled) Pathscale (lp45020.f:48) LOOP WAS VECTORIZED. (lp45020.f:48) Non-contiguous array "C(_BLNK__.0.0)" reference exists. Loop was not vectorized. 68
  • 69. Rewrite ( 71) DO 45021 I = 1,N ( 72) F(I) = A(I) + .5 ( 73) 45021 CONTINUE ( 74) ( 75) DO 45022 J = 1, 10 ( 76) DO 45022 I = 1, N ( 77) D(I,J) = B(J) * F(I) ( 78) 45022 CONTINUE ( 79) ( 80) DO 45023 K = 1, 5 ( 81) DO 45023 J = 1, 10 ( 82) DO 45023 I = 1, N ( 83) C(K,I,J) = D(I,J) * E(K) ( 84) 45023 CONTINUE 69
  • 70. PGI 73, Generated an alternate loop for the inner loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop 78, Generated 2 alternate loops for the inner loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop 82, Interchange produces reordered loop nest: 83, 84, 82 Loop unrolled 5 times (completely unrolled) 84, Generated vector sse code for inner loop Generated 1 prefetch instructions for this loop Pathscale (lp45020.f:73) LOOP WAS VECTORIZED. (lp45020.f:78) LOOP WAS VECTORIZED. (lp45020.f:78) LOOP WAS VECTORIZED. (lp45020.f:84) Non-contiguous array "C(_BLNK__.0.0)" reference exists. Loop was not vectorized. (lp45020.f:84) Non-contiguous array "C(_BLNK__.0.0)" reference exists. Loop was not vectorized. 70
  • 71. 71
  • 72. Big Loop ( 52) C THE ORIGINAL ( 53) ( 54) DO 47020 J = 1, JMAX ( 55) DO 47020 K = 1, KMAX ( 56) DO 47020 I = 1, IMAX ( 57) JP = J + 1 ( 58) JR = J - 1 ( 59) KP = K + 1 ( 60) KR = K - 1 ( 61) IP = I + 1 ( 62) IR = I - 1 ( 63) IF (J .EQ. 1) GO TO 50 ( 64) IF( J .EQ. JMAX) GO TO 51 ( 65) XJ = ( A(I,JP,K) - A(I,JR,K) ) * DA2 ( 66) YJ = ( B(I,JP,K) - B(I,JR,K) ) * DA2 ( 67) ZJ = ( C(I,JP,K) - C(I,JR,K) ) * DA2 ( 68) GO TO 70 ( 69) 50 J1 = J + 1 ( 70) J2 = J + 2 ( 71) XJ = (-3. * A(I,J,K) + 4. * A(I,J1,K) - A(I,J2,K) ) * DA2 ( 72) YJ = (-3. * B(I,J,K) + 4. * B(I,J1,K) - B(I,J2,K) ) * DA2 ( 73) ZJ = (-3. * C(I,J,K) + 4. * C(I,J1,K) - C(I,J2,K) ) * DA2 ( 74) GO TO 70 ( 75) 51 J1 = J - 1 ( 76) J2 = J - 2 ( 77) XJ = ( 3. * A(I,J,K) - 4. * A(I,J1,K) + A(I,J2,K) ) * DA2 ( 78) YJ = ( 3. * B(I,J,K) - 4. * B(I,J1,K) + B(I,J2,K) ) * DA2 ( 79) ZJ = ( 3. * C(I,J,K) - 4. * C(I,J1,K) + C(I,J2,K) ) * DA2 ( 80) 70 CONTINUE ( 81) IF (K .EQ. 1) GO TO 52 ( 82) IF (K .EQ. KMAX) GO TO 53 ( 83) XK = ( A(I,J,KP) - A(I,J,KR) ) * DB2 ( 84) YK = ( B(I,J,KP) - B(I,J,KR) ) * DB2 ( 85) ZK = ( C(I,J,KP) - C(I,J,KR) ) * DB2 ( 86) GO TO 71 5/10/2010 72
  • 73. Big Loop ( 87) 52 K1 = K + 1 ( 88) K2 = K + 2 ( 89) XK = (-3. * A(I,J,K) + 4. * A(I,J,K1) - A(I,J,K2) ) * DB2 ( 90) YK = (-3. * B(I,J,K) + 4. * B(I,J,K1) - B(I,J,K2) ) * DB2 ( 91) ZK = (-3. * C(I,J,K) + 4. * C(I,J,K1) - C(I,J,K2) ) * DB2 ( 92) GO TO 71 ( 93) 53 K1 = K - 1 ( 94) K2 = K - 2 ( 95) XK = ( 3. * A(I,J,K) - 4. * A(I,J,K1) + A(I,J,K2) ) * DB2 ( 96) YK = ( 3. * B(I,J,K) - 4. * B(I,J,K1) + B(I,J,K2) ) * DB2 ( 97) ZK = ( 3. * C(I,J,K) - 4. * C(I,J,K1) + C(I,J,K2) ) * DB2 ( 98) 71 CONTINUE ( 99) IF (I .EQ. 1) GO TO 54 ( 100) IF (I .EQ. IMAX) GO TO 55 ( 101) XI = ( A(IP,J,K) - A(IR,J,K) ) * DC2 ( 102) YI = ( B(IP,J,K) - B(IR,J,K) ) * DC2 ( 103) ZI = ( C(IP,J,K) - C(IR,J,K) ) * DC2 ( 104) GO TO 60 ( 105) 54 I1 = I + 1 ( 106) I2 = I + 2 ( 107) XI = (-3. * A(I,J,K) + 4. * A(I1,J,K) - A(I2,J,K) ) * DC2 ( 108) YI = (-3. * B(I,J,K) + 4. * B(I1,J,K) - B(I2,J,K) ) * DC2 ( 109) ZI = (-3. * C(I,J,K) + 4. * C(I1,J,K) - C(I2,J,K) ) * DC2 ( 110) GO TO 60 ( 111) 55 I1 = I - 1 ( 112) I2 = I - 2 ( 113) XI = ( 3. * A(I,J,K) - 4. * A(I1,J,K) + A(I2,J,K) ) * DC2 ( 114) YI = ( 3. * B(I,J,K) - 4. * B(I1,J,K) + B(I2,J,K) ) * DC2 ( 115) ZI = ( 3. * C(I,J,K) - 4. * C(I1,J,K) + C(I2,J,K) ) * DC2 ( 116) 60 CONTINUE ( 117) DINV = XJ * YK * ZI + YJ * ZK * XI + ZJ * XK * YI ( 118) * - XJ * ZK * YI - YJ * XK * ZI - ZJ * YK * XI ( 119) D(I,J,K) = 1. / (DINV + 1.E-20) ( 120) 47020 CONTINUE ( 121) 5/10/2010 73
  • 74. PGI 55, Invariant if transformation Loop not vectorized: loop count too small 56, Invariant if transformation Pathscale Nothing 5/10/2010 74
  • 75. Re-Write ( 141) C THE RESTRUCTURED ( 142) ( 143) DO 47029 J = 1, JMAX ( 144) DO 47029 K = 1, KMAX ( 145) ( 146) IF(J.EQ.1)THEN ( 147) ( 148) J1 = 2 ( 149) J2 = 3 ( 150) DO 47021 I = 1, IMAX ( 151) VAJ(I) = (-3. * A(I,J,K) + 4. * A(I,J1,K) - A(I,J2,K) ) * DA2 ( 152) VBJ(I) = (-3. * B(I,J,K) + 4. * B(I,J1,K) - B(I,J2,K) ) * DA2 ( 153) VCJ(I) = (-3. * C(I,J,K) + 4. * C(I,J1,K) - C(I,J2,K) ) * DA2 ( 154) 47021 CONTINUE ( 155) ( 156) ELSE IF(J.NE.JMAX) THEN ( 157) ( 158) JP = J+1 ( 159) JR = J-1 ( 160) DO 47022 I = 1, IMAX ( 161) VAJ(I) = ( A(I,JP,K) - A(I,JR,K) ) * DA2 ( 162) VBJ(I) = ( B(I,JP,K) - B(I,JR,K) ) * DA2 ( 163) VCJ(I) = ( C(I,JP,K) - C(I,JR,K) ) * DA2 ( 164) 47022 CONTINUE ( 165) ( 166) ELSE ( 167) ( 168) J1 = JMAX-1 ( 169) J2 = JMAX-2 ( 170) DO 47023 I = 1, IMAX ( 171) VAJ(I) = ( 3. * A(I,J,K) - 4. * A(I,J1,K) + A(I,J2,K) ) * DA2 ( 172) VBJ(I) = ( 3. * B(I,J,K) - 4. * B(I,J1,K) + B(I,J2,K) ) * DA2 ( 173) VCJ(I) = ( 3. * C(I,J,K) - 4. * C(I,J1,K) + C(I,J2,K) ) * DA2 ( 174) 47023 CONTINUE ( 175) ( 176) ENDIF 5/10/2010 75
  • 76. Re-Write ( 178) IF(K.EQ.1) THEN ( 179) ( 180) K1 = 2 ( 181) K2 = 3 ( 182) DO 47024 I = 1, IMAX ( 183) VAK(I) = (-3. * A(I,J,K) + 4. * A(I,J,K1) - A(I,J,K2) ) * DB2 ( 184) VBK(I) = (-3. * B(I,J,K) + 4. * B(I,J,K1) - B(I,J,K2) ) * DB2 ( 185) VCK(I) = (-3. * C(I,J,K) + 4. * C(I,J,K1) - C(I,J,K2) ) * DB2 ( 186) 47024 CONTINUE ( 187) ( 188) ELSE IF(K.NE.KMAX)THEN ( 189) ( 190) KP = K + 1 ( 191) KR = K - 1 ( 192) DO 47025 I = 1, IMAX ( 193) VAK(I) = ( A(I,J,KP) - A(I,J,KR) ) * DB2 ( 194) VBK(I) = ( B(I,J,KP) - B(I,J,KR) ) * DB2 ( 195) VCK(I) = ( C(I,J,KP) - C(I,J,KR) ) * DB2 ( 196) 47025 CONTINUE ( 197) ( 198) ELSE ( 199) ( 200) K1 = KMAX - 1 ( 201) K2 = KMAX - 2 ( 202) DO 47026 I = 1, IMAX ( 203) VAK(I) = ( 3. * A(I,J,K) - 4. * A(I,J,K1) + A(I,J,K2) ) * DB2 ( 204) VBK(I) = ( 3. * B(I,J,K) - 4. * B(I,J,K1) + B(I,J,K2) ) * DB2 ( 205) VCK(I) = ( 3. * C(I,J,K) - 4. * C(I,J,K1) + C(I,J,K2) ) * DB2 ( 206) 47026 CONTINUE ( 207) ENDIF ( 208) 5/10/2010 76
  • 77. Re-Write ( 209) I = 1 ( 210) I1 = 2 ( 211) I2 = 3 ( 212) VAI(I) = (-3. * A(I,J,K) + 4. * A(I1,J,K) - A(I2,J,K) ) * DC2 ( 213) VBI(I) = (-3. * B(I,J,K) + 4. * B(I1,J,K) - B(I2,J,K) ) * DC2 ( 214) VCI(I) = (-3. * C(I,J,K) + 4. * C(I1,J,K) - C(I2,J,K) ) * DC2 ( 215) ( 216) DO 47027 I = 2, IMAX-1 ( 217) IP = I + 1 ( 218) IR = I – 1 ( 219) VAI(I) = ( A(IP,J,K) - A(IR,J,K) ) * DC2 ( 220) VBI(I) = ( B(IP,J,K) - B(IR,J,K) ) * DC2 ( 221) VCI(I) = ( C(IP,J,K) - C(IR,J,K) ) * DC2 ( 222) 47027 CONTINUE ( 223) ( 224) I = IMAX ( 225) I1 = IMAX - 1 ( 226) I2 = IMAX - 2 ( 227) VAI(I) = ( 3. * A(I,J,K) - 4. * A(I1,J,K) + A(I2,J,K) ) * DC2 ( 228) VBI(I) = ( 3. * B(I,J,K) - 4. * B(I1,J,K) + B(I2,J,K) ) * DC2 ( 229) VCI(I) = ( 3. * C(I,J,K) - 4. * C(I1,J,K) + C(I2,J,K) ) * DC2 ( 230) ( 231) DO 47028 I = 1, IMAX ( 232) DINV = VAJ(I) * VBK(I) * VCI(I) + VBJ(I) * VCK(I) * VAI(I) ( 233) 1 + VCJ(I) * VAK(I) * VBI(I) - VAJ(I) * VCK(I) * VBI(I) ( 234) 2 - VBJ(I) * VAK(I) * VCI(I) - VCJ(I) * VBK(I) * VAI(I) ( 235) D(I,J,K) = 1. / (DINV + 1.E-20) ( 236) 47028 CONTINUE ( 237) 47029 CONTINUE ( 238) 5/10/2010 77
  • 78. PGI 144, Invariant if transformation Loop not vectorized: loop count too small 150, Generated 3 alternate loops for the inner loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop Generated vector sse code for inner loop Generated 8 prefetch instructions for this loop 160, Generated 4 alternate loops for the inner loop Generated vector sse code for inner loop Generated 6 prefetch instructions for this loop Generated vector sse code for inner loop ooo 5/10/2010 78
  • 79. Pathscale (lp47020.f:132) LOOP WAS VECTORIZED. (lp47020.f:150) LOOP WAS VECTORIZED. (lp47020.f:160) LOOP WAS VECTORIZED. (lp47020.f:170) LOOP WAS VECTORIZED. (lp47020.f:182) LOOP WAS VECTORIZED. (lp47020.f:192) LOOP WAS VECTORIZED. (lp47020.f:202) LOOP WAS VECTORIZED. (lp47020.f:216) LOOP WAS VECTORIZED. (lp47020.f:231) LOOP WAS VECTORIZED. (lp47020.f:248) LOOP WAS VECTORIZED. 5/10/2010 79
  • 80. LP47020 2500 2000 Original PS-Quad 1500 Restructured PS-Quad Original PS-Dual MFLOPS Restructured PS-Dual Original PGI-Dual Restructured PGI-Dual 1000 Original PGI-Quad Restructured PGI-Quad 500 0 0 50 100 150 200 250 Vector Length 5/10/2010 80
  • 81. Original ( 42) C THE ORIGINAL ( 43) ( 44) DO 48070 I = 1, N ( 45) A(I) = (B(I)**2 + C(I)**2) ( 46) CT = PI * A(I) + (A(I))**2 ( 47) CALL SSUB (A(I), CT, D(I), E(I)) ( 48) F(I) = (ABS (E(I))) ( 49) 48070 CONTINUE ( 50) PGI 44, Loop not vectorized: contains call Pathscale Nothing 5/10/2010 81
  • 82. Restructured ( 69) C THE RESTRUCTURED ( 70) ( 71) DO 48071 I = 1, N ( 72) A(I) = (B(I)**2 + C(I)**2) ( 73) CT = PI * A(I) + (A(I))**2 ( 74) E(I) = A(I)**2 + (ABS (A(I) + CT)) * (CT * ABS (A(I) - CT)) ( 75) D(I) = A(I) + CT ( 76) F(I) = (ABS (E(I))) ( 77) 48071 CONTINUE ( 78) PGI 71, Generated an alternate loop for the inner loop Unrolled inner loop 4 times Used combined stores for 2 stores Generated 2 prefetch instructions for this loop Unrolled inner loop 4 times Used combined stores for 2 stores Generated 2 prefetch instructions for this loop Pathscale (lp48070.f:71) LOOP WAS VECTORIZED. 5/10/2010 82
  • 83. LP48070 3500 3000 2500 Original PS-Quad Restructured PS-Quad 2000 Original PS-Dual MFLOPS Restructured PS-Dual Original PGI-Dual 1500 Restructured PGI-Dual Original PGI-Quad Restructured PGI-Quad 1000 500 0 0 50 100 150 200 250 300 350 400 450 500 Vector Length 5/10/2010 83
  • 84. Cache-Blocking OPTIMIZING YOUR CODE July 2009 Slide 84
  • 85. What is Cache Blocking? Cache blocking is a combination of strip mining and loop interchange, designed to increase data reuse. Takes advantage of temporal reuse: re-reference array elements already referenced Good blocking will take advantage of spatial reuse: work with the cache lines! Many ways to block any given loop nest Which loops get blocked? What block size(s) to use? Analysis can reveal which ways are beneficial But trial-and-error is probably faster May 10 Slide 85
  • 86. Cache Use in Stencil Computations j=1 j=8 2D Laplacian i=1 do j = 1, 8 do i = 1, 16 a = u(i-1,j) + u(i+1,j) & - 4*u(i,j) & + u(i,j-1) + u(i,j+1) end do end do Cache structure for this example: Each line holds 4 array elements Cache can hold 12 lines of u data No cache reuse between outer loop i=16 iterations 120 30 18 15 13 12 10 9 7 6 4 3 May 10 Slide 86
  • 87. Blocking to Increase Reuse j=1 j=8 Unblocked loop: 120 cache misses i=1 Block the inner loop do IBLOCK = 1, 16, 4 do j = 1, 8 i=5 do i = IBLOCK, IBLOCK + 3 a(i,j) = u(i-1,j) + u(i+1,j) & - 2*u(i,j) & + u(i,j-1) + u(i,j+1) i=9 end do end do end do i=13 Now we have reuse of the “j+1” data 80 20 12 10 11 9 8 7 6 4 3 May 10 Slide 87
  • 88. Blocking to Increase Reuse j=1 j=5 One-dimensional blocking reduced i=1 misses from 120 to 80 Iterate over 4 4 blocks do JBLOCK = 1, 8, 4 i=5 do IBLOCK = 1, 16, 4 do j = JBLOCK, JBLOCK + 3 do i = IBLOCK, IBLOCK + 3 a(i,j) = u(i-1,j) + u(i+1,j) & i=9 - 2*u(i,j) & + u(i,j-1) + u(i,j+1) end do end do i=13 end do end do Better use of spatial locality (cache 15 13 12 10 60 30 18 17 16 11 9 8 7 6 4 3 lines) May 10 Slide 88
  • 89. Obligatory GEMM discussion Matrix-matrix multiply (GEMM) is the canonical cache-blocking example Operations can be arranged to create multiple levels of blocking Block for register Block for cache (L1, L2, L3) Block for TLB No further discussion here. Interested readers can see Any book on code optimization Sun’s Techniques for Optimizing Applications: High Performance Computing contains a decent introductory discussion in Chapter 8 Insert your favorite book here Gunnels, Henry, and van de Geijn. June 2001. High-performance matrix multiplication algorithms for architectures with hierarchical memories. FLAME Working Note #4 TR- 2001-22, The University of Texas at Austin, Department of Computer Sciences Develops algorithms and cost models for GEMM in hierarchical memories Goto and van de Geijn. 2008. Anatomy of high-performance matrix multiplication. ACM Transactions on Mathematical Software 34, 3 (May), 1-25 Description of GotoBLAS DGEMM
  • 90. What Could Go Wrong? “I tried cache-blocking my code, but it didn’t help” You’re doing it wrong. Your block size is too small (too much loop overhead). Your block size is too big (data is falling out of cache). You’re targeting the wrong cache level (?) You haven’t selected the correct subset of loops to block. The compiler is already blocking that loop. Prefetching is acting to minimize cache misses. Computational intensity within the loop nest is very large, making blocking less important.
  • 91. A Real-Life Example: NPB MG Multigrid PDE solver Class D, 64 MPI ranks do i3 = 2, 257 Global grid is 1024 1024 do i2 = 2, 257 1024 do i1 = 2, 257 Local grid is 258 258 258 ! update u(i1,i2,i3) Two similar loop nests account for ! using 27-point stencil end do >50% of run time end do 27-point 3D stencil end do There is good data reuse along leading dimension, even without blocking i2+1 i2 cache lines i2-1 i3+1 i3 i3-1 i1-1 i1 i1+1
  • 92. I’m Doing It Wrong Block the inner two loops Mop/s/proces Block size Creates blocks extending along i3 direction s unblocked 531.50 do I2BLOCK = 2, 257, BS2 do I1BLOCK = 2, 257, BS1 16 16 279.89 do i3 = 2, 257 do i2 = I2BLOCK, & 22 22 321.26 min(I2BLOCK+BS2-1, 257) do i1 = I1BLOCK, & 28 28 358.96 min(I1BLOCK+BS1-1, 257) ! update u(i1,i2,i3) 34 34 385.33 ! using 27-point stencil end do 40 40 408.53 end do end do 46 46 443.94 end do end do 52 52 468.58 58 58 470.32 64 64 512.03 70 70 506.92
  • 93. That’s Better Block the outer two loops Mop/s/proces Block size Preserves spatial locality along i1 direction s unblocked 531.50 do I3BLOCK = 2, 257, BS3 do I2BLOCK = 2, 257, BS2 16 16 674.76 do i3 = I3BLOCK, & min(I3BLOCK+BS3-1, 257) 22 22 680.16 do i2 = I2BLOCK, & min(I2BLOCK+BS2-1, 257) 28 28 688.64 do i1 = 2, 257 ! update u(i1,i2,i3) 34 34 683.84 ! using 27-point stencil end do 40 40 698.47 end do end do 46 46 689.14 end do end do 52 52 706.62 58 58 692.57 64 64 703.40 70 70 693.87
  • 94. Tuning Malloc OPTIMIZING YOUR CODE July 2009 Slide 94
  • 95. GNU Malloc GNU malloc library malloc, calloc, realloc, free calls Fortran dynamic variables Malloc library system calls Mmap, munmap =>for larger allocations Brk, sbrk => increase/decrease heap Malloc library optimized for low system memory use Can result in system calls/minor page faults 95
  • 96. Improving GNU Malloc Detecting “bad” malloc behavior Profile data => “excessive system time” Correcting “bad” malloc behavior Eliminate mmap use by malloc Increase threshold to release heap memory Use environment variables to alter malloc MALLOC_MMAP_MAX_ = 0 MALLOC_TRIM_THRESHOLD_ = 536870912 Possible downsides Heap fragmentation User process may call mmap directly User process may launch other processes PGI’s –Msmartalloc does something similar for you at compile time 96
  • 97. Google TCMalloc Google created a replacement “malloc” library “Minimal” TCMalloc replaces GNU malloc Limited testing indicates TCMalloc as good or better than GNU malloc Environment variables not required TCMalloc almost certainly better for allocations in OpenMP parallel regions There’s currently no pre-built tcmalloc for Cray XT, but some users have successfully built it. 97
  • 98. Memory Allocation: Make it local Linux has a “first touch policy” for memory allocation *alloc functions don’t actually allocate your memory Memory gets allocated when “touched” Problem: A code can allocate more memory than available Linux assumed “swap space,” we don’t have any Applications won’t fail from over-allocation until the memory is finally touched Problem: Memory will be put on the core of the “touching” thread Only a problem if thread 0 allocates all memory for a node Solution: Always initialize your memory immediately after allocating it If you over-allocate, it will fail immediately, rather than a strange place in your code If every thread touches its own memory, it will be allocated on the proper socket July 2009 Slide 98
  • 99. Using Libraries OPTIMIZING YOUR CODE July 2009 Slide 99
  • 100. Using Libraries Cray’s Scientific Libraries team has worked to optimize common library calls for each architecture There are more library routines than developers, so tell us what’s important Let the wrappers choose the right library for you As long as you have xtpe-<arch> loaded, the wrappers will pick the best library for you Linking against the wrong library can dramatically reduce performance Library calls are tuned for general cases, if you have a particular size, they may be able to do better GEMMs are tuned for square matrices, if yours aren’t square, they may be able to help you do better. July 2009 Slide 100
  • 101. NCCS: [email protected] NICS: [email protected] Contact Your Liaison Jeff Larkin: [email protected] The Best Optimization Technique: SEEK HELP May 10 Slide 101