Direct QR factorizations for tall-and-skinny matrices in MapReduce architectures (IEEE BigData)
1 like•298 views
This document describes algorithms for computing the QR factorization of tall-and-skinny matrices in MapReduce architectures. It introduces direct and indirect Tall-Skinny QR (TSQR) algorithms, as well as techniques like iterative refinement and pseudo-iterative refinement to improve stability. A performance model is developed showing the algorithms are I/O bound and take a number of passes over the data. Experimental results demonstrate the direct TSQR algorithm has better stability and performance than other approaches.
![Matrix representation 8
We have matrices, so what are the key-value pairs?
A =
2
1:0 0:0
2:4 3:7
0:8 4:2
9:0 9:0
664
3
775
!
2
(1; [1:0; 0:0])
(2; [2:4; 3:7])
(3; [0:8; 4:2])
(4; [9:0; 9:0])
664
3
775
(key, value) ! (row index, row)](https://image.slidesharecdn.com/ieee-bigdata-2013-140912163158-phpapp02/85/Direct-QR-factorizations-for-tall-and-skinny-matrices-in-MapReduce-architectures-IEEE-BigData-8-320.jpg)
![c example: (x, y, z) coordinates and model number:
((47570,103.429767811242,0,-16.525510963787,iDV7), [0.00019924
-4.706066e-05 2.875293979e-05 2.456653e-05 -8.436627e-06 -1.508808e-05
3.731976e-06 -1.048795e-05 5.229153e-06 6.323812e-06])
Figure: Aircraft simulation data. Aero/Astro Department, Stanford](https://image.slidesharecdn.com/ieee-bigdata-2013-140912163158-phpapp02/85/Direct-QR-factorizations-for-tall-and-skinny-matrices-in-MapReduce-architectures-IEEE-BigData-10-320.jpg)
![nement step by Qs = AR1
1
I R = R1Rs , QTQ I for ill-conditioned A
I Theory on why this works, need 100n log n rows
[Mahoney 2011], [Avron, Maymounkov, and Toledo 2010],
[Ipsen and Wentworth 2012]
We call this Pseudo-Iterative Re](https://image.slidesharecdn.com/ieee-bigdata-2013-140912163158-phpapp02/85/Direct-QR-factorizations-for-tall-and-skinny-matrices-in-MapReduce-architectures-IEEE-BigData-28-320.jpg)
Direct QR factorizations for tall-and-skinny matrices in MapReduce architectures (IEEE BigData)
- 1. Direct QR factorizations for tall-and-skinny matrices in MapReduce architectures Austin Benson ICME, Stanford University David Gleich (Purdue) and Jim Demmel (UC-Berkeley) A Q R m n m n n n IEEEDATA October 8, 2013
- 2. Contributions 2 I Numerically stable and scalable algorithm for QR and SVD of tall-and-skinny matrices in MapReduce I Performance and stability tradeos of several methods I Performance model: prediction within a factor of two I Code: https://github.com/arbenson/mrtsqr
- 3. MapReduce overview 3 Two functions that operate on key value pairs: (key; value) map ! (key; value) (key; hvalue1; : : : ; valueni) reduce ! (key; value) shue stage between map and reduce to sort values by key.
- 4. MapReduce overview 4 The programmer implements: I map(key, value) I reduce(key, h value1, : : :, valuen i) Handled by MapReduce framework, e.g., Hadoop: I shue I load balancing I reading and writing data I data serialization I fault tolerance I ...
- 5. MapReduce Example: ColorCount 5 (key, value) input is (image id, image) 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 shuffle Map 1 Reduce 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 5 2 1 1 4 2 def ColorCountMap(key , val ) : for pixel in val : yield ( pixel , 1) def ColorCountReduce(key , vals ) : total = sum( vals ) yield (key , total )
- 6. Why MapReduce? (for scientists) 6 MapReduce is restrictive! Why Bother? I Easy I load balancing I structured data I/O I fault tolerance I cheap clusters with large data storage Hadoop may not be the best option... Generate lots of data on supercomputer Post-process and analyze on MapReduce cluster
- 7. Tall-and-skinny matrices 7 What are tall-and-skinny matrices? m n n m A Examples: rows are data samples; blocks of A are images from a video; Krylov subspaces; unrolled tensors
- 8. Matrix representation 8 We have matrices, so what are the key-value pairs? A = 2 1:0 0:0 2:4 3:7 0:8 4:2 9:0 9:0 664 3 775 ! 2 (1; [1:0; 0:0]) (2; [2:4; 3:7]) (3; [0:8; 4:2]) (4; [9:0; 9:0]) 664 3 775 (key, value) ! (row index, row)
- 9. Matrix representation: an example 9 Scienti
- 10. c example: (x, y, z) coordinates and model number: ((47570,103.429767811242,0,-16.525510963787,iDV7), [0.00019924 -4.706066e-05 2.875293979e-05 2.456653e-05 -8.436627e-06 -1.508808e-05 3.731976e-06 -1.048795e-05 5.229153e-06 6.323812e-06]) Figure: Aircraft simulation data. Aero/Astro Department, Stanford
- 11. Tall-and-skinny matrices 10 Tall-and-skinny: m n n m A Slightly more rigorous de
- 12. nition: It is cheap to pass O(n2) data to all processors.
- 13. Quick QR and SVD review 11 A Q R VT n n n n n n n n A U n n n n Σ n n Figure: Q, U, and V are orthogonal matrices. R is upper triangular and is diagonal with decreasing, nonnegative entries.
- 14. Tall-and-skinny QR 12 A Q R m n m n n n Tall-and-skinny (TS): m n. QTQ = I .
- 15. TS-QR ! TS-SVD 13 A Q R Σ VT Q UR U R is small, so computing its SVD is cheap.
- 16. Why Tall-and-skinny QR and SVD? 14 1. Regression with many samples 2. Principle Component Analysis (PCA) 3. Model Reduction Pressure, Dilation, Jet Engine Figure: Dynamic mode decomposition of the screech of a jet. Joe Nichols, University of Minnesota.
- 17. Cholesky QR 15 Cholesky QR ATA = (QR)T (QR) = RTQTQR = RTR I Computing ATA in MapReduce is easy and well-studied. I We call this Cholesky QR.
- 18. Cholesky QR: Getting Q 16 Q = AR1. A A1 R-1 map R Q1 A2 R-1 map Q2 A3 R-1 map Q3 A4 R-1 map Q4 emit emit emit emit distribute Local MatMul
- 19. Stability problems 17 I Can get Q = AR1 I Problem: Columns can be far from orthogonal and ATA squares condition number (data later) I Idea: Use a more advanced algorithm.
- 20. Communication-avoiding TSQR 18 A = 2 A1 A2 A3 A4 664 3 775 | {z } 8n4n = 2 Q1 664 Q2 Q3 Q4 3 775 | {z } 8n4n 2 R1 R2 R3 R4 664 3 775 | {z } 4nn = =Q z2 }| { Q1 664 Q2 Q3 Q4 3 775 | {z } 8n4n ~Q |{z} 4nn |{Rz} nn Demmel et al. 2008
- 21. Communication-avoiding TSQR 19 A = 2 A1 A2 A3 A4 664 3 775 | {z } 8n4n = 2 Q1 664 Q2 Q3 Q4 3 775 | {z } 8n4n 2 R1 R2 R3 R4 664 3 775 | {z } 4nn Ai = QiRi can be computed in parallel. If we only need R, then we can throw out the intermediate Qi factors.
- 22. MapReduce TSQR 20 S(1) A A1 A3 A3 R1 map A2 R emit 2 map R emit 3 map A4 R emit 4 map shuffle S1 A2 reduce S2 R2,1 emit reduce R2,2 emit emit shuffle AS23 reduce R2,3 emit Local TSQR identity map SA(22 ) reduce R emit Local TSQR Local TSQR Figure: S(1) is the matrix consisting of the rows of all of the Ri factors. Similarly, S(2) consists of all of the rows of the R2;j factors.
- 23. MapReduce TSQR: Getting Q 21 I Again: have R, want Q A = QR ! Q = AR1 I We call this method Indirect TSQR. I Problem: Q can be far from orthogonal (again).
- 24. Indirect TSQR: Iterative Re
- 25. nement 22 Iterative re
- 26. nement: repeat TSQR for a more orthogonal Q A A1 R-1 map R Q1 A2 R-1 map Q2 A3 R-1 map Q3 A4 R-1 map Q4 emit emit emit emit distribute TSQR Q Q1 R1 -1 map R1 Q1 Q2 R1 -1 map Q2 Q3 R1 -1 map Q3 Q4 R1 -1 map Q4 emit emit emit emit distribute Local MatMul Local MatMul Iterative Refinement step
- 27. Indirect TSQR: a randomized approach 23 I Idea: Take a small sample of rows of A and form Rs s , Qs ! R1, Q = QsR1 I Re
- 28. nement step by Qs = AR1 1 I R = R1Rs , QTQ I for ill-conditioned A I Theory on why this works, need 100n log n rows [Mahoney 2011], [Avron, Maymounkov, and Toledo 2010], [Ipsen and Wentworth 2012] We call this Pseudo-Iterative Re
- 29. nement
- 31. nement 24 A A1 Rs -1 map Rs Q1 A2 Rs -1 map Q2 A3 Rs -1 map Q3 A4 Rs -1 map Q4 emit emit emit emit distribute TSQR Q Q1 R1 -1 map R1 Q1 Q2 R1 -1 map Q2 Q3 R1 -1 map Q3 Q4 R1 -1 map Q4 emit emit emit emit distribute Local MatMul Local MatMul Iterative Form Qs Refinement step A1 TSQR Form Rs (In the implementation, combine AR1 s and TSQR in one pass)
- 32. Direct TSQR 25 Why is computing truly orthogonal Q dicult in MapReduce? I Orthogonality is a global property, but we compute locally. I Can only label data via keys and
- 33. le names.
- 34. Communication-avoiding TSQR 26 A = 2 A1 A2 A3 A4 664 3 775 | {z } 8n4n = 2 Q1 664 Q2 Q3 Q4 3 775 | {z } 8n4n 2 R1 R2 R3 R4 664 3 775 | {z } 4nn = 2 664 Q1 Q2 Q3 Q4 3 775 | {z } 8n4n 2 Q1;2 Q2;2 Q3;2 Q4;2 664 3 775 | {z } 4nn |{Rz} nn = 2 Q1Q1;2 Q2Q2;2 Q3Q1;2 Q4Q1;2 664 3 775 | {z } 8nn |{Rz} nn = QR
- 35. Gathering Q 27 2 R1 R2 R3 R4 664 3 775 | {z } n#(mappers)n = 2 Q1;2 Q2;2 Q3;2 Q4;2 664 3 775 | {z } n#(mappers)n |{Rz} nn I Idea: Compute QR (n #(mappers) rows) in serial. I Idea: Pass Qi ;2 (n rows each) in second pass to reconstruct Q. I We call this Direct TSQR
- 36. Direct TSQR: Steps 1 and 2 28 A A1 map R1 Q1 emit emit A2 map R2 Q2 emit emit A3 map R3 Q3 emit emit A4 map R4 Q4 emit emit First step R1 R2 R3 R4 Q1,2 Q2,2 Q3,2 Q4,2 R reduce emit emit emit emit emit Second step shuffle
- 37. Direct TSQR: Step 3 29 Q1 Q1,2 emit map Q Q2 Q2,2 emit map Q Q3 Q3,2 emit map Q Q41 Q4,2 emit map Q Q12 Q22 Q32 Q42 distribute Third step
- 38. Stability 30 0 2 10 0 10 −2 10 −4 10 −6 10 −8 10 −10 10 −12 10 −14 10 −16 10 2 10 4 10 6 10 8 10 10 10 12 10 14 10 16 10 10 (A) k 2 2 ||QTQ − I|| Numerical stability: 10,000x10 matrices Indir. TSQR + PIR Dir. TSQR Indir. TSQR Indir. TSQR + IR Chol. Chol. + IR
- 39. Performance model 31 I Only count reads and writes I Streaming benchmark for read and write bandwidth of system I Within a factor of two of experimental data for all algorithms I I/O dominates runtime I Algorithms take same time as a few passes over data
- 40. Performance 32 4B x 4 (134.6 GB) 2.5B x 10 (193.1 GB) 600M x 25 (112.0 GB) 500M x 50 (183.6 GB) 150M x 100 (109 GB) Matrix size 7000 6000 5000 4000 3000 2000 1000 0 Time to solution (seconds) Performance of QR algorithms on MapReduce Chol Indir TSQR Chol + PIR Indir TSQR + PIR Chol + IR Indir TSQR + IR Direct TSQR
- 41. Direct TSQR: recursive extension 33 2 R1 R2 R3 R4 664 3 775 | {z } n#(mappers)n TSQR ! 2 Q1;2 Q2;2 Q3;2 Q4;2 664 3 775 | {z } n#(mappers)n |{Rz} nn I n #(mappers) rows is too large ! recurse
- 42. Direct TSQR: recursive performance 34 0 50 100 150 200 6000 4000 2000 0 number of columns running time (s) 150M rows 0 50 100 150 200 250 8000 6000 4000 2000 0 number of columns running time (s) 100M rows 0 50 100 150 200 250 300 15000 10000 5000 0 number of columns running time (s) 50M rows no recursion recursion no recursion recursion no recursion recursion
- 43. End 35 Contributions: I Numerically stable and scalable QR I Performance and stability tradeos I Performance model I Code: https://github.com/arbenson/mrtsqr Contact: I [email protected]