Abstract image of a woman sitting on books and studying on a laptop

Code dive 2019 kamil witecki - should i care about cpu cache

K
Kamil Witecki

The document discusses the importance of CPU cache, focusing on latency and performance metrics within memory hierarchies. It presents various examples of cache mapping techniques, including direct-mapping and set-associative caching, addressing their consequences on efficiency and speed of data access. Additionally, it analyzes the impact of alignment and structure layout on cache performance, providing empirical data on different object retrieval methods.

Software
1 of 71
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
Should I care about CPU cache? Kamil Witecki
Disclaimers
Key notes Relative latency
Key notes Relative latency Organization
Key notes Relative latency Organization Profits?
What is average DRAM latency? 0 5 10 15 20 25 0 500 1000 1500 2000 2500 speed (MT s ) latency(ns)
And end to end? row 0 row N . . . data address control
And end to end? row 0 row N . . . data address control activeinactive
And end to end? row 0 row N . . . data address control activeinactive
And end to end? row 0 row N . . . data address control activeinactivelatency
And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive
And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive more latency
And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive more latency DRAM controller
And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive more latency DRAM controller CPU core
And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive more latency DRAM controller CPU core end to end latency: 50-100ns
Is 50ns a lot? What Duration Reference 50ns
Is 50ns a lot? What Duration Reference 50ns 1 clock cycle 50ns
Is 50ns a lot? What Duration Reference 50ns 1 clock cycle@2MHz 50ns Fun fact: Clock rate of 8080 CPU
Is 50ns a lot? What Duration Reference 50ns 1 clock cycle@2MHz 50ns 1 clock cycle@1GHz 1ns
Is 50ns a lot? What Duration Reference 50ns 1 clock cycle@2MHz 50ns 1 clock cycle@1GHz 1ns 1 clock cycle@5GHz 0.2ns
Is 50ns a lot? What Duration Reference 50ns 1 clock cycle@2MHz 50ns 1 clock cycle@1GHz 1ns 1 clock cycle@5GHz 0.2ns Fun fact: 1 heartbeat vs boiling 1 liter of water
And how to counter that? row 0 row N . . . data address control activeinactive inactiveactive DRAM controller CPU core end to end latency: 50-100ns
And how to counter that? row 0 row N . . . data address control activeinactive inactiveactive DRAM controller CPU core end to end latency: 50-100ns Cache
And how to counter that? row 0 row N . . . data address control activeinactive inactiveactive DRAM controller CPU core end to end latency: 50-100ns Cache DRAM <-> Cache latency: 50-100ns
And how to counter that? row 0 row N . . . data address control activeinactive inactiveactive DRAM controller CPU core end to end latency: 50-100ns Cache DRAM <-> Cache latency: 50-100nsCache latency: 1 clock cycle
Memory hierarchy L1 64KiB L2 512KiB L3 8MiB RAM 64MiB
Memory hierarchy L1 64KiB L2 512KiB L3 8MiB RAM 64MiB
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ?
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ?
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ?
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1 Consequences: - complexity grows with number of ways Consequences: - 15% less cache misses Consequences: - avoids N-parallel stalls
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1 Consequences: - complexity grows with number of ways Consequences: - 15% less cache misses Consequences: - avoids N-parallel stalls
Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1 Consequences: - complexity grows with number of ways Consequences: - 15% less cache misses Consequences: - avoids N-parallel stalls
Cache line L1
Cache line L1 Cache line: 32B
Cache line L1 Cache line: 32B
Cache line L1 Cache line: 32B
Cache line L1 Cache line: 32B
Cache line L1 Cache line: 32B
Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];
Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];
Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];
Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];Alignment: 4B -> int32_t;
Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];Alignment: 4B -> int32_t;
Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];Alignment: 4B -> int32_t;Alignment: 32B -> alignas(32) int32_t x[4];
Cache line - write consequences Cache line: 32B Object size: 4B Alignment: 4B
Cache line - write consequences Cache line: 32B Object size: 4B Alignment: 4B
Cache line - write consequences Cache line: 32B Object size: 4B Alignment: 4B Cache line: 32B (invalidated)
Keeping caches hot // data: vector of objects composed of: // int32_t type , string name , data* parent , // map<string , string > params // task: find by type
Naive C-style! // data: vector of objects composed of: // int32_t type , string name , data* parent , // map<string , string > params s t r u c t data { i n t 3 2 _ t type ; s t r i n g name ; o b s e r v e r _ p t r <data> p a r en t ; map<s t r i n g , s t r i n g > params ; } ; // task: find by type (vector is sorted by type) pair <s i z e _ t , s i z e _ t > f i n d _ b y _ t y p e ( v e c t o r <data> const & x , key type ) { auto r = equal_range ( begin ( x ) , end ( x ) , type ) ; r e t u r n { r . f i r s t − begin ( x ) , r . second − begin ( x ) } ; }
Layout s i z e o f ( data ) ; // : 64B o f f s e t o f ( type ) ; // : 0B o f f s e t o f ( name ) ; // : 8B o f f s e t o f ( p ar e n t ) ; // : 40B o f f s e t o f ( params ) ; // : 48B a l i g n o f ( data ) ; // : 64B //Cache line : 64B Type #1 — Name #1 Parent #1 Params #1 Type #2 — Name #2 Parent #2 Params #2
C++ style! // data: vector of objects composed of: // int32_t type , string name , data* parent , // map<string , string > params s t r u c t data { s t r i n g name ; o b s e r v e r _ p t r <data> p a r en t ; map<s t r i n g , s t r i n g > params ; } ; boost : : flat_map <i n t 3 2 _ t , data >:: equal_range ; std : : map<i n t 3 2 _ t , data >:: equal_range ; std : : unordered_map<i n t 3 2 _ t , data >:: equal_range ;
AMD AthlonTM II X3 440, 12GiB RAM DDR3-1333, clang 8.0.1-3 0 2500 5000 7500 10000 1e+04 1e+06 1e+08 Number of objects Time(ns) colour flat-map-clang map-clang naive-clang unordered-map-clang
AMD RyzenTM 1600X, 16GiB RAM DDR4-3200, clang 8.0.1-3 0 2500 5000 7500 10000 1e+04 1e+06 1e+08 Number of objects Time(ns) colour flat-map-clang map-clang naive-clang unordered-map-clang
Will separate array do better? s t r u c t a r r { v e c t o r <i n t 3 2 _ t > types ; v e c t o r <entry > e n t r i e s ; } ; pair <s i z e _ t , s i z e _ t > f i n d _ b y _ t y p e ( a r r const & d , i n t 3 2 _ t type ) { auto b = begin ( d . types ) ; auto r = equal_range ( b , end ( d . types ) , type ) ; r e t u r n { r . f i r s t − b , r . second − b } ; }
Layout s i z e o f ( type ) ; // : 8B a l i g n o f ( data ) ; // : 4B //Cache line : 64B Type #1 Type #2 Type #3 Type #4 Type #5 Type #6 Type #7 Type #8 Type #9 Type #10 Type #11 Type #12 Type #13 Type #14 Type #15 Type #16
AMD AthlonTM II X3 440, 12GiB RAM DDR3-1333, clang 8.0.1-3 0 2500 5000 7500 10000 1e+04 1e+06 1e+08 Number of objects Time(ns) colour flat-map-clang map-clang naive-clang optimized-clang unordered-map-clang
AMD RyzenTM 1600X, 16GiB RAM DDR4-3200, clang 8.0.1-3 0 2500 5000 7500 10000 1e+04 1e+06 1e+08 Number of objects Time(ns) colour flat-map-clang map-clang naive-clang optimized-clang unordered-map-clang
NOT BAD
Instructions (1.0e+07, 2.5e+07, 5.0e+07) 95614956149561495614111611161116111613461346134613468698698698691241124112411241 0 500 1000 1500 Instr values types flat-map map naive optimized unordered-map
L1 uses & misses 870587058888115115156156115115 1739517395122122130130227227118118 0 100 200 300 L1 misses L1 uses values types flat-map map naive optimized unordered-map
LL miss rate 7.77.77.77.78.68.68.68.69.29.29.29.215.315.315.315.39.89.89.89.8 0 5 10 15 L3 miss rate values types flat-map map naive optimized unordered-map
Questions and Answers Kamil.Witecki@nokia.com
Bibliograpy I Micron Technology, Inc. Speed vs. latency. White paper, Micron Technology, Inc. David A. Patterson and John L. Hennessy. Computer Organization and Design, Fifth Edition: The Hardware/Software Interface. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 5th edition, 2013.

More Related Content

PDF
A Comparative Analysis between SHA and MD5 algorithms
Er Piyush Gupta IN ⊞⌘
 
PDF
RDF Streams and Continuous SPARQL (C-SPARQL)
Artificial Intelligence Institute at UofSC
 
PPT
Ch12
Joe Christensen
 
PPTX
Secure Hash Algorithm
Vishakha Agarwal
 
PDF
SHA-1 backdooring & exploitation
Ange Albertini
 
PPTX
NDD Project presentation
ahmedmishfaq
 
PPTX
SHA- Secure hashing algorithm
Ruchi Maurya
 
PDF
Sha
ha123
 
A Comparative Analysis between SHA and MD5 algorithms
Er Piyush Gupta IN ⊞⌘
 
RDF Streams and Continuous SPARQL (C-SPARQL)
Artificial Intelligence Institute at UofSC
 
Ch12
Joe Christensen
 
Secure Hash Algorithm
Vishakha Agarwal
 
SHA-1 backdooring & exploitation
Ange Albertini
 
NDD Project presentation
ahmedmishfaq
 
SHA- Secure hashing algorithm
Ruchi Maurya
 
Sha
ha123
 

What's hot (18)

PDF
Hash functions MD5 and RIPEMD 160
chutinhha
 
PDF
The SHA Hashing Algorithm
Bob Landstrom
 
PPT
Secure hashing algorithm
Karteek Paruchuri
 
PDF
SHA 1 Algorithm
Shiva RamDam
 
PPTX
Message Authentication using Message Digests and the MD5 Algorithm
Ajay Karri
 
PPTX
Data streaming algorithms
Hridyesh Bisht
 
PPT
Hash crypto
Harry Potter
 
PPTX
Basic explanation to md5 implementation in C
Sourav Punoriyar
 
PPT
Hash& mac algorithms
Harry Potter
 
PPTX
Hash Techniques in Cryptography
Basudev Saha
 
PPTX
MD5
rokham khawaja
 
PPTX
Secure Hashing Techniques - Introduction
Udhayyagethan Mano
 
PPTX
Message digest 5
Tirthika Bandi
 
PPT
Hash Function & Analysis
Pawandeep Kaur
 
PDF
MD-5 : Algorithm
Sahil Kureel
 
PPTX
Chacha ppt
VikramSingh1378
 
PPT
Computer Networking : Principles, Protocols and Practice - lesson 1
Olivier Bonaventure
 
PPT
Hashing
Hossain Md Shakhawat
 
Hash functions MD5 and RIPEMD 160
chutinhha
 
The SHA Hashing Algorithm
Bob Landstrom
 
Secure hashing algorithm
Karteek Paruchuri
 
SHA 1 Algorithm
Shiva RamDam
 
Message Authentication using Message Digests and the MD5 Algorithm
Ajay Karri
 
Data streaming algorithms
Hridyesh Bisht
 
Hash crypto
Harry Potter
 
Basic explanation to md5 implementation in C
Sourav Punoriyar
 
Hash& mac algorithms
Harry Potter
 
Hash Techniques in Cryptography
Basudev Saha
 
MD5
rokham khawaja
 
Secure Hashing Techniques - Introduction
Udhayyagethan Mano
 
Message digest 5
Tirthika Bandi
 
Hash Function & Analysis
Pawandeep Kaur
 
MD-5 : Algorithm
Sahil Kureel
 
Chacha ppt
VikramSingh1378
 
Computer Networking : Principles, Protocols and Practice - lesson 1
Olivier Bonaventure
 
Hashing
Hossain Md Shakhawat
 
Ad

Similar to Code dive 2019 kamil witecki - should i care about cpu cache (20)

PPT
Memory Optimization
Wei Lin
 
PPT
Memory Optimization
guest3eed30
 
PDF
Caching in (DevoxxUK 2013)
RichardWarburton
 
PPTX
Code and Memory Optimisation Tricks
Sperasoft
 
PPTX
Code and memory optimization tricks
DevGAMM Conference
 
PPTX
Data oriented design and c++
Mike Acton
 
PPT
Gdc03 ericson memory_optimization
brettlevin
 
PPT
cache memory
NAHID HASAN
 
PDF
Caching in
RichardWarburton
 
PPT
04 Cache Memory
Jeanie Delos Arcos
 
PPTX
#GDC15 Code Clinic
Mike Acton
 
PPT
04 cache memory
dilip kumar
 
PPT
Memory organization including cache and RAM.ppt
bansidhar11
 
PPT
Memory Organization and Cache mapping.ppt
bansidhar11
 
PDF
Cache-Memory for university courses at PG
Saurabh846965
 
PPT
chapter6.ppt
2020CE19
 
PPT
Lec9 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- Memory part 1
Hsien-Hsin Sean Lee, Ph.D.
 
PPT
cache memory.ppt
MUNAZARAZZAQELEA
 
PPT
cache memory.ppt
MUNAZARAZZAQELEA
 
PPTX
Cache recap
Hoang Nguyen
 
Memory Optimization
Wei Lin
 
Memory Optimization
guest3eed30
 
Caching in (DevoxxUK 2013)
RichardWarburton
 
Code and Memory Optimisation Tricks
Sperasoft
 
Code and memory optimization tricks
DevGAMM Conference
 
Data oriented design and c++
Mike Acton
 
Gdc03 ericson memory_optimization
brettlevin
 
cache memory
NAHID HASAN
 
Caching in
RichardWarburton
 
04 Cache Memory
Jeanie Delos Arcos
 
#GDC15 Code Clinic
Mike Acton
 
04 cache memory
dilip kumar
 
Memory organization including cache and RAM.ppt
bansidhar11
 
Memory Organization and Cache mapping.ppt
bansidhar11
 
Cache-Memory for university courses at PG
Saurabh846965
 
chapter6.ppt
2020CE19
 
Lec9 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- Memory part 1
Hsien-Hsin Sean Lee, Ph.D.
 
cache memory.ppt
MUNAZARAZZAQELEA
 
cache memory.ppt
MUNAZARAZZAQELEA
 
Cache recap
Hoang Nguyen
 
Ad

Recently uploaded (20)

PPTX
Airline CRS | Airline CRS Systems | CRS System
yugababu033
 
PDF
AI-Powered Fuzz Testing: The Future of QA
Shubham Joshi
 
PDF
PDF-XChange Editor Plus 10.7.0.398.0 Crack Free Download Latest 2025
imang66g
 
PDF
Type Class Derivation in Scala 3 - Jose Luis Pintado Barbero
José Luis Pintado Barbero
 
PDF
Top 10 Software Development Trends to Watch in 2025 🚀.pdf
KodekX
 
PDF
Multiverse AI Review 2025: Access All TOP AI Model-Versions!
Moshiur Rahman Jisan
 
PDF
DNT Brochure 2025 – ISV Solutions @ D365
sai vignesh
 
PPTX
Tech Workshop Escape Room Tech Workshop
muthuiyad
 
PDF
The Dynamic Duo Transforming Financial Accounting Systems Through Modern Expe...
Triforce Global
 
PDF
EaseUS PDF Editor Pro 6.2.0.2 Crack with License Key 2025
halimaanam8
 
PDF
BoxLang Dynamic AWS Lambda - Japan Edition
Ortus Solutions, Corp
 
PDF
MCP Security Tutorial - Beginner to Advanced
Harish Ramadoss
 
PPTX
4Seller: The All-in-One Multi-Channel E-Commerce Management Platform for Glob...
4Seller Multi-channel Ecommerce Tool
 
PDF
novaPDF Pro 11.9.482 Crack + License Key [Latest 2025]
viktamscs
 
PDF
Practical Indispensable Project Management Tips for Delivering Successful Exp...
Kandarp Bhatt
 
PPTX
Cybersecurity-and-Fraud-Protecting-Your-Digital-Life.pptx
balaji9945191
 
PDF
Workplace Software and Skills - OpenStax
tranviettoan19672001
 
PPTX
Matchmaking for JVMs: How to Pick the Perfect GC Partner
Tier1 app
 
PDF
AI/ML Infra Meetup | Beyond S3's Basics: Architecting for AI-Native Data Access
Alluxio, Inc.
 
PDF
Cloud Native Aachen Meetup - Aug 21, 2025
roehlclemens
 
Airline CRS | Airline CRS Systems | CRS System
yugababu033
 
AI-Powered Fuzz Testing: The Future of QA
Shubham Joshi
 
PDF-XChange Editor Plus 10.7.0.398.0 Crack Free Download Latest 2025
imang66g
 
Type Class Derivation in Scala 3 - Jose Luis Pintado Barbero
José Luis Pintado Barbero
 
Top 10 Software Development Trends to Watch in 2025 🚀.pdf
KodekX
 
Multiverse AI Review 2025: Access All TOP AI Model-Versions!
Moshiur Rahman Jisan
 
DNT Brochure 2025 – ISV Solutions @ D365
sai vignesh
 
Tech Workshop Escape Room Tech Workshop
muthuiyad
 
The Dynamic Duo Transforming Financial Accounting Systems Through Modern Expe...
Triforce Global
 
EaseUS PDF Editor Pro 6.2.0.2 Crack with License Key 2025
halimaanam8
 
BoxLang Dynamic AWS Lambda - Japan Edition
Ortus Solutions, Corp
 
MCP Security Tutorial - Beginner to Advanced
Harish Ramadoss
 
4Seller: The All-in-One Multi-Channel E-Commerce Management Platform for Glob...
4Seller Multi-channel Ecommerce Tool
 
novaPDF Pro 11.9.482 Crack + License Key [Latest 2025]
viktamscs
 
Practical Indispensable Project Management Tips for Delivering Successful Exp...
Kandarp Bhatt
 
Cybersecurity-and-Fraud-Protecting-Your-Digital-Life.pptx
balaji9945191
 
Workplace Software and Skills - OpenStax
tranviettoan19672001
 
Matchmaking for JVMs: How to Pick the Perfect GC Partner
Tier1 app
 
AI/ML Infra Meetup | Beyond S3's Basics: Architecting for AI-Native Data Access
Alluxio, Inc.
 
Cloud Native Aachen Meetup - Aug 21, 2025
roehlclemens
 

Code dive 2019 kamil witecki - should i care about cpu cache

  • 1. Should I care about CPU cache? Kamil Witecki
  • 5. Key notes Relative latency Organization Profits?
  • 6. What is average DRAM latency? 0 5 10 15 20 25 0 500 1000 1500 2000 2500 speed (MT s ) latency(ns)
  • 7. And end to end? row 0 row N . . . data address control
  • 8. And end to end? row 0 row N . . . data address control activeinactive
  • 9. And end to end? row 0 row N . . . data address control activeinactive
  • 10. And end to end? row 0 row N . . . data address control activeinactivelatency
  • 11. And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive
  • 12. And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive more latency
  • 13. And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive more latency DRAM controller
  • 14. And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive more latency DRAM controller CPU core
  • 15. And end to end? row 0 row N . . . data address control activeinactivelatency inactiveactive more latency DRAM controller CPU core end to end latency: 50-100ns
  • 16. Is 50ns a lot? What Duration Reference 50ns
  • 17. Is 50ns a lot? What Duration Reference 50ns 1 clock cycle 50ns
  • 18. Is 50ns a lot? What Duration Reference 50ns 1 clock cycle@2MHz 50ns Fun fact: Clock rate of 8080 CPU
  • 19. Is 50ns a lot? What Duration Reference 50ns 1 clock cycle@2MHz 50ns 1 clock cycle@1GHz 1ns
  • 20. Is 50ns a lot? What Duration Reference 50ns 1 clock cycle@2MHz 50ns 1 clock cycle@1GHz 1ns 1 clock cycle@5GHz 0.2ns
  • 21. Is 50ns a lot? What Duration Reference 50ns 1 clock cycle@2MHz 50ns 1 clock cycle@1GHz 1ns 1 clock cycle@5GHz 0.2ns Fun fact: 1 heartbeat vs boiling 1 liter of water
  • 22. And how to counter that? row 0 row N . . . data address control activeinactive inactiveactive DRAM controller CPU core end to end latency: 50-100ns
  • 23. And how to counter that? row 0 row N . . . data address control activeinactive inactiveactive DRAM controller CPU core end to end latency: 50-100ns Cache
  • 24. And how to counter that? row 0 row N . . . data address control activeinactive inactiveactive DRAM controller CPU core end to end latency: 50-100ns Cache DRAM <-> Cache latency: 50-100ns
  • 25. And how to counter that? row 0 row N . . . data address control activeinactive inactiveactive DRAM controller CPU core end to end latency: 50-100ns Cache DRAM <-> Cache latency: 50-100nsCache latency: 1 clock cycle
  • 26. Memory hierarchy L1 64KiB L2 512KiB L3 8MiB RAM 64MiB
  • 27. Memory hierarchy L1 64KiB L2 512KiB L3 8MiB RAM 64MiB
  • 28. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3
  • 29. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ?
  • 30. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ?
  • 31. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ?
  • 32. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient
  • 33. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal
  • 34. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line
  • 35. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1
  • 36. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1
  • 37. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1
  • 38. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1 Consequences: - complexity grows with number of ways Consequences: - 15% less cache misses Consequences: - avoids N-parallel stalls
  • 39. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1 Consequences: - complexity grows with number of ways Consequences: - 15% less cache misses Consequences: - avoids N-parallel stalls
  • 40. Associativity or how to map memory Must be fast, die-size and power efficient Ln+1 Ln 0 1 2 3 4 5 6 7 0 1 2 3 Example: Direct-mapping Selection: address mod 4 Consequences: ? Consequences: - simple - fast and die-size, power efficient Consequences: - good best case - optimal sequence traversal Consequences: - worst case - jumping every nth line Example: 2-way set associative Selection: address mod 2 And then? Example: 2-way set associative Selection: address mod 2 Then: Least Recently Used Consequences: ? 0 1 Consequences: - complexity grows with number of ways Consequences: - 15% less cache misses Consequences: - avoids N-parallel stalls
  • 47. Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];
  • 48. Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];
  • 49. Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];
  • 50. Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];Alignment: 4B -> int32_t;
  • 51. Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];Alignment: 4B -> int32_t;
  • 52. Cache line - alignment consequences Cache line: 32B Object size: 4B Alignment: 2B -> alignas(2) std::byte x[4];Alignment: 4B -> int32_t;Alignment: 32B -> alignas(32) int32_t x[4];
  • 53. Cache line - write consequences Cache line: 32B Object size: 4B Alignment: 4B
  • 54. Cache line - write consequences Cache line: 32B Object size: 4B Alignment: 4B
  • 55. Cache line - write consequences Cache line: 32B Object size: 4B Alignment: 4B Cache line: 32B (invalidated)
  • 56. Keeping caches hot // data: vector of objects composed of: // int32_t type , string name , data* parent , // map<string , string > params // task: find by type
  • 57. Naive C-style! // data: vector of objects composed of: // int32_t type , string name , data* parent , // map<string , string > params s t r u c t data { i n t 3 2 _ t type ; s t r i n g name ; o b s e r v e r _ p t r <data> p a r en t ; map<s t r i n g , s t r i n g > params ; } ; // task: find by type (vector is sorted by type) pair <s i z e _ t , s i z e _ t > f i n d _ b y _ t y p e ( v e c t o r <data> const & x , key type ) { auto r = equal_range ( begin ( x ) , end ( x ) , type ) ; r e t u r n { r . f i r s t − begin ( x ) , r . second − begin ( x ) } ; }
  • 58. Layout s i z e o f ( data ) ; // : 64B o f f s e t o f ( type ) ; // : 0B o f f s e t o f ( name ) ; // : 8B o f f s e t o f ( p ar e n t ) ; // : 40B o f f s e t o f ( params ) ; // : 48B a l i g n o f ( data ) ; // : 64B //Cache line : 64B Type #1 — Name #1 Parent #1 Params #1 Type #2 — Name #2 Parent #2 Params #2
  • 59. C++ style! // data: vector of objects composed of: // int32_t type , string name , data* parent , // map<string , string > params s t r u c t data { s t r i n g name ; o b s e r v e r _ p t r <data> p a r en t ; map<s t r i n g , s t r i n g > params ; } ; boost : : flat_map <i n t 3 2 _ t , data >:: equal_range ; std : : map<i n t 3 2 _ t , data >:: equal_range ; std : : unordered_map<i n t 3 2 _ t , data >:: equal_range ;
  • 60. AMD AthlonTM II X3 440, 12GiB RAM DDR3-1333, clang 8.0.1-3 0 2500 5000 7500 10000 1e+04 1e+06 1e+08 Number of objects Time(ns) colour flat-map-clang map-clang naive-clang unordered-map-clang
  • 61. AMD RyzenTM 1600X, 16GiB RAM DDR4-3200, clang 8.0.1-3 0 2500 5000 7500 10000 1e+04 1e+06 1e+08 Number of objects Time(ns) colour flat-map-clang map-clang naive-clang unordered-map-clang
  • 62. Will separate array do better? s t r u c t a r r { v e c t o r <i n t 3 2 _ t > types ; v e c t o r <entry > e n t r i e s ; } ; pair <s i z e _ t , s i z e _ t > f i n d _ b y _ t y p e ( a r r const & d , i n t 3 2 _ t type ) { auto b = begin ( d . types ) ; auto r = equal_range ( b , end ( d . types ) , type ) ; r e t u r n { r . f i r s t − b , r . second − b } ; }
  • 63. Layout s i z e o f ( type ) ; // : 8B a l i g n o f ( data ) ; // : 4B //Cache line : 64B Type #1 Type #2 Type #3 Type #4 Type #5 Type #6 Type #7 Type #8 Type #9 Type #10 Type #11 Type #12 Type #13 Type #14 Type #15 Type #16
  • 64. AMD AthlonTM II X3 440, 12GiB RAM DDR3-1333, clang 8.0.1-3 0 2500 5000 7500 10000 1e+04 1e+06 1e+08 Number of objects Time(ns) colour flat-map-clang map-clang naive-clang optimized-clang unordered-map-clang
  • 65. AMD RyzenTM 1600X, 16GiB RAM DDR4-3200, clang 8.0.1-3 0 2500 5000 7500 10000 1e+04 1e+06 1e+08 Number of objects Time(ns) colour flat-map-clang map-clang naive-clang optimized-clang unordered-map-clang
  • 67. Instructions (1.0e+07, 2.5e+07, 5.0e+07) 95614956149561495614111611161116111613461346134613468698698698691241124112411241 0 500 1000 1500 Instr values types flat-map map naive optimized unordered-map
  • 68. L1 uses & misses 870587058888115115156156115115 1739517395122122130130227227118118 0 100 200 300 L1 misses L1 uses values types flat-map map naive optimized unordered-map
  • 69. LL miss rate 7.77.77.77.78.68.68.68.69.29.29.29.215.315.315.315.39.89.89.89.8 0 5 10 15 L3 miss rate values types flat-map map naive optimized unordered-map
  • 71. Bibliograpy I Micron Technology, Inc. Speed vs. latency. White paper, Micron Technology, Inc. David A. Patterson and John L. Hennessy. Computer Organization and Design, Fifth Edition: The Hardware/Software Interface. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 5th edition, 2013.