[네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주
9 likes21,425 views
This document discusses load balancing problems in container clusters and proposes using the IPVS Maglev hashing scheduler to provide an efficient and reliable load balancer. It describes how traditional load balancing approaches like static routing and ECMP can cause issues like disruption when containers are added or removed. The Maglev hashing algorithm is presented as a solution to provide consistent hashing for high availability. The document outlines implementing Maglev hashing in the IPVS module of the Linux kernel to leverage netfilter for packet processing and forwarding while avoiding issues of traditional load balancers. This achieves an efficient and reliable load balancer that can hash consistently without connection loss, even if load balancers fail.
![Hashing Disruption
192.68.0.1 Hashing(IP) % 2
Container[0]
Container[1]
192.68.0.2
192.68.0.3
192.68.0.4
Hashing(IP) % 2
Hashing(IP) % 2
Hashing(IP) % 2](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-23-320.jpg)
![Hashing Disruption
192.68.0.1 Hashing(IP) % 4
192.68.0.2
192.68.0.3
192.68.0.4
Hashing(IP) % 4
Hashing(IP) % 4
Hashing(IP) % 4
Container[0]
Container[1]
Container[2]
Container[3]](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-24-320.jpg)
![Consistent Hash Ring
192.68
.0.2
Container[0]
192.68
.0.3
Container[2]
192.68
.0.4
Container[0]
Container[3]
Container[2]
192.68
.0.1
Container[1]
Container[3]
Container[1]
Guarantee
to remap
K/n only](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-27-320.jpg)
![Load Balancing
192.68
.0.2
Container[0]
192.68
.0.3
Container[2]
192.68
.0.4
Container[0]
Container[3]
Container[2]
192.68
.0.1
Container[1]
Container[3]
Container[1]
Efficient
Load
Balancing](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-28-320.jpg)
![Population
Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf
Population
Permutationtable ( 3x 7 )
B1 B0 B1 B0 B2 B2 B0
Lookup_table ( size =7 )
B0 B1 B2
3 0 3
0 2 4
4 4 5
1 6 6
5 1 0
2 3 1
6 5 2
[0] [1] [2] [3] [4] [5] [6]
Assign backends by
preference list](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-32-320.jpg)
![Load Balancing
Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf
B1 B0 B1 B0 B2 B2 B0
each backend will receive an
almost equal numberof
connections.
[0] [1] [2] [3] [4] [5] [6]
L4-LB
L4-LB
B0 =3 * connection
B1 =2 * connection
B2 =2 * connection](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-33-320.jpg)
![5-Tuples Hash
192.68.0.2
IPVS
IPVS MH
IPVS
IPVS MH
IPVS
IPVS MH
Container[0]
Container[1]
Container[2]
Container[3]
IPVS MH is no need to track connection table
for forwarding packets to same destination.](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-48-320.jpg)
![TCP Connection Loss
192.68.0.2
Container[0]
Container[1]
Container[2]
Container[3]
L4-LB
1. Connection
Established With
Container[2]
2. A Load
Balancer Down
3. ECMP Disruption.
Forward packet
another l4
4. Container[0] has no
idea about connection
with container[2]
5. Container[0] Send RST.
Connection Closed](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-52-320.jpg)
![TCP Connection Loss 2
192.68.0.3
Container[0]
Container[1]
Container[2]
Container[3]
L4-LB
L4-LB
1. Connection
Established With
Container[3]
2. A Load
Balancer Down
3. ECMP Disruption.
Forward packet
another l4 although
the serving LB is
alive. 4. Container[1] has no
idea about connection
with container[3]
5. Container[1] Send RST.
Connection Closed
L4-LB](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-53-320.jpg)
![No Loss TCP Connection
192.68.0.2
Container[0]
Container[1]
Container[2]
Container[3]
L4-LB
IPVS
1. Connection
Established With
Container[2]
2. A Load
Balancer Down
3. ECMP Disruption.
Forward packet
another l4
4. IPVS MH can forward
the packet to same
destination by hashing.
5. Continue the
Established connection.
IPVS MH](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-58-320.jpg)
![No Loss TCP Connection 2
192.68.0.3
Container[0]
Container[1]
Container[2]
Container[3]
L4-LB
1. Connection
Established With
Container[3]
2. A Load
Balancer Down
3. ECMP Disruption.
Forward packet
another l4 although
the serving LB is
alive.
5. Continue the
Established connection.
IPVS
IPVS
IPVS MH
4. IPVS MH can forward
the packet to same
destination by hashing
With no connection info.](https://image.slidesharecdn.com/ipvslinuxkernel-180905043834/85/Maglev-Hashing-Scheduler-in-IPVS-Linux-Kernel-59-320.jpg)
[네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주
- 1. Maglev Hashing Scheduler in IPVS, Linux Kernel Inju Song Software Engineer, Container Platform Team NAVER
- 2. CONTENTS Load balancing problems in Container Cluster Maglev Consistent Hashing IPVS Maglev Hashing Scheduler Build A Efficient and Reliable Load Balancer Q & A
- 5. Container Host Container Host Container Host Container Host Container Cluster There are many containers...
- 6. Load balancing Problems in Container Cluster
- 7. Static Routing L4-LB L4-LB L4-LB L4-LB
- 8. Container Host Container Host Container Host Container Host Container Cluster Added / Removed Frequently
- 9. Static Routing Problem Container Host(Past) Container Host(Current) L4-LB L4-LB L4-LB L4-LB
- 10. Container Cluster N Too many containers . . . . . . Container Cluster 5 Container Cluster 4 Container Cluster 3 Container Cluster 2 Container Cluster 1
- 11. Guarantee Load Balancing L4-LB Container Host Container Host Container Host Container Host L4-LB
- 13. Load Balancer Down Container Host Container Host Container Host Container Host L4-LB
- 14. Efficient and Reliable Load Balancing
- 16. Large Traffic Balancing L4-LB Container Host Container Host Container Host Container Host L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB
- 19. ECMP L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB L4-LB Equal Cost Multi Path
- 20. Hash Based Routing L4-LB L4-LB 1.1.1.1 L4-LB L4-LB 1.1.1.2 PATH= Hashing( IP.Src, IP.Dst, IP.Protocol, Port.Src, Port.Dst ) Mod N
- 21. Efficient and Reliable Load Balancing BGP/ ECMP
- 22. Internet L4-LB L4-LB L4-LB If a load balancer down L4-LB L4-LB L4-LB L4-LB
- 23. Hashing Disruption 192.68.0.1 Hashing(IP) % 2 Container[0] Container[1] 192.68.0.2 192.68.0.3 192.68.0.4 Hashing(IP) % 2 Hashing(IP) % 2 Hashing(IP) % 2
- 24. Hashing Disruption 192.68.0.1 Hashing(IP) % 4 192.68.0.2 192.68.0.3 192.68.0.4 Hashing(IP) % 4 Hashing(IP) % 4 Hashing(IP) % 4 Container[0] Container[1] Container[2] Container[3]
- 25. ECMP is A Good Load Balancing, but.. Not Reliable Not scalable High Availability
- 26. Consistent Hashing for High Availability
- 29. Google’s Network Load Balancer Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf
- 30. Google’s Maglev Features Consistent Hashing Backend Selection Packet Processing Packet Forwarding Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf
- 31. Permutation B0 B1 B2 3 0 3 0 2 4 4 4 5 1 6 6 5 1 0 2 3 1 6 5 2 Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Backend = 3 Table Size = 7 Permutate Hashing Permutationtable ( 3x 7 ) Preference list of Backends Lookup_table ( size =7,empty )
- 32. Population Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Population Permutationtable ( 3x 7 ) B1 B0 B1 B0 B2 B2 B0 Lookup_table ( size =7 ) B0 B1 B2 3 0 3 0 2 4 4 4 5 1 6 6 5 1 0 2 3 1 6 5 2 [0] [1] [2] [3] [4] [5] [6] Assign backends by preference list
- 33. Load Balancing Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf B1 B0 B1 B0 B2 B2 B0 each backend will receive an almost equal numberof connections. [0] [1] [2] [3] [4] [5] [6] L4-LB L4-LB B0 =3 * connection B1 =2 * connection B2 =2 * connection
- 34. Minimal disruption Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Population Permutationtable ( 2x 7 ) B0 B2 3 3 0 4 4 5 1 6 5 0 2 1 6 2 Assign backends by preference list Before B1 B0 B1 B0 B2 B2 B2 B1 is Removed Re-Permutate Hashing Lookup_table ( size =7 ) After B1 -> B0 B0 B1 -> B0 B0 B2 B2 B2 Same Value By hashing Lookup_table ( size =7 )
- 35. Hard to implement Google’s Maglev Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf Backend SelectionPacket ProcessingPacket Forwarding
- 37. Internet L4-LB L4-LB L4-LB L4-LB Network Software L4 Load Balancer
- 38. Using Netfilter in Linux Kernel Nftables – Wikipedia| https://en.wikipedia.org/wiki/Nftables
- 39. Replace Maglev With IPVS Consistent Hashing Backend Selection: FUNCTION Packet Processing: NETFILTER Packet Forwarding: NAT, DR, IP Tunneling
- 40. Pluggable Scheduler Module Weighted Round Robin Round Robin Source Hashing least connection
- 42. Maglev Scheduling Module Maglev Hashing Scheduling Round Robin Source Hashing least connection Weighted Round Robin
- 43. Maglev Hashing Table Size Maglev: A Fast and Reliable Software Network Load Balancer | https://www.usenix.org/system/files/conference/nsdi16/nsdi16-paper-eisenbud.pdf ... Lookup_table Size=251 ... Lookup_table Size=131071 Disruption % VS Memory usage
- 44. Optimize your Memory Usage Kernel Configuration -> Networking support -> Networking options -> Netfilter -> IPVS -> Maglev hashing table size IPVS MaglevHashing SchedulerKConfig | https://github.com/torvalds/linux/blob/master/net/netfilter/ipvs/Kconfig#L290
- 45. L4-LB L4-LB L4-LB L4-LB Canary Release Strategy New Service Old Service 5% 95%
- 46. Weight Based Balancing Container Host Container Host Container Host Container Host L4-LB W: 10 W: 40 W: 80
- 47. Destination Fallback Container Host Container Host Container Host Container Host L4-LB W: 10 W: 40 -> 0 W: 80
- 48. 5-Tuples Hash 192.68.0.2 IPVS IPVS MH IPVS IPVS MH IPVS IPVS MH Container[0] Container[1] Container[2] Container[3] IPVS MH is no need to track connection table for forwarding packets to same destination.
- 50. Internet L4-LB L4-LB L4-LB A load balancer down L4-LB L4-LB L4-LB L4-LB
- 51. ECMP Hashing Disruption L4-LB L4-LB 1.1.1.2 L4-LB L4-LB 1.1.1.3 PATH= Hashing() Mod ( N - 1 ) L4-LB L4-LB 1.1.1.4 L4-LB L4-LB 1.1.1.1
- 52. TCP Connection Loss 192.68.0.2 Container[0] Container[1] Container[2] Container[3] L4-LB 1. Connection Established With Container[2] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 4. Container[0] has no idea about connection with container[2] 5. Container[0] Send RST. Connection Closed
- 53. TCP Connection Loss 2 192.68.0.3 Container[0] Container[1] Container[2] Container[3] L4-LB L4-LB 1. Connection Established With Container[3] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 although the serving LB is alive. 4. Container[1] has no idea about connection with container[3] 5. Container[1] Send RST. Connection Closed L4-LB
- 55. High Availability Container Host Container Host Container Host Container Host L4-LB
- 56. Build A Efficient and Reliable Load Balancer
- 57. Efficient and Reliable Load Balancer IPVS MH IPVS MH IPVS MH IPVS MH IPVS MH Add IP of containers dynamically To IPVS Add IP of containers dynamically To IPVS
- 58. No Loss TCP Connection 192.68.0.2 Container[0] Container[1] Container[2] Container[3] L4-LB IPVS 1. Connection Established With Container[2] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 4. IPVS MH can forward the packet to same destination by hashing. 5. Continue the Established connection. IPVS MH
- 59. No Loss TCP Connection 2 192.68.0.3 Container[0] Container[1] Container[2] Container[3] L4-LB 1. Connection Established With Container[3] 2. A Load Balancer Down 3. ECMP Disruption. Forward packet another l4 although the serving LB is alive. 5. Continue the Established connection. IPVS IPVS IPVS MH 4. IPVS MH can forward the packet to same destination by hashing With no connection info.
- 60. At least 1 Load Balancer up, IPVS MH IPVS MH IPVS MH IPVS MH IPVS MH Add IP of containers dynamically To IPVS Add IP of containers dynamically To IPVS
- 61. Can Hash Consistently With No Connection Loss IPVS MH IPVS MH IPVS MH IPVS MH IPVS MH Add IP of containers dynamically To IPVS Add IP of containers dynamically To IPVS
- 62. Efficient and Reliable Load Balancing BGP/ ECMP IPVS Maglev Hashing Scheduler
- 63. Required Linux Kernel >= 4.18 See Also: https://github.com/torvalds/linux/blob/master/net/netfilter/ipvs Choose M in Kernel menuconfig to use IP_VS_MH
- 64. Required Kernel Parameters echo 1 > /proc/sys/net/ipv4/vs/sloppy_tcp See Also: https://github.com/torvalds/linux/blob/master/Documentation/networking/ipvs-sysctl.txt echo 2 > /proc/sys/net/ipv4/vs/conn_reuse_mode
- 65. Q & A