Generic Resource Manager - László Vadkerti, András Kovács

generic
resource
manager
András KovÁcs (Andras.kovacs@ericsson.com)
LászlÓ Vadkerti (laszlo.vadkerti@Ericsson.com)
A manager we would like :)

Generic resource manager for DPDK | Public | © Ericsson AB 2015 | 2015-09-29 | Page 2
Last year…

by merriam-webster:
› something that a country has and can use to increase its wealth
› a supply of something (such as money) that someone has and can use
when it is needed
› a place or thing that provides something useful
by wikipedia (computing):
› A resource, or system resource, is any physical or virtual
component of limited availability within a computer system. [...]
Every internal system component is a resource. […].
What is a resource?

› CPU
› Memory (hugepage, cache, ivshm, memzone)
› Virtual address
› Packet pool
› Network/Virtual Device (port/queue)
› Lcore
› Instance
› HW accelerator
› And there are lots of other resources coming… (threads,
protocol stacks, etc)
typical dpdk
resources

› Better control over resources
› Hide “real” environment from applications
› Support migration
› Keep track of resources/usage (High Availability)
› Prioritize resources
› Access control
› Avoid code duplication
Why?

› Resource pools (resources with similar attributes) e.g.
Servers -> Compute nodes -> Virtual Machines -> CPU/memory/interface/lcore/pktpool
Resource pools
MEM
CPU
I/F
CPU CPU CPU CPU CPU
CPU CPU CPU CPU CPU
CPU CPU CPU CPU CPU
CPU CPU CPU CPU CPU
CPU CPU CPU CPU CPU
CPU
CPU
CPU
CPU
CPU
CPU CPU CPU CPU CPU CPU

MEM
CPU
MEM
CPU
MEM
CPU
I/F
Resource pools
CPU CPU
CPU CPU

MEM
CPU
MEM
CPU
MEM
CPU
I/F
Resource pools
CPU
CPU
attribute 1
attribute 2

› cpualias foreground {
› cpumask = "2"
› }
› pktpool pktpool_fg {
› cpualias = foreground
› num_pkts = 8K
› num_cached_pkts = 64
› }
Config EXAMPLE
PKT
POOL
CPU

› cpumask = "2,12"
› }
› num_pkts = 8K
› }
Config EXAMPLE
PKT
POOL
CPU CPU
S1S0

› cpumask = "2,12"
› }
› num_pkts = 8K
type = numa
› }
Config EXAMPLE
S1S0
PKT
POOL
PKT
POOL
CPU CPU

› cpumask = "2,4,12,14"
› }
› num_pkts = 8K
type = numa
› }
Config EXAMPLE
S1S0
PKT
POOL
PKT
POOL
CPU CPU
CPUCPU

› cpumask = "2,4,12,14"
› }
› num_pkts = 8K
type = exclusive
› }
Config EXAMPLE
PKT
POOL
S1S0
PKT
POOL
PKT
POOL
PKT
POOL
CPU CPU
CPUCPU

› resource allocation (static or on-demand)
› chain of pools in case of dynamic allocation (example)
RESOURCE allocation

› e.g. a native workstation with 4 sockets, memory,
interfaces.
› resmgr starts, autodetects (CPUs, HT siblings, cache sizes,
etc) every resources we can use.
› resmgr: daemon/binary/library
› reading configuration and creating a free resource
database (our choice is xattr)
› supporting popup cpu, memory, interface, etc
› linking/referencing resources (cpu table, HT table, NUMA,
L3 cache (CAT/cache QoS), etc)
RESMGR at work

› Is the relation between virtual ports, virtual queues and
named packet pools. Somewhat similar to pipeline model
configuration.
packet domain

› Allocating resources
› Per-application configuration
› Application instances see only assigned resources (like a
VM)
DPDK DOMAIN
(APPlication DOMAIN)

existing Memory
management example
process1
process2
1G
1G
1G
1G
1G
1G
1G
1G
DPDK memory
DPDK
socket

› Simple memory management API
› Transparent NUMA awareness for applications
– Even in a non-NUMA aware guest VM environment
› Flexible, configuration, re-configuration
› Provide a more granular way of placing objects in memory
– e.g. to control the number of TLB cache entry usage
› Memory partitioning
– Physically contiguous memory, hugepages etc.
– Partitioning across application instances
– Shared memory support
memory management
goals

› Named and indexed memory partition (shared memory)
with defined properties such as
– Location, e.g. NUMA node(s)
– Size of partition, e.g. 4GB/1G, 512MB/2M hugepages
– Physical contiguity
– Type (future improvement), e.g.
› Hugepage memory (only possible type today)
› Mmap-ped file
› Inter-vm shared memory
› Distributed memory
› etc.
what is a memdomain?

› Is a pool of shared memory resources (memdomains)
› Is a named group of memory partitions with different
location attributes
› Three main types
– DEFAULT
› No location preference, only a single memdomain in the pool
– NUMA
› One memdomain per involved NUMA node
– EXCLUSIVE
› One memdomain per application instance
what is a memdomain
pool?

MEMDOMAIN
Configuration
› Can be compared to partitioning a hard drive where
– Available hugepage memory is the hard drive,
e.g. one drive per NUMA node
– Memdomains can be compared to disk partitions
– Memzones can be compared to files
› Should be a system engineering task to find the best model
working for the specific application model

MEMDOMAIN Simple
Application API
› Initialize library (oai_memcfg_libinit)
– Register a unique name to the process (High Availability, debug)
– Initializes DPDK memory, grabs hugepages etc.
› Attach to named memory partition (oai_memcfg_attach)
– Using pre-configured memdomain pool name
– Automatic memdomain selection from pool (per instance, NUMA)
– Maps memdomain (memzones) into virtual address space

MEMDOMAIN
ConfiguratioN EXAMPLE
CPUALIAS
cpualias all {
cpumask = "0-31" # all available CPU
}
CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU

MEMDOMAIN
ConfiguratioN EXAMPLE
CPUALIAS
cpualias all {
cpumask = "0-31" # all available CPU
}
cpualias foreground {
cpumask = "2-31:2" # even cpus excluding cpu0
}

MEMDOMAIN
CONFIGURATION EXAMPLE
DEFAULT TYPE
# -------------------------------
# Shared memory accessed by
# all instances
# -------------------------------
memdomain App_Shared_Memory {
type = default
cpualias = "all"
alloc_memzone = true
size {
huge_2M = 512M
huge_1G = 0
}
}
Instance1
S1S0
MEM
DOMAIN
Instance2

memdomain API ATTACH
EXAMPLE
DEFAULT TYPE
Instance1:
- oai_memcfg_libinit(“AppProcess1”);
- addr1 = oai_memcfg_attach(“App_Shared_Memory”,
OAI_MEMCFG_MD_INDEX_AUTO,
&len,
OAI_MEMCFG_MZ_FLAGS_NONE);
Instance2:
- addr2 = oai_memcfg_attach(“App_Shared_Memory”,
&len,
› Explanation:
– addr1 in process1 will point to the same memory as addr2 in process2

MEMDOMAIN
NUMA TYPE
# ----------------------------------------
# Per NUMA node shared memory.
# App instances tied to the same NUMA
# are sharing the same memory partition.
# ----------------------------------------
memdomain App_NUMA_Shared {
type = numa
cpualias = "all"
size {
is_per_numa = true
huge_2M = 0
huge_1G = 1G
}
}
Instance1
S1S0
MEM
DOMAIN
MEM
DOMAIN
Instance2
Instance3

EXAMPLE
NUMA TYPE
Instance1:
- pin to vcpu on numa node0 (lcore_assign, pktpthread_setaffinity_np)
- addr1 = oai_memcfg_attach(“App_NUMA_Shared”,
&len,
Instance2:
&len,
Instance3:
&len,
› Explanation:
– addr1 in instance1 will point to the partition allocated in NUMA node0 while
addr2 in instance2 and addr3 in instance3 will point to the same partition allocated in NUMA
node1

memdomain
EXCLUSIVE TYPE
# -----------------------------------
# Per App instance private memory.
# Every instance has its own memory
# partition.
# -----------------------------------
memdomain App_Thread_Local {
type = excl
cpualias = "foreground"
size {
is_per_cpu = true
huge_2M = 0
huge_1G = 1G
}
}
Instance1
S1S0
MEM
DOMAIN
MEM
DOMAIN
MEM
DOMAIN
Instance2
instance3

EXAMPLE
EXCLUSIVE TYPE
Instance1:
- pin to vcpu2 on numa node0 (lcore_assign, pktpthread_setaffinity_np)
- addr1 = oai_memcfg_attach(“App_Thread_Local”,
&len,
Instance2:
&len,
Instance3:
&len,
› Explanation:
– Addr1, addr2 and addr3 will point to different memory addresses
– addr1 allocated on NUMA node0 while addr2 and addr3 allocated on NUMA node1

Advanced MEMCFG API
› Create memdomain pools in a transaction
– Creating a pool of memory partitions
(partitions are auto aligned based on user's location or requested
layout)
– Creating multiple memdomain pools in a transaction allows better
placement (e.g. prioritizing physically contiguous memory)
› Attach to memdomain (memory partition)
– Attach by memdomain name and index, returning md handle
(index could be auto if instance location is known e.g. by lcoreid)
– Could also MMAP whole partition into virtual address space
› Get number of memory partitions in a memdomain pool
– Allows attaching to all partitions by index within a pool
e.g. for per NUMA table replication

Advanced MEMCFG API
› Reserve named memzones by memdomain handle
– Memzone name must only be unique within the memdomain,
same name can be used in another memdomain
e.g. it allows attaching to replicas using the same name
– supports fragments if physical contiguity is not requested
(virtually cont)
› Lookup memzone by memdomain handle
– Looks up memzone within a memdomain returning mz handle
› Attach to memzone by memzone handle
– Attaches to a memzone, e.g. mapping into requested/reserved
virtual address space as readonly/readwrite

Memdomain example
process1
process2
1G
1G
1G
1G
1G
1G
1G
1G
DPDK memory
memory
partition
1
memory
partition
2

› Support non-DPDK applications
› Memory types for rte_malloc
› DPDK certification (for VM or hardware)
› GUI or config file for dpdk startup
parameters (RESMGR config alternative)
› Visual resource allocator (GParted style)
› DPDK OS / DPDK distro
› standby instances
› NEW emulated QEMU device
Future ideas

Generic Resource Manager - László Vadkerti, András Kovács

Generic Resource Manager - László Vadkerti, András Kovács

More Related Content

What's hot (20)

Similar to Generic Resource Manager - László Vadkerti, András Kovács (20)

Recently uploaded (20)

Generic Resource Manager - László Vadkerti, András Kovács