Effective service and resource management with systemd
17 likes•3,955 views
This document provides an overview of using systemd to manage services effectively. It discusses defining service behavior and types, handling timeouts and failures, tightening security, and automating monitoring and management. The key steps outlined are to define expected behavior, plan for the unexpected, tighten security early, and automate monitoring. Various systemd directives and options are explained for tasks like controlling resources, dependencies, reloading, and failure handling.
![Type=simple (the default)
systemctl start foo.service systemctl stop foo.service
ExecStart=/usr/bin/foo
/etc/systemd/system/foo.service
Considered started for dependencies
Considered stopped for dependencies
[Service]
ExecStart=/usr/bin/foo
# systemctl daemon-reload](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-5-320.jpg)
![Type=oneshot
systemctl start foo.service systemctl stop foo.service
*Unless RemainAfterExit=true
*
ExecStart=/usr/bin/foo
[Service]
Type=oneshot
ExecStart=/usr/bin/foo
RuntimeMaxSec=30
/etc/systemd/system/foo.service
RuntimeMaxSec=30](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-6-320.jpg)
![Type=forking
systemctl start foo.service
systemctl stop foo.service
ExecStart...
PIDFile=/var/run/foo.pid
[Service]
Type=forking
ExecStart=/usr/bin/foo
PIDFile=/var/run/foo.pid
TimeoutStartSec=30
/etc/systemd/system/foo.service
TimeoutStartSec=30](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-7-320.jpg)
![Type=notify
systemctl start foo.service systemctl stop foo.service
ExecStart...
[Service]
Type=notify
ExecStart=/usr/bin/foo
TimeoutStartSec=30
NotifyAccess=all ⬅maybe
/etc/systemd/system/foo.service
Called from daemon:
systemd-notify --ready
Best of
All
Types](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-8-320.jpg)
![KillMode=control-group (the default)
systemctl stop foo.service
[Service]
ExecStart=/usr/bin/foo
KillMode=control-group
TimeoutStopSec=30
/etc/systemd/system/foo.service
PID=100
101
102
103
…or “Oprah’s Favorite Signals”
SIGTERM
PID=100
101
102
103
SIGKILL
TimeoutStopSec=30](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-10-320.jpg)
![KillMode=none
systemctl stop foo.service
[Service]
ExecStart=/usr/bin/foo
KillMode=none
ExecStop=/usr/bin/fooctl
stop
/etc/systemd/system/foo.service
PID=100
101
102
103
PID=100
101
102
103
No CleanupExecStop=/usr/bin/fooctl stop](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-11-320.jpg)
![KillMode=process
systemctl stop foo.service
[Service]
ExecStart=/usr/bin/foo
KillMode=process
/etc/systemd/system/foo.service
PID=100
101
102
103
SIGTERM PID=100
101
102
103
No Cleanup](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-12-320.jpg)
![KillMode=mixed
systemctl stop foo.service
[Service]
ExecStart=/usr/bin/foo
KillMode=mixed
TimeoutStopSec=30
/etc/systemd/system/foo.service
PID=100
101
102
103
SIGTERM PID=100
101
102
103
SIGKILL
TimeoutStopSec=30
Best
for
Most](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-13-320.jpg)
![ExecReload=
systemctl reload foo.service
[Service]
ExecStart=/usr/bin/foo
ExecReload=/bin/kill -HUP $MAINPID
/etc/systemd/system/foo.service
Use Me
ExecReload=/bin/kill -HUP $MAINPID](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-14-320.jpg)
![WantedBy=
Implicit in late bootup:
systemctl start multi-user.target
[Service]
ExecStart=/usr/bin/foo
[Install]
WantedBy=multi-user.target
/etc/systemd/system/foo.service
Use Me
# systemctl enable foo.service
Added to transaction by wants:
systemctl start foo.service
multi-user.target completes startup
Operations in systemd happen in transactions, which are ordered sets of jobs.
…the successor to runlevels](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-16-320.jpg)
![Other Dependencies
Inclusion
These dependencies will add more units to a
transaction. There is no effect on ordering.
● Requires=bar.service
○ If foo.service is starting, starting bar.service
will also happen. A failure to start bar.service
will cause the entire transaction to fail.
○ Inverse of RequiredBy=
● Wants=bar.service
○ A weak form of Requires=. If bar.service fails
to start, the transaction will still succeed.
○ Inverse of WantedBy=
● Also=bar.service
○ When foo.service is enabled to start by
default, bar.service will also be enabled.
Ordering
These dependencies will order units in the
transaction. They will not add specified units if
they are not already in the transaction.
● Before=bar.service
○ If bar.service is in the same transaction, bar.
service will not begin starting until foo.
service is finished starting.
● After=bar.service
○ If bar.service is in the same transaction, foo.
service will not begin starting until bar.
service is finished starting.
[Unit]
Requires=bar.service
After=bar.service
...
/etc/systemd/system/foo.service](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-17-320.jpg)
![Pantheon is a Chef Shop
template '/etc/systemd/system/foo.service' do
mode '0644'
source 'foo.service.erb'
end
service 'foo.service' do
provider Chef::Provider::Service::Systemd
supports :status => true, :restart => true, :reload => true
action [ :enable, :start ]
end](https://image.slidesharecdn.com/effectiveserviceandresourcemanagementwithsystemd-160330192739/85/Effective-service-and-resource-management-with-systemd-29-320.jpg)
Effective service and resource management with systemd
- 1. Effective Service + Resource Management with systemd Adventures running millions of systemd services for
- 2. About Me and Pantheon ● Production users of systemd since 2011 ● Millions of units in deployment across hundreds of servers ● Committer since 2012 ● Focus has been on journal logging, control group scalability, and general systemd scalability
- 3. The Basic Steps 1 Define expected behavior and control 2 Plan for the unexpected 3 Tighten security 4 Manage, monitor, and automate
- 4. Service Types 1 Define expected behavior and control
- 5. Type=simple (the default) systemctl start foo.service systemctl stop foo.service ExecStart=/usr/bin/foo /etc/systemd/system/foo.service Considered started for dependencies Considered stopped for dependencies [Service] ExecStart=/usr/bin/foo # systemctl daemon-reload
- 6. Type=oneshot systemctl start foo.service systemctl stop foo.service *Unless RemainAfterExit=true * ExecStart=/usr/bin/foo [Service] Type=oneshot ExecStart=/usr/bin/foo RuntimeMaxSec=30 /etc/systemd/system/foo.service RuntimeMaxSec=30
- 7. Type=forking systemctl start foo.service systemctl stop foo.service ExecStart... PIDFile=/var/run/foo.pid [Service] Type=forking ExecStart=/usr/bin/foo PIDFile=/var/run/foo.pid TimeoutStartSec=30 /etc/systemd/system/foo.service TimeoutStartSec=30
- 8. Type=notify systemctl start foo.service systemctl stop foo.service ExecStart... [Service] Type=notify ExecStart=/usr/bin/foo TimeoutStartSec=30 NotifyAccess=all ⬅maybe /etc/systemd/system/foo.service Called from daemon: systemd-notify --ready Best of All Types
- 9. Service Shutdown and Reloading 1 Define expected behavior and control
- 10. KillMode=control-group (the default) systemctl stop foo.service [Service] ExecStart=/usr/bin/foo KillMode=control-group TimeoutStopSec=30 /etc/systemd/system/foo.service PID=100 101 102 103 …or “Oprah’s Favorite Signals” SIGTERM PID=100 101 102 103 SIGKILL TimeoutStopSec=30
- 11. KillMode=none systemctl stop foo.service [Service] ExecStart=/usr/bin/foo KillMode=none ExecStop=/usr/bin/fooctl stop /etc/systemd/system/foo.service PID=100 101 102 103 PID=100 101 102 103 No CleanupExecStop=/usr/bin/fooctl stop
- 12. KillMode=process systemctl stop foo.service [Service] ExecStart=/usr/bin/foo KillMode=process /etc/systemd/system/foo.service PID=100 101 102 103 SIGTERM PID=100 101 102 103 No Cleanup
- 13. KillMode=mixed systemctl stop foo.service [Service] ExecStart=/usr/bin/foo KillMode=mixed TimeoutStopSec=30 /etc/systemd/system/foo.service PID=100 101 102 103 SIGTERM PID=100 101 102 103 SIGKILL TimeoutStopSec=30 Best for Most
- 14. ExecReload= systemctl reload foo.service [Service] ExecStart=/usr/bin/foo ExecReload=/bin/kill -HUP $MAINPID /etc/systemd/system/foo.service Use Me ExecReload=/bin/kill -HUP $MAINPID
- 15. Dependencies and Transactions 1 Define expected behavior and control
- 16. WantedBy= Implicit in late bootup: systemctl start multi-user.target [Service] ExecStart=/usr/bin/foo [Install] WantedBy=multi-user.target /etc/systemd/system/foo.service Use Me # systemctl enable foo.service Added to transaction by wants: systemctl start foo.service multi-user.target completes startup Operations in systemd happen in transactions, which are ordered sets of jobs. …the successor to runlevels
- 17. Other Dependencies Inclusion These dependencies will add more units to a transaction. There is no effect on ordering. ● Requires=bar.service ○ If foo.service is starting, starting bar.service will also happen. A failure to start bar.service will cause the entire transaction to fail. ○ Inverse of RequiredBy= ● Wants=bar.service ○ A weak form of Requires=. If bar.service fails to start, the transaction will still succeed. ○ Inverse of WantedBy= ● Also=bar.service ○ When foo.service is enabled to start by default, bar.service will also be enabled. Ordering These dependencies will order units in the transaction. They will not add specified units if they are not already in the transaction. ● Before=bar.service ○ If bar.service is in the same transaction, bar. service will not begin starting until foo. service is finished starting. ● After=bar.service ○ If bar.service is in the same transaction, foo. service will not begin starting until bar. service is finished starting. [Unit] Requires=bar.service After=bar.service ... /etc/systemd/system/foo.service
- 18. Controlling Resources 1 Define expected behavior and control
- 19. Control Groups Options for Resources Absolute Limits ● MemoryLimit= ○ Caution: Certain limits cause further allocation for a group to use swap, impacting system performance. ● TasksMax= ○ Maximum combined processes and threads, including kernel threads. ● BlockIOReadBandwidth= ○ Limits reading block I/O to the specified bytes per second. ● BlockIOWriteBandwidth= ○ Limits writing block I/O to the specified bytes Relative Controls and More ● CPUShares= ○ When under contention, CPU is allocated by the kernel proportionally using the number for this service versus the combined shares of all others. ● BlockIOWeight= ○ When under contention, block I/O is allocated by the kernel proportionally using the number for this service versus the combined weights of all others. ● nftables for network traffic ○ Not configured in systemd, but nftables can leverage systemd’s control groups for traffic shaping and other rules.
- 20. Using Traditional ulimit/rlimit Options ● CPU ○ LimitCPU= ○ LimitNPROC= ○ LimitRTPRIO= ○ LimitRTTIME= ○ LimitNICE= ● Disk ○ LimitCORE= ● Memory ○ LimitDATA= ○ LimitFSIZE= ○ LimitSTACK= ○ LimitMSGQUEUE= ○ LimitAS= ○ LimitRSS= ○ LimitMEMLOCK= ● Other ○ LimitSIGPENDING= ○ LimitNOFILE= ○ LimitLOCKS=
- 21. Handling Timeouts and Abnormal Exits 2 Plan for the unexpected
- 22. Directives for Detecting and Responding to Failure Detecting Failure ● SuccessExitStatus= ○ Whitelist of exit codes and signals to indicate a normal exit. Defaults to zero and the usual process signals for healthy processes. ● RestartPreventExitStatus= ○ Blacklist of exit codes and signals to not trigger restarts. Useful to restart on most failures but not unrecoverable ones like a bad configuration. ● RestartForceExitStatus= ○ The opposite of the previous option. ● StartLimitInterval= and StartLimitBurst= ○ Thresholds at which attempted failure recovery becomes a stickier failure. Responding to Failure ● Restart= ○ Allows many options, but on-failure is probably best for most cases. ● FailureAction= ○ Supports options like rebooting or shutting down the system on service failure. ● StartLimitAction= ○ Same as FailureAction= but triggered when StartLimit… thresholds get hit. ● systemctl reset-failed ○ Resets status units marked as failed.
- 23. Built-In Service Monitoring with Watchdog Services ● WatchdogSec= ○ Configures the maximum interval for the healthy service to ping systemd. ● $WATCHDOG_USEC and $WATCHDOG_PID ○ Environmental variables set for a service that is expected to provide systemd with watchdog pings. ● systemd-notify WATCHDOG=1 ○ CLI; the most basic way for a service to send systemd a watchdog ping. ● sd_notify(0, “WATCHDOG=1”); ○ A better way that requires linking to a systemd library. Overall System ● RuntimeWatchdogSec= ○ Configures the maximum interval for systemd to ping the hardware watchdog service (if it exists). If the hardware fails to receive an expected ping, it will reboot the system. ● ShutdownWatchdogSec= ○ Bounds the time the watchdog hardware is willing to wait for a clean shutdown for the triggered reboot.
- 24. Dropping Privileges and Access Early 3 Tighten security
- 25. Dropping Privileges and Access Early ● Hardening options that mostly just work ○ User=<service-user> ○ PrivateTmp=true ○ PrivateDevices=true ○ ProtectSystem=full ○ ProtectHome=read-only ○ NoNewPrivileges=true ○ MountFlags=private ○ SystemCallArchitectures=native ○ SecureBits=noroot noroot-locked ● Restrict visible directories ○ ReadWriteDirectories= ○ ReadOnlyDirectories= ○ InaccessibleDirectories= ○ RootDirectory= runs the service in chroot ● Whitelist capabilities and system calls ○ AmbientCapabilities= ○ CapabilityBoundingSet= ○ SystemCallFilter= ○ SystemCallErrorNumber=EPERM tests filters in a non-enforcing mode ● Control sockets ○ RestrictAddressFamilies= ○ PrivateNetwork=true, which is best combined with socket activation ● Bridge to mandatory access control (MAC) ○ SELinuxContext= ○ AppArmorProfile= ○ SmackProcessLabel=
- 26. Monitoring 4 Manage, monitor, and automate
- 27. Monitor at the Box Level Plug a systemctl call into your monitoring tool: # systemctl --state=failed --all 0 loaded units listed. To show all installed unit files use 'systemctl list-unit-files'.
- 28. Automation 4 Manage, monitor, and automate
- 29. Pantheon is a Chef Shop template '/etc/systemd/system/foo.service' do mode '0644' source 'foo.service.erb' end service 'foo.service' do provider Chef::Provider::Service::Systemd supports :status => true, :restart => true, :reload => true action [ :enable, :start ] end
- 30. Questions? Follow Ups? Reach out to me @DavidStrauss. Want to get more hands-on? We’re hiring! pantheon.io/careers