Ksplice - Keep your Database systems up to date with no downtime
Download as PPTX, PDF•2 likes•516 views
This document provides an overview of Oracle's Ksplice technology, which allows for installing Linux kernel security updates and patches without requiring a reboot. Ksplice was originally created by researchers at MIT and later acquired by Oracle. It allows critical systems like databases to remain online while installing kernel updates. The document discusses how Ksplice works, demonstrates its use, and compares it to alternative live patching technologies.
![Ksplice - Ksplice SNMP Plugin
▪ Use Oracle Enterprise Manager 12c/13c(?) to monitor status of Ksplice on your systems.
▪ Query the status of Ksplice using SNMP command snmpwalk:
[root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatus
KSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: outofdate
▪ [root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable
▪ KSPLICE-UPTRACK-MIB::kspliceavailName.12 = STRING: [ur5of4nd]
KSPLICE-UPTRACK-MIB::kspliceavailName.13 = STRING: [ue4dtk2k]
KSPLICE-UPTRACK-MIB::kspliceavailName.14 = STRING: [wy52x339]
Details: http://docs.oracle.com/cd/E37670_01/E39380/html/ch01s11s01.html](https://image.slidesharecdn.com/oraclelinuxksplicev8-170417101640/85/Ksplice-Keep-your-Database-systems-up-to-date-with-no-downtime-24-320.jpg)
Ksplice - Keep your Database systems up to date with no downtime
- 1. Oracle Linux Ksplice KEEP YOUR DATABASE SYSTEMS UP TO DATE WITH NO DOWNTIME
- 2. Luis & Yves ▪ Luís Marques ▪ Oracle ACE ▪ Lisbon ▪ http://lcmarques.com / @drune / [email protected] ▪ Yves Colin ▪ Oracle ACE Associate ▪ Paris ▪ http://ycolin.wordpress.com / @ycolin / [email protected]
- 3. ksplice - A piece of history (before Oracle) ▪ Created by Jeffrey Arnold, Tim Abbott, Waseem Daher, and Anders Kaseorg at MIT ▪ Released as open source (GPL v2 license) - Latest open source version: 0.9.9 ▪ Ksplice (as company) started in 2008 and got 700 customers ▪ Support for RHEL, CentOS, Fedora, Debian, Ubuntu, CloudLinux and Scientific Linux ▪ Announced in LKML as “A system for rebootless kernel security updates” – (23 April 2008) (http://lwn.net/Articles/279378/)
- 4. ksplice - A piece of history (after Oracle) ▪ Oracle acquired Ksplice in 2011. ▪ ▪ Offers feature only to Oracle Linux Premier Support customers ▪ Support for Oracle Linux (UEK, RH compatible kernels) via ULN – (also supports RHEL as legacy customers(?)) ▪ Support for Ubuntu and Fedora “Desktop” versions as “free community protection” ▪ Strategic move for Oracle: mission critical systems run Linux (eg: Databases, Exadata, ODA)
- 5. ksplice – Why use it? Allows to install security and reliability patches without reboot and without downtime Downtime (and other) dilemmas: ▪ Huge security Linux kernel hole discovered and few Oracle single instance without DR? ▪ Factory Control Systems, Energy Management System Critical Applications ▪ Rollback an update without another emergency reboot? ▪ SLA or Security?
- 6. ksplice – Installation Kernel Patching – Manual (Recommended) # wget -N https://www.ksplice.com/uptrack/install-uptrack # sh install-uptrack ACCESS_KEY # uptrack-upgrade –y Kernel Patching - Automatic Mode # sh install-uptrack ACCESS_KEY –autoinstall or edit /etc/uptrack/uptrack.conf and change autoinstall = yes
- 7. ksplice – Your first live demo with ksplice Install and view all kernel updates available # uptrack-show –-available # uptrack-show # uptrack-install <ksplice_id> -y # uptrack-upgrade -y # uptrack-uname –a
- 8. ksplice – Your first live demo with ksplice You can patch your system without any reboot but remove them rebootless as well # uptrack-remove <ksplice_id> # uptrack-remove –all # uptrack-uname –a
- 10. ksplice and Oracle Database ▪ Ksplice doesn’t patch any Oracle shared libraries or binaries ▪ You want to patch your Database that is running of top of ext4 (?) ▪ Oracle instrumentation doesn’t catch it (?) ▪ Nothing in “Wait event” interface ▪ A few more cpu cycle are recorded during and after the patch, but is negligible. ext4_file_open needs to be patched but it is in use by Oracle and your OS The thread that can run the function is suspended for <1ms – No CPU can execute it Execution is resumed <1ms
- 11. Ksplice ,Exadata and AWS ▪ Email from Ksplice support about Exadata: ▪ Check against OVM on EXADATA version 12.1.2.2.1.160119 ▪ Tested under Redhat 7.3 EC2 image ▪ Possible on AWS AMI customized XEN kernel (“Supported for legacy customers”)
- 12. Ksplice – Under the wood
- 13. Kplice – How patch is created? (1/2) Source Code of Running Kernel Slice Source Code of Patch Ksplice Build Environment: • Pre Object Code • Post Object Code Ksplice-id.ko Ksplice-id-vmlinux- new.ko Ksplice-id-vmlinux- old.ko Identify modified functions Object Level Code difference Patch Created Slice
- 14. Kplice – How patch is created? (2/2) ▪ Ksplice compares code at object code level ▪ In this way Ksplice patch creation could be completely automated(in most of the cases) ▪ Ksplice compiles code(patched/non patched) with special gcc options -ffunction-sections - fdata-sections
- 15. Patch creation magic (1/2)
- 16. Kplice – How patch is applied? Connect to the repository Download the Splice (tar.gz) Files in /var/cache/uptrack/ Linux/$arch/$kernel Ksplice-id.ko core module interface (userland – kernel land) Ksplice-id-vmlinux-new.ko Ksplice-id-vmlinux-old.ko Interaction via /sys/modules/ksplice-id/ksplice Inspect thread – patched functions in execution? (safety check) Execute stop_machine() to suspend thread Patched code loaded in memory Patch applied Thread execution resumed
- 17. Kplice – How patch is applied (in deep) ? Compare Running Code in memory to pre-code(safety check) Resolve Undefined Symbol in Post Code/Patched code (where are they located in running kernel?) pre matching phase Find the patched function in running kernel Modified Old Function to insert a JMP instruction Patch applied stop_machine() : disable CPU interruption and set the patching process in High priority scheduling mode Patched Function Check kernel stack thread
- 18. Patch creation magic (2/2)
- 19. Ksplice - Alternatives ▪ Kpatch (RedHat) / kGraft (Suse): ▪ Uses ftrace to do the patching ▪ Kpatch uses stop_machine() to load new code (like Ksplice) ▪ Kgraft works on thread level, as soon as the threads reaches a point it switches to a new function ▪ KernelCare (CloudLinux) ▪ Proprietary software
- 20. Kpatch ▪ Kpatch > LivePatch (RedHat) April 2015 Kernel 4.0 release – Foundations for LIVE PATCHING May 2014 Submitted Linux Kernel Mainline February 2014 Public release GPLv2 June 2014 RHEL 7 – Preview Canonical has a service called LivePatch that takes advantage of Kernel 4.x LIVE PATCHING
- 21. Ksplice – Web Management tool ▪ Webinterface available at: http://uptrack.ksplice.com or https://status- ksplice.oracle.com ▪
- 22. Ksplice – Web Management tool ▪ Webinterface available at: http://uptrack.ksplice.com or https://status- ksplice.oracle.com
- 23. Ksplice - Uptrack offline client ▪ Remove the need for servers to connect to “public” Ksplice server ▪ Create a local YUM mirror of Ksplice mirror and allow your servers to connect to it ▪ Or, Oracle bundles all available Ksplice patches for each supported kernel version to one RPM package and install in every server using a pen drive ▪ A new patch is available, a new RPM is available ▪ You cannot use the Web management tool. ▪ Details for offline client: https://docs.oracle.com/cd/E37670_01/E37355/html/ol_olcsetup_ksplice.html
- 24. Ksplice - Ksplice SNMP Plugin ▪ Use Oracle Enterprise Manager 12c/13c(?) to monitor status of Ksplice on your systems. ▪ Query the status of Ksplice using SNMP command snmpwalk: [root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatus KSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: outofdate ▪ [root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable ▪ KSPLICE-UPTRACK-MIB::kspliceavailName.12 = STRING: [ur5of4nd] KSPLICE-UPTRACK-MIB::kspliceavailName.13 = STRING: [ue4dtk2k] KSPLICE-UPTRACK-MIB::kspliceavailName.14 = STRING: [wy52x339] Details: http://docs.oracle.com/cd/E37670_01/E39380/html/ch01s11s01.html
- 25. Watch and learn..to tune wait events
- 26. Watch and learn..to tune wait events
Editor's Notes
- #5: 1502322.1 (NOTE EXALOGIC) Investigate for Exadata, Exalogic and ODA
- #7: You kernel will be updated automatically now as new update comes. Could be done afterwards in ksplice configuration file as wellç. Only patches will be automatically installed the ksplice client is not automatically upgraded. Questions: Autoinstall when it happens??
- #8: uptrack-install <ksplice_id> -y -> ksplice_id depends on the previous installed ksplice_ids showed in uptrack-show. (Dependencies) Methodology: Test the bug, install the patch, test the bug, install all updates
- #9: Methodology: Test the bug, remove the patch, test the bug, remove all updates
- #10: Disable automatic patch application and rollback patches: uptrack-remove –all - sh exploit.sh: build and compiles exploit.c - Exploit on perf (linux profiler). ./exploit 0 Now we become root sh installexploitpatch.sh - exit - Try exploit again (core dump expected) - It will take care of dependencies and changed the effective kernel: uptrack-uname -r vs uname -r
- #11: GLIBC library patching also demonstrated no issues
- #12: GLIBC library patching also demonstrated no issues
- #14: Pre Object Code / Post Object Code Object code is the the output of the compiler (machine language) not yet linked
- #15: Pre Object Code / Post Object Code
- #16: createpatch.sh ksplice-apply splice……..tar.gz cat /proc/uptime
- #17: Ksplice-<kspliceid>.ko : module in charge of loading and unloading patched code Ksplice-<kspliceid>_vmlinux-new.ko : module containing the new patched functions Ksplice-<kspliceid>_vmlinux-old.ko : module containing the previous functions
- #18: The pre matching phase does not need exported symbol table at all. It finds the same function or a function in the same scope and resolve symbol address in the running kernel. Replacement needs to be atomic Stop_machine The caller can specify a non-sleeping 12 * function to be executed on a single or multiple cpus preempting all 13 * other processes and monopolizing those cpus until it finishes.
- #19: lsmod cat /proc/kallsyms | grep uptime_proc_show perf probe --add uptime_proc_show perf record -e probe:uptime_proc_show -e probe:uptime_proc_show_1 -aR sleep 15 & cat /proc/uptime perf script
- #20: http://www.zdnet.com/article/no-reboot-patching-comes-to-linux-4-0/
- #26: apply the patch ksplice-apply ksplice-00000000.tar.gz undo the patch ksplice-undo 00000000