What is Linux kernel live patching?

Linux kernel live patching is a way to apply critical and important security patches to a running Linux kernel, without the need to reboot or interrupt runtime.

There is a difference between a patch and an update. An update is a new, more minor version of a package and it can contain bug fixes, performance improvements, new features, edits at the command line, and other enhancements. A patch is a partial snippet of code (usually the differing output between two versions of a package or file) that fixes a vulnerability in the existing version. These patches fix vulnerabilities without latency so that the existing implementation can run more securely and system administrators can hold off on rebooting until the next regular maintenance window.

In theory, live patching could be used for any patch for the running kernel, including regular bug fixes and enhancements. In practice, the priority function for live patching has been major security vulnerabilities, since these are fixes that shouldn’t be delayed. With Red Hat® Enterprise Linux®, we have focused on security patches and kernel live patching is considered a security feature (with management benefits for administrators).

Without live patching, applying a patch requires the reboot of the patched service or app in order to load the new changes of the patched version. This is true even for patches applied to the Linux kernel itself, creating a conundrum for Linux system administrators: They need to apply imperative security updates, but that means unscheduled reboots to the linux server, including latency and downtime.

This was the predicament encountered by a student system administrator at MIT, Jeff Arnold. He had a system with a pending security update and chose to delay addressing the vulnerability. That system ended up being hacked. In 2008, Arnold decided to address the challenge he’d faced by creating what he called "rootless kernel security updates" — a way of applying and loading security patches to a system as it was running. That initial project (ksplice) was the creation of the concept of Linux kernel live patching.

Dynamic software updates are a challenge in computing because of the complexity of dealing with things like stateless versus stateful services, persistence, data management, transactions, and following commands—all things that are at the very foundation of how a service or application operates.

What Arnold and subsequent projects for Linux kernel live patching did was use tools within the kernel space to load kernel modules.

Linux system operations are divided into two sections: the user space (where all the services and applications operate), and the kernel space (where the core system operations are). The kernel is an intermediary for all of the applications to access hardware resources like the CPU and storage. Along with the kernel itself, administrators can create custom kernel modules to extend or modify functionality, and these kernel modules can be loaded and executed dynamically, even after boot.

What kernel live patching tools do is create a kernel module out of the patched code, then using the ftrace (function trace) tool to route from the obsolete function to the new replacement function, patch module, or patched function.

Figure 1: How kernel live patching works

Ksplice was the first project for live patching the Linux kernel; however, ksplice was sold to Oracle and eventually changed to a closed-source tool. Other development teams began trying to come up with open source projects that could replace ksplice, with two slightly different projects launching in 2014: kpatch from Red Hat and kgraft from SuSE. Ultimately, for the good of the Linux kernel community, Red Hat and SuSE developers worked together to create livepatch, which is a common layer within the Linux kernel that allows people to develop compatible kernel live patching tools.

The key thing to remember is that patches are specifically used for addressing security risks. One of the challenges that sysadmins have is that they not only need to account for security patching in Linux systems, they also need to make sure that they are meeting uptime requirements. This means they may not be able to take systems offline outside preset maintenance windows.  This is a constant tension that is only increasing.

According to the Red Hat Security Risk Report, there was a massive spike of identified security issues (called common vulnerabilities and exposures or CVEs) from 2019 to 2020—up about 155% (1,313 to 2,040). Although that dropped slightly in 2021 (1,596), it was still 22% higher than 2019. Most of this has been an increase in moderate severity vulnerabilities; important CVEs have stayed roughly steady and critical vulnerabilities have been dropping.

Kernel updates and patches are released every 6 weeks, and then minor updates of Red Hat Enterprise Linux come out every six months with all the security patches included. The ability for live patching service to be done to the Linux kernel without a reboot removes the need to decide which policy to follow.

CVEs related to the kernel are one of the top concerns, with four of the top 10 most-viewed CVEs in 2021 relating to the kernel.

Being able to perform Linux kernel security patching live is much more than a convenience. It is an important tool for security teams to proactively address security vulnerabilities, keep kernel functions running, and maintain the safety of their systems.

Kernel live patching is one specific tool to manage security, but it is far from the only one available with Red Hat Enterprise Linux. Red Hat Enterprise Linux includes different tools for managing security, such as SELinux for permissions and access control and system roles to help automate configuration and tasks.

There is no single solution to create a "secure environment." Threats are always changing and evolving, with different attack vectors emerging and different targets pinpointed. Security starts at the operating system level—even the very source code—and works up through the technology stack and throughout the lifecycle, which is why having a variety of tools is essential.

Manage kernel live patching from the Red Hat Enterprise Linux web console to greatly reduce the complexity of performing critical maintenance. This new web console capability provides a simplified interface for both highly-skilled and non-experienced administrators to apply kernel updates without having to use command line tooling.

Red Hat Insights is available with any Red Hat Enterprise Linux subscription, providing a unified management experience for addressing security concerns. This includes:

• Visual dashboards that cover your entire infrastructure, showing vulnerable systems and patch status, allowing you to find and fix critical bugs without a separate subscription to Red Hat Satellite Server.
• Relevant CVEs and other updates.
• Playbooks for automated Linux security patching.
• Both defined security profiles and custom profiles to manage system configuration.
• Using baseline configs and flagging systems which diverge from the baselines.

Linux kernel live patching came to exist when a need was identified and an open project created to work collaboratively with organizations and community members, creating a solution that would benefit the community. This open way of working is at the core of Red Hat’s value of subscription principles: collaboration, transparency, and a focus on improving the technology and user experience.