Kernel Module Packages

Kernel Module Packages (KMPs) are a way for anyone to package kernel modules in such a way that they are cleanly handled in case of kernel updates.

kABI stability

A loaded kernel module is part of the Linux kernel and has access to all data structures of the kernel. As such it needs to be compiled with exactly the right kernel configuration the right compiler flags and -- most importantly -- the right kernel sources. In real life, only the exported symbols are normally accessed -- but there are >6000 exported kernel symbols in recent 2.6 kernels.

Due to the fact that kernel internal interfaces (kABI) are quite volatile and change frequently, kernel updates can quite easily change some of the interfaces. This will break the kernel modules that have been compiled for the older version. Fortunately this is detected by symbol versioning, thus preventing the module from loading into the kernel and thus preventing data corruption.

History

This close connection between the kernel and its modules made distributors (incl. SUSE) compile them all together and provide them all in one binary kernel RPM. Every kernel module (mostly drivers) that was needed by users needed to be included in the kernel, so it would be recompiled on every kernel security update.

This approach of handling kernel modules has come to its limits:

• Despite the willingness to accept modules in the tree, third party kernel modules existed, and no provisions were made to handle this correctly in case of kernel updates. In the worst case, this could lead to a system that was not bootable any more after a kernel update -- clearly not the best user experience.
• Modules can not always be provided. There are legal restrictions, issues with code quality or portability or simply missing possibilities to test (be it due to missing hardware or due to missing manpower or knowledge).
• Sometimes, kernel modules just are not ready in time a distribution is cut -- a way to provide modules/drivers later (or updated versions of existing modules) is highly useful then.

Enter Kernel Module Packages

The idea is to cleanly integrate kernel modules that are not part of the distributor-provided kernel RPM. Anyone with the right knowledge can create a kernel module that matches a certain kernel version and works with it. The real difficult problem to solve is to maintain the module-provided functionality on a kernel security update. This is achieved by reusing the module if possible and by requiring a recompiled modules if needed.

To address this, the following steps are taken on a kernel update:

• detect third party kernel modules
• for every module, check whether it still works with the new kernel (by looking at all the needed symbols and their versions)
  • if it still works, make sure it can and will be reused by the new kernel
  • if it is not compatible with the new kernel anymore
    • try to download a new kernel module RPM
    • if this is not available display an error message (and by default abort the update)

This is all that the user gets to see -- once kernel module packages are installed, updates will automatically be downloaded and installed if needed.

In case of kABI changes, a mere recompilation of the module against the right kernel sources does do the job -- it is SUSE's policy not to require code changes (read do kAPI changes) in security updates. This even applies to SLES service packs, though exceptions are not completely impossible there. But as even this simple recompilation can create a bit of work, SUSE does watch changes closely nowadays and tries hard to avoid kABI changes for security/maintenance updates for SLES and SUSE Linux. However, SLES service packs will typically change a good part of the kABI -- and openSUSE factory kernels are expected to have kABI changes very often, as the upstream development is tracked there.

Kernel Module Package format

The kernel module packages are normal RPMs that contain the kernel modules (.ko files) and a bit of metadata. For the third party kernel modules to be identified, updates to be found, etc., the kernel module packages need to conform to packaging conventions. Packaging conventions are documented separately for the older Code10 solution and for newer one for Code11.

Like any other RPM, the kernel module packages should have a GnuPG signature. The updater will check it and complain if the package is not signed, signed by an unknown party or is corrupted. On initial installation, the install documentation or install script should thus recommend to import the GnuPG signature key of the module package provider into the RPM keyring to mark that RPM package provider as trusted.

Initial Module Installation

The above deals with how third party kernel modules can survive kernel updates without creating trouble for the user. But how can additional modules be found initially?

• The module packages can be part of a product. Vendors may include the RPMs on driver CDs or whatever the normal software distribution channel is. For Code10, a tiny script that registers an update source and offers to import the GnuPG package signature key into the RPM keyring are recommended.
• The module packages can be offered for download. RPMs can just be downloaded or the download location can just be entered as installation and update source during installation time (or later) in the installer/updater.
  • The user can find the download sites by searching the vendor's site, reading documentation from the vendor or from others, following hints from openSUSE, search engines, etc.
  • To save search work, a better automated mechanism that saves the searching work for the end user using a software/driver portal where software providers could register additional installation/update sources which are offered by YaST optionally at installation time is in preparation. Offering packages as function of the detected hardware is also part of the design.

The packaging format specification also takes care that modules that have been created for a different kernel version, but are compatible with the one that's installed on the system will get used for the installed kernel.

Support and Certification

Support and Certification are only relevant for the Enterprise products (SLES, NLD/SLED, OES). Support in this context refers to the ability of customers to open support calls and get guaranteed individual attention (including engineering efforts) to the problem they are reporting.

A kernel module has access to the kernel address space and can easily put the stability and reliability of the operating system at risk. Modules unknown to SUSE have the status "unsupported" and upon loading them will generate a warning message to the syslog and "taint" the kernel. If a user has such an unsupported kernel module loaded, SUSE can not do very much to help with kernel problems. Instead the problem should be reproduced without such a module loaded to be debuggable.

However, it is possible for a module provider to get the module in the "eXternally supported" status; for this it's necessary that he works closely with SUSE. The appropriate support channel interfaces need to be in place to allow SUSE support and the module provider to work jointly on a customer issue. The kernel module package needs to be built according to the rules specified above -- best by having it built by SUSE's build service (SUSEl Partner Engineering). Modules built this way can also be tested and certified and thus become part of certified systems.