Just over six months ago, I was gearing up for Kernel Recipes 2023 and submitted a talk proposal about how KernelCI could help drive a transition in Linux upstream kernel testing towards automation. While this was a hot topic in the 2022 edition, it’s a small conference and the focus had shifted in 2023 with no talks dedicated to testing so I’m now writing this post instead. Since then, I’ve also left the KernelCI project as the disconnect between the stated goals and the day-to-day reality of where efforts were being spent throughout the years had eventually drained my motivation. Here’s a silver lining.

The abstract from the 2023 talk now feels like a breath of fresh air contained in a time capsule. While releasing it here certainly helps me move on to other horizons, I believe it can also help articulate the principles underpinning the project’s next chapters.

KernelCI, Or What You Will

…was the proposed title for the talk, and this was the abstract:

It has been a long journey for KernelCI since its initial incarnation as a Linaro offshoot to bring about some automated build coverage for the ARM upstream kernel. And dare I say, some boot testing with a few loose boards on some shelves distributed around the world. Then it was chosen among a handful of candidates to become the main source of automated test reports for the whole kernel when it joined the Linux Foundation in 2019. That was no small mission.

Let 2023 be the year when finally all the stars align and a lot of effort starts bearing fruit. The best part of it is that it’s all up to you, the users, to pick it up when you feel it’s ripe: kernel developers, maintainers, OEMs, SoC manufacturers, enthusiasts, students, interns… The key to it all hinges on a new, modern, carefully (albeit probably not perfectly) designed API and a powerful infrastructure to provide solutions to what we hope were the kind of problems that needed one. Yet another CI bot? Sort of; it is an automated pipeline with capabilities to build lots of kernels and run lots of tests in the Cloud and on real hardware, anywhere really, and track regressions and send emails etc. But what truly sets it apart is that you can sign up and create your own user account, get an API token and then plug in your own tools to combine it with your own development workflow. As a developer or maintainer, it offers you a smooth transition from manually running your usual test scripts or any other checks you might have all the way to the ethereal power of full automation. And if you felt so inclined, your tests could also be run as part of the main pipeline and to everyone’s benefit.

Does it sound too good to be true? You be the judge of that. Or put another way, it’s what you make of it! Discussing test-driven kernel releases last year with the audience was very productive so let’s do this again.

Epilogue

The new KernelCI API is not yet in production, and I’m not sure where it currently stands compared to the original timeline as only the first milestone has actually been met so far. Work is still on-going and there’s hope it will eventually be launched along with the new web dashboard by the end of 2024. But more importantly, what was the key idea behind this new system which I wanted to convey through my talk?

Kernel developers, from junior contributors to large subsystem maintainers, tend to work in a very hands-on manner: you build your own kernel or module, run it with some manual tests and iterate locally while code changes are sent over email. There are many reasons why this is the case, it’s harder to set up a development environment for an operating system kernel than for a website or a mobile app. The Linux kernel supports so many different kinds of devices and has been around for so many years that you can’t just force the whole community to adopt an off-the-shelf DevOps package such as GitLab. Still, kernel developers don’t live in a cave but are on the cutting edge of technology and this topic keeps coming back. There just isn’t an obvious solution; KernelCI’s efforts are an attempt to find one.

As described in the abstract, it’s about providing a continuum between the current typical manual workflow and a fully automated one. Individuals and subsystems can then choose how fast and to which degree turn the “automation” dial to benefit from it without causing unnecessary disruption in an already very complex ecosystem.

For example

In a typical scenario, a maintainer has a script to run a number of tests and produce results before applying patches on a branch for the next kernel release. One first step could be to define this as a KernelCI job and run it with the kci command line tool, in the exact same runtime environment as when running the script in a shell. A critical difference is that the results could be sent to the KernelCI API under the user’s account to make them available to the public and get extra features such as regression tracking and reports with virtually no added cost. A second step would be to run the same tests but say, in a container or an on-demand VM or some remote hardware test pool. This is made possible thanks to the runtime abstraction layer in the new KernelCI implementation. Once enabled, these tests could be scheduled to run automatically by the main KernelCI pipeline which constantly monitors upstream Linux kernel trees and incoming patches on mailing lists.

Having a demo to showcase all this here would have been great, and it could already be done to some extent even without the API in production and a new web dashboard, but that’s beyond the scope of this blog post. It may also be a bit too much to call it an exercise for the reader. So I’ll just leave this idea here as a seed and perhaps come back every year or so to watch it grow.

About the author

Guillaume was a software engineer at Collabora for seven years while also Chair of the KernelCI Linux Foundation project for three years. A large part of this time was dedicated to KernelCI with code contributions deep into bootloaders all the way up to web services, laying the foundations to set up a sustainable organization with the Board & TSC and growing ties with the upstream Linux kernel community.

Last modified on 2024-01-29