This story started back in February when I was still finalising the core VIXI features ahead of the Renelick v1.0-rc1 release. While doing this I happened to notice a KUnit failure in linux-next which was causing some headaches in the community as per this email from Mark Brown: RCU stalls running KUnit on mainline. It appeared to me as a perfect opportunity to evaluate the VIXI bisection capabilities in a genuinely useful way since it was facing a real-world use case. The issue at hand turned out to be fully reproducible with the standard KUnit tests running in Tekton as-is which was a very good start.

From there a series of events I could not have anticipated started to unfold. It led to the implementation of new VIXI features and for Linus Torvalds to help fix a nasty kernel bug. Let’s retrace this whole journey step by step.

First Things First

Square one was to simply trigger a pipeline on the next-20260218 tag. This worked as expected as it did a build followed by KVM boot testing and KUnit which indeed failed to complete. It wasn’t a KUnit test case failure per se but the kernel crashed half-way through, roughly always around the same point and always with the same error message. I had to rework part of the job to get a result data node at every step to capture this as a failure, which after a re-run got caught as a regression by a delta task as expected (see last post for more details). This in turn triggered a test on the base commit from mainline which provided the initial revision range for an automated bisection.

After about 15 iterations, it finally landed on a merge commit: Merge branch 'kernel-7.0.misc' into vfs.all. This was real progress and rather encouraging as a first shot but not quite satisfactory yet. So I started digging around in the Git history to work out how to narrow this further down. In fact, it felt like I had been there many times before as it’s actually a pretty usual scenario when bisecting linux-next with over a hundred development branches being merged together every day. I ended up rebasing a small incoming set of commits to handcraft a linear history. The base commit was passing and the HEAD was failing: it was time to run a second bisection.

Same player shoot again

This time round it picked up an actual potential culprit, if I may call it so as one always needs to be cautious with plain bisection results: pidfs: convert rb-tree to rhashtable. Alright, we seemed to be getting somewhere with a finger pointing at pidfs. But was it really where the root cause was? That patch didn’t seem to be doing anything obviously wrong. I then instrumented the kernel with some good old printk in a few key locations and saw that it was trying to use a pointer that looked dodgy to say the least. To be honest, it smelled like a use-after-free bug or maybe a buffer overflow rewriting the pointer. However, I also wanted to be more than honest so I did what you’re supposed to do when you’re not familiar with a kernel subsystem and just reported the issue as-is by replying to the original patch email thread – follow that link for the rabbit hole discussion with all the nerdy details about a function pointer going astray.

Note about containers

This was also a good opportunity to mention how to reproduce the problem with the in-tree scripts/container tool and an available container image for the first time. More on that in a future blog post!

Fixed by Linus Torvalds

If you’re used to kernel development, this part will probably feel like business as usual. Still it’s worth quickly going through the sequence of events that followed, for the record.

So a couple of days later, Christian Brauner replied with what he had found to be a use-after-free issue in kthread and a draft fix for it: make_task_dead() & kthread_exit(). In short, quoting the email:

Ugh, yuck squared.

IIUC, the bug is a UAF in free_kthread_struct(). It wasn't easy
detectable until the pidfs rhashtable conversion changed struct pid's
size and field layout.

Mark Brown tested the change with some tweaks and I then verified that manually too with the same container image that the VIXI jobs use for KUnit and replied to the email accordingly to close the loop. And then Linus Torvalds proposed a different way to fix this:

Ugh.

So this is nasty, but I really detest the suggested fix. It just
smells wrong to have that affinity_node cleanup done in two different
places depending on how the exit is done.

…ugh indeed, with an alternative patch. From then on it’s really just kernel dev as usual, with Christian’s reply:

Anyway, let's just take what you proposed and slap a commit message on it.

Quoting the aforementioned slapped commit message:

Date: Thu, 26 Feb 2026 10:43:55 +0100
Subject: [PATCH] kthread: consolidate kthread exit paths to prevent use-after-free

Guillaume reported crashes via corrupted RCU callback function pointers
during KUnit testing. The crash was traced back to the pidfs rhashtable
conversion which replaced the 24-byte rb_node with an 8-byte rhash_head
in struct pid, shrinking it from 160 to 144 bytes.

[...]

Reported-by: Guillaume Tucker <gtucker@gtucker.io>
Tested-by: Guillaume Tucker <gtucker@gtucker.io>
Tested-by: Mark Brown <broonie@kernel.org>
Tested-by: David Gow <davidgow@google.com>
Cc: <stable@vger.kernel.org>
Link: https://lore.kernel.org/all/20260224-mittlerweile-besessen-2738831ae7f6@brauner
Co-developed-by: Linus Torvalds <torvalds@linux-foundation.org>
Fixes: 4d13f4304fa4 ("kthread: Implement preferred affinity")
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Christian Brauner <brauner@kernel.org>

It all started by trying out VIXI bisections for the first time and ended up with a personal mention in a critical fix co-authored by Linus Torvalds. How did that happen again? At any rate, this was pretty encouraging from a VIXI point of view and I must say it did give me more confidence to also push forward with the Renelick v1.0-rc1 milestone and the Beta Testing deployment. Now VIXI is monitoring some kernel trees continuously as a real-world Renelick application.

And it was fixed happily forever after along with a CVE report and stable backports too amidst the usual purring sound of the upstream kernel dev cogs churning away.

Rock & Roll patching 🎸

Back to VIXI land, as showcased in Demo No.4 it’s now possible to provide arbitrary patch files in all the tasks including automated bisections. While it’s always been on the VIXI roadmap, I actually implemented it while investigating this issue as I wanted to apply a workaround and the proposed fixes on top of various base revisions. No need to repeat what the detailed demo already covers in depth, let’s just take a quick look at the experiments I ran with it in this particular case.

I first made a workaround to avoid the crash when calling call_rcu() in pidfs. This was clearly not a real fix as the issue was still being investigated but it avoided the problem by releasing the PID without relying on an RCU call. I wanted to trigger some KUnit jobs with this patch applied on a new revision so I picked next-20260219. However, the workaround patch only applied after the commit found by the bisection. This would work for next-20260219 but not on the base mainline revision and as such, the delta task would fail to apply it. So I also supplied the bisection result commit as a patch along with another one it depended on so to have a small series that would apply on mainline. But then, that wouldn’t apply on top of next anymore since it already had the first two patches. This is a common use case, for example to verify a fix or isolate different issues on a same branch. So I implemented an extra feature when applying patches: the Rock & Roll flag or rnr.

As per the VIXI Demo No.4:

For lack of a better name, rnr stands for Rock & Roll which accurately describes the feature’s attitude. If applying a patch with git am fails, this will try again using patch --no-backup-if-mismatch -r- -p1 directly to pick up any pieces that might apply, discard the ones that don’t, commit what’s left and carry on.

Without the flag, failing to apply a patch will be reported as an error since in regular cases one would want to ensure the patch is applied e.g. when testing changes from mailing lists before they get merged in a maintainer tree. However, in this case we just want to use it as an extra hint for delta and bisection tasks which will be replaying the same jobs with different revisions where all patches may not always apply.

In practice, starting the KUnit task could be done this way (using my own tree here as the linux-next tags get deleted after a few weeks):

rk run tekton kunit \
   tree=gtucker \
   url="https://gitlab.com/gtucker/linux.git" \
   revision=next-20260219 \
   flags=rnr \
   orch=vixi \
   patches="https://storage.gtucker.io/vixi/patches/0001-pid-only-take-pidmap_lock-once-on-alloc.patch https://storage.gtucker.io/vixi/patches/0002-pidfs-convert-rb-tree-to-rhashtable.patch https://storage.gtucker.io/vixi/patches/0003-HACK-skip-call_rcu.patch"

Also worth noting, orch=vixi is to tell the orchestrator to follow its own rules on the results which means running a delta task to detect regressions compared to mainline.

So this ran and completed, the first two patches were already applied so they were skipped and the workaround worked around the problem. Then the orchestrator replayed this task on the mainline base commit with the same series of patches and this time the three of them were applied – all as expected.

Here’s the result node created by delta task on the mainline base commit showing the patch files that were applied:

rk node get 69b4170cb5f0d51ab9dbd2d7 -kpath -kdata.revision -kdata.result

{
  "path": [
    "next:v6.19-418-gd08008f19610:checkout",
    "next:v6.19-418-gd08008f19610:build",
    "next:v6.19-418-gd08008f19610:exec",
    "kunit"
  ],
  "data": {
    "revision": {
      "sha1": "d08008f196107a80c4e88b866d594b88a56ceaa9",
      "describe": "v6.19-418-gd08008f19610",
      "subject": "Merge tag 'asoc-fix-v7.0-merge-window' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus",
      "tree": "next",
      "url": "https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git",
      "digest": "df6b888d3422425bed440fc7582a7cba02e965115dc5fa77552e916fded53b1c",
      "patches": [
        {
          "url": "https://storage.gtucker.io/vixi/patches/0001-pid-only-take-pidmap_lock-once-on-alloc.patch",
          "subject": "pid: only take pidmap_lock once on alloc"
        },
        {
          "url": "https://storage.gtucker.io/vixi/patches/0002-pidfs-convert-rb-tree-to-rhashtable.patch",
          "subject": "pidfs: convert rb-tree to rhashtable"
        },
        {
          "url": "https://storage.gtucker.io/vixi/patches/0003-HACK-skip-call_rcu.patch",
          "subject": "HACK skip call_rcu()"
        }
      ]
    },
    "result": "pass"
  }
}

This wasn’t strictly required to report the initial issue but it helped confirm there wasn’t anything else going on and it’s definitely going to be helpful going forward.

Lessons learnt

Looking back at it a few months later meant it took me quite a while to untangle the whole story again. Taking a fresh look is always a great opportunity to capture some lessons learnt from a more objective standpoint. Overall, it confirmed many of the assumptions and design ideas I’ve gathered over the years to build a more advanced automated kernel testing tool for upstream. There is now a good foundation for the next set of features, starting with email interactions: monitoring mailing lists (no Rock & Roll here) and sending reports.

VIXI is now live and monitoring some upstream trees (mainline and next) so let’s see what other adventures lie ahead. A well known characteristic of upstream kernel development is that while there are literally thousands of commits pouring in through every merge window, each individual feature development cycle has its own temporality which oftentimes spans across many months if not years. The real power comes from the massive parallelism involved, and this extends to how fixes are made too.

Appendix: Timeline

Here’s the timeline of all the events and data I gathered for this blog post:

Last modified on 2026-06-16