gtucker.io

VIXI – kernel bisection with Renelick

Mon, 02 Feb 2026 00:00:00 +0000

gtucker.io https://gtucker.io/posts/2026-02-02-vixi-and-renelick/ -

First of all, let’s go though some brief introductions. VIXI stands for Virtual Kernel Continuous Integration and is the main topic of this post. Renelick started as a fork of the new KernelCI stack in early 2024 and has kept evolving ever since into a general purpose data-driven automation framework. VIXI is essentially an application powered by Renelick to continuously test the upstream Linux kernel in virtual environments. It is now reaching the end of its proof-of-concept phase and as such deserves a highlight. A secondary goal is to validate the Renelick API with a comprehensive use case before making a v1.0 release candidate, which is due very soon. I’ll cover that in another post once it has crossed the finish line. For now, let’s focus on the VIXI results.

Context

Building upon all the lessons learnt from working on KernelCI for several years, VIXI brings together a more advanced solution aiming to go above and beyond what classic automated testing typically provides. The scope is the upstream Linux kernel, which is very different from vertically-integrated products. Rather than being just one component among many others in a monolithic stack, the kernel is a moving part within a virtually endless spectrum of configurations. Another critical difference with regular product development is the way changes get merged upstream, with hundreds of patches sent every day via mailing lists and through a hierarchy of maintainers. The concept of CI/CD doesn’t really apply in the usual sense, it would only work if there were a small number of officially supported platforms with a designated user-space stack and compiler toolchain – which would tend to go against the spirit of a neutral, common mainline kernel.

Dynamic orchestration

Extraordinary workflows require extraordinary solutions. With such an open-ended system under test, the key is to rely on dynamic orchestration to activate particular coverage according to the circumstances: area of the code being changed, previous results, available resources etc. Starting with a fast path to get initial results from lightweight tests, the orchestrator can then dynamically schedule additional runs depending on their outcome. Say, if a warning was detected at boot time in a particular device driver, a selection of especially relevant tests can be run with debugging and instrumentation options enabled. Conversely, long-running tests may be scheduled only if the system booted fine in the first place within the allocated resource budget – no need to start a full range of test suites on a system that won’t even boot. While the logic isn’t currently quite there yet, VIXI is being designed with this vision in mind.

One primitive for dealing with such scenario is implemented by VIXI as the “replay” feature which enables any test to be run again while changing one or more parameters. Every time some results come in, the current proof-of-concept will run a post-processing delta task which will compare them against some reference revision to detect new failures. This could be expanded in the future by running the replay with a different compiler version, CPU architecture, hardware platform etc.

When testing a maintainer branch, the base revision in mainline should be tested too in order to determine whether a failure was already there. Then extra checks need to be run to reduce false positives: is the “bad” revision really bad and the “good” one really good? If so, this will provide a reliable range for running an automated bisection and try to identify the individual commit that caused the test to fail. This logic is already implemented in the current VIXI PoC as detailed in the demo section below.

Bisection works well for high-volume branches and in particular linux-next as it integrates new commits from all the subsystems on a daily basis. Another dimension to the problem is testing patches from mailing lists before they get pushed on a Git branch. In this case, a better approach for short series is to test each patch individually but the same concept of comparing results applies. Although the proof-of-concept doesn’t do that yet, some initial trials have been run with an IMAP inbox and it will be enabled in incremental steps.

Demo time

Early experiments have given way to a few demos which show the progress made so far:

From Git to KUnit

The first demo is to lay down the foundation: Git checkout using mirrors within a Kubernetes deployment, a kernel build followed by a KVM boot test using KubeVirt as well as KUnit runs with results parsed into a tree of Renelick database nodes.

Delta and Git bisect

Building upon this, the second demo showcases the delta and bisect tasks. They can automatically compare results and run Git bisections when KUnit test regressions are detected. This only worked for KUnit as it showed the challenge of being able to automatically reproduce arbitrary test results with a different kernel revision.

Automated replay

Then the third demo introduced the “replay” feature which enabled delta and bisect to deal with any kind of workload, not just KUnit. The data nodes now include the information needed to recreate the whole series of tasks which were run to produce them, with can be done while changing some parameters such as the Git revision.

All things combined

The outcome is a first orchestrator which listens for a simple trigger event pointing at a Git kernel revision. Here’s how all the various tasks unfold from there:

trigger: event sent manually with the Git tree name and revision to test
kbuild: kernel build run as a Tekton pipeline
kboot: kernel boot with the artifacts produced by kbuild using KubeVirt
kunit and parser: KUnit run in Tekton followed by a task to parse the JSON results in a plain Kubernetes pod

The kbuild, kboot and kunit + parser tasks eventually produce some database nodes with test results which can be consumed by the delta task. The orchestrator will therefore schedule a delta for each incoming set of results following the rule set in its configuration file:

# compute delta between test results
- task: delta
  channel: node
  match:
    kind: vixi.group
    op: populate
  scheduler: inline-vixi

Note: The inline-vixi scheduler is for running lightweight tasks directly in Python. For more details, see the Renelick documentation.

Each delta task will look for results from the base revision between the branch being tested and mainline. If it can’t find any, it will get the tests replayed there first in order to be able to detect regressions such as in the example above with KUnit. Once the base revision results are in, it will create a node for each new test failure detected. The orchestrator will then start a bisect task for each of them as per this other rule:

# run a Git bisection based on found delta
- task: bisect
  channel: node
  match:
    kind: vixi.delta
    op: populate
  scheduler: inline-vixi

This logic works independently of the underlying tasks and can reproduce any test results automatically: kbuild, kunit or kboot. The data included within the result nodes contains the parameters required for replaying each task that produced them. As such, the sequence will differ depending on the type of test:

kbuild is a single task which directly produces some pass/fail results
kboot is run following kbuild results as it needs its artifacts as inputs
kunit is followed by the kunit-parser task to produce results

The example in the diagram above shows the kunit + parser case although kbuild and kbuild + kboot have also been verified as part of the demos. In fact, any number of tasks can be chained this way using the input-nodes task attribute to describe how to feed the results of a previous task into the next one. For example, with kboot which takes its inputs from kbuild:

"input-nodes": {
  "kbuild": {
    "id": "69734c80a855520ff4115a9c",
    "kind": "vixi.group",
    "name": "kbuild"
  },
  "image": {
    "id": "69734c67a855520ff4115a96",
    "kind": "artifact-v1",
    "name": "bzImage"
  },
  "modules": {
    "id": "69734c78a855520ff4115a9a",
    "kind": "artifact-v1",
    "name": "modules.tar.gz"
  }

Bisection results

At the end of this long string of events a successful bisect task will eventually find the commit that caused a test to fail and create its own node too. Here’s the Git history from the branch that was being bisected in demo 3, with an artificial KUnit test failure in the middle:

b1d2fdbfdf5f HACK BISECT 4.noop
9e1c130d734e HACK BISECT 3.noop
76819e0f306e HACK fail KUnit overflow.shift_sane_test
71746a491c0f HACK BISECT 1.noop
f8f9c1f4d0c7 Linux 6.19-rc3

The head of the branch b1d2fdbfdf5f was tested first, then the base revision v6.19-rc3 followed by the actual bisection which landed on the expected commit.

What’s especially interesting to mention here is the continuous data path which goes all the way from the Git checkout to the final bisection result. It can be seen in the last node that was created, earlier ones will have a shorter path of course:

"path": [
  "gtucker:v6.19-rc3-4-gb1d2fdbfdf5f:checkout",
  "gtucker:v6.19-rc3-4-gb1d2fdbfdf5f:build",
  "gtucker:v6.19-rc3-4-gb1d2fdbfdf5f:exec",
  "kunit",
  "overflow",
  "shift_sane_test",
  "shift_sane_test:break",
  "bisect:76819e0f306e1d1682ffcdeba5cd408c0eeb85c5"
],

This list contains the names of the nodes following their parent relationships in the tree. The checkout is the root node with no parent. Then the build and KUnit execution are added as child nodes, followed by the parsed results under kunit. There are actually over 800 nodes with individual test cases in a single tree of this kind, here we just see the ones that led to a failure. The shift_sane_test:break node was created by the delta task with the regression data. Finally, its child node comes from the bisect task with the SHA1 of the Git commit that was found.

I won’t go into all the intricate details of how this came to be as that is covered more in depth by the demos themselves. To get a peek under the hood, you can take a look at the Renelick pub/sub event monitor log. Here’s an excerpt with just a few hand-picked lines from it:

I  2026-01-12T17:18:39.505717+00:00 VIXI   trigger       gtucker                               git                 linux-6.19-rc3-kunit-fail
I  2026-01-12T17:18:39.874351+00:00 TASK   add           0365a602-7c45-40ad-880a-c04795ee53db  tekton              kunit
I  2026-01-12T17:19:05.451992+00:00 NODE   add           69652d0967ff704f772f8ce2              vixi.kunit.step     gtucker:v6.19-rc3-4-gb1d2fdbfdf5f:checkout
I  2026-01-12T17:22:05.844831+00:00 NODE   add           69652dbd67ff704f772f8cfe              vixi.kunit.step     gtucker:v6.19-rc3-4-gb1d2fdbfdf5f:exec
I  2026-01-12T17:22:06.531947+00:00 NODE   add           69652dbe67ff704f772f8d00              artifact-v1         kunit.json
I  2026-01-12T17:22:15.510677+00:00 TASK   add           9ba6a2d9-72a1-49e4-bcfc-aca0325874b7  inline-vixi         kunit-parser
I  2026-01-12T17:22:18.566051+00:00 NODE   populate      69652dc867ff704f772f8d02              batch:1             kunit
I  2026-01-12T17:22:18.942489+00:00 TASK   add           72e89f23-7f09-4cca-835f-11bb96ee3f68  inline-vixi         delta
I  2026-01-12T17:22:21.821146+00:00 NODE   populate      69652dcd67ff704f772f9080              batch:1             shift_sane_test:break
I  2026-01-12T17:22:22.251833+00:00 TASK   add           0c8ab94d-290f-48be-8090-683c0da56c5d  inline-vixi         bisect
I  2026-01-12T17:22:23.532738+00:00 VIXI   bisect        v6.19-rc3-4-gb1d2fdbfdf5f             check:bad           b1d2fdbfdf5f9d869bea4714c3e00c5cb3faea38
I  2026-01-12T17:23:49.670967+00:00 VIXI   bisect        v6.19-rc3-4-gb1d2fdbfdf5f             check:good          f8f9c1f4d0c7a64600e2ca312dec824a0bc2f1da
I  2026-01-12T17:25:18.153491+00:00 VIXI   bisect        v6.19-rc3-4-gb1d2fdbfdf5f             iter  1/2           76819e0f306e1d1682ffcdeba5cd408c0eeb85c5
I  2026-01-12T17:26:43.290082+00:00 VIXI   bisect        v6.19-rc3-4-gb1d2fdbfdf5f             iter  2/2           71746a491c0f6557d43047dc9eca5f0a5613f37c
I  2026-01-12T17:28:09.578434+00:00 VIXI   bisect        v6.19-rc3-4-gb1d2fdbfdf5f             found               76819e0f306e1d1682ffcdeba5cd408c0eeb85c5
I  2026-01-12T17:28:09.907141+00:00 NODE   add           69652f2967ff704f772f9115              vixi.bisect         bisect:76819e0f306e1d1682ffcdeba5cd408c0eeb85c5
I  2026-01-12T17:28:10.478395+00:00 NODE   add           69652f2a67ff704f772f9116              artifact-v1         bisect.txt

Next steps

The VIXI experiments have proven to be rather conclusive so far. The replay feature is implemented entirely at the application level using the standard Renelick API primitive operations. This is good news as it shows the API can remain simple while enabling a variety of complex use cases.

It’s also encouraging to see that demo 3 which ran a Kunit bisection lasted only about 10 minutes, with an extra kernel build running in parallel on the same worker followed by a boot test on a separate bare-metal machine with KVM enabled. The Renelick API instance, the orchestrator and the workers are all running on different machines with remote connections so this is a fairly realistic topology for an actual deployment – at least in terms of latencies.

One current limitation is that the delta task only compares results with the base revision in mainline, not previous runs on the same branch. So this will be the next improvement there. A more ambitious goal is to use Scalpel and make bisection an integral part of the dynamic orchestration logic using statistics.

Another issue is that while KubeVirt is great for scaling large workloads in the Cloud, the overhead of starting VMs and the general design isn’t too appopriate for short-lived tests such as booting kernels. Some more low-level Runtime implementations using QEMU and Cloud APIs directly would help there.

Overall, this sets the stage for the upcoming roadmap. Once Renelick v1.0-rc1 is released and hosted on a public instance, VIXI can be enabled as a continuous service: monitoring linux-next once a day, mainline every hour, patches from a couple of mailing lists e.g. KUnit and Rust-for-Linux to start with, sending email reports, showing results clearly on the frontend etc. There is also the open topic of sharing some of the bisection logic with KernelCI via a common Python package, which would be closing a very long loop.

Last but not least, the builds and KUnit tests in VIXI are all using kernel.org compiler toolchains as generic container images. This fits into the bigger picture of having them maintained upstream as per the mailing list discussion and the scripts/container tool that just landed in linux-next.

- https://gtucker.io/posts/2026-02-02-vixi-and-renelick/ - 2021-2026 Guillaume Tucker

Digital Substation Pt.3 – Deployment

Mon, 01 Dec 2025 00:00:00 +0000

gtucker.io https://gtucker.io/posts/2025-12-01-digisub-pt3/ -

The time has come to wrap up this blog series about Digital Substations. More than a conclusion, it totally feels like a new beginning. The past few months have been a rich adventure, meeting and learning from many people via many projects and at many events as I was still going through my paces. I can definitely confirm that there is a huge potential for open-source developers from all horizons who wish to join the Energy sector and entire communities are being formed under our very eyes. I’m very much looking forward to the next steps in my journey. But first, let’s see where this has taken me so far.

OSPOlogyLive at RTE

The last event I attended was OSPologyLive at the RTE Campus in Lyon. I must begin with some very special thanks to all the organisers and to Maxime and Boris in particular for your warm welcome, the guided tour and for being such inspirational figures.

It was a double-headed event: OSPO in general and LF Energy in particular. As I’ve always been a staunch open source activist, I felt at home in OSPO land and it also helped me position myself as an independent contributor. The presentations and discussions were all of very high quality with many interactive sessions. While I can only recommend and praise this yearly meetup, let’s now return to the hot topic of Energy.

No electrical hazard

This is fine.

Electricity being transported at 400kV is not to be taken lightly, but what can possibly go wrong with running virtual machines on a plain development laptop? The tour around the RTE Transfo Campus included the miniature laser-cut electrical grid shown at the top of this article and the training ground for electrical line engineering work among many other things. It has helped me put things in perspective with my software hat on.

A key takeaway is that this isn’t just about electrical grid equipment manufacturers dealing with voltages that require a wide safety perimeter. The open-source software deployed in electrical substations can be run by anyone and virtually anywhere. Hobbyists can take part and uncover new professional avenues. Researchers and students can advance cutting-edge technology with small-scale labs. The long history and lessons learnt that have shaped the FOSS culture also apply here: while not everyone in the Energy sector is running Linux on their laptop yet, people know about licenses, best practices and the value of building a shared platform in the open.

SEAPATH

Pursuing our RTE tour, we then stopped by this rack of servers running SEAPATH for evaluation purposes. It’s designed to continuously monitor the electrical lines connected to a substation and “pull the plug” if a fault condition is met. I’m over-simplifying, this is much more complex than just a peak detector with a fixed threshold. Every millisecond counts in the event of something going wrong: waveform distortion, voltage surges etc. can damage large amounts of infrastructure, cause fires and explosions. But software-wise, it’s just regular streams of data flowing in and out - under tight real-time constraints. And of course, this can be emulated with artificial sensor data and circuit breaker models. Some substations already feature early deployments of SEAPATH while the most critical safety functions are still handled in dedicated hardware for now. Other TSOs in various countries are doing the same, largely with their own and often proprietary solutions. With a lifetime of at least 25 years, the pace and magnitude of this kind of entreprise is titanic. In comparison, writing code is fast.

The community around this project still needs to take off although it is continuously gathering momentum. More and more contributors are coming together, chiefly under the wing of RTE via LF Energy: Savoir-Faire Linux has been involved for many years and Red Hat just joined the party. This is the second key takeaway from the event: while each player is being assigned a particular role, where should I stand?

Personal roadmap

As an independent developer and long-time advocate of open source and free software best practices, I can see many ways to foster a wider engagement and extend the project beyond the corporate boundaries drawn by the big actors. This naturally boils down to the various business models involved. I don’t provide complete enterprise solutions or manage data centres and Cloud services. There’s only so much a single individual can do. However, I have extensive freedom of action and benefit from a privileged, neutral position to be an active catalyst in the community.

My previous experience in Linux kernel development and automated testing also brings a valuable element to the picture. Maintaining production deployments for 25 years is hard. We live in a connected world and security fixes in particular need to be constantly back-ported to keep a system safe. I witnessed this first-hand when ChromeOS extended its kernel support lifetime via KernelCI. Upstream long-term stable kernel releases last at most 5 years and CIP maintains specific use-cases for up to 10 years so production kernels always require a significant amount of effort.

Putting all this together, this is what my current personal Energy roadmap looks like:

Open Stack

My area of interest has now narrowed down to the Digital Substation software stack. As such, I would like to provide a development setup with ideally all components available as open-source. This would start with a SEAPATH setup that can be run on a development laptop rather than dedicated servers - I have a nested KVM PoC coming soon for this 🚀

Then as mentioned in my previous post, the IEC 61850 standard is lacking in open-source library implementation. So I’m looking forward to contributing to the Open Energy Tools project and include this in the development stack via a base image with some demo application containers and VMs.

Community Catalyst

By not being tied to the particular interests and decision process of any established organisation, I am literally walking in the community developer’s shoes. As such, I have the agency to connect the dots in the bigger picture. In practice, this can mean writing more blog posts like this one and speaking at events but also help managing communication channels (Discord, IRC, mailing list etc.) in a neutral way. I also don’t have the burden of proprietary license and confidential information that could taint software development. Put another way, my only bias is towards FOSS indeed.

Kernel Support

Going a bit further on the technical side, I can also put my deep expertise to good use with long-term Linux kernel testing and support for production deployments. As I’m currently working on VIXI, an R&D project for automated kernel testing in virtual environments, a particular use case would be to run the SEAPATH development stack mentioned earlier with rt-stable and any other relevant upstream branches. This could catch a large amount of issues before doing full-blown testing on real hardware targets. Keeping the same kernel base version for more than a few years isn’t really practical, the real answer is to be able to upgrade in the field. Doing this requires a high standard of quality to avoid regressions and minimise the hours of engineering time spent porting downstream patches.

See you at FOSDEM

The journey continues while I’m going back to where it all started. The Energy devroom has been announced for the forth time at FOSDEM 2026 (schedule to be finalised mid-December) so it’ll be good to see you there if you can attend. And as usual, please feel free to send any queries you may have via email. In the meantime, keep energised ⚡

- https://gtucker.io/posts/2025-12-01-digisub-pt3/ - 2021-2026 Guillaume Tucker

Digital Substation Pt.2.1 – LFE Summit

Mon, 29 Sep 2025 00:00:00 +0000

gtucker.io https://gtucker.io/posts/2025-09-29-digisub-pt2.1/ -

Attending LF Energy Summit Europe was a really inspiring experience. The event is growing very quickly every year which shows how much interest there is in the industry for open-source solutions. It was my second Summit and I’m now able to navigate my way through the various projects, companies and individuals – none of them were known to me just 18 months ago. Writing my previous post really helped me getting prepared for the event and I arrived with many questions in mind. I’m glad I’ve found some answers and met numerous people to discuss all these topics and beyond. For now, let’s focus on the ones I raised last time and what I’ve learnt since then.

Reference Stack

💡 What would be needed to create a complete open-source reference solution?

Having a fully open-source reference stack is definitely something many people are aiming for. It reminds me of the story of the early days of GNU, starting with just a C library before adding a kernel (Linux), a compiler (GCC), applications (glib, GTK) etc. The demo as seen in the main picture with the Savoir-Faire Linux team was a great way to showcase where things currently stand and how SEAPATH provides a very promising basis for an open-source reference stack. In a nutshell, it’s a hypervisor which addresses all the real-time and quality of service requirements for running applications in a digital substation. I’ll definitely be spending more time on this in the coming weeks.

However, the upper layers of the stack are still missing. As I anticipated, application VMs are all proprietary. Having some reference ones would be very useful, even if they aren’t suitable as-is for production deployments. They would first of all facilitate development and validation of the lower layers, then potentially provide building blocks for the real world. As community efforts are growing, I believe it should eventually materialise so I’ll be keeping an eye on that too.

As a bonus, there is also a path to having an open hardware reference design for at least some components with initiatives such as OwnTech. So the stack can grow both ways around SEAPATH, all the way from the hardware to application VMs.

IEC 61850 Libraries

💡 How about creating a non-official, open standard for IEC 61850?

One key building block of digital substation services is an implementation of the IEC 61850 communication protocols. This is a somewhat thorny topic as it’s a proprietary international standard surrounded by many legal grey areas. In my previous post, I suggested writing a new Rust crate under the MIT license. While in Aachen, I was delighted to meet Jakob Vogelsang who happens to be working on the OpenEnergyTools project which does just that!

I’m looking forward to start contributing very soon even though I don’t currently have access to the standard itself. This may in fact be all for the better if it can avoid a few legal issues. The details of how it’s all going to unfold still need to be worked out. At least we’ve been able to confirm that having an open-source implementation of the standard is permitted. It seems a bit simpler than some reverse-engineered GPU driver projects I’ve seen such as Panfrost which ended up being a huge success in spite of the countless pitfalls that paved the way there. Now, electrical equipment destined to be used in actual substations still has to meet safety regulations as per IEC standards too but that’s an orthogonal issue. Developers aren’t responsible for deploying the grid.

Real-Time Requirements

The topic of punching a hole through the ISO networking layers to get data packets delivered directly and in real time doesn’t transpire as an obvious source of concern. This doesn’t mean it will never be, a recurring theme at the Summit was that security is paramount but there is currently little clarity over how to achieve it or how secure the current systems are. So it might be an oversight, time will tell. There must have been some form of security audits in the past, maybe the results just haven’t been shared publicly. The industry tends to focus on safety first as with traditional substations and it takes a while to address the additional risks that the digital world brings to the table.

💡 How can one create a virtual digital substation for development?

Still, using the networking hardware rather than a dedicated digital CAN-like bus for GOOSE and sample values (SV) seems like a practical approach for keeping things simple. As long as these packets don’t disrupt the regular services running on TCP and the whole approach doesn’t pose serious security risks then I suppose it’s fine. At least it makes it easier to implement the rest of the stack so it definitively helps with development. Again, this is where I’m expecting SEAPATH to shine.

Also I’ve heard that the standards tend to be rather heavy-weight, so it may be justified to have more streamlined, optimised communication within some well-defined areas of substations and rely on the established protocols primarily for the sake of interoperability – food for thought.

Last Leg

I now have all the elements lined up for the third and last part of this blog series, except maybe that I haven’t been exposed to the hard reality of electrical grids yet. So I’ll continue down the same avenue with more hands-on experiments, maybe running SEAPATH with some basic hardware. What are the challenges faced when deploying and maintaining a digital substation in the field? How does regular grids compare with railway networks or other specialised use-cases? Let’s see where this takes us.

- https://gtucker.io/posts/2025-09-29-digisub-pt2.1/ - 2021-2026 Guillaume Tucker

Digital Substation Pt.2 - Insights

Tue, 09 Sep 2025 00:00:00 +0000

gtucker.io https://gtucker.io/posts/2025-09-09-digisub-pt2/ -

The energy world is overwhelmingly large. This blog series draws a line around digital electrical substations which are themselves a very rich topic with many avenues to explore. The first part covered some basic concepts along with my first impressions as an open-source software developer. It’s now time to step in a bit further and focus on two specific areas: the open-source part of the energy ecosystem and how the concept of software-defined networking (SDN) applies to substations.

As I’m still only at the beginning of this long journey, there are many questions that remain to be answered in this blog post. I’ve highlighted them with lightbulbs 💡 and they are like “breakpoints” where further debugging is needed.

Ecosystem

The edges of any ecosystem are always fuzzy so let’s start from the core. While I’ve been actively looking for hidden gems, it turns out that the key player by far in this area is the Linux Foundation Energy initiative. It acts as an umbrella to drive a variety of industry members towards open collaboration. Like I wrote before, the open-source communities in the energy sector at large and around substations in particular still seem to have a long way to go to catch up with other industries such as data centres and consumer electronics - which makes such collaborative efforts all the more valuable.

LF Energy is currently made up of roughly 50 members and 50 projects which all deserve some attention. I’ve arbitrarily cherry-picked a handful that appear to be the most relevant ones for this blog post. And let’s not forget to keep an eye open for other entities outside this circle.

Standards

Before we dive into a sea of lines of code, an important aspect to keep in mind is international standards. While they might seem rather daunting at first, they are also the backbone of the industry as they help ensure interoperability. Many open-source projects provide concrete implementations of them. It all essentially revolves around the now-famous IEC 61850. If you’ve never heard of it, the written interview from the technical committee lead provides a quick intro.

The most relatable parts of the standards for software engineers are messaging protocols (MMS, GOOSE, SV) as well as the substation configuration language (SCL). Here’s a small cheat sheet:

MMS: general-purpose messages with a topology of abstract objects inherited from the manufacturing world (ISO 9506)

GOOSE: low-latency publisher / subscriber events operating at the MAC address level (OSI level 2) in substation LANs

SV: sampled values, they are the digital equivalent of analogue signals

SCL: machine-readable XML dialect to describe a substation topology

The IEC standards are all proprietary. Copies of the PDF documents can be purchased from the International Electrotechnical Comission website. While this has been common practice for many decades, it also shows an obvious gap to fill when it comes to open source. I couldn’t even find canonical links with definitions to the acronyms mentioned earlier other than some lightweight Wikipedia entries or independent articles such as this one. The non-profit UCA user group goes in this direction but isn’t quite a true open-source community effort either.

To clarify what I have in mind, let’s take the example of how the graphics industry has adopted a now well-known and successful open standard approach (OpenGL, Vulkan) via the Khronos group. These standards are entirely open and available free of charge, including the certification test suites so anyone can run them on their GPU drivers. Only the final, authoritative certification is not free and is typically needed only by hardware vendors in the supply chain for end-user products.

💡 How about creating a non-official, open standard for IEC 61850?

As a side note, it’s also worth mentioning that earlier communication protocols such as Modbus are still around and they keep coming up in the literature. Electrical grid infrastructure naturally has a huge inertia due to the costs and amount of effort involved with replacing any part of it, and they’re built to last. A good example is this 2019 slide deck from Siemens. But let’s stay focused on the more modern solutions here.

Projects

How well do these international standards translate into currently available open-source tools? First, let’s enumerate a few. Starting with the LF Energy projects, here are the ones I picked:

SEAPATH: hypervisor for digital substations
- Resources: GitHub, wiki
- Runs on Yocto and Debian with Ansible to deploy new VMs on top
Power Grid Model: grid modeling and profiling
- Resources: GitHub, PyPI
- Comes with some nice examples
PowSyBl: Power System Blocks
- Resources: GitHub, documentation
- All done in Java code, it would seem useful to have some data file format
- Here’s a nice tutorial among a few others
CoMPAS: Configuration Modules
- Resources: GitHub, wiki
- A docker compose setup is available
- Some tutorial videos are available too
- Looks great, finding my way around the project wasn’t very easy at first

Then here are a few independent ones which I found to be equally interesting:

libiec61850: IEC 61850 library written in C
- Resources: GitHub, documentation
- Under the GPL v3, appears to be the only open-source one available
- Part of MZ Automation
OpenSCD: Open Substation Communication Designer
- Resources: GitHub, wiki
- Editor for SCL files, see the demo
- Has a downstream fork for CoMPAS
VeraGrid (formally GridCal): electrical grid simulation
- Resources: GitHub, PyPI
- While this is potentially highly relevant here, substation modeling still remains to be done
OpenEnergyTools: toolbox related to digital substations and grids
- Provides Typescript tools to handle SCL data
- Nice community initiative, still early days

Each project mentioned above as well as many others would require at least a dedicated blog post and several hours of playing around to do them justice. So as a tl;dr summary, I found libiec61850 to be pretty neat and a great way to get started with the standard from a hands-on point of view. Reading the code already sheds some light on what it’s all about. Then I found VeraGrid to be of very high quality, but that’s slightly outside the physical substation realm. Other simulation and modeling tools can be used locally too, more or less easily. The biggest issue I have with those that relate directly to substations is how to use them in practice without having a real electrical grid at hand. I’m sure there must be ways to set up lightweight development environments but it didn’t seem too obvious. For example, I haven’t found sample Ansible deployment files for SEAPATH with typical VM instances other than in the release procedure.

💡 How can one create a virtual digital substation for development?

Stacking things up

Also I’m struggling to understand how things fit together. In particular, I find it hard to see how these projects can be composed to form a functional stack. Say, where does VeraGrid sit compared to Power Grid Model and PowSyBl, or OpenSCD compared to CoMPAS-SCT? The mapping between electricity transportation and distribution networks on one hand and software projects on the other seems rather fuzzy too, but that may just be because I’m missing some information and expertise there. Then I don’t know to which extent libiec61850 is used in real-world substations, I would expect such use-cases to be somewhat publicly visible due to the GPL license.

For the sake of the exercise, let’s try and put together all the pieces I’ve gathered so far. A digital substation has a set of software services running on top of a dedicated local network (LAN) infrastructure. This is done via a hypervisor such as SEAPATH and isolated virtual machines which provide the different functions. The substation topology for the underlying electrical equipment such as circuit breakers etc. is described using the SCL language which OpenSCD or CoMPAS SCT can help manage. Critical communication within the substation’s LAN is done via GOOSE, mostly to protect the equipment against faults or surges. Sensors send data as samples values (SV) which are used by monitoring systems and the control logic to trigger actions. Routable-GOOSE (R-GOOSE) is used to open remote connections via the regular IP layer but without real-time guarantees. All the rest of the communication is handled via industry-standard MMS messages. Aside from grid modeling and simulation, this is all I can think of right now that applies to digital substations.

What stands out is that some areas seem to have competing projects while there are large gaps in other areas. What are the actual applications running in substations and implementing the control logic? Probably this is driven mostly by industry members with a rather narrow set of requirements. So I would expect all the missing parts to have proprietary implementations, a bit like consumer electronics or web technologies have open-source low-level building bricks to provide a platform for deploying proprietary applications.

💡 What would be needed to create a complete open-source reference solution?

My 2¢

While I have spent quite a bit of time looking into each of them, I’m still mostly commenting here on the “out-of-box” experience as a newcomer. So I may be missing some obvious things, but I suspect others like me might have to jump over the same hurdles. My general feeling is that there is a very strong momentum behind all these projects although the community around it is still in its infancy. I may be projecting too much, but what seems to be needed now is to expand the spectrum of people involved. New contributors with some relevant experience from different backgrounds and enough time available would help bring things to maturity. It reminds me of where embedded electronics was before Android took off around 2012 and Linux became the main OS in the industry. It’s true that while virtually anyone can get a computer and run plain software on it, the hardware involved with electrical grids means it’s bound to be destined to a much reduced user base. Still, smart homes and domestic generators such as solar panels are becoming increasingly common. As such, there is a potential for consumer-level nanogrids at the household scale. Having homes connected to the grid from a data point of view is increasingly common and raises awareness about energy usage. The main incentives for consumers may be keeping an eye on bills, where the electricity is coming from as well as its associated carbon footprint while preserving privacy.

I realise this has raised more questions than answers, which again probably shows where I lack some knowledge but also where the potential for growing the community side of the ecosystem might lie. Let’s now take a look at the second topic of this blog post and jump to a different starting point which does have a well-established software ecosystem.

Software Defined

Data centres route information through their networks with elaborate and dynamic rules. Similarly, substations route electricity. However, drawing an accurate parallel between these two kinds of routing is far from trivial. Network data packets are abstract entities that can be held in memory and carry their own headers, which are used to build sophisticated rules. An electrical energy flow has a very real physical dimension. A broken TCP connection doesn’t sound as dangerous as an electrical arc flash explosion. I suppose some concepts still apply, such as load balancing for data traffic and load shedding with power generators. We may also see an equivalent of the data plane in the electrical concept I’d like to call power plane. Generally speaking, even if the analogy with data routing only goes so far, the idea of describing both kinds of rules and deploying them dynamically via software is equally valid. So the first takeaway is that software-defined substations need their own dedicated frameworks and can’t just reuse networking ones for routing energy. Rather, we may want to take a closer look at how modern-day SDN came to be over the past twenty years or so. There are probably many lessons to be learnt from it and ideas to guide progress in software-defined substations (and material for more blog posts…).

Substation networking

The second aspect is that substations are built on top of computer networks. As such, SDN can totally be used to define their topologies. It’s not directly about how to route electricity any more but how to set up the control systems that do. In this respect, it is clearly related to data centres. One might even argue that a digital substation contains a small, quirky, dedicated data centre. Above all, network security is the most important aspect as digital substations become exposed to the Internet. Control logic gets run in isolated VMs on top of specialised hypervisors and standard security rules can be enforced when exposing them to the Internet: firewalls, encryption, authentication etc.

💡 Should networking, energy flows or both be software defined?

Duck the GOOSE

Still, some specific features are only found in digital substations. The main one that comes to my mind is the concept of low-latency events with the IEC 61850 GOOSE specification. To guarantee delivery time, it bypasses the TCP, UDP and IP parts of the network stack (i.e. OSI layers 3 and 4) to directly talk at the link level (layer 2) using MAC addresses. While this of course gives fine-grained access to the infrastructure, it comes at a high cost: no IP routing tables, specific security considerations and non-standard raw communication software sockets as per the libiec61850 implementation. Note that security might be simplified and even strengthened by skipping the IP layer, but I fear it’s not that simple when it comes to preventing escalation once an intrusion has occurred.

At any rate, I was quite surprised to learn about this and I’m sure it would also make other software engineers raise an eyebrow. For example, to the best of my knowledge, this doesn’t happen in the automotive industry which has similar challenges. Instead, a dedicated CAN bus is used to transfer sensor and actuator messages which translate easily to UDP for high-level applications such as car dashboards. A real-time operating system or RTOS such as QNX (proprietary) is typically used for handling the logic of critical functions. Why not use something similar in digital substations? Say, a CAN-like bus over fiber optics? Maybe it’s just down to practical considerations. Still there seems to be a disconnect with the wider standard IEC 61508 about the safety of electrical systems, but again I’m not an expert in that field. Then I found out about the IEC 61850-90-5 extension which basically does GOOSE on top of UDP or TCP, known as R-GOOSE as it’s routable to other IP networks, without the same time-critical considerations which puts it back in SDN land.

Fun times ahead

These are all insights into the hard problems of digital substations and I’m only scratching the surface. There’s no doubt that every technical design decision was rooted in sound expertise. I’ve tried to highlight the parts that I found to be the hardest to grasp and more generally speaking, how a more open-source culture could be fostered. We’ve seen how the Linux kernel really became probably the most successful open-source project, by allowing anyone to take part. How an unknown contributor ended up writing the basis for IPv6 support or how reverse-engineering enthusiasts have shifted entire business models. Acting on the same plane as major corporations, universities and everything in between is really what the free software philosophy is all about - beyond the plain distribution terms outlined in the license.

This week I’m attending my second LF Energy Summit Europe. It’s a fun, welcoming event and a great way to get introduced to the ecosystem. I might find some answers to my questions and probably will come up with many more. That will give me yet again plenty of things to cover for the third and last post of this series. Wrapping this one up now on my way to Aachen, traveling entirely by train, is a great way of immersing myself into the Energy world.

- https://gtucker.io/posts/2025-09-09-digisub-pt2/ - 2021-2026 Guillaume Tucker

Digital Substation Pt.1 - Groundwork

Mon, 07 Jul 2025 00:00:00 +0000

gtucker.io https://gtucker.io/posts/2025-07-07-digisub-pt1/ -

Electricity has been a well-known commodity for over a century. When I was a student learning electronics and power applications in the early 2000s, renewable energy sources were very scarce. Generators could be found in off-grid locations and while some research was underway, not enough funding was being allocated to enable large-scale deployments. My main motivation when I entered university was to help reshape how energy is being produced in a more sustainable way, but it really was an uphill battle. So I focused more on my second passion which is open-source software.

Fast-forward twenty years or so: the temperature is still rising and the world is changing fast. In the meantime, I have acquired experience in embedded electronics, software in general and open source in particular. One of the reasons for choosing to work as freelance was to dedicate some time and effort to learning more about renewable energy as the electricity industry is now undergoing a major transformation. It was a pleasure to attend FOSDEM 2024 and spend a day at the Energy Devroom to discover so many high-profile initiatives. As this is a large open-source software conference, it’s no surprise that many in the room were also new to the energy topic which is rooted in hardware and power electronics.

First post of three

While discussing things with others at FOSDEM and then later at the LF Energy Summit, I accepted the challenge when Boris suggested to me that I write a blog post about Digital Substations and all the software that goes with it - here we are today, and thank you so much for reviewing it! In particular, it’s really important to spread the word about this emerging sector which hasn’t yet fully matured its open-source culture. And it’s a fascinating journey for me to join the dots between my academic knowledge in electronics and my experience in software. I’m sure many others will enjoy taking part in shaping the future of energy too, the open-source way.

Then I realised it wouldn’t fit in a single blog post, so I decided to break it down into a series of three. This first one is to set the stage. The next one is going to be about how to contribute and get on board as a software developer. The last one is to go back to real-world deployments and look into modernising their equipment and workflows. The huge inertia involved with electrical grids means it can show a much higher resistance to change than pure software products. A useful parallel is how data centres have evolved using software-defined networking so we’ll touch on this topic too. Now, let’s dive in.

Traditional Substations

An electrical substation is basically a transformer with switches. Its main role is to route high-voltage electricity coming from power plants via transportation networks and distribute it to low-voltage consumer households. One of the critical missions involved is to balance the flow of electricity between producers and consumers to avoid surges or voltage drops in the grid. This isn’t a new problem and traditional substations have been dealing with it since the early days of electricity. However, things are changing radically now for two main reasons: renewable energy sources are more diverse and electricity consumption keeps increasing.

Classic substation model

Solar energy production can vary widely based on sunshine, and the same goes for wind and water turbines. Generally speaking, these sources are smaller and more distributed than traditional large-scale power plants based on oil, coal or nuclear. As such, the grid is becoming more complex and backup generators or spinning reserves which typically rely on gas are often required to ensure a continuous supply. The substations have to quickly switch generators on and off the grid accordingly.

Historically, electricity was mostly about light and industrial applications. Electrical appliances are now much more widespread within domestic households and the transition to renewable energy means more electrical vehicles need to be charged. On top of this, data centres powering the backbones of the Internet and Artificial Intelligence are growing exponentially. It’s very likely that some breakthroughs will make them a lot more efficient such as quantum computing, but it’s reasonable to assume that energy needs aren’t going to decrease over the coming years. Nuclear fusion is also probably not for this decade although it has been making a few giant leaps forward recently, and AI may help it progress further. Having small, distributed reactors is seen as a promising way forward to build a more reliable infrastructure. So in any case, electrical grids aren’t getting any simpler.

The traditional substation has now become a limiting factor to the expansion of new energy solutions. To address these growing challenges, they are gradually being replaced with their digital counterparts.

Digital Substations

While essentially based on the same electrical hardware with transformers, circuit breakers and switches, the digital substation differs greatly in how it operates and how it controls its routing. Traditional substations rely on hardware sensors and actuators connected with copper wires and have a fixed, basic control logic. In the digital world, sensors are IoT devices and communication is done over Ethernet or fibre optic. One connection carrying all the data and control commands replaces many wires with analogue voltages. Flexible algorithms and AI models replace hard-wired circuits to enable the more advanced functionality required in a modern grid.

First introduced in the early 2000s, which happens to coincide with my university years although I didn’t learn about them until recently, nearly 50% of substations world-wide are now digital. Their adoption keeps growing, especially in China and the most developed countries. It’s expected that by 2030, all newly built substations would be digital.

Standards

Ensuring interoperability is paramount to the success of digital substations and electrical grids in general. As of today, the main standard is IEC 61850 which is a very large technical document. A key section of the specifications is how services running in substations should communicate with each other. A related initiative is the Open Platform Communications foundation which is more about standard tools whereas IEC 61850 describes the protocols. From a classic open-source software point of view, one may think it would be interesting to start by looking at the Substation Configuration Language as this is a software-like description of how a substation is meant to operate. However, it’s not that simple. It’s actually pretty hard to find a full SCL example, even for illustrative purposes. The best public thing I’ve found so far is the IEC_61850_SCL_2_Nodeset repository which is 10 years old. Here’s a snippet from a file describing a protection relay:

<?xml version="1.0" encoding="utf-8"?>
<SCL xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xsi:schemaLocation="http://www.iec.ch/61850/2003/SCL SCL.xsd" xmlns="http://www.iec.ch/61850/2003/SCL">
  <Header id="P145" version="1.94" revision="" toolID="AREVA IEC61850 Px40 Modelling Tool" nameStructure="IEDName">
    <Text source="">IED Capability Description for P145 Feeder protection relay (based on P143 with XCPU2)</Text>
  </Header>
  <Communication>
    <SubNetwork name="NONE" type="8-MMS">
      <ConnectedAP iedName="P145_CLT" apName="AP1">
        <Address>
          <P type="IP">144.58.243.226</P>
          <P type="IP-SUBNET">255.255.255.0</P>
          <P type="IP-GATEWAY">144.58.243.1</P>
        </Address>
[...]
        <PhysConn type="Plug">
          <P type="Type">10BaseT</P>
          <P type="Plug">RJ45</P>
        </PhysConn>
      </ConnectedAP>
      <BitRate unit="b/s" multiplier="">100</BitRate>
    </SubNetwork>
[...]

Basically, these XML formats aren’t meant to be used directly by humans. That’s why there are so many tools around to manipulate them with GUIs which is also a typical part of the MS Windows corporate culture. Ergonomic tools are great, and having them based on configuration text files that humans can comprehend is even better. Being able to work on this without having to pay for a full IEC 61850 training would help open things up to the rest of the world. This is one area where the open-source community has thrived and where the electrical industry can improve. From my naive developer point of view, I would have expected a friendly YAML or even TOML intermediate format with command line tools to manipulate them and generate the XML for the machines. I guess this is something to explore further in the next blog post.

Tools

To be fair, there are actually many great open-source modern tools in the energy sector such as OCF’s Quartz Solar Forecast or Santiago’s GridCal. These typically live outside of the electrical grid infrastructure, to process solar data or simulate a grid as per the examples above. Let’s clarify the scope here and focus only on substations for now.

The Linux Foundation Energy umbrella project hosts a number of open-source tools and solutions in this area. In particular, CoMPAS is directly about IEC 61850 tooling and SEAPATH provides a real-time hypervisor for hosting virtualised digital substations. However, these haven’t generated too much interest from the wider open-source software community so far. There is a huge unresolved potential there. The XML file issue I mentioned earlier is just one small symptom of the general status quo, with MS-DOS batch scripts and proprietary tools used internally by electrical corporations.

In the day of open-source hardware boards e.g. OwnTech and CPUs with RISC-V as well as Linux servers powering the majority of data centres, wouldn’t it be great to have electrical grid workflows running on FOSS and fully open digital substation designs?

Challenges

Every digital system carries cybersecurity risks. It’s quite hard to hack into a traditional substation, in fact it can be very dangerous for a physical attacker given the high voltage hazards involved. Within a connected world, a substation then becomes a lot more vulnerable and a prized remote target for terrorists or strategic military operations.

Security is another area where it can learn from other software-oriented industries. A number of things have already been put in place such as virtualised substations and containers. This appears to be developing quickly and hopefully it will also pick up some of the momentum from other applications that rely heavily on open-source software. It appears to be an obvious way to bridge that gap and attract engineers familiar with these issues to the energy realm. Still it will probably require some amount of cultural shift from major players. It took years for Microsoft to fully embrace open source software, time will tell whether the electricity sector will be able to follow a similar trend.

Data Centres

The Open Compute Project is a typical example of a successful transition in another type of infrastructure: data centres. Coupled with software-defined networking, the hardware routers, switches and firewalls have been replaced with virtual machines decoupled from their underlying hardware which are all standard servers. This means that the network topology can change dynamically and continuously to match evolving needs and adjust when the underlying hardware is being upgraded with minimum downtime. How can we do this with substations?

Transformers are based on a very basic principle with multiple coils on a same magnetic core. As such, they would seem much less likely to go through upgrades than computer hardware. This is true to a large extent as many traditional substations have been operational for decades with a minimum amount of maintenance. The trouble is when the grid starts changing with new power sources being connected and an increasing demand in energy consumption. Expanding an existing substation or replacing parts of it to address these changes also means updating the way it operates. Being able to do it in a continuous way and with the ability to dynamically adjust the topology and energy profile to route electricity differently in real-time is what is at stake here.

Networking is about routing bits of information, which are effectively tiny amounts of energy. Here we have a similar problem at a different scale, with a lot of energy and a small but critical amount of information to orchestrate it. In fact, data centres and digital substations are closely related already. Some compute applications require their dedicated electricity generator: AI, Bitcoin mining (putting aside its unbearable carbon footprint for a moment), military deployments and large technology campuses such as Samsung Digital City.

Chaotic compute resource usage also impacts how large amounts of electricity get routed and consumed. There are many sensors in a data centre to monitor temperature, humidity, power supply voltages etc. which is also the case in a substation. In the end, new technologies require new ways of managing energy and new energy grids borrow ideas from these new technologies. In the same way that a motherboard has its own CPU voltage regulators, one might imagine how a digital substation could merge with a data centre to create an optimal architecture.

Taking part ⚡

Where shall we go next? In particular, as we’re dealing with a huge industry, how can one embark into this adventure and start contributing in a way that is both useful and fulfilling?

In the next blog post, we’ll take a closer look at software-defined networking and its related tools to compare them against digital substations and see what could be done there. We’ll also take a deeper dive into the existing tools backed by LF Energy and other open-source projects that have already gathered some community momentum. Maybe this will prove me wrong on some of the concerns I raised here if some burgeoning solutions already exist and also give us the key to improving things like SCL file handling and foster open-source adoption as a whole.

In the meantime, do feel free to reach out as always with questions, feedback and ideas by email.

- https://gtucker.io/posts/2025-07-07-digisub-pt1/ - 2021-2026 Guillaume Tucker

First year in Freelance

Thu, 27 Feb 2025 00:00:00 +0000

gtucker.io https://gtucker.io/posts/2025-02-27-first-year/ -

You may be starting off in freelance yourself or just curious to know what it means. Here’s the story of my first year with some lessons learnt and a glimpse into what might be coming next.

It was sunny this February at FOSDEM as per the picture above. It reminded me of when I was there last year during my early days as self-employed and the long trail of events I have been through since then.

Ready, Set…

It all started when I left Collabora at the end of 2023. While I had been contemplating such a move for a while, there was no way I could really find the time and space to be ready for it while having a full-time job. Building a business model and getting all the necessary things lined up are simply part of freelance itself and can’t be completed beforehand. The key thing is to become convinced of your potential and how it should allow you to strive in a particular environment. It is also crucial to take some risk mitigation measures and prepare the ground: trainings, savings, fallback scenarios, side jobs, living cost reduction alternatives etc.

Another very, very important goal is to properly wrap up any activity you need to stop. In my case, I had a long notice period and I was working on KernelCI which is an open-source Linux Foundation project. As such, I first focused on internal handover duties with my colleagues and then had no real deadlines left for completing the community side of things. In practice that meant making my own timeline for transferring responsibilities and maintainer roles to other people as well as helping find a new Chair for the KernelCI TSC.

Go!

Day 0 is when any previous employment ends and a new journey begins. The amount of time and mental space it instantly brings can be a blessing and a curse. Unless you have already found some early customers, the apparent calm could easily become a storm of anxiety. Although everyone will get a different experience from it, I believe the best approach is to put healthy routines in place and keep moving forward. There’s a fine line between taking your time to reflect constructively and becoming idle. It’s a great challenge and an opportunity to expand your self-motivation which can only make you stronger.

Each individual will need a tailored solution. I found it to be a good time to brainstorm and gather as many ideas as possible about places to go next. While planning won’t make any of it actually happen by itself, it’s a productive way of being prepared for when an unexpected opportunity arises. Having worked for several startups before, I felt some commonalities in the mindset in terms of being very reactive and flexible. Still, the aim is a different one since it’s bound to be a solo journey and the scope can be much wider than having a laser-focused pitch to get a product or business idea over the line. And finally, to balance things out against free-flowing inspiration, it’s crucial to have checklists to go through and keep making concrete progress every single day.

Let’s take a quick look at a handful of items that paved my way here:

1. Admin

An obvious but necessary initial step is to adopt the most appropriate self-employed status for the work. This will vary widely between countries and depends on the nature of your business model. In my case I opted for the micro-entreprise framework which is a common choice in France with simplified procedures.

Other admin duties include bookkeeping, insurance, banking, taxes, having a dedicated company address to receive mail when working from home and lots of other small things along those lines. Overall, I don’t think there should be any major hurdle or hard decisions to take in doing this. It can be quite time consuming at first, then it should become easier by putting a good process in place to avoid repeating the same manual tasks over and over. Getting professional advice and support is money well worth spending too.

2. KernelCI handover

Throughout the first quarter of 2024, I had to regularly keep in touch with the KernelCI community and attend meetings in order to complete the transition. I eventually opted to simply remain a member of the TSC to contribute my past experience and bring some variety following my move.

To conclude my direct involvement in the project, I was invited to give an online talk as part of the Linux Foundation Mentorship sessions: KernelCI Travel Guide 2024. This was a good way to assess the project’s situation and for me to consider my next steps as an independent contributor. I then ended up making a fork of the code to continue pushing its design further with a more general-purpose automation tool - more on that in an upcoming blog post!

3. Energy

When I started my MEng degree in electronics, I had to complete a survey which included a question about future work aspirations. I already knew I wanted to work in the energy sector and especially in sustainable ways to generate electricity. There were of course already a few things going on in this field about 20 years ago but it was mostly research or anecdotal projects. The technology wasn’t there yet as it hadn’t been receiving that much investment and attention. Things have changed drastically since then with the electric vehicle revolution and a raising awareness of the climate emergency.

This is when having a chance to make a career turn can be useful for realigning the nature of your work with your true aspirations. It might feel like something you should have embraced earlier but it’s never too late to make a change. Years of experience in any domain will always be valuable as a unique personal strength. In my case, I could refresh my power electronics academic knowledge and build upon the open-source expertise I have acquired. The Energy devroom at FOSDEM 2024 was a major event for the industry but also on my personal journey as I could see new horizons opening up and engage with an emerging community. It simply joined all the dots.

4. Consultancy

Leaving all the new avenues I may be able to explore on one side, the overarching goal is to provide consultancy services. As such, I felt I needed to rely on the full range of knowledge I had gathered through life to put all the odds on my side. Signing the first contract is a fantastic milestone as it’s a tangible realisation of what was once nothing but a faint, abstract idea. This is also when the vulnerability of leaving the comfort of a nice and stable job in a privately-funded company brings true gratification.

Keeping a sustained source of income is the real deal though as that proves the viability of the whole enterprise. I can’t stress enough how grateful I am to have been given such a wonderful chance to achieve this. Without further ado, let me introduce you to…

Source.dev

Some might say that timing is everything. I would argue that it’s akin to how a photographer would dedicate years to eventually know how to spontaneously take a truly great snapshot at an unpredictable time and place. Stepping into self-employment with the agency that it brings automatically led me to seize the day in unexpected ways. That is when I renewed contact with Serban, a former colleague of mine from my days at ARM who was about to launch a startup with David whom he had met at Google. It was Summer 2024 and the beginning of Source.dev.

In short, the main goal is to facilitate Android product development and long-term support for device ecosystem companies, from OEMs to chip providers. This also enables them to deal with new regulations aiming to extend consumer products’ lifespans. It is all made possible thanks to the unique expertise accumulated over the years by the Source.dev co-founders.

Had I not published a blog post about my professional move, I probably would never have appeared on their radar. Also, I privately hold the belief that what might seem like an anecdotal, long-forgotten event played a key role in this. Serban once asked me to port some micro-benchmarks from Javascript to Java after randomly bumping into me at a Linaro Connect conference about 10 years ago. I completed this for an MD5 use-case on the same day with my lightweight ARM Chromebook in a hacking room and it definitely had a positive impact on our mutual trust. Spending several years bringing KernelCI to maturity in the meantime obviously also helped a lot.

All these things are intertwined as the world is a continuum. Regardless of the actual causes, the outcome is that I am now contributing my fair share to this high-profile customer with an ongoing part-time contract.

Momentum

Once things have been set into motion, it’s all about fostering that momentum and being always on the lookout for new developments. Having a stable revenue with strong customer relationships makes it possible to carve out some time and space for this. It’s also a way to reconnect with former colleagues and extend your network. It helps to feel part of something bigger since humans are social animals and as a result, it will naturally generate new business leads too.

I am now looking forward to discovering what lies ahead in 2025. Among all the hardship throughout the world and global uncertainties, it’s important to be able to focus on what matters at an individual scale. Finally, feel free to reach me via email or LinkedIn as I would love to know your thoughts and your own stories too.

- https://gtucker.io/posts/2025-02-27-first-year/ - 2021-2026 Guillaume Tucker

kernel.org containers

Mon, 30 Sep 2024 00:00:00 +0000

gtucker.io https://gtucker.io/posts/2024-09-30-korg-containers/ -

Many moons ago, some discussions were being held around having common container images with toolchains maintained by the upstream Linux kernel community. It didn’t quite happen back then, TuxMake was launched by Linaro soon after with images forked from the KernelCI ones and they have been diverging ever since. Where do things stand now?

A renewed interest

There does seem to be a trend forming in 2024. The debate around using GitLab CI for testing the upstream kernel has been revived and TuxMake has now started supporting containers with the kernel.org toolchains. It looked like all the stars were finally lining up so I picked this topic up again and took the time to articulate it with a proposal: first as an email discussion followed by a Linux Plumbers talk. Both have been well received with some constructive feedback and ideas. Concrete steps are now forming down that path; let’s take a look at a potential way forward.

Yellow Brick Road

Lots of things have already been done to improve development workflows and quality control for the Linux kernel. Some of them are directly available upstream and have become second nature such as Git, test frameworks (kselftest, KUnit) and HTML documentation. As the software world keeps turning, new tools become de facto standard among all developers and the upstream kernel community is no exception - within reason, of course. One of the most obvious ones are containers: Docker was voted #1 used tool on the 2023 Stack Overflow survey and again in 2024.

How can the Linux kernel benefit from containers? Most people are using them to run builds and tests in reproducible, controlled, containerized environments. If an automated test system finds an issue, having a consistent way for developers to reproduce it under the exact same conditions is paramount when looking for a fix.

This also applies to cutting edge kernel development, for example with features in linux-next moving hand-in-hand with particular compiler changes - typically Clang and more recently Rust, but also RISC-V support and new architectures among other topics. It’s also a neat way of working with multiple compiler toolchain versions.

The broader concept here is to create a continuum between individual developers and automated services. For toolchain containers, this can be achieved via a number of small steps.

📦 Packages

A rather straightforward improvement would be to create standard packages with the kernel.org toolchains e.g. .deb, .ipk or .rpm in addition to the existing tarballs. The main benefit would be to facilitate adding or removing them while taking care of dependencies automatically. The LLVM toolchains page provides an informal list of things to be installed manually which could typically be formalised with regular packaging meta-data.

It would of course take a bit of maintenance work to keep them alive as well as some additional storage space if several formats were applied to all the packages. Maybe only the most popular ones could be maintained initially and then coverage would expand as needed. Actually producing the packages can be easily achieved using tools such as Open Build Service.

This is somewhat an optional step as all the subsequent ones can be done regardless; having packages would just make things a bit easier.

🚢 Containerfiles

Container images are created using Containerfiles, or Dockerfiles. Since the OCI has defined an open and authoritative image specification, there is no vendor lock-in when using containers. We’ll just refer to Containerfiles in a generic way going forward.

Proof of Concept

As an experiment to see how viable this would be, I’ve started a kernel.org containers repository which can build an initial set of images. They include x86 host toolchains with GCC 14 for arm64 and x86_64 targets as well as Clang 18 from the kernel.org toolchains. The images are automatically built using GitLab CI and stored in the project’s container registry. For example, to use the default gcc image from within a kernel source tree:

$ docker run -it -v $PWD:/home/kbuild/linux -w /home/kbuild/linux \
    registry.gitlab.com/gtucker/korg/gcc \
    make defconfig bzImage -j$(nproc)
[... pulling image the first time ...]
[... HOSTCC ...]
*** Default configuration is based on 'x86_64_defconfig'
#
# configuration written to .config
#
[... build goes on ...]
  BUILD   arch/x86/boot/bzImage
Kernel: arch/x86/boot/bzImage is ready  (#1)

A full kernel can be built although the dependencies for any extras aren’t included yet (kselftest, perf, documentation etc.).

To rebuild the images from scratch locally and have your own tags:

$ git clone https://gitlab.com/gtucker/korg.git
$ cd korg
$ make PREFIX=korg-
  BUILD   korg-toolchain-base
  BUILD   korg-clang:18.1.8-x86
  TAG     korg-clang:18
  TAG     korg-clang
  BUILD   korg-gcc:14.2-x86-x86
  TAG     korg-gcc:14
  TAG     korg-gcc
  BUILD   korg-gcc:14.2-x86-arm64
  TAG     korg-gcc:14-arm64
  TAG     korg-gcc:arm64

Then the previous example becomes:

$ docker run -it -v $PWD:/home/kbuild/linux -w /home/kbuild/linux korg-gcc make defconfig
*** Default configuration is based on 'x86_64_defconfig'
#
# No change to .config
#

Each image is a combination of host architecture, target architecture and compiler version. Only a subset of the kernel.org toolchains are covered for now as a proof-of-concept. It’s also possible to use Podman instead of Docker, see the project’s README for more details.

Upstream

The aim is to make upstream-friendly container definitions to prepare an RFC patch series and continue the discussion with the community on mailing lists. Typically, this could be added in tools/containers. Particular care has been taken in the following respects:

the Containerfile is vendor agnostic
image builds are managed with Make
external dependencies are kept to standard Debian (other distros may be used)
only kernel.org toolchains are being installed

⚙️ Git Hooks

There are ongoing discussions around adding a gitlab-ci.yml file upstream, and the proof-of-concept repository uses GitLab CI to automatically build and push Docker images. However, this is unlikely to meet broad adoption as things currently stand. A more community-oriented approach would be using Git hooks: they’re vendor neutral as purely based on Git, optional and up to each maintainer to enable as they want in their own trees.

I went ahead and made an experimental post-receive hook which blocks the client on a git push while images are being built. This isn’t quite good enough for the real world, it should instead be starting a task in the background with maybe an email reply when it’s done, but it’s the most basic way of wiring things up and ItWorks™:

$ git push
Enumerating objects: 67, done.
Counting objects: 100% (67/67), done.
Delta compression using up to 8 threads
Compressing objects: 100% (59/59), done.
Writing objects: 100% (67/67), 12.69 KiB | 764.00 KiB/s, done.
Total 67 (delta 26), reused 0 (delta 0), pack-reused 0
remote: Building Docker images [6852976572e37c9b7e340f65155d9190c32c9f31]
remote: 6852976572e3 Makefile: rename base to korg-toolchain-base
remote:   BUILD   korg-toolchain-base
remote:   BUILD   gtucker/clang:18.1.8-x86
[...more BUILD and TAG...]
remote: Pushing Docker images
remote:   PUSH    gtucker/clang:18
[...more PUSH...]
remote:   PUSH    gtucker/korg-gcc:latest

The logic of that hook could easily be wrapped into another runtime environment, for example with a Docker-in-Docker container which is what the GitLab CI pipeline does. Even simpler, it could just be started as a detached process although this of course has some security implications. It could also start a pipeline somewhere, for example in Tekton. Each kernel maintainer or developer could pick and choose what works for them.

🗄️ Container registry

Building upon the concept of running Git hooks to automatically update container images, the next logical step would be to set them up on git.kernel.org with either a dedicated repository or in kernel trees if merged upstream such as mainline, linux-next and some maintainer trees. There would then need to be an appropriate registry where to store these images. Still keeping in mind the same concerns of avoiding vendor lock-in and using standard tools, hosting a dedicated registry.kernel.org would seem to make more sense than relying on Docker Hub or GitLab. A popular solution for doing this would be Harbor. The container registry protocol is also specified by the OCI so other alternatives may be adopted seamlessly.

🪄 Makefile

This last step was motivated by an important point which Nathan raised as part of the initial email thread.

On Mon, Jul 08, 2024 at 22:30:31AM -0700, Nathan Chancellor wrote:

Having first party Dockerfiles could be useful but how would they be used? Perhaps building a kernel in such a container could be plumbed into Kbuild, such that the container manager could be invoked to build the image if it does not exist then build the kernel in that image? This might be a lofty idea but it would remove a lot of the friction of using containers to build the kernel so that more people would adopt it?

I haven’t looked into actually building the container image directly as a side-effect of running make. However I did some experiments to wrap the build in a container by providing the image name as per this patch:

0001-RFC-Makefile-wrap-in-container-if-CONTAINER-is-defin.patch

which is also available on GitLab.

For example, from within a kernel tree:

$ git fetch https://gitlab.com/gtucker/linux.git linux-6.11-make-container --depth=1
$ git checkout FETCH_HEAD
$ mkdir -o build/x86
$ make CONTAINER=registry.gitlab.com/gtucker/korg/gcc:14 O=build/x86 defconfig
Running in docker registry.gitlab.com/gtucker/korg/gcc:14
Unable to find image 'registry.gitlab.com/gtucker/korg/gcc:14' locally
14: Pulling from gtucker/korg/gcc
[...]
Digest: sha256:928a79af1de7550ffa1e0955177c6ef11b6a2b49334d4bced3f41c60b37a31e2
Status: Downloaded newer image for registry.gitlab.com/gtucker/korg/gcc:14
make[1]: Entering directory '/src/build/x86'
  GEN     Makefile
  HOSTCC  scripts/basic/fixdep
[...]
*** Default configuration is based on 'x86_64_defconfig'
#
# configuration written to .config
#
make[1]: Leaving directory '/src/build/x86'

Then after that it’s kernel building as usual:

$ make CONTAINER=registry.gitlab.com/gtucker/korg/gcc:14 O=build/x86 bzImage -j$(nproc)
Running in docker registry.gitlab.com/gtucker/korg/gcc:14
make: warning: -j2 forced in submake: resetting jobserver mode.
make[1]: Entering directory '/src/build/x86'
  GEN     Makefile
[...]

If you’ve built the container images locally or have your own tags, you may even reduce the example to its most minimal form:

$ make CONTAINER=korg-gcc defconfig
$ make CONTAINER=korg-gcc

There are of course already a few known issues with this:

docker -it uses CRLF DOS line endings

Dropping the interactive mode should be possible, this was a quick way of propagating Ctrl-C to the container.

uid may not be the current user (1000)

A user id mapping or namespace would be needed to address this.

The source tree is mounted directly

Preventing containers from modifying the source tree would be better, typically with overlays.

The if / else in the top-level Makefile is rather invasive

Having a separate Makefile would mean zero changes when not using containers.

Other glitches

There are a few more rough edges: using the -j option causes a submake warning, only one target can be specified per command line (e.g. no bzImage modules) and interrupting the build with Ctrl-C quite doesn’t work reliably (related to the docker -it issue).

Still, it’s an interesting step in facilitating adoption of upstream-supported build containers. The next question is going to be whether this may fly and if it’s worth pursuing efforts accordingly. Having the actual container images available would seem like a prerequisite so I’ll work on that first.

Going Further

While mentioning these ideas to people at Plumbers, I thought I might meet some resistance as this could seem a bit too disruptive. In fact, the opposite happened and all I got was encouragement and more ideas to go even beyond builds. For example, adding container images with QEMU to quickly test the kernel images (how about make boot CONTAINER=korg-qemu:x86 then?) and others with test dependencies pre-installed such as kselftest and in particular for the eBPF special case.

We’ll see where this takes us, probably there are going to be many more aspects to take into consideration but the fun is in the journey. Looking forward to some follow-up email discussions, and feel free to reach out directly on gtucker@gtucker.io as always.

- https://gtucker.io/posts/2024-09-30-korg-containers/ - 2021-2026 Guillaume Tucker

micro-entreprise

Mon, 06 May 2024 00:00:00 +0000

gtucker.io https://gtucker.io/posts/2024-05-06-micro-entreprise/ -

Spring is here. After a long wave of April showers, I’m now very pleased to announce that the garden plot I was allocated a month ago is going well and a few shoots are starting to show up. Equally importantly, I’ve finally established the framework for my upcoming work initiatives. It’s a steep climb, almost vertical, although a very rewarding one. Steady progress eventually leads to new heights and can open new horizons.

Overview

Essentially, micro-entrepreneur is a French flavour in the self-employed category with simplified procedures. So I’m open for business, officially since 1st May 2024. That’s for the status, but what about the actual work?

First of all, I’ll carry on doing open-source software development in the foreseeable future.

Secondly, I’m kickstarting some R&D in the field of renewable energy which should start bearing fruit over the coming months.

Lastly, a few extra activities may also come into play such as designing hardware kits, running workshops and education.

I’d like to take the opportunity here to include a section in French since I now live in Tours. It’s basically a summary of the English parts.

En Français

Jardin familial / Allotment plot

Après avoir vécu pendant quinze ans au Royaume-Uni, j’ai découvert beaucoup de choses changées à mon retour en été 2022. Je ne conaissais pas ameli.fr, j’ai dû faire beaucoup de démarches souvent lentes, manuelles et via divers organismes isolés. J’ai pourtant eu la chance de pouvoir garder mon emploi à distance avec mon employeur, ce qui m’a grandement simplifié la tâche. Malgré tout, j’ai finalement décidé de franchir le cap et commencer ma nouvelle activité en freelance ce 1er Mai 2024.

Il y a de quoi écrire de longs articles sur les différences de cultures ainsi que sur les procédures administratives. Je me limiterai ici au fait que le prix du vin en France est comparable à celui de la bière en Angleterre, et à une liste des étapes principales pour démarrer une entreprise individuelle: choix du statut, solution de domiciliation, définition des activités & business plan, formulaires INPI, assurance RC Pro, compte bancaire…

Mais venons-en au sujet principal. Prendre le temps de faire une pause profesionnelle permet de revoir ses priorités et redonner du sens à son activité. C’est aussi l’occasion de rétablir un équilibre avec sa vie personelle, surtout après un changement important de situation. Quelles conclusions en ai-je donc tirées?

Il me semble qu’être prêt à s’adapter aux changements rapides est devenu la priorité. Cela se traduit dans mon cas à choisir certains domaines et y déveloper une base pour catalyser des opportunités futures - dont certaines déjà en vue. Concrètement, après vingt années de dévelopement logiciel et particulièrement du libre, il est naturel que je poursuive dans cette voie: noyau Linux, devops, Web3… Entre temps, une passion qui n’a pas cessé de m’animer depuis mes études d’ingénieur en électronique se retrouve maintenant en plein essor: les énergies renouvelables. C’est donc mon second terrain d’action, plein de potentiel comme j’ai pu en témoigner à FOSDEM.

Une autre façon de valoriser son expérience est de la partager, en particulier par le biais de l’enseignement: interventions en milieu universitaire, ateliers, tutoriels… C’est aussi un bon moyen en tant que travailleur indépendant pour créer des connexions et être actif dans une communauté, tant à l’échelle locale qu’à distance. Un atout supplémentaire dont je bénéficie est la possibilité de le faire en anglais.

Strategy

Of course, it’s hard to predict exactly how things are going to pan out. The overall strategy at this stage is to focus on a number of particular areas to prepare the ground for longer-term opportunities - with a couple already in sight.

Taking the time to slow down during a career break makes it possible to reflect upon personal priorities and renew a sense of purpose. This is true for every aspect of someone’s life such as family and health, but let’s focus on work for now. Seven years ago, a similar approach led me to invest fully into open-source software when I joined Collabora. The stakes are very high today with so many disruptive events reshaping the world. Renewable energy is finally receiving massive amounts of interest while artificial intelligence is also taking off exponentially. How to deal with such macroscopic things as an individual?

Software

Given my past experience in open-source software consultancy, it’s only natural that I keep this momentum going. The big difference now is that I can open up new avenues. To start with, I’ll resume actual Linux kernel development and most likely extrapolate from some of the devops automation work I did around KernelCI to other use-cases. I’m also fully learning the Rust language, diving deeper into machine learning technicalities and investigating Web3 technologies such as Polkadot.

Software is everywhere though, and the next main application domain I want to focus on is energy.

Photovoltaic harvesting devkit

Energy

Exactly twenty years ago, in 2004, I was answering a particular question while filling an entry questionnaire for my MEng electronics degree about career aspirations. In my case, it was all to do with renewable energy and contributing to a more sustainable future. While it was probably seen more as a niche or nerdy activist idea back then, we’ve now come a long way.

It’s so exciting to be able to put into practice all my dormant hardware skills on top of the software ones: power electronics, automation, sensors, signal processing… And since open-source concepts have now become mainstream too, as witnessed at the FOSDEM Energy devroom there’s a lot to be done both in software and hardware to reinvent this century-old industry with open standards.

Likely topics for some future blog posts are going to be micro-grids, electrical substations, machine learning applied to energy production and transport, software tooling at every level of the stack and hardware kits. But that’s enough buzzwords for today.

Connecting the dots

Another important aspect of freelancing is communicating and engaging with social networks. While working remotely, some typical means to achieve this would be taking part in conferences and events which can be online or face-to-face, writing articles and creating educational material such as tutorials.

Other ways to reach out to new communities would of course be contributing technical value (code, design, discussions…) but also teaching and running workshops. One particular way for me to put my experience to good use in France would be via meetups in English around tech topics, and lectures.

As always, you’re more than welcome to get in touch and share any thoughts you might have via email.

Speak soon!

- https://gtucker.io/posts/2024-05-06-micro-entreprise/ - 2021-2026 Guillaume Tucker

Contact Details

Wed, 01 May 2024 00:00:00 +0000

gtucker.io https://gtucker.io/contact/ -- https://gtucker.io/contact/ - 2021-2026 Guillaume Tucker