Quality Engineering

The openQA test suite for openSUSE and SLE has a test module called tests/console/sudo.pm. It verifies that sudo works: passwords, shells, sudoers rules, environment isolation. Basic stuff. It runs tens of thousands of times per year and takes about 9 minutes each time. That adds up.

Where the time goes

There is no single bottleneck. The test uses expect to interact with password prompts. Every sudo call goes through credential cache reset, process spawn, password entry, and result verification. It does this 20 times because the test runs the full suite twice with slightly different sudoers configurations.

A few years ago I wrote a blog post about a small hackweek project called rdepends. The idea was simple: given a package, find out what other packages depend on it, recursively. It lived in my home project on the build service and it was a useful but rough tool.

Since then a few things happened that made me revisit it.

Zypper got the feature

Back when I started this project, zypper had no built-in way to ask “what depends on package X?”. You could ask the other direction easily with zypper info --requires, but not the reverse. I raised this with the zypper developers and Benjamin Zeller implemented the --requires-pkg flag in zypper 1.14.33. So now you can do:

In the first post I described the setup: funkoverage eBPF tracing, a podman container running openQA, and the first coverage report with 141 binary targets. That was the starting line. This is what happened next.

Daemon shimming

The biggest limitation in the first round was daemons. Services like sshd, cups, postgresql, and rpcbind couldn’t be shimmed because funkoverage’s wrapper broke systemd’s service lifecycle. Two specific problems: the shim used SIGKILL instead of forwarding SIGTERM to the child process, so daemons couldn’t clean up sockets on restart. And the shim didn’t relay sd_notify, so Type=notify services timed out on start.

Let’s be clear about what this is and what it isn’t.

A typical openSUSE Tumbleweed installation has roughly 2,600 ELF executables in /usr/bin and /usr/sbin (more or fewer depending on the installation pattern). Our test repository has 2,343 test modules. We measure coverage on 141 of those binaries using 98 of those test modules. That’s a fraction of the whole picture and we’re not claiming this is the total test coverage of the distribution.

Back when I was working at Canonical on the Ubuntu phone project, test-driven development was the default and natural way to work. One thing I noticed and genuinely liked was how Ubuntu and Debian ship functional tests alongside many of their packages. Not only build-time unit tests hidden in the CI pipeline, but installable test binaries that you can run against the actual packages on an actual system. They are named *-tests or *-test, and they live in the archive next to the production packages.

I gave a proprietary tool a fair chance - and was not disappointed

The views expressed in this post are my own and do not represent the position of my employer. This post is not a commercial endorsement of QASE or any of its products.

I very seldom write about proprietary solutions and software. Not because I have any problem with them, but simply because I usually prefer open source solutions, and during my daily work and hobby projects I use almost exclusively open source software.