One Dimensional Reversible Automata

I created a demo for the GFXPrim library. It implements and displays a nearest-neighbor, one-dimensional, binary cell automata. Additionally it implements a reversible automata, which is almost identical except for a small change to make it reversible. The automata is displayed over time in two dimensions, time travels from top to bottom. Although in the reversible case time could be played backwards.

A reversible rendition of rule 73

The automata works as follows:

• Each cell has a state, which is on or off, black or white, boolean etc.
• At each time step, the state of a cell in the next step is chosen by a rule.
• The rule looks at a cell’s current value and the values of its left and right neighbors.
• There are 2³ = 8 possible state combinations (patterns) for 3 binary cells.
• A rule states which patterns result in a black cell in the next time step.
• There are 2⁸ = 256 possible rules. That is, 256 unique combinations of patterns.

Rule 105

So a pattern is a 3 digit binary number, where each digit corresponds to a cell. The middle digit is the center cell, the high order bit the left cell, the low order bit the right cell. A rule can be display by showing a row of patterns and a row of next states.

Above is rule 110, 0x6e or 01101110. It essentially says to match patterns 110, 101, 011, 010, 001. Where a pattern match results in the cell being set to 1 at the next time step. If no pattern is matched or equivalently, an inactive pattern is matched, then the cell will be set to 0.

Again note that each pattern resembles a 3bit binary number. Also the values of the active patterns resemble an 8bit binary number. We can use this to perform efficient matching of the patterns using binary operations.

Let’s assume our CPU natively operates on 64bit integers (called words). We can pack a 64 cell automata into a single 64bit integer. Each bit corresponds to a cell. If a bit is 1 then it is a black cell and 0 for white. In this case we are using integers as bit fields. We don’t care about the integer number the bits can represent.

Rule 94 Reversible

The CPU can perform bitwise operations on all 64bits in parallel and without branching. This means we can perform a single operation 64 times in parallel.¹

If we rotate (wrapped >>²) all bits to the right by one, then we get a new integer where the left neighbor of a bit is now in its position. Likewise if we shift all bits to the left, then we get an integer representing the right neighbors. This gives us 3 integers where the left, center and right bits are in the same position. For example, using only 8bits:

Each pattern can be represented as a 3bit number, plus a 4th bit to say whether it is active in a given rule. As we want to operate on all 64bits at once in the left, right and center bit fields. We can generate 64bit long masks from the value of each bit in a given pattern.

So if we have a pattern where the left cell should be one, then we can create a 64bit mask of all ones. If it should be zero, then all zeroes. Likewise for the center and right cells. The masks can be xor’ed³ (^) with the corresponding cell fields to show if no match occurred. That is, if the pattern is one and the cell is zero or the cell is one and the pattern is zero. We can invert this (~) to give one when a match occurs.

To see whether all components (left, right, center) of a pattern matches we can bitwise and (&) them together. We can then bitwise or⁴ (|) the result of the pattern matches together to produce the final values.

Rule 193, inverted rule 110

If we wish to operate on an automata larger than 64 cells, then we can combine multiple integers into an array. After performing the left and right shifts, we get the high or low bit from the next or previous integers in the array. Then set the low and high bits of the right and left bit fields. In other words we chain them together using the end bits of the left and right bit fields.

For illustration purposes, below is the kernel of the the automata algorithm.

/* If bit n is 1 then make all bits 1 otherwise 0 */
#define BIT_TO_MAX(b, n) (((b >> n) & 1) * ~0UL)

/* Numeric representation of the current update rule */
static uint8_t rule = 110;

/* Apply the current rule to a 64bit segment of a row */
static inline uint64_t ca1d_rule_apply(uint64_t c_prev,
                                       uint64_t c,
                                       uint64_t c_next,
                                       uint64_t c_prev_step)
{
    int i;
    /* These are wrapping shifts when c_prev == c or c_next == c */
    uint64_t l = (c >> 1) ^ (c_prev << 63);
    uint64_t r = (c << 1) ^ (c_next >> 63);
    uint64_t c_next_step = 0;

    for (i = 0; i < 8; i++) {
        uint64_t active = BIT_TO_MAX(rule, i);
        uint64_t left   = BIT_TO_MAX(i, 2);
        uint64_t center = BIT_TO_MAX(i, 1);
        uint64_t right  = BIT_TO_MAX(i, 0);

        c_next_step |=
            active & ~(left ^ l) & ~(center ^ c) & ~(right ^ r);
    }

    /* The automata becomes reversible when we use c_prev_state */
    return c_next_step ^ c_prev_step;
}

To make the automata “reversible” an extra step can be added. We look at a cell’s previous (in addition to the current, left and right) and if it was one then invert the next value. This is equivalent to xor’ring the previous value with the next.

Rule 193 again, but reversible

It is not entirely clear to me what the mathematical implications are of being reversible. However it is important to physics and makes some really cool patterns which mimic nature. Also entropy and the second rule of themodynamics, yada, yada…

The automata definition is taken from Stephen Wolfram’s “A new kind of science”. He proposes at least one obvious⁵ C implementation using arrays of cells. He also provides a table of binary expressions for each rule. E.g. rule 90 reduces to just the l^r binary expression. It may be possible for the compiler to automatically reduce my implementation to these minimal expressions.

To see why, let’s consider rule 90 for each pattern.

Rule 90 reversible

01011010 = 90

First for pattern 000.

  active & ~(left ^ l) & ~(center ^ c) & ~(right ^ r);
= 0 & ~(0 ^ l) & ~(0 ^ c) & ~(0 ^ r);
= 0.`

Active is zero so the whole line reduces to zero. Now for 001. Note that 1 here actually means ~0UL, that is 64bit integer max.

   1 & ~(0 ^ l) & ~(0 ^ c) & ~(1 ^ r);
= ~l & ~c & r.

As expected pattern 001 matches l=0, c=0, r=1. Let’s just list the remaining patterns or’ed together in their reduced state. Then reduce that further. Note that the for loop in ca1d_rule_apply will be unrolled by the compiler when optimising for performance. It’s also quite clear that c_next_step is dependant on an expression from the previous iteration or zero. So all the pattern match results will get or’ed together.

  l & c & ~r | l & ~c & ~r | ~l & c & r | ~l & ~c & r;
= l & ~r | ~l & r;
= l ^ r.

See on the top row that (l & c & ~r | l & ~c & ~r) or’s together c and not c. So we can remove it. Then we get an expression equivalent to xor’ring l and r.

In theory at least, the compiler can see that rule only has 256 values and create a reduced version of ca1d_rule_apply for each value. Whether it actually does is not of much practical concern when the rendering code is the bottle neck. However it’s interesting to see if the compiler can deduce the best solution or whether anything trips it up.

Judging from the disassembly from gcc -O3 -mcpu=native -mtune=native, it may actually do this. Additionally it vectorizes the code packing four 64bit ints at a time into 256bit registers and operating on those. I don’t know which part of the code it is vectorising or how. It’s possible that what I think is the rule being reduced is something related to vectorisation.

Rule 210

To render the automata we take the approach of iterating over each pixel in the image. We calculate which cell the pixel falls inside and set the color of the pixel to that of the cell. That’s it.

/* Draws a pixel */
static inline void shade_pixel(gp_pixmap *p, gp_coord x, gp_coord y,
                               gp_pixel bg, gp_pixel fg)
{
    gp_pixel px;
    size_t i = (x * (64 * width)) / p->w;
    size_t j = (y * height) / p->h;
    size_t k = 63 - (i & 63);
    uint64_t c = steps[gp_matrix_idx(width, j, i >> 6)];

    c = BIT_TO_MAX(c, k);
    px = (fg & c) | (bg & ~c);

    gp_putpixel_raw(p, x, y, px);
}

GFXPrim makes drawing very simple. The above code is fast enough for my purpsoses, but a significant improvement can be had. Integer division is much slower than floating point multiplication on most newer CPUs. It’s actually much faster (2x at least) on my CPU to calculate a pair of ratios in floating point, then convert them back to integers.

However, you may ask why we are even drawing on the CPU in the first place? This is because GFXPrim targets embedded systems with no graphics processor. Additionally the CPU may not even support floating point natively. So integer division may actually be faster in this case. Still better would be to limit the size of the pixmap to be 2^x larger than the dimensions of the automata, where x ∈ ℕ then we can use shifts instead of division.

Rule 210 Reversible

Rule 105 Reversible

This is a reimplementation of my Bitbanging 1D Reversible Automata in WGSL so that it runs on GPUs inside a computation shader. If you are not familiar with 1D automata or bitbanging techniques then please see the original article.

WebGPU is still not available on some browsers at the time of writing. On my Linux machine I had to set --enable-unsafe-webgpu --enable-features=Vulkan on Chromium to enable it. However it worked out-of-the-box with Chrome on my phone and ancient Mac Book Air.

I’ve included some screen-shots throughout the article if you can’t get it work.

Automata viewer

Screen size: Mobile Desktop

Pan X: 0.0

Pan Y: 0.5

Zoom: 2

Rule: 105

Seed: 114322622 Value:

Reversible

If you find a particular rule doesn’t produce interesting output then try dragging the seed value to a negative value. This will populate all of the starting bit fields instead of just one.

You can find the JavaScript source here and the WGSL source here. In addition you can just open your browser’s developer console and get the files there. I didn’t use an build tools, it’s just vanilla JS.

WebGPU background

For some time I have wanted to embed the original automata viewer in my website. The original is written in C, so I thought I would try compiling it to WASM, but this wasn’t very fun.

At the same time I have been interested in learning to do something on the GPU and even if the GPU is not particularly well suited to the calculations involved it should be plenty fast enough.

Originally I tried using WebGL, but after some research I couldn’t see a way to perform the calculation in a vertex or fragment shader without multiple render calls. It may very well be possible, but it’s not obvious how to use the result of a previous calculation on a vertex or fragment without multiple stages to a pipeline.

Rule 178 reversible

WebGPU on the other hand has computation shaders which are perfectly suited to this type of calculation. It’s very new so I’d preferred to have steered clear of it, but it seemed to be the path of least resistance and by far the most interesting.

Probably I could get JavaScript to go fast enough to do the calculation while dealing with 1D automata. However using WebGPU paves the way for multi-dimensional automata or just displaying 1D automata in 3D.

WebGPU has its own Rust flavoured shader language that is called WGSL due to some drama with Apple and perhaps other reasons. It’s tempting to talk about this, but it’s a waste of time for both me and you. The language itself though seems pretty good although I don’t have much to base this upon.

I learnt enough about WebGPU to make this from WebGPU Fundamentals, the WebGPU specification and the WGSL spec. I also found Compute Toys useful and noticed that someone had already uploaded a 1D automata albeit not a reversible one.

WGSL automata implementation

I didn’t think I would be able to achieve quite the same result as my original implementation. Indeed only 32-bit ints are supported by WebGPU and there is no vectorisation. However none of that is needed on GPU and we can still do bit banging.

I don’t know if GPUs are limited in terms of integer arithmetic compared to floating point, but it makes no difference here. I’d have to line up more automata than could be displayed for performance to be an issue.

Rule 233 reversible

Below is the compute shader which calculates the next value for each automaton 16 x 32 automata at a time. There are 32 bits in an integer and the work group size is 16 by default. Meaning that the GPU will do 16 u32 computations in parallel.

Most GPUs can do at least 64 computations in parallel, but it is a bit pointless because we run out of pixels. We could write the automata activations to a much larger texture and sample multiple texels for each pixel, but I suspect it wouldn’t be all that interesting.

override stride: u32 = 16;

struct Params {
  width: u32,
  height: u32,
  rule: u32,
  reversible: u32,
  zoom: u32,
};

@group(0) @binding(0) var<storage, read_write> cells: array<u32>;
@group(0) @binding(1) var<uniform> params: Params;

// Take bit n of m and if it is 1 return all 1s, if 0 return all 0s
@must_use fn bit_to_max(m: u32, n: u32) -> u32 {
  let z: u32 = 0;

  return ((m >> n) & 1) * ~z;
}

@compute @workgroup_size(stride) fn cmp(
  @builtin(global_invocation_id) id: vec3u
) {
  let rule = params.rule;
  let i = id.x;
  let cols = stride;
  let rows = arrayLength(&cells) / cols;

  // Wrap the u32 bitfields; if we are on the first or last index
  let left_i = select(i - 1, cols - 1, i == 0);
  let right_i = select(i + 1, 0, i == cols - 1);

  // The cells before the current ones, used for reversible automata
  var prev: u32 = 0;

  for (var j: u32 = 1; j < rows; j++) {
    let row_off = (j - 1) * cols;
    let center = cells[row_off + i];
    // move the cell-bits on the left and right into the center
    // the bits at the edge are taken from the neighboring bitfields
    let left  = (center >> 1) | (cells[row_off + left_i ] << 31);
    let right = (center << 1) | (cells[row_off + right_i] >> 31);

    var result: u32 = 0;

    // for each of the 8 3-bit patterns...
    for (var k: u32 = 0; k < 8; k++) {
      // is the cell (de)activated for this pattern?
      let on = bit_to_max(rule, k);
      // check if the left, center and right cell-bits match the pattern
      // it's useful to remember that we are working on 32 cells at once
      let l = ~(bit_to_max(k, 2) ^ left);
      let c = ~(bit_to_max(k, 1) ^ center);
      let r = ~(bit_to_max(k, 0) ^ right);

      // set cell-bits to active if pattern is active and left,
      // right and center cell-bits all matched
      result |= l & c & r & on;
    }
    // for reversible automata...
    result ^= prev;

    cells[j * cols + i] = result;
    workgroupBarrier();

    prev = select(center, 0, params.reversible == 0);
  }
}

Rule 82 reversible

The automata are drawn to the screen with a vertex and pixel shader combination. The colour gradient in the background is the result of the vertex colours being blended together. I didn’t start out with the intention of there being a gradient, but putting colours on the vertices was in the tutorial and I kept it.

struct Verts {
  @builtin(position) pos: vec4f,
  @location(0) color: vec4f,
};

@vertex fn vs(@builtin(vertex_index) i : u32) -> Verts {
  let pos = array(
    vec2f(-1, 1),
    vec2f(-1,-1),
    vec2f( 1,-1),
    vec2f(-1, 1),
    vec2f( 1, 1),
    vec2f( 1,-1),
  );
  let col = array(
    vec4f(0.1, 0.5, 1, 1),
    vec4f(0.1, 1, 0.5, 1),
    vec4f(0.1, 0.5, 1, 1),
  );

  return Verts(vec4f(pos[i], 0.0, 1.0), col[i % 3]);
}

@fragment fn fs(verts: Verts) -> @location(0) vec4f {
  let fields = stride;
  let cols = (32 * fields) / params.zoom;
  let col_width = f32(params.height) / f32(cols);
  let rows = (arrayLength(&cells) / fields) / params.zoom;
  let row_height = f32(params.width) / f32(rows);
  let i_off = ((32 * fields) - cols) / 2;

  // get the cell's column index
  let i = u32(floor(verts.pos.y / col_width)) + i_off;
  // get the field index; i / 32
  let f = i >> 5;
  // get the bit index; i % 32
  let b = 31 - (i & 31);
  let j = u32(floor(verts.pos.x / row_height));
  let a = 0.2 + 0.8 * f32(((cells[j * fields + f]) >> b) & 1);

  return a * verts.color;
}

The vertex shader positions the vertices into a rectangle made out of two triangles. Each fragment then gets a color value that is the result of interpolating between the nearest vertices to where the fragment was located on a triangle.

Rule 82r again, but with different params

The automata is drawn by changing the alpha value for each fragment depending whether it falls inside an active cell. When the number of fragments equal the number of cells, then it is one cell per fragment and therefor it is one bit per cell.

Conclusion

I am excited to get something working with WebGPU and while it is complicated to get computation shaders running, I think there is a lot of potential there. When I get chance I’d like to look into higher dimensional automata and how these could be computed and displayed with WebGPU. I seem to remember that adding dimensions makes finding interesting rules far more difficult, but it would be cool to see something in 3D.

Rule 250 reversible

Rule 58 reversible

Rule 18 reversible

Avantirc.com

• Has embedded systems roles
• Some contract roles?
• Some remote roles

ioassociates.co.uk

• Has contract roles
• Mainly hybrid
• Seem confused

Don’t like reading? Video accompaniment.

Fuzzy Sync

Fuzzy Sync is a library we have developed for the Linux Test Project (LTP). It allows us to synchronise events in time, thus triggering bugs which are dependant on the outcome of one or more data races. Additionally it can introduce randomised delays (the fuzzy part) to achieve outcomes that would otherwise be extremely rare.

Cyril Hrubis previously introduced the library on the kernel.org blog along with one of the tests we created it for. In this article I will present a contrived data race which demonstrates why a delay is sometimes needed in order to synchronise the critical sections.

Update: I extracted Fuzzy Sync from the LTP into a standalone library.

For reference the Fuzzy Sync library is just a single header file which you can see here. There is more documentation than code. It is entirely a user-land library, not tied to Linux in any way, although it does use a few other parts of the LTP library so it is not currently stand-alone. It certainly does not inject errors or delays into the kernel or instrument it.

Despite the fact I think this is an obvious thing to do, if you are writing an exploit or just writing regression tests involving a data race, I couldn’t find a library which already did it. Although I have since found various tools and whatnot with something similar embedded; for example the Syzkaller executor has system call collision.

Note that when I say ‘rare’, I mean rare in my local context. In the wild someone will accidentally (or deliberately for that matter) discover a way to trigger it on a regular basis. There are a lot of systems in the wild running a lot of different software.

The Race

We have a simple race between two threads A and B, trying to read and write to the same variable.

Let’s look at the race code, most of the boiler plate is omitted and just the main loops for threads A and B are shown along with some variable definitions. You can see the full source here.

static struct tst_fzsync_pair pair;
static volatile char winner;

The data race revolves around the winner global variable which we access in both threads. Roughly speaking; setting it as volatile tells the compiler both that the value can change at any time and also setting the value may have side effects. In this case, it prevents the compiler from removing the if statement in the code below. Static analysis will show that winner is never equal to 'B' unless the variable can change at any time.

pair contains the Fuzzy Sync libraries state and is passed to most library functions. It is accessed from both threads as well, but is protected with memory barriers and the atomic API, which is far preferable to marking it as volatile. Generally speaking, you shouldn’t just mark stuff as volatile and then access it from multiple threads like any other variable. However it makes the demo code easier to read and I’m only running it on x86 so…

while (tst_fzsync_run_a(&pair)) {
    winner = 'A';

    tst_fzsync_start_race_a(&pair);
    if (winner == 'A' && winner == 'B')
        winner = 'A';
    tst_fzsync_end_race_a(&pair);
}

This is the loop for thread A, tst_fzsync_run_a() returns true if the test should continue and false otherwise. tst_fzsync_start_race_a() and tst_fzsync_end_race_a() delineate the block of code where we think the race can happen. Usually this is just a single statement, such as a function or system call, which is external to our reproducer code. Usually we would have some more complex setup between ...run_a/b() and ...start_race_a/b(), if we didn’t then it might be better to dispense with Fuzzy Sync and just use a pair of plain loops. Which is actually the case here, but for now let’s just pretend something complex and time consuming needs to be done before each race.

Before the race begins we set winner to A, then during the race we branch on the seemingly impossible condition

winner == 'A' && winner == 'B'`

Clearly if winner is 'A' then it is not 'B', unless the value of winner changes half way through the statement. Which it can because this is actually two statements in C and will be compiled to a number of assembly instructions. In other words it is not atomic.

0x00000000004055d2 <+1026>: movzbl 0x1b297(%rip),%eax        # 0x420870 <winner>
0x00000000004055d9 <+1033>: cmp    $0x41,%al
0x00000000004055db <+1035>: je     0x4057e8 <run+1560>

... <500+ lines of fuzzy sync assembly redacted> ...

0x00000000004057e8 <+1560>: movzbl 0x1b081(%rip),%eax        # 0x420870 <winner>
0x00000000004057ef <+1567>: cmp    $0x42,%al
0x00000000004057f1 <+1569>: jne    0x4055e1 <run+1041>
0x00000000004057f7 <+1575>: movb   $0x41,0x1b072(%rip)        # 0x420870 <winner>
0x00000000004057fe <+1582>: jmpq   0x4055e1 <run+1041>

This is the x86_64 assembly which checks and sets winner. The first 2 instructions load winner and compare it to 'A', the third will then jump to the second half of the statement if it is equal to 'A'. The second half is much like the first. Importantly we have two loads on winner (the two movzbls) which chronologically happen a few instructions apart.

Interestingly GCC has decided to move the second half of the statement towards the end of the program. I guess this is related to some CPU instruction cache optimisation, hint to branch prediction, register juggling or something else. If the behavior of our program is dependant on a data race, then this kind of optimisation can have a significant effect on the program behavior. A data race outcome which is highly improbable if the kernel is compiled with one compiler version may be highly probable when compiled with another. In this particular case, it won’t make a difference because the timings are dominated by nanosleep.

struct timespec delay = { 0, 1 };

while (tst_fzsync_run_b(&pair)) {
    tst_fzsync_start_race_b(&pair);
    nanosleep(&delay, NULL);
    winner = 'B';
    tst_fzsync_end_race_b(&pair);
}

We have a similar loop for thread B, the big difference is we make a system call and just set winner to 'B'. nanosleep here will attempt to sleep for one nanosecond, which basically means it won’t sleep at all as context switching takes far longer than a nanosecond, so by the time it gets into kernel land it will have already spent too long.

In a real test we usually have one or more syscalls in A and B. The race happens somewhere deep in the kernel and all we can do is ‘collide’ the system calls. Possibly one thread’s system calls may take much longer to reach the race critical section than the other. This is what we are simulating here; we have a very large difference in the time it takes A to reach the race critical section compared to B.

To make matters worse the race window is very short relative to the time it takes to context switch in and out of the kernel. We could move the sleep to before tst_fzsync_start_race_b, but often in a real reproducer the ideal place to demarcate the race window is somewhere inside the kernel.

Visualising the race

The race has been run 100,000 times and timestamps taken of when A and B start and finish. The differences between A and B are shown below. The red circle indicates where A won the race, that is, set winner = 'A'. In this case A only won once.

Each time difference is represented by a small black dot.

Difference Between Starts

That is the time difference between when A exited tst_fzsync_start_race_a() and when B exited tst_fzsync_start_race_b(). Initially both A and B start at about the same time; after about ~13000 loops the random delay of A is introduced.

Strictly speaking, the possible delay range includes a small delay to B as well, so that A ends as B starts, but A is so short we can’t see it on this graph.

You can see that when the start of B is delayed enough we may hit the race condition. It is close to the upper bound of the delay range.

Difference Between Ends

On the left of this graph we can see that B is consistently ~55000 nanoseconds behind A by the finish. If you look very closely you will find outliers where this is not the case, but we would have to rerun the race for a long time for A to win (on my setup).

The delay is calculated once a minimum number of loops have been completed and the mean deviation of various timings settles down. On the left you can see there is some natural jitter, but not much relative to the overall difference in end times. So the delay is calculated after relatively few loops. If the natural jitter were relatively large, then we wouldn’t need to artificially introduce more.

Once the delay is introduced we can see that A and B are far more likely to finish at the same time. The race window for A to win is almost precisely at a difference of 0.

Visualising the race 2

Below is a more abstract (and patently not-to-scale) visualisation of the two major epochs of Fuzzy Sync execution.

Race Time Diagrams

In the first case we are just using spin locks which act as synchronising barriers for entering and exiting the race (see tst_fzsync_pair_wait() in the code).

When one thread reaches the end of the race before the other, it counts the number of times it spins waiting. We use this value later to calculate the delay range.

The second diagram shows the ideal delay for 'A' to win the race, however we actually calculate a delay range and randomly pick values from it. In this example we know exactly where the race occurs so we could dispense with some of the stochasticity and narrow the delay range to just attempt the synchronisation shown.

However, for most tests, we don’t have this level of knowledge about the race condition on all possible kernel versions and configurations where the bug exists. In fact we sometimes don’t know much at all about the call tree and timings within the kernel, we just know that calling syscall X and Y at the same time makes bad things happen.

For an attacker focusing on a particular system though, you can assume that they would be able to narrow the delay range down to a value which allows them to trigger the race in much fewer iterations. Fuzzy Sync allows this in some cases using tst_fzsync_pair_add_bias() which I may cover in a future article.

Reproducers using Fuzzy Sync

At the time of writing there are several LTP tests using Fuzzy Sync:

• cve-2014-0196
• cve-2016-7117
• cve-2017-2671
• pty03 (cve-2020-14416)
• snd_seq01 (cve-2018-7566)
• snd_timer01 (cve-2017-1000380)
• bind06 (cve-2018-18559)
• inotify09
• shmctl05
• sendmsg03 (cve-2017-17712)
• setsockopt06 (cve-2016-8655)
• setsockopt07 (cve-2017-1000111)
• timerfd_settime02 (cve-2017-10661)
• fanout01 (cve-2017-15649)

From minimal VMs to AI/ML GPU rigs with Kubernetes

A recurring theme throughout my career is frustration at not being able to easily utilise a remote machine or a local VM.

In some cases this was because I had a local VM running a stripped down kernel with very little userland in it. Creating such an environment was useful for quickly building and redeploying the Linux kernel, but left very little in userland to work with. It moved a lot of stuff outside the machine where the software is being deployed.

In others I had a full blown Linux distro on a remote machine, but quickly getting updated versions of my software on to there is still a pain. Often involving SSH’ing into a remote machine and doing a Git pull or opening a file in Vim/Emacs and tweaking it.

Even worse are occasions where I had to upload a Docker image to a repository just so I can run the software in a Kubernetes cluster. In this case there are a bunch of tools to help (e.g. Skaffold). However the mind boggles at how complicated anything involving Kubernetes can get.

In pretty much all cases I have a local source repository with some code in. That could be a Linux Test Project test or an LLM example for the Prem Kubernetes operator. I then need to get that software built and into a remote machine which has the appropriate hardware or OS level software.

There are plenty of CI/CD options out there, but they lack the kind of tight feedback loop I want in any given situation. I certainly don’t want to involve GitHub or GitLab when tweaking one line of code in a series of experiments.

There are some tools that do a very good job, but typically require a fair amount of setup. To the extent that I think it’s more effort than its worth and resort to some variation of logging in over serial and doing things manually.

Ayup: A solution for some of this

What I generally want in these cases is a tool I can easily install and connect to on these systems that does the full CI/CD in a very rapid manner. Meaning it has to be statically compiled with just the kernel as its dependency and it has to have an easy, secure connection mechanism and it needs to cache builds.

Importantly it needs to do the good stuff by default. It’s amazing what Nix, Docker or Kubernetes can do. The tools and the features are all there, it’s just they’re not put together in the right way for the user experience I envision.

So here is Ayup, a build and deployment tool based on Buildkit and Containerd. It’s initial focus is on AI/ML projects, but in theory there’s nothing stopping it from deploying an LTP test.

Presently it doesn’t bundle Buildkit or Containerd, but that’s what a bunch of Kubernetes distros do and it results in a 200MB executable. This can be dumped as a single executable on a system and off we go.

There are a bunch of other problems that Ayup is trying to tackle that you can see on Prem’s blog.

Intro

Don’t get me wrong HTTP/2 is a clear improvement on HTTP1.x, but where is the entry level version? It’s certainly not HTTP/3, that’s for sure.

The general wisdom amongst any new technology group, such as Zig, is to stick with text based HTTP and hide behind a proxy. This is the logical thing to do because HTTP1.1 has a lower barrier to entry. Primarily because of HPACK which we’ll get into in a moment.

To see why HTTP1.1 is an attractive option, take a look at my article on Zig Vs C - Minimal HTTP server. It’s relatively easy to slap together a non-compliant HTTP server. My static file server is an extreme example, but practically you can aim for partial compliance then hide behind a normalising HTTP proxy.

In theory and ostensibly in practice the proxy takes whatever awful HTTP is thrown at you from the internet, then converts it into some manageable subset. It can also handle TLS termination, so you can leave all that bad jazz to a third party.

There are two problems with this. The least important being performance and the most being security.

In my semi-sophisticated opinion, both issues have the same root cause. Essentially the length of a HTTP message is not known until you parse a variable length list of variable length headers.

In theory you should then know how long the body is. However there is confusion over what specifies the body length which leads to request smuggling.

Request smuggling and desync attacks are enabled by the presence of proxies. Allegedly the problem gets worse with the introduction of HTTP/2. However at the end of the linked article it states:

If you’re setting up a web application, avoid HTTP/2 downgrading - it’s the root cause of most of these vulnerabilities. Instead, use HTTP/2 end to end.

OK, so how hard can it be to implement HTTP/2 then? This is something I was excited to find out about. Not least because it is an excuse to try out Zig for implementing network protocols. With the eventual goal of doing some security research into crusty old tech using exciting new tech (a vague plan of mine).

Zig isn’t just exciting in my opinion; the traffic to my blog indicates it is the most interesting thing. I also believe it is practical. Surprisingly my original HTTP server still compiles and runs on the latest Zig from Git. It seems the build API has changed quite a bit, but for an unstable language it is quite stable if you stay away from new features.

Here is some sample output upon a request by Curl; it’s not pretty:

zig run src/self-serve2.zig -- ~/portfolio/public
info: Listening on 127.0.0.1:9001; press Ctrl-C to exit...
info: Accepted Connection from: 127.0.0.1:48274
info: <<< Got preface!
info: >>> Sending server preface
info: <<< http2.FrameHdr{ .length = 18, .type = http2.FrameType.settings, .flags = http2.FrameFlags{ .settings = http2.SettingsFlags{ .ack = false, .unused = 0 } }, .r = false, .id = 0 } http2.Payload{ .settings = http2.SettingsPayload{ .settings = { 0, 3, 0, 0, 0, 100, 0, 4, 2, 0, 0, 0, 0, 2, 0, 0, 0, 0 } } }
info:     http2.Setting{ .maxConcurrentStreams = 100 }
info:     http2.Setting{ .initialWindowSize = 33554432 }
info:     http2.Setting{ .enablePush = false }
info: <<< http2.FrameHdr{ .length = 4, .type = http2.FrameType.windowUpdate, .flags = http2.FrameFlags{ .unused = 0 }, .r = false, .id = 0 } http2.Payload{ .windowUpdate = http2.WindowUpdatePayload{ .r = false, .windowSizeIncrement = 33488897 } }
info: <<< http2.FrameHdr{ .length = 44, .type = http2.FrameType.headers, .flags = http2.FrameFlags{ .headers = http2.HeadersFlags{ .endStream = true, .unused1 = false, .endHeaders = true, .padded = false, .unused2 = false, .priority = false, .unused3 = 0 } }, .r = false, .id = 1 } http2.Payload{ .headers = http2.HeadersPayload{ .headerBlockFragment = { 130, 4, 141, 98, 49, 216, 90, 61, 45, 58, 83, 88, 150, 246, 105, 191, 134, 65, 138, 160, 228, 29, 19, 157, 9, 184, 248, 0, 31, 122, 136, 37, 182, 80, 195, 203, 129, 112, 255, 83, 3, 42, 47, 42 }, .hdec = hpack.Decoder{ .from = { ... }, .to = { ... }, .table = hdrIndx.Table{ ... } } } }
info:     :method => GET
info:     :path => /barely-http2-zig
info:     :scheme => http
info:     :authority => localhost:9001
info:     user-agent => curl/8.0.1
info:     accept => */*
info: *** Opening barely-http2-zig.html
info: >>> Sending OK headers
info: >>> Sending DATA http2.FrameHdr{ .length = 16384, .type = http2.FrameType.data, .flags = http2.FrameFlags{ .data = http2.DataFlags{ .endStream = false, .unused = 0, .padded = false } }, .r = false, .id = 1 }
info: >>> Sending DATA http2.FrameHdr{ .length = 16384, .type = http2.FrameType.data, .flags = http2.FrameFlags{ .data = http2.DataFlags{ .endStream = false, .unused = 0, .padded = false } }, .r = false, .id = 1 }
info: >>> Sending DATA http2.FrameHdr{ .length = 16384, .type = http2.FrameType.data, .flags = http2.FrameFlags{ .data = http2.DataFlags{ .endStream = false, .unused = 0, .padded = false } }, .r = false, .id = 1 }
info: >>> Sending DATA http2.FrameHdr{ .length = 16384, .type = http2.FrameType.data, .flags = http2.FrameFlags{ .data = http2.DataFlags{ .endStream = false, .unused = 0, .padded = false } }, .r = false, .id = 1 }
info: >>> Sending DATA http2.FrameHdr{ .length = 16384, .type = http2.FrameType.data, .flags = http2.FrameFlags{ .data = http2.DataFlags{ .endStream = false, .unused = 0, .padded = false } }, .r = false, .id = 1 }
info: >>> Sending DATA http2.FrameHdr{ .length = 12857, .type = http2.FrameType.data, .flags = http2.FrameFlags{ .data = http2.DataFlags{ .endStream = true, .unused = 0, .padded = false } }, .r = false, .id = 1 }
info: >>> Sent 94777 file bytes

HPACK

It turned out the main obstacle to slapping together a half-arsed HTTP/2 parser and writing a victory blog post declaring it took me only N hours, is the header compression scheme.

This, in addition to the stream dependency and encryption talk, is probably what frightens people away from HTTP/2.

There is no way that you can skip over HPACK decoding unless the other side is equally as lazy. Initially I thought I had found a way of doing it by setting the decoder table size to zero. However this is scuppered by an allowance for the client to start sending frames before it has received any settings (because latency).

In fact even if you could avoid implementing the decoder table, you would still have to deal with Huffman encoded strings and variable length integers. Although these things are standard computer science, so there is plenty of good material to fall back on.

I did however fuss over avoiding memory allocations and implement all the data structures myself. It’s worth pointing out that the Zig standard library has some nice data structures and functions for doing stuff like this.

Let’s take a look at the thing which I tried hardest to avoid, the decoder table. This thing provides some nice compression for site specific headers. If you are lost as to what I am talking about then do a search on HPACK. There are some articles waxing lyrical about the greatness of HPACK. Essentially though it remembers headers, in full or part, so that they don’t have to be resent for multiple requests.

/// The content of the table entries; A FIFO buffer.
///
/// It's a buffer with capacity 3x the size of the minimum table size
/// required by HPACK. This allows us to keep adding entries in
/// contiguous chunks with only an occasional copy.
///
/// New entries are added before start and their length subtracted
/// from start.  When start gets below the minimum table size,
/// everything is shifted backwards.
///
/// Only 2X the table size would be needed except that a new entry can
/// reference the index of an entry which is about to be removed.
const HdrData = struct {
    /// The start of the first entry (most recently added)
    start: u16 = 2 * 4096,
    /// Where the start was at the previous copy to shift everything backwards.
    /// Needed to correct indexes for copied items.
    prevStart: u16 = 3 * 4096,
    /// The length of the current entries
    len: u16 = 0,
    /// The data
    vec: [3 * 4096]u8 = undefined,
};

/// An entry in the table data. Sort of like a slice, but using
/// 16-bit indexes.
const HdrPtr = struct {
    start: u16,
    nameLen: u16,
    valueLen: u16,
};

/// An inner table indexing the table's entries. Needed because the
/// entries are uneven.
const HdrIndx = struct {
    start: u8 = 127,
    len: u8 = 0,
    vec: [256]HdrPtr = undefined,
};

/// The encapsulating Table struct because I forgot that files in Zig
/// are structs.
pub const Table = struct {
    data: HdrData = HdrData{},
    indx: HdrIndx = HdrIndx{},
    size: u16 = 4096,

    fn capacity(self: *Table) u16 {
        return self.data.vec.len / 3;
    }

    /// Get an entry from the table. The returned struct is borrowed
    /// and needs to be copied if it is to be used after it has been
    /// evicted from the table.
    pub fn get(self: *Table, i: u8) !HdrConst {
        const data = &self.data;
        const indx = &self.indx;
        const slen = STATIC_INDX.len;

        if (i == 0)
            return error.InvalidIndexZero;

        if (i < slen)
            return STATIC_INDX[i];

        if (i - slen >= indx.len) {
            return error.IndexTooBig;
        }

        const hdr = indx.vec[indx.start + (i - slen)];
        const start = if (hdr.start < data.start)
            2 * self.capacity() + (hdr.start - data.prevStart)
        else
            hdr.start;

        const value_start = start + hdr.nameLen;

        return .{
            .name = data.vec[start..value_start],
            .value = data.vec[value_start .. value_start + hdr.valueLen],
        };
    }

    /// The length of the table according to the HPACK spec
    fn nominalLen(self: *Table, name: []const u8, value: []const u8) usize {
        const estimated_overhead = 32 * (1 + @as(usize, self.indx.len));

        return self.data.len + name.len + value.len + estimated_overhead;
    }

    /// Add an entry to the table. The name and value arguments can
    /// point to an existing entry which will evict itself.
    pub fn add(self: *Table, name: []const u8, value: []const u8) !void {
        const data = &self.data;
        const indx = &self.indx;

        while (self.nominalLen(name, value) > self.size) {
            if (indx.len == 0)
                return;

            const last = indx.vec[indx.start + indx.len - 1];

            data.len -= last.nameLen + last.valueLen;
            indx.len -= 1;
        }

        if (indx.start == 0) {
            mem.copy(HdrPtr, indx.vec[128..], indx.vec[0..128]);
            indx.start = 128;
        }

        indx.len += 1;
        indx.start -= 1;

        const hdr = &indx.vec[indx.start];
        hdr.nameLen = @truncate(u16, name.len);
        hdr.valueLen = @truncate(u16, value.len);

        data.start -= hdr.nameLen;
        data.start -= hdr.valueLen;
        hdr.start = data.start;

        data.len += hdr.nameLen;
        data.len += hdr.valueLen;

        const value_start = hdr.start + hdr.nameLen;

        mem.copy(u8, data.vec[hdr.start..value_start], name);
        mem.copy(u8, data.vec[value_start .. value_start + hdr.valueLen], value);

        if (data.start < self.capacity()) {
            mem.copyBackwards(
                u8,
                data.vec[2 * self.capacity() ..],
                data.vec[data.start .. data.start + data.len],
            );
            data.prevStart = data.start;
            data.start = 2 * self.capacity();
        }
    }
};

This stores the full name and value of each header in a buffer that takes up 3x the space of the nominal table size. This is not great memory usage, but it’s relatively simple and hopefully cache efficient. Because the memory accesses are likely to be close together.

The table is of a fixed size of 4096, the minimum required by HTTP/2. Zig would allow this to be changed easily in a variety of ways. I just haven’t bothered to do it.

In the end it’s not a lot of code, although it’s the type of code which can be a pain to debug. Zig’s built in tests helped with that. You can run them with:

$ zig test src/hdrIndx.zig

Each file in the src directory has its own tests. Continuing with HPACK let’s look at integer decoding. Each header field name and value is prepended with a variable length integer describing the fields length.

What’s more this integer encoding allows some bits in the first byte of the encoding to be used for flags or whatever. So the integer value actually starts part way through the first byte. The first bit on the remaining bytes (if any) is then used as a stop bit.

This allows for infinitely large integers, but I don’t allow that because I’m a spoil sport.

/// Get the unsigned type big enough to count the bits in T. Needed
/// because Zig constrains the right hand side of a shift to an
/// integer only big enough to perform a full shift. Which is only u3
/// for u8 (for e.g.).
///
/// Meanwhile I don't know a way to specify this type other than to
/// construct it like this.
fn ShiftSize(comptime T: type) type {
    const ShiftInt = Type{
        .Int = .{
            .signedness = std.builtin.Signedness.unsigned,
            .bits = comptime std.math.log2_int(u16, @bitSizeOf(T)),
        },
    };

    return @Type(ShiftInt);
}

fn decodeInt(comptime T: type, comptime n: u3, buf: *[]const u8) !T {
    const prefix = (1 << n) - 1;
    var b = buf.*[0];
    var i: T = b & prefix;

    if (i < prefix) {
        buf.* = buf.*[1..];
        return i;
    }

    var j: ShiftSize(T) = 1;
    while ((j - 1) * 7 < @bitSizeOf(T)) : (j += 1) {
        b = buf.*[j];

        i += @as(T, (b & 0x7f)) << (7 * (j - 1));

        if (b < 0x80)
            break;
    } else {
        return UnpackError.IntTooBig;
    }

    buf.* = buf.*[j + 1 ..];
    return i;
}

It’s been a while since I wrote this and boy am I glad I wrote that stuff about the shift size. This function takes comptime parameters which allow different functions to be generated depending on the type it returns and how many bits are ignored (n).

Interestingly it seems that Zig forces you to use the minimum sized type to perform a shift. Which I think caught a few mistakes. The ShiftSize function is constructing the necessary sized type to shift T that was passed into decodeInt.

So indeed, Zig allows constructing arbitrary types from ordinary code at comptime.

Something else to note about this code is that buf is a pointer to a slice. The syntax buf.* dereferences the pointer. I update the slice to chop off the bits that were used decoding the integer. I’m not sure this is an advisable thing to do.

Now let’s look at string encodings, which display the other type of compression employed by HPACK.

/// Decode a string which if it is not Huffman encoded is fairly
/// straight forward.
///
/// If it is Huffman encoded then we have to deal with the fact
/// Huffman codes are not byte aligned and are variable length.
///
/// We could put the huffman codes in a binary tree and lookup one bit
/// at a time. However I doubt this is the right place to start on
/// common CPUs.
///
/// So instead we shift (at most) the next four bytes into a
/// buffer. Then compare the first bits of the first byte to the
/// shortest huffman codes. If it doesn't match any, then move on to
/// longer codes until we are comparing all four bytes.
///
/// I haven't done any research into the fastest methods of Huffman
/// decoding. This is just a first approximation.
fn decodeStr(from: *[]const u8, to: *[]u8) ![]const u8 {
    const huffman = from.*[0] & 0x80 == 0x80;
    const len = try decodeInt(u16, 7, from);
    const str = from.*[0..len];

    from.* = from.*[len..];

    if (!huffman)
        return str;

    var i: u16 = 0;
    var j: u32 = 0;
    var k: u16 = 0;
    var c = [_]u8{0} ** 5;

    all: while (i < len) {
        mem.copy(u8, &c, str[i..std.math.min(i + 5, str.len)]);

        const j_rem = @truncate(u3, j);

        var l: u3 = 0;
        while (j_rem > 0 and l < c.len - 1) : (l += 1) {
            c[l] <<= j_rem;
            c[l] |= c[l + 1] >> @truncate(u3, 8 - @as(u4, j_rem));
        }

        var glen: u5 = 0;
        const dehuff = decode: for (DEHUFF) |group| {
            glen = 1 + ((group.len - 1) >> 3);

            if (glen != group.codes[0].code.len)
                return error.GlenWrongLen;

            const bits_left = str.len * 8 - j;
            if (bits_left < group.len) {
                if (glen > 1 or j_rem < 1)
                    return error.HuffNoMatchInputEndedEarly;

                const pad_mask = @truncate(u8, @as(u16, 0xff00) >> @truncate(u4, bits_left));
                if (c[0] & pad_mask == pad_mask)
                    break :all
                else
                    return error.HuffInvalidPadding;
            }

            const glen_rem = @truncate(u3, group.len);
            const last_mask = if (glen_rem == 0)
                0xff
            else
                @truncate(u8, @as(u16, 0xff00) >> glen_rem);
            const last = c[glen - 1];

            for (group.codes) |huff| {
                if (huff.code[glen - 1] != last_mask & last)
                    continue;

                if (mem.eql(u8, huff.code[0 .. glen - 1], c[0 .. glen - 1]))
                    break :decode huff;
            }
        } else {
            return error.HuffNoMatch;
        };

        j += dehuff.len;
        i = @truncate(u16, j / 8);

        to.*[k] = dehuff.sym;
        k += 1;
    }

    const ret = to.*[0..k];
    to.* = to.*[k..];

    return ret;
}

The first four lines of the function decode the string if it is not Huffman encoded. HPACK is actually relatively simple in the sense that the Huffman encoding is static. So I didn’t have spend long reading my algorithms text book and could put it back under the table leg, thus restoring stability.

There is a big static table in src/huff.zig which I sort at comptime. From a Zig point of view it makes use of the loop labels and break statements. Which are good. There is lots of use of the @truncate builtin. Possibly this is wrong in some cases and it should be @intCast or something. The later having a safety check I believe.

Zig prevents some bit banging techniques common to C. Because it results in undefined behaviour. The end result being lots of uses of builtins instead of shifts and masking. Of course you can still use the wrong builtins.

Finally, for HPACK, let’s look at the Decoder struct which provides an iterator interface for incoming headers. I’ll show how it’s used first:

fn serveFiles(h2c: *http2.NetConnection, dir: fs.Dir) !void {

    ...

                var path_buf: [fs.MAX_PATH_BYTES]u8 = undefined;
                var path: ?[]const u8 = null;

                while (headers.next()) |h| {
                    std.log.info("    {s} => {s}", h);

                    if (mem.eql(u8, ":path", h.name) and h.value.len <= path_buf.len) {
                        mem.copy(u8, &path_buf, h.value);
                        path = path_buf[0..h.value.len];
                    }
                } else |err| {
                    if (err != error.EndOfData) return err;
                }

                if (path) |p|
                    try sendFile(h2c, dir, frame.hdr.id, p)
                else
                    return error.DidntFindThePathHeader;
    ...

Above we can see that headers.next() is called and if we find the path header we copy it. Most HTTP implementations I have seen will allocate strings for every header and pass them up to a higher level Framework. We could do that too, but I think it’s important to keep the option of just skipping over stuff we don’t care about.

Also some headers we do care about we could deal with during the parsing without performing a copy. Below is the implementation which correlates with this part of the spec.

/// An iterator that takes I/O buffers, a decoding table and returns
/// header entries. The table is mutated so you can't run the iterator
/// twice.
pub const Decoder = struct {
    /// buffer containing the encoded data
    from: []const u8,
    /// A scratch buffer for decoded headers
    to: []u8,
    table: hdrIndx.Table = hdrIndx.Table{},

    pub fn init(from: []const u8, to: []u8) Decoder {
        return .{
            .from = from,
            .to = to,
        };
    }

    pub fn newData(self: *Decoder, from: []const u8, to: []u8) void {
        self.from = from;
        self.to = to;
    }

    /// Get the next header. The contents of the header may be
    /// borrowed from the scratch buffer or the table's buffer.
    pub fn next(self: *Decoder) !HdrConst {
        if (self.from.len < 1)
            return error.EndOfData;

        const table = &self.table;
        const tag = self.from[0];
        const repr = try HdrFieldRepr.from(tag);

        switch (repr) {
            .indexed => {
                const i = try decodeInt(u8, 7, &self.from);
                return table.get(i);
            },
            .indexedNameAddValue => {
                const i = try decodeInt(u8, 6, &self.from);
                const ihdr = try table.get(i);
                const hdr = .{
                    .name = ihdr.name,
                    .value = try decodeStr(&self.from, &self.to),
                };

                try table.add(hdr.name, hdr.value);

                return hdr;
            },
            .indexedNameLitValue => {
                const i = try decodeInt(u8, 4, &self.from);
                const ihdr = try table.get(i);
                const hdr = .{
                    .name = ihdr.name,
                    .value = try decodeStr(&self.from, &self.to),
                };

                return hdr;
            },
            .addNameAddValue, .litNameLitValue => {
                self.from = self.from[1..];

                const hdr: HdrConst = .{
                    .name = try decodeStr(&self.from, &self.to),
                    .value = try decodeStr(&self.from, &self.to),
                };

                if (repr == .addNameAddValue)
                    try table.add(hdr.name, hdr.value);

                return hdr;
            },
        }
    }
};

Couldn’t they have just made a HTTP1.2 that encoded the headers in RESP strings or something?

Frames

A HTTP/2 connection is made up of a series of frames with various types of payload. The frame header is always the same and not too complicated. Initially I tried decoding the header by declaring a packed struct for it and doing a @ptrCast or @bitCast.

If it worked it would look something like this:

const HdrData = packed struct {
    length: u24,
    type: FrameType, // or u8
    flags: FrameFlags,
    r: bool,
    id: u31,
};

...
    var hdr = @ptrCast(HdrData, buf[n..n+9]);
    // Then switch the endianess

The major issue is the endianess needs switching from network byte order to native for length and perhaps id. There are functions to help with reading in integers of different endianess. However it didn’t seem worth trying to do that and a pointer cast.

Then there is FrameFlags, which it is convenient to define as a tagged union for printing. To my knowledge Zig doesn’t define where the tag is or allow you to specify it. We can’t tell it that a packed union has its tag immediately before the union field content.

So I feel this code could be cleaner, but also that I am fussing over a minor details. The below is how a Frame is decoded and encoded.

/// All HTTP/2 traffic is made up of frames with a fixed sized header
/// of the same format. Each frame specifies its type and payload
/// length. On an abstract level this makes parsing HTTP/2 traffic
/// easy.
///
/// The only complicatin in Zig being the interactiong between
/// endianess and packed u24. Otherwise we could declare the flags as
/// u8, mark the struct as packed then do a single @ptrCast. I tried
/// something like this, but got in a mess and settled on the below.
pub const FrameHdr = struct {
    /// Length of the frame's payload
    length: u24,
    /// How we should interpret everything that follows
    type: FrameType,
    flags: FrameFlags,
    /// A reserved bit, which we can set to 1 as an act of rebellion.
    r: bool = false,
    /// The stream ID or zero if this frame applies to the connection.
    id: u31,

    pub fn from(buf: *const [9]u8) FrameHdr {
        const ftype = @intToEnum(FrameType, buf[3]);

        return .{
            .length = mem.readIntBig(u24, buf[0..3]),
            .type = ftype,
            .flags = switch (ftype) {
                .headers => .{ .headers = @bitCast(HeadersFlags, buf[4]) },
                .settings => .{ .settings = @bitCast(SettingsFlags, buf[4]) },
                .windowUpdate => .{ .unused = buf[4] },
                else => .{ .unknown = buf[4] },
            },
            .r = @bitCast(bool, @truncate(u1, buf[5] >> 7)),
            .id = @intCast(u31, mem.readIntBig(u32, buf[5..9]) & 0x7fffffff),
        };
    }

    pub fn to(self: FrameHdr, buf: []u8) []const u8 {
        mem.writeIntBig(u24, buf[0..3], self.length);
        buf[3] = @enumToInt(self.type);
        buf[4] = switch (self.flags) {
            .data => |flags| @bitCast(u8, flags),
            .headers => @bitCast(u8, self.flags.headers),
            .settings => @bitCast(u8, self.flags.settings),
            else => unreachable,
        };
        // r is always 0
        mem.writeIntBig(u32, buf[5..9], @intCast(u32, self.id));

        return buf[0..9];
    }
};

There are a number of frame types which are listed below.

const FrameType = enum(u8) {
    data,
    headers,
    priority,
    rstStream,
    settings,
    pushPromise,
    ping,
    goAway,
    windowUpdate,
    continuation,
    _,
};

These mostly have different payloads. The structure of the payloads varies depending what flags are specified in the frame header. They are mostly not too complicated in terms of layout though. The exception being the headers of course.

Let’s look at the settings payload decoder. It would be possible to ignore the settings completely. However I was curious to see what Curl would send.

const SettingId = enum(u16) {
    headerTableSize = 0x1,
    enablePush,
    maxConcurrentStreams,
    initialWindowSize,
    maxFrameSize,
    maxHeaderListSize,
};

const Setting = union(SettingId) {
    headerTableSize: u32,
    enablePush: bool,
    maxConcurrentStreams: u32,
    initialWindowSize: u31,
    maxFrameSize: u24,
    maxHeaderListSize: u32,
};

/// Settings limit what we can send to the other side. This is
/// essentially an iterator which returns a tagged u32
const SettingsPayload = struct {
    settings: []const u8,
    used: usize,

    pub fn init(buf: []const u8) SettingsPayload {
        return .{ .settings = buf, .used = 0 };
    }

    pub fn next(self: *SettingsPayload) !Setting {
        if (self.settings.len - self.used == 0)
            return error.EndOfData;

        if (self.settings.len - self.used < 6)
            return error.UnexpectedEndOfData;

        const buf = self.settings[self.used..][0..6];
        self.used += 6;

        const id = mem.readIntBig(u16, buf[0..2]);
        const val = mem.readIntBig(u32, buf[2..]);

        return switch (id) {
            1 => .{ .headerTableSize = val },
            2 => .{ .enablePush = @bitCast(bool, @intCast(u1, val)) },
            3 => .{ .maxConcurrentStreams = val },
            4 => .{ .initialWindowSize = @intCast(u31, val) },
            5 => .{ .maxFrameSize = @intCast(u24, val) },
            6 => .{ .maxHeaderListSize = val },
            else => error.NoIdeaWhatThatSettingIs,
        };
    }
};

It is another iterator. Again it returns a tagged union which is nice because we can switch on it and it is formatted for printing automatically.

By this point I was no longer passing around pointers to slices. Instead I had (re)discovered that functions in a Zig struct can begin with a self argument. I could then just keep a slice and used argument on the struct. It occurred to me while writing this, that I don’t even need used.

const SettingsPayload = struct {
    settings: []const u8,

    pub fn init(buf: []const u8) SettingsPayload {
        return .{ .settings = buf };
    }

    pub fn next(self: *SettingsPayload) !Setting {
        if (self.settings.len == 0)
            return error.EndOfData;

        if (self.settings.len < 6)
            return error.UnexpectedEndOfData;

        const buf = self.settings[0..6];

        self.settings = self.settings[6..];

        const id = mem.readIntBig(u16, buf[0..2]);
        const val = mem.readIntBig(u32, buf[2..]);

        return switch (id) {
            1 => .{ .headerTableSize = val },
            2 => .{ .enablePush = @bitCast(bool, @intCast(u1, val)) },
            3 => .{ .maxConcurrentStreams = val },
            4 => .{ .initialWindowSize = @intCast(u31, val) },
            5 => .{ .maxFrameSize = @intCast(u24, val) },
            6 => .{ .maxHeaderListSize = val },
            else => error.NoIdeaWhatThatSettingIs,
        };
    }
};

It’s not like I haven’t used a language with slices in before. However there does seem to be some kind of mental block. Perhaps because they look similar to arrays.

Connection

HTTP/2 organises things into connections and streams within connections. Some things are connection level and other things are stream level.

Also a connection maps to a TCP/TLS connection… I think. At least for the current purposes a stream is just an ID number we need to remember when responding to a headers frame containing a get method.

There are no explicit request/response objects. However a request can be thought of as a sequence of headers, continuation and data frames on a single stream. There seems to be some leeway here though which could be another barrier to adoption.

For now though, we just have a Connection object which provides a low level interface for getting frames.

/// HTTP/2's idea of a connection which wraps around the underlying
/// stream.  Usually the underlying stream will be a TCP connection,
/// but could by anything which provides the Reader and Writer
/// interfaces e.g. a file or buffer with captured frame data in.
///
/// This is essentially an iterator interface to the underlying
/// data. Which recurses into iterators for the various types of frame
/// payload.
///
/// It only allocates memory during init and we can reuse the
/// connection object by calling reinit. This preempts the No. 1
/// performance issue I have seen in most open source libraries.
///
/// One should assume that any pointers it returns are to buffers it
/// allocated at init time. So their lifetime is only until the next
/// call to nextFrame[Hdr]. Long lived data therefor needs to be
/// copied. Whether this is an issue depends on the use case.
pub fn Connection(comptime Reader: type, comptime Writer: type) type {
    const preface = [_]u8{
        0x50, 0x52, 0x49, 0x20, 0x2a, 0x20, 0x48, 0x54, 0x54, 0x50, 0x2f,
        0x32, 0x2e, 0x30, 0x0d, 0x0a, 0x0d, 0x0a, 0x53, 0x4d, 0x0d, 0x0a,
        0x0d, 0x0a,
    };

    return struct {
        const Self = @This();

        allocator: std.mem.Allocator,
        frame_in: []u8,
        have: usize = 0,
        used: usize = 0,

        headers_in: []u8,
        hdec: hpack.Decoder,
        frame_out: []u8,

        reader: Reader,
        writer: Writer,

        pub fn init(
            a: std.mem.Allocator,
            buf_len: usize,
        ) !Self {
            return .{
                .allocator = a,
                .frame_in = try a.alloc(u8, buf_len),
                .headers_in = try a.alloc(u8, buf_len),
                .hdec = .{ .from = undefined, .to = undefined },
                .frame_out = try a.alloc(u8, buf_len),
                .reader = undefined,
                .writer = undefined,
            };
        }

        pub fn reinit(self: *Self, r: Reader, w: Writer) void {
            self.have = 0;
            self.used = 0;
            self.reader = r;
            self.writer = w;

            mem.set(u8, self.frame_in, 0);
            mem.set(u8, self.frame_out, 0);
            mem.set(u8, self.headers_in, 0);
        }

        pub fn deinit(self: *Self) void {
            self.allocator.free(self.frame_in);
            self.allocator.free(self.frame_out);
            self.allocator.free(self.headers_in);
        }

        fn read(self: *Self, needed: usize) ![]const u8 {
            var have = self.have - self.used;
            const in = self.frame_in[self.used..];

            const len = if (have < needed)
                try self.reader.readAtLeast(in, needed - have)
            else
                0;

            have += len;
            self.have += len;

            if (have == 0)
                return error.EndOfData;

            if (have < needed)
                return error.UnexpectedEndOfData;

            self.used += needed;

            return in[0..needed];
        }

        /// Start the HTTP/2 connection as the server and assuming
        /// "prior knowledge". This is a simple case of reading in the
        /// magic string (preface) sent by the client and sending a
        /// settings frame.
        pub fn start(self: *Self) !void {
            const pface = try self.read(preface.len);

            if (mem.eql(u8, &preface, pface))
                std.log.info("<<< Got preface!", .{})
            else {
                std.log.info("<<< Expected preface, bug got\n: {s}", .{pface});
                return error.InvalidPreface;
            }

            std.log.info(">>> Sending server preface", .{});
            const empty_settings = FrameHdr{
                .length = 0,
                .type = .settings,
                .flags = .{ .settings = .{ .ack = false } },
                .id = 0,
            };
            try self.writer.writeAll(empty_settings.to(self.frame_out));
        }

        /// Lower level iterator which returns just the frame
        /// header. Potentially this can be used to skip over
        /// uninteresting frames.
        pub fn nextFrameHdr(self: *Self) !FrameHdr {
            return FrameHdr.from((try self.read(9))[0..9]);
        }

        /// Returns a slightly higher level Frame payload iterator and
        /// frame header object. Still pretty low level. We'd probably
        /// want to abstract this into streams and abstract streams
        /// into requests and responses.
        pub fn nextFrame(self: *Self) !Frame {
            const hdr = try self.nextFrameHdr();

            return Frame.init(
                hdr,
                &self.hdec,
                try self.read(hdr.length),
                self.headers_in,
            );
        }

        ...
}

There is a simplified listener in the main library file which can be run with $ zig run src/http2.c. This uses the connection as follows.

pub const NetConnection = Connection(std.net.Stream, std.net.Stream);

fn serve(h2c: *NetConnection) !void {
    try h2c.start();

    while (h2c.nextFrame()) |frame| {
        std.log.info("<<< {} {}", frame);

        var payload = frame.payload;

        switch (payload) {
            .settings => |*settings| {
                while (settings.next()) |setting| {
                    std.log.info("    {}", .{setting});
                } else |err| {
                    if (err != error.EndOfData) return err;
                }
            },
            .headers => |*headers| {
                while (headers.next()) |h| {
                    std.log.info("    {s} => {s}", h);
                } else |err| {
                    if (err != error.EndOfData) return err;
                }

                std.log.info(">>> Sending 200 OK and end stream", .{});

                try h2c.sendHeaders(.{
                    .end_stream = true,
                    .stream_id = frame.hdr.id,
                }, &.{.{
                    .indexed = .status200,
                }});
            },
            else => {},
        }
    } else |err| {
        if (err != error.EndOfData)
            return err;
    }
}

So again it is an iterator that goes over the incoming frames from any stream. I have omitted the sendHeaders() function, but it just sends a headers frame back with :status => 200 and END_STREAM set. This closes the stream and Curl seems to go away happy.

Zelf Zerve 2

Finally I converted my static site server to use barely HTTP/2. It’s currently pretty useless due to the lack of TLS, which I’ll come to in a minute. To my knowledge browsers refuse to use HTTP/2 without TLS. So all we can do is use Curl.

I suppose one part that might be interesting is the use of std.os.sendfile to populated data frames.

    while (send_total < file_len) {
        const len_left = file_len - send_total;
        const frame_len = std.math.min(max_frame_len, len_left);
        const data_hdr = http2.FrameHdr{
            .length = @intCast(u24, frame_len),
            .type = .data,
            .flags = .{ .data = .{
                .endStream = len_left == frame_len,
                .padded = false,
            } },
            .id = stream_id,
        };
        var data_buf: [9]u8 = undefined;

        std.log.info(">>> Sending DATA {}", .{data_hdr});
        try h2c.writer.writeAll(data_hdr.to(&data_buf));

        var send_len: usize = 0;
        while (send_len < frame_len) {
            send_len += try std.os.sendfile(
                h2c.writer.handle,
                body_file.handle,
                send_total,
                frame_len - send_len,
                zero_iovec,
                zero_iovec,
                0,
            );
        }

        send_total += send_len;
    }

This is a bit more complicated than the HTTP1.1 version, but that is only because we split the data across multiple frames. It would probably be worse in HTTP1.1 if we started using chunked encoding.

Because we are working on a low level we can send the frame headers then copy the file (or page cache) data to the socket buffer in the kernel. Meaning we never have to buffer the file data in user land.

This may even be possible if we are using TLS if we can use a Linux crypto socket. Speaking of which…

TLS (TODO)

I was quite shocked that Zig already has a TLS implementation in the standard library. Presently though only the client is implemented. There is an issue open to create the server.

HTTP/2 connections are meant to be established with TLS and the ALPN extension. I suppose some clients may support HTTP/2, but not ALPN however I imagine Chrome and Firefox do. So hopefully I can find time to implement that or someone else will.

Related

• Zig Vs C - Minimal HTTP server
• Minimal Linux VM cross compiled with Clang and Zig
• Override libc’s malloc with Zig

This is an article primarily for friends and family; a warning from someone you know.

TLDR; in the unlikely event block-chain solves a real problem, no crypto token holders will profit from it. Purchasing tokens is gambling.

On a basic technical level Bitcoin is awful and has been superseded by Monero¹. The only practical use of Bitcoin was criminal activity. This is past tense because the authorities are now aware of how easily traceable it is. I have not owned any significant amount of Bitcoin for a long time. The electricity and silicon costs alone are reason enough for me to dump it. It’s depressing even having to talk about it.

So this article is not about Bitcoin. I have opinions about Bitcoin, but I’m an expert on moving bytes around, checking bits are in the corrected order, that sort of thing. I don’t know about whatever it is that keeps bitcoin going. For that you should consult Nassim Taleb’s Bitcoin “Black paper”.

This isn’t about other pure crypto “currencies”, like Monero or Reserve, either. Perhaps these can function as a medium of exchange or a store of value.². It is doubtful, but that is not something I can talk about authoritatively. So this is about block-chains and their associated tokens e.g. Ethereum, Solana etc.

The narrative is that these are going to revolutionise the world. Much like the internet has done and even the movement of compute resources to the Cloud. This is going to start in places like Africa for things like tracking the distribution of people and goods.

Of course, people living in Europe have difficulty verifying anything that happens in Africa. This is due to basic physical and cultural barriers. It is difficult to know that something is really happening in a remote part of the world. Even a place such as New York which has an excellent internet connection and is easily accessible.

There is an obvious disconnect between sensory data being recorded digitally and what is actually taking place. Not to mention how one chooses to interpret complex data. I have not the faintest idea how block-chain can help with these issues. It doesn’t sit at the interface between the physical and the virtual.

It’s not clear if block-chain can help with recording data, it’s a distributed data storage and transformation technology. There is a whole load of hardware and software in between the physical world and the block-chain.

Recording data requires a physical interaction with the world followed by a series of transmissions. Eventually the data is stored somewhere, which could be a block-chain, a combination of block-chain and other technologies or just a regular old database with some cryptography thrown in.

Transmission may require a number of hops, with transformations applied to the data along the way. This could be because the source data is too large and needs reducing. Perhaps some kind of electronic tampering can be detected using cryptography. Perhaps multiple parties need to vote on something to agree it happened. This is not only plausible, it already exists and it doesn’t require a block-chain.

Block-chains, by their very nature, require a large network to operate. Otherwise they are not distributed. Public block-chains require the public internet. By the time data reaches the public internet, there has been plenty of opportunity to tamper with it. If the local networks are not to be trusted for some reason. Then the data must already be protected cryptographically.

Still it’s possible there is a place in the world for slower semi-validated distributed databases with a fee structure built in. Which is essentially the nature of block-chains. I haven’t ruled it out as a possibility. Nor do I need to, I can sit back and wait for the technology to mature, not giving it too much time or attention.

What I can rule out, is that owning crypto tokens is an investment. At best it is speculative gambling. Something I am not above. However I much prefer to speculate on things which have some kind of connection to reality. This is just a pure game of betting on which crypto coin has the best narrative to sell or will be pumped by a big player next.

What I don’t see is a reason why the end users of a block-chain would pay sky high network fees instead of starting a new network. Public block-chains are necessarily limited in the volume of transactions they can perform. Validation and synchronisation are necessarily expensive operations in terms of computer resources. Therefor a high fee has to be paid for each transaction to pay back those who bought into the network as well as those who provided the computational resources.

Spinning up some “validator”, compute and storage nodes is relatively trivial. Just select some cloud vendors and fire up some preconfigured virtual machine images.

All of the software is Open Source and made so that independent parties can run it. Third party services, such as oracles, that use the block-chain are made to be interoperable. There is no lock-in to a particular block-chain or even to block-chain at all.

If you don’t trust cloud vendors, you can buy some preconfigured boxes and stick them wherever. If you can’t afford that then you also can’t afford to pay token owners enough to justify massive valuations. If you are not technical enough, you are not technical enough to protect your private keys. You will have to trust a third party somewhere down the line.

To cut a long story short, I can’t see a logical reason why a group of end users, people who are getting shit done, are going to pay for some tokens. If they need that technology, they will find the cheapest way of obtaining it. Paying back token holders is a useless expense and is easily avoidable.

There are no technical “network effects”, that would ensure one block chain network becomes dominant. It’s relatively easy for users to access multiple block-chains at once. It’s easy to transfer tokens between chains and even convert “smart contracts”, from one chain to another.

Possibly one chain could become dominant if network fees stayed the same as the number of users increased. In such a case though there has to be losers and it’s going to be passive token holders.

Some of these arguments could be falsified by an instance of a block chain application with stable profitability. However I can not find a single non-circular block-chain project with significant end users paying for the service.

Neither can I see a reason why I would personally use block-chain or web3 except as a free promotion. Even then I wouldn’t trust it with sensitive data. I’d ensure I can run my infra on regular cloud or bare metal as well.

This pains me, because I absolutely love the idea of “comoditizing” cloud services and giving AWS a kick in the teeth. I really wanted web3 to be viable, but looking at it pragmatically, it appears to be unnecessary complication.

Ironically block-chains are transparent, which I really like. You can peer in at everything happening through a block-chain explorer. Look at the programs being executed and what they do. Usually this is just maintaining the network itself or some Dex/NFT/game nonsense. All stuff that circles back around to block-chain.

At this point the only way to invest in block-chain is to learn the tech and try to use it. I started on that path and found a small amount of tech mixed with huge amounts of bullshit. There is always some bullshit with new tech, but the ratio here is not good.

The environment appears very conducive to making money as a freelance software developer. The problem is the level of horse shit crosses over from exaggeration and enthusiasm into outright fraud. Operating in such an environment while maintaining some sense of ethics, is more than my little heart could bear.

Frankly we would not be talking about block-chain if it weren’t for the speculative mania around Cryptos. Friends and family don’t ask me about conflict-free-replicated-data-types or cryptographically verifiable computations in general.

NFTs are the purest form of bollocks imaginable. There is nothing technically preventing anyone from copying them. Their value is a pure social construct. Frankly not too dissimilar to physical paintings and other types of art. There is nothing concrete for me to say about them.

However they probably show what block-chain really is, a social construct. The manifestation of belief in technology. NFTs are obviously bollocks and proud of it. This is fine by me. Where other types of block-chain token fall foul is they claim to be something other than art.

The closer I look at such claims, the more they appear to be chimerical. Fantasy beasts made up of parts that don’t make sense. I can’t rule out in absolute terms that some of this technology will come to fruition. Occasionally “putting wings on a horse and seeing if it flies”, works.

The probability though is small and the probability that token holders will reap the profits is zero. I have made quite a profit on Crypto myself, but my interest is rapidly waning as I continue to fail to find anything concrete at its base.

So I’m selling up and sharing my findings as fair warning. Be careful giving any of those fuckers your money.

Having spent considerable time trying to create a “CGroup” compatability layer. In addition to figuring out why a counter in the memory CGroup was underflowing. I now know for sure that I do not know what Linux Control Groups are. Before I thought maybe I did, but after a much deeper investigation, it’s clear I do not.

With a little bit of luck this article will help tip you over the edge. Perhaps it is time for a career change? Have you ever considered making things out of wood?

Joking aside, Linux Control Groups are trees of groups. A group can have processes or other groups inside it. Normally it can’t have both unless it is the root group. Internally the Kernel provides an interface to the CGroup hierarchy (or tree(s)).

So we have a generic hierarchy of groups and processes. This is taken advantage of by Controllers. Usually these encapsulate control of various resources. In particular the memory and CPU controllers allow one to restrict the amount of memory and CPU time groups can use.

Internally the kernel provides a standard interface for developing controllers. So for each controller we get a roughly similar interface. Both internally and in user land. In user land we get a file based interface, usually located at /sys/fs/cgroup.

Each controller has wildly different knobs, represented by files. Each file can produce and consume arbitrary data. Although all the files I have seen are text based.

Something to keep in mind; there are many controller specific details. Some resources require mutual exclusion while others can be “over committed”. Details such as these can break abstractions. Linux does not hold back features because they have “leaky” abstractions. This is a reason for its success and a source of confusion.

Linux also has the following maxim: do not break user land. This creates an interesting scenario when a regrettable interface is introduced. Which appears to be what happened with CGroups V1.

The first interface allowed each controller to have its own hierarchy. In V2 this was simplified to a single hierarchy. Many other things were changed as well. Including many details of the controller interfaces.

Because the kernel can’t break user land. It now must support both. It also supports hybrid configurations. Where both V1 and V2 are active at once. This is possibly because controllers are being migrated to V2 piecemeal. So some controllers are missing altogether from V2. Meanwhile others are missing features in V2.

Each controller must be exclusively mounted as V1 or V2. However we may have a mixture of different V1 and V2 controllers. It’s not clear to me whether anyone needs a hybrid setup at this point. However it is being used by various distributions.

The Linux Test Project has many tests which rely on CGroups. A lot of these were, and many still are, limited to CGroups V1. Perhaps foolishly, we decided to create a compatability layer. Below is what the author wrote in tst_cgroup.h.

The LTP CGroups API tries to present a consistent interface to the many possible CGroup configurations a system could have.

You may ask; “Why don’t you just mount a simple CGroup hierarchy, instead of scanning the current setup?”. The short answer is that it is not possible unless no CGroups are currently active and almost all of our users will have CGroups active. Even if unmounting the current CGroup hierarchy is a reasonable thing to do to the sytem manager, it is highly unlikely the CGroup hierarchy will be destroyed. So users would be forced to remove their CGroup configuration and reboot the system.

This perhaps deserves some emphasis. We need to test with specific CGroup controls, but we also have to play nice with init. There is unshare(CLONE_CGROUP_NAMESPACE) which may help, but it requires root.

The core library tries to ensure an LTP CGroup exists on each hierarchy root. Inside the LTP group it ensures a ‘drain’ group exists and creats a test group for the current test. In the worst case we end up with a set of hierarchies like the follwoing. Where existing system-manager-created CGroups have been omitted.

      (V2 Root)       (V1 Root 1)     ...     (V1 Root N)
          |                |                      |
        (ltp)            (ltp)        ...        (ltp)
       /     \          /     \                  /    \
  (drain) (test-n) (drain)  (test-n)  ...     (drain)  (test-n)

V2 CGroup controllers use a single unified hierarchy on a single root. Two or more V1 controllers may share a root or have their own root. However there may exist only one instance of a controller. So you can not have the same V1 controller on multiple roots.

It is possible to have both a V2 hierarchy and V1 hierarchies active at the same time. Which is what is shown above. Any controllers attached to V1 hierarchies will not be available in the V2 hierarchy. The reverse is also true.

Note that a single hierarchy may be mounted multiple times. Allowing it to be accessed at different locations. However subsequent mount operations will fail if the mount options are different from the first.

The user may pre-create the CGroup hierarchies and the ltp CGroup, otherwise the library will try to create them. If the ltp group already exists and has appropriate permissions, then admin privileges will not be required to run the tests.

Because the test may not have access to the CGroup root(s), the drain CGroup is created. This can be used to store processes which would otherwise block the destruction of the individual test CGroup or one of its descendants.

The test author may create child CGroups within the test CGroup using the CGroup Item API. The library will create the new CGroup in all the relevant hierarchies.

There are many differences between the V1 and V2 CGroup APIs. If a controller is on both V1 and V2, it may have different parameters and control files. Some of these control files have a different name, but similar functionality. In this case the Item API uses the V2 names and aliases them to the V1 name when appropriate.

Some control files only exist on one of the versions or they can be missing due to other reasons. The Item API allows the user to check if the file exists before trying to use it.

Often a control file has almost the same functionality between V1 and V2. Which means it can be used in the same way most of the time, but not all. For now this is handled by exposing the API version a controller is using to allow the test author to handle edge cases. (e.g. V2 memory.swap.max accepts “max”, but V1 memory.memsw.limit_in_bytes does not).

So what does this API look like? Below is an example taken from the docs.

#include "tst_test.h"
#include "tst_cgroup.h"

static const struct tst_cgroup_group *cg;

static void run(void)
{
    ...
    // do test under cgroup
    ...
}

static void setup(void)
{
    tst_cgroup_require("memory", NULL);
    cg = tst_cgroup_get_test_group();
    SAFE_CGROUP_PRINTF(cg, "cgroup.procs", "%d", getpid());
    SAFE_CGROUP_PRINTF(cg, "memory.max", "%lu", MEMSIZE);
    if (SAFE_CGROUP_HAS(cg, "memory.swap.max"))
        SAFE_CGROUP_PRINTF(cg, "memory.swap.max", "%zu", memsw);
}

static void cleanup(void)
{
    tst_cgroup_cleanup();
}

struct tst_test test = {
    .setup = setup,
    .test_all = run,
    .cleanup = cleanup,
    ...
};

This works quite nicely for the memory CGroup. Most of the time we can just translate V2 names to V1 names. There are things V2 accepts when V1 does not. For example V2 allows “max” to be written to memory.max, but V1 does not allow it to be written to memory.limit_in_bytes.

For other CGroups we are looking at some bigger issues. The following function sets the “bandwidth” of a CPU CGroup in cfs_bandwidth01. The bandwidth is the amount of CPU time used in a given period. So it is a two dimensional value.

static void set_cpu_quota(const struct tst_cgroup_group *const cg,
              const float quota_percent)
{
    const unsigned int period_us = 10000;
    const unsigned int quota_us = (quota_percent / 100) * (float)period_us;

    if (TST_CGROUP_VER(cg, "cpu") != TST_CGROUP_V1) {
        SAFE_CGROUP_PRINTF(cg, "cpu.max",
                   "%u %u", quota_us, period_us);
    } else {
        SAFE_CGROUP_PRINTF(cg, "cpu.cfs_period_us",
                  "%u", period_us);
        /* Actually cpu.cfs_quota_us, but we translate it */
        SAFE_CGROUP_PRINTF(cg, "cpu.max",
                   "%u", quota_us);
    }

    tst_res(TINFO, "Set '%s/cpu.max' = '%d %d'",
        tst_cgroup_group_name(cg), quota_us, period_us);
}

Note that we must branch on the CGroup Controller version. In V1 two files were used to represent the two values representing the bandwidth. In V2 these were combined. Presently our translation layer can’t handle something like this. It’s not entirely clear if it is needed. Often the extra complication to the library code is not worth saving some lines in the tests.

Likely there are many other corner cases. On the plus side, we are now able to run some tests on way more setups. Practically speaking the change from V1 to V2 did break user land. At least it broke LTP. Although to be fair LTP is not a real user. Still some tests stopped working because of the introduction of V2. Well technically the adoption of V2 configs by init systems like Systemd broke LTP…

So you have a large C code base. It has its own libraries, rules and context. You see the same mistakes being repeated. Not general C coding mistakes, but errors unique to your project.

They look like the kind of mistakes which could be detected at “compile time”. In fact, they may only be detectable at compile time. Put another way, we think we can find bugs in the source code without running it. This is commonly referred to as “static analysis”.

The Linux Test Project is a large and eccentric C code base. It has its own library and rules. We most definitely see the same mistakes, again and again. Despite having spent years developing the LTP, the present author still makes those mistakes.

Beyond simple regular expressions and syntactic checks. Writing static analysis tools is hard. Some languages come equipped with self analysis or reflection. Needless to say, this is not the case for C.

While we can justify a significant investment in developing checks. It can not be on the order of creating a new compiler. Much code review, feedback delays and bugs can be eliminated with checks. However the effort saved by static analysis, should not be all spent on developing static analysis.

Luckily C does have tooling available to develop semantic checks and perform various types of analysis. For the LTP we investigated a range of tools. This article will review those tools and explain our choices.

For the time being we have chosen Sparse to power our primary tool. This is not the most powerful, nor arguably the most user friendly. It is the most self-contained and small enough to vendor in.

Background

Program representation

There are many ways to represent a computer program. With different abstractions and structures. One may also partially evaluate a program with given assumptions. Theoretically there is no difference between transforming a program and running it. They are both computations.

For the sake of this review and the simple minds of quality assurance engineers, we just need to roughly identify what level each tool works on.

1. Text
2. Abstract Syntax Tree (AST)
3. Linearised Intermediate Representation (IR)
4. Exploded Graph or IR with state tracking

The first level is the raw source code. Secondly we have the AST which is fairly well known. Then we have IR and IR with state. IR is a generic term which we shall take some liberties with.

C compiler’s usually convert the AST into a collection of linear instruction blocks (basicblocks). These blocks are linked together into a graph or network (Control Flow Graph). The links (graph edges) represent function calls, jumps or branches.

The instructions within a block are sequential. Meanwhile one may go between blocks in whatever order the logic allows. Usually compilers also convert the IR into Single Static Assignment (SSA) form. Meaning each variable is only assigned to once. New variable names are generated for each additional assignment.

In this article we use IR to mean approximately the above. If you look in the compiler of a functional language you may find something utterly different. Also most compilers tend to use a different IR for assembly generation. We are not concerned with that.

IR with state is where we are given a range of possible values for each variable. This means we may also be given multiple versions of the code for each branch which sets a variable differently. Thus resulting in an “exploded” CFG.

If that makes no sense to you, then imagine being able to see multiple parallel universes. With the caveat that each universe could be further split into more exact possibilities.

For many checks, the AST is not an ideal level of abstraction. Even just figuring out if a memory location is modified from the AST can be difficult. There are lots of syntactic constructs which will cause a store instruction to be issued.

The reason compilers have “linearized” IR, is to cut out a bunch of syntactic details. Making it easier to perform optimisations and machine code generation. Sometimes the AST is the best place to do certain types of analysis, but often not.

LTP Requirements

The review of each tool is heavily influenced by the LTP’s requirements. With a different project in mind, one may come to very different conclusions.

The LTP project can not tolerate more barriers to development. We have contributors from many different organisations. All using different Linux distros. There are even some downstream forks of LTP on other operating systems.

The test API is very large, there is a lot to learn for a new contributor. Not to mention that the thing we are testing is exceedingly complicated.

There are many things we want to create checks for. However we have begun by trying to enforce the following rule:

The test author is guaranteed that the test API will not modify the
TST_ERR and TST_RET. This prevents silent errors where the return
value and errno are overwritten before the test has chance to
check them.

These global variables are used by the TEST() macro and similar. These are intended for exclusive use by the test author. However they were being written to by library code.

Possibly there is somme way to define these variables as const in library code, but not in test code. In general though we want an extensible way of performing checks. This seemed as good a place as any to start.

The goal is to enforce these checks and for contributors to run them locally. Both to save reviewer time and to save contributor time. It is much less expensive to correct mistakes before sending a patch.

If checks are not mandatory, then they tend to be forgotten about. Or it becomes one person’s job to run and maintain them. This then means the check have to be robust. We need to avoid false positives and allow checks to be disabled sometimes.

Tool overview

We took a look at the following tools.

• GCC
  • Analyzer
  • Plugins
• Smatch
• Clang
  • Clang Plugins
  • Clang analyzer
  • Clang Tidy
  • Clang LibTooling
  • libclang
• Coccinelle
• Sparse
• Tree-sitter

There are more out there. These are not the only ones we found even. We just have limited time and resources.

We did not investigate any proprietary tools. It is expected that LTP developers can freely download, modify and run any mandatory development tools.

The amount of time and effort assigned to each tool was not equal. They have been listed roughly in the amount of progress made before giving up. With tools such as GCC I quickly abandoned them. This is as much due to the nature of the LTP as the tool in question.

GCC

The GCC compiler is available on practically every Linux distribution and desktop OS. It is the main compiler used to build LTP. We don’t need to worry about parsing problems and non-standard C when using GCC.

Some of the older parts of the LTP are quite disgusting. They are not compliant with any C standard, but GCC has accepted them. We of course want to remove this code, but in the meantime we have to deal with it.

GCC Analyzer

The GCC Analyzer appears to be powerful if inaccurate. It tracks program control and data flow. Including between procedures, which is often absent in analysers of this type. This means you can see an approximation of what values a variable may take at any given point in the program.

At the time we investigated it, there did not seem to be any way to extend it. So we couldn’t use it to develop LTP specific checks without forking GCC.

It did find some general errors. Such as null pointer dereferences. It also found some false positives and missed other errors.

GCC Plugins

GCC has plugins which allow one to interfere with various compilation passes. They provide access to more than one type of intermediate representation used by GCC.

The access is both read and write. So we could also create our own instrumentation, that is, insert runtime checks.

Primarily there are two reasons for discarded this option. Firstly GCC’s code base is nearly opaque. Secondly plugins appear to be version dependent. Possibly GCC’s internal representation does not change much. However even small changes would create issues when the resident compiler “expert” is not around.

This means we would have high up front cost and ongoing maintenance. This is a shame because we can always rely on LTP developers to have GCC.

Smatch

The Smatch analyser is in the same league as the GCC Analyzer and Clang Analyzer. It can do inter-procedural control flow and state tracking. At least if you can figure out how to operate it.

To get the full might of Smatch one needs to generate a database. This required quite some time and fiddling for the LTP. It was never quite clear if this was fully working. On the other hand, it found some general bugs without any false positives.

To extend Smatch we could either fork it or submit LTP specific tests upstream. It is clear how a new check is added to Smatch. It is less obvious how to construct the check logic.

Smatch now uses Sparse to parse the C AST. Otherwise it seems to be its own beast. Possibly the only analyser of this type which can find bugs in the Linux kernel without producing huge amounts of noise.

We discarded it for now because of the high level of friction. In the future we may able to swap our checks from Sparse to Smatch.

Clang

The LLVM C frontend. Clang is essentially part of LLVM. All of the tools below are in the LLVM mono repository. It appears that they all get wrapped up into LLVM releases.

While LLVM and Clang are supported by every major Linux distribution. It is often a much older version on stable releases. We also found that compiling against LLVM on multiple distributions is inconvenient.

LLVM comes with the llvm-config utility to help figure out what compiler flags are needed and such. This itself comes in multiple versions on some distributions. There isn’t necessarily a default version either.

Unsurprisingly building LLVM and Clang from source is quite time consuming. So we can not sidestep distribution package issues by vendoring it in to the LTP.

Clang can output LLVM IR. We could read this and perform checks on it. We did not see an easy way to do this. So it was not properly investigated.

Clang Plugins

Like GCC, Clang has plugins. These appear to be based on the same interface(s) as Clang Tidy and LibTooling described below.

Clang Analyzer

The Clang Analyzer is another powerful analyser capable of tracking state. It is comparable to GCC’s analyser described earlier. Less so to Smatch which is far more self-contained. Out of the analysers, Clang appears to produce the most false positives.

Unlike GCC it has an extention mechanism. It’s not clear how well supported or popular this is. It appears that analyser extensions do not get inter-procedural state information.

It was dismissed primarily for the same reasons as LibTooling and Clang Tidy. Although it provides an exploded graph instead of AST matching. The analyser appears to be accessible through Clang Tidy and LibTooling.

This is a very attractive option for those already invested in the LLVM ecosystem.

Clang Tidy

Checks developed with the clang-tidy command need to be added to the LLVM mono repository. Other project specific checks have been added to upstream. So perhaps LTP specific checks would also be accepted.

The issue for us is the time between a check being accepted into LLVM upstream and the check being available to all LTP contributors. Considering the frequency of LLVM releases and stable distribution releases. It could be years before we can demand test developers run the checker.

Demanding our contributors download and compile LLVM is not reasonable. So we can dismiss the Clang Tidy approach.

Clang LibTooling

Clang has an unstable C++ interface and a stable C interface. LibTooling represents the C++ interface.

As the C++ interface is not stable, any checks written with it will need to be adapted for each LLVM release. Although Clang and LLVM are much less opaque than GCC. We still can’t afford that kind of maintenance.

The LTP is also written in C not C++. This is only a minor point, but it does save some effort to use C throughout.

libclang

This is the stable C interface. It is a wrapper for the C++ interface. The main advantage is that functions are only added, not changed or removed.

It appears that the interface’s primary clients are text editors. Specifically to allow features like auto completion. The primary header is even called Index.h.

It does provide some access to the AST. This is done through a relatively simple and well documented API. Combined with its stability promises, we decided this was enough to give it a serious try.

Below is the code which performs the check in version three of the patch series. Code for printing errors and such has been removed.

#include <clang-c/Index.h>

/* The rules for test, library and tool code are different */
enum ltp_tu_kind {
    LTP_LIB,
    LTP_OTHER,
};

/* Holds information about the TU which we gathered on the first pass */
static struct {
    enum ltp_tu_kind tu_kind;
} tu_info;

static int cursor_cmp_spelling(const char *const spelling, CXCursor cursor)
{
    CXString cursor_spelling = clang_getCursorSpelling(cursor);
    const int ret = strcmp(spelling, clang_getCString(cursor_spelling));

    clang_disposeString(cursor_spelling);

    return ret;
}

static int cursor_type_cmp_spelling(const char *const spelling, CXCursor cursor)
{
    CXType ctype = clang_getCursorType(cursor);
    CXString ctype_spelling = clang_getTypeSpelling(ctype);
    const int ret = strcmp(spelling, clang_getCString(ctype_spelling));

    clang_disposeString(ctype_spelling);

    return ret;
}

/*
 * Check if the TEST() macro is used inside the library.
 *
 * This check takes an AST node which should already be known to be a
 * macro expansion kind.
 *
 * If the TU appears to be a test executable then the test does not
 * apply. So in that case we return.
 *
 * If the macro expansion AST node is spelled TEST, then we emit an
 * error. Otherwise do nothing.
 */
static void check_TEST_macro(CXCursor macro_cursor)
{
    if (tu_info.tu_kind != LTP_LIB)
        return;

    if (!cursor_cmp_spelling("TEST", macro_cursor)) {
        emit_check_error(macro_cursor,
               "TEST() macro should not be used in library");
    }
}

/* Recursively visit each AST node and run checks based on node kind */
static enum CXChildVisitResult check_visitor(CXCursor cursor,
                         attr_unused CXCursor parent,
                         attr_unused CXClientData client_data)
{
    CXSourceLocation loc = clang_getCursorLocation(cursor);

    if (clang_Location_isInSystemHeader(loc))
        return CXChildVisit_Continue;

    switch (clang_getCursorKind(cursor)) {
    case CXCursor_MacroExpansion:
            check_TEST_macro(cursor);
        break;
    default:
        break;
    }

    return CXChildVisit_Recurse;
}

static void collect_info_from_args(const int argc, const char *const *const argv)
{
    int i;

    for (i = 0; i < argc; i++) {
        if (!strcmp("-DLTPLIB", argv[i])) {
            tu_info.tu_kind = LTP_LIB;
        }
    }
}

int main(const int argc, const char *const *const argv)
{
    CXIndex cindex = clang_createIndex(0, 1);
    CXTranslationUnit tu;
    CXCursor tuc;
    enum CXErrorCode ret;

    tu_info.tu_kind = LTP_OTHER;
    collect_info_from_args(argc, argv);

    ret = clang_parseTranslationUnit2(
        cindex,
        /*source_filename=*/NULL,
        argv + 1, argc - 1,
        /*unsaved_files=*/NULL, /*num_unsaved_files=*/0,
        CXTranslationUnit_DetailedPreprocessingRecord,
        &tu);

    if (ret != CXError_Success) {
        emit_error("Failed to parse translation unit!");
        return 1;
    }

    tuc = clang_getTranslationUnitCursor(tu);

    clang_visitChildren(tuc, check_visitor, NULL);

    /* Stop leak sanitizer from complaining */
    clang_disposeTranslationUnit(tu);
    clang_disposeIndex(cindex);

    return error_flag;
}

The above code uses Clang to create an AST from a C file (Translation Unit; TU). Then recurses into the AST, checking the type (kind) of each node (cursor). If we find a node of a kind we can check, then we call a checking function on it.

The LTP build system passes the same flags it would pass to the compiler. In addition we add -resource-dir $(shell $(CLANG) -print-resource-dir). Because libclang can not find the compiler’s resource directory.

The resource directory contains some compiler specific headers and libraries. The clang command is able to find it automatically. The code which performs this search is not in the Clang library.

We search the arguments for -DLTPLIB which tells us if we are compiling the test library. The TEST() macro check only applies to the test library. In a previous version we looked at the code itself to decide if the the file were a test.

/* If we find `struct tst_test = {...}` then record that this TU is a test */
static void info_ltp_tu_kind(CXCursor cursor)
{
    CXCursor initializer;

    if (clang_Cursor_hasVarDeclGlobalStorage(cursor) != 1)
        return;

    if (cursor_cmp_spelling("test", cursor))
        return;

    if (cursor_type_cmp_spelling("struct tst_test", cursor))
        return;

    initializer = clang_Cursor_getVarDeclInitializer(cursor);

    if (!clang_Cursor_isNull(initializer))
        tu_info.tu_kind = LTP_TEST;
}

Apart from being more complicated, the problem here was clang_Cursor_getVarDeclInitializer. This function was only introduced in LLVM 12. Meanwhile stable Ubuntu was on LLVM 10. It’s not clear how to achieve the same thing without this function.

There is another problem with our TEST() check. The actual requirement is to ensure the variables TST_ERR and TST_RET are not written to. Determining from the AST if a variable is written to is awkward enough. In libclang’s case it seems to be impossible. The necessary information is not exposed.

The amount of friction simply integrating with libclang is probably enough for us to have dismissed it. Even if that were not the case though, there is too much stuff missing for it to be useful.

If you can use LLVM at all, it is better to use the C++ interface.

Coccinelle

Also known as the spatch command. It is described as a semantic patch tool. It implements a pattern matching language which looks somewhat like a C code “diff”.

These patterns match against the syntax, semantics and control flow of C code. Under the hood Coccinelle operates on one or more IRs of the C program. However the user is not exposed to that. We are given a quirky language which looks like a Git commit to some C code.

Apparently a Coccinelle semantic patch is compiled into Control Tree Logic (CTL) and this is matched against some representation of the C code. This is perhaps analogous to how a regular expression is compiled into an automata and the automata matches the input text.

As far as we can tell, Coccinelle does not track state automatically. It does understand control flow however. Limited state tracking can be added using Python or OCaml snippets. These may be attached at certain points in the matching process.

All in all, You can be forgiven for thinking it works by magic. The tool has multiple papers and presentations. There is quite a bit of documentation. Still it is difficult to grasp. One suspects this is due to some misconceptions and communication issues. Perhaps the notes below will help.

1. There is no plain text or C code in a semantic patch. It all has meaning specified by the domain specific language. It looks like C code mixed with some special symbols, but it is not.

2. Matching takes the control flow into consideration. You can specify that all branches must match. Or that one or more matches exists.

3. You can match against the spelling of variables and other syntactic details. However it is primarily matching against the deeper structure of the program.

With these things in mind you may have more of a chance understanding the documentation.

smatch does not have helpful error messages. The implementation is also opaque to us (more on that later). So the process of writing a semantic patch is often blind trial and error, mixed with reading the docs and examples.

That said it is a truly wonderful tool. We made a lot of progress in a short time. Below is a semantic patch which both finds and (almost) fixes TEST() macro usages.

// Find and fix violations of rule LTP-002

// Set with -D fix
virtual fix

// Find all positions where TEST is _used_.
@ depends on !fix exists @
@@

* TEST(...);

// Below are rules which will create a patch to replace TEST usage
// It assumes we can use the ret var without conflicts

// Fix all references to the variables TEST modifies when they occur in a
// function where TEST was used.
@ depends on fix exists @
@@

 TEST(...)

 ...

(
- TST_RET
+ ret
|
- TST_ERR
+ errno
|
- TTERRNO
+ TERRNO
)

// Replace TEST in all functions where it occurs only at the start. It
// is slightly complicated by adding a newline if a statement appears
// on the line after TEST(). It is not clear to me what the rules are
// for matching whitespace as it has no semantic meaning, but this
// appears to work.
@ depends on fix @
identifier fn;
expression tested_expr;
statement st;
@@

  fn (...)
  {
-   TEST(tested_expr);
+   const long ret = tested_expr;
(
+
    st
|

)
    ... when != TEST(...)
  }

// Replace TEST in all functions where it occurs at the start
// Functions where it *only* occurs at the start were handled above
@ depends on fix @
identifier fn;
expression tested_expr;
statement st;
@@

  fn (...)
  {
-   TEST(tested_expr);
+   long ret = tested_expr;
(
+
    st
|

)
    ...
  }

// Add ret var at the start of a function where TEST occurs and there
// is not already a ret declaration
@ depends on fix exists @
identifier fn;
@@

  fn (...)
  {
+   long ret;
    ... when != long ret;

    TEST(...)
    ...
  }

// Replace any remaining occurrences of TEST
@ depends on fix @
expression tested_expr;
@@

-   TEST(tested_expr);
+   ret = tested_expr;

This has been merged into the LTP. However we determined that Coccinelle can not be forced upon LTP contributors. Despite the fact Coccinelle is stable and has been around for years. We ran into distribution issues. It seems that at least the Gentoo package is lacking a maintainer.

We suspect this has little to do with Coccinelle itself. The issue is that it is written in OCaml. Package maintainers struggle with OCaml projects. It is easy to see why, as our attempts to learn the basics of OCaml were fraught with issues.

For a tool as good as Coccinelle, some of us are willing to learn a new language. If it were Haskell, for example, we’d not have a problem fixing the occasional issue.

However everyones’ patience ran out with OCaml. Being functional when we are primarily working on C does not help. However the main issue is that many distributions are not maintaining the packages properly. So it is often difficult just to get the REPL and compiler running.

I personally have no opinion on whether it is a good language. I didn’t get far enough to decide that. It seems to be the case though that people are not interested in it. Meanwhile we want to get some static analysis done, not revive a struggling language.

We still merged the Coccinelle scripts into the LTP. They provide a useful example of how to automate changes with spatch. We haven’t found another option for making these kinds of changes. Using Clang Tidy is extremely laborious compared to writing a semantic patch.

Sadly it has to be dismissed as our primary checker due to the OCaml ecosystem.

Sparse

Sparse is a stand alone C parser library. It produces an AST and linearised IR consisting of basicblocks. In fact it can produce executable x86 code or LLVM IR. So it is essentially a compiler.

Unlike most C compilers however it is very simple. It is not designed to produce fast code, nor can it parse everything GCC can. It only parses C and is not concerned with C++.

Sparse itself can be compiled relatively quickly and has few dependencies. It doesn’t take long to clone it with Git either. This meant we were able to vendor it in as a Git module.

Some disapprove of vendoring and Git modules. Unfortunately the Sparse package available on most distributions is not useful to us. Sparse is linked statically and the package only contains an executable for use with the Linux kernel. There is no dynamic library. Of course someone can change that, but it would take time to propagate downstream.

The IR is relatively easy to traverse and write checks against. The documentation is maybe a little sparse. However with some knowledge about compilers, it’s not too hard to understand the code. It is written in a similar style to the kernel and LTP. Albeit with some quirks.

Below is the full checker program sans some boilerplate.

/* The rules for test, library and tool code are different */
enum ltp_tu_kind {
    LTP_LIB,
    LTP_OTHER,
};

static enum ltp_tu_kind tu_kind = LTP_OTHER;

/* Check for LTP-002
 *
 * Inspects the destination symbol of each store instruction. If it is
 * TST_RET or TST_ERR then emit a warning.
 */
static void check_lib_sets_TEST_vars(const struct instruction *insn)
{
    static struct ident *TST_RES_id, *TST_ERR_id;

    if (!TST_RES_id) {
        TST_RES_id = built_in_ident("TST_RET");
        TST_ERR_id = built_in_ident("TST_ERR");
    }

    if (insn->opcode != OP_STORE)
        return;
    if (insn->src->ident != TST_RES_id &&
        insn->src->ident != TST_ERR_id)
        return;

    warning(insn->pos,
        "LTP-002: Library should not write to TST_RET or TST_ERR");
}

static void do_basicblock_checks(struct basic_block *bb)
{
    struct instruction *insn;

    FOR_EACH_PTR(bb->insns, insn) {
        if (!bb_reachable(insn->bb))
            continue;

        if (tu_kind == LTP_LIB)
            check_lib_sets_TEST_vars(insn);
    } END_FOR_EACH_PTR(insn);
}

static void do_entrypoint_checks(struct entrypoint *ep)
{
    struct basic_block *bb;

    FOR_EACH_PTR(ep->bbs, bb) {
        do_basicblock_checks(bb);
    } END_FOR_EACH_PTR(bb);
}

/* Compile the AST into a graph of basicblocks */
static void process_symbols(struct symbol_list *list)
{
    struct symbol *sym;

    FOR_EACH_PTR(list, sym) {
        struct entrypoint *ep;

        expand_symbol(sym);
        ep = linearize_symbol(sym);
        if (!ep || !ep->entry)
            continue;

        do_entrypoint_checks(ep);

        if (dbg_entry)
            show_entry(ep);
    } END_FOR_EACH_PTR(sym);
}

static void collect_info_from_args(const int argc, char *const *const argv)
{
    int i;

    for (i = 0; i < argc; i++) {
        if (!strcmp("-DLTPLIB", argv[i]))
            tu_kind = LTP_LIB;
    }
}

int main(int argc, char **argv)
{
    struct string_list *filelist = NULL;
    char *file;

    /* ... Disable a bunch of inbuilt checks ... */

    do_output = 0;

    collect_info_from_args(argc, argv);

    process_symbols(sparse_initialize(argc, argv, &filelist));
    FOR_EACH_PTR(filelist, file) {
        process_symbols(sparse(file));
    } END_FOR_EACH_PTR(file);

    report_stats();
    return 0;
}

Unlike the Clang and Coccinelle checks, this actually checks the variables themselves. We traverse the IR and look for writes to them. This will catch some additional cases where we write to the variables without using the macros.

It may be possible to fool it somehow. It does have the issue that library header files are considered part of the test code. We have only just begun to use Sparse so this will likely be modified over time.

For now we don’t try to do anything with the AST, we just look at the IR. Unlike Clang, Sparse does not save information about macro expansions. They do not show up as nodes in the AST. It appears that preprocessing is performed without saving any details. We may need to change this.

Sparse has many built-in checks and warnings. We have disabled most of them for now. In some cases they are kernel specific. In other cases they have been adopted by GCC and Clang which produce prettier warnings. Mostly though we just need to clean up the LTP, then we can enable them.

Sparse also introduces some attributes (e.g. __attribute__(address_space(name))) which may be useful or not. Attributes are a way of extending C which does not interfere with compilers that do not support them. The kernel uses them to prevent functions and variables being used in certain ways.

Tree-sitter

Since writting this article and adopting Sparse I discovered Tree-sitter thanks to Weggli. Weggli is a very fast “semantic search tool” inspired by Coccinelle amongst others. I’d say it’s more of an AST matcher. As far as I know it doesn’t have the control flow analysis features of Coccinelle. On the plus side I can see myself contributing to it as it is written in Rust. It’s also much faster than Coccinelle and easy to install.

Just go and try it, it should only take 5 minutes if you are willing to install it with Rust’s cargo command. I often use it now for searching the Linux tree as it tends to find things that clangd doesn’t because compile_commands.json doesn’t have some files in it due to the build configuration.

However for the purposes of the LTP checker, it’s Tree-sitter that is really interesting. Tree-sitter ticks a lot of the boxes in our requirements: it generates zero dependency parsers written in C. These can easily be vendored in.

It supports many languages including C and Bash. We also have many tests written in Shell and a Shell test API. So there is also the possiblity of producing LTP specific checks for Shell as well.

It only operates at the AST level, but is vastly easier to understand than Sparse’s AST. For one thing it has a nice CLI for interactively inspecting ASTs and even a web based playground. The C API appears to have some proper documentation and looks straightforward compared to Sparse.

The problem is the lack of linearization. Some things are just much easier with some IR, that’s why it exists. There is also the fact we already have something working in Sparse. Still I would not rule out us using Tree-sitter.

Conclusion

Going forwards we will continue to develop Sparse as our main tool. We may still need to abandon it. Perhaps the checks we really want will be too difficult. Personally, I will also continue to use Coccinelle, especially for “evolutionary development”.

There is a huge amount of great software here. Which took a lot of hard work by smart people. As usual with open source it is rough around the edges. In the end we chose the solution which we are mostly likely able to fix ourselves. Also the solution least likely to need fixing once implemented.

Depending on how things progress, I will be back to write about using Sparse and Coccinelle. Please send any suggestions, praise or insults via the contact details below.

So you have a large C code base. It has its own libraries, rules and context. You see the same mistakes being repeated. Not general C coding mistakes, but errors unique to your project.

They look like the kind of mistakes which could be detected at “compile time”. In fact, they may only be detectable at compile time. Put another way, we think we can find bugs in the source code without running it. This is commonly referred to as “static analysis”.

The Linux Test Project is a large and eccentric C code base. It has its own library and rules. We most definitely see the same mistakes, again and again. Despite having spent years developing the LTP, the present author still makes those mistakes.

Beyond simple regular expressions and syntactic checks. Writing static analysis tools is hard. Some languages come equipped with self analysis or reflection. Needless to say, this is not the case for C.

While we can justify a significant investment in developing checks. It can not be on the order of creating a new compiler. Much code review, feedback delays and bugs can be eliminated with checks. However the effort saved by static analysis, should not be all spent on developing static analysis.

Luckily C does have tooling available to develop semantic checks and perform various types of analysis. For the LTP we investigated a range of tools. This article will review those tools and explain our choices.

For the time being we have chosen Sparse to power our primary tool. This is not the most powerful, nor arguably the most user friendly. It is the most self-contained and small enough to vendor in.

Background

Program representation

There are many ways to represent a computer program. With different abstractions and structures. One may also partially evaluate a program with given assumptions. Theoretically there is no difference between transforming a program and running it. They are both computations.

For the sake of this review and the simple minds of quality assurance engineers, we just need to roughly identify what level each tool works on.

1. Text
2. Abstract Syntax Tree (AST)
3. Linearised Intermediate Representation (IR)
4. Exploded Graph or IR with state tracking

The first level is the raw source code. Secondly we have the AST which is fairly well known. Then we have IR and IR with state. IR is a generic term which we shall take some liberties with.

C compiler’s usually convert the AST into a collection of linear instruction blocks (basicblocks). These blocks are linked together into a graph or network (Control Flow Graph). The links (graph edges) represent function calls, jumps or branches.

The instructions within a block are sequential. Meanwhile one may go between blocks in whatever order the logic allows. Usually compilers also convert the IR into Single Static Assignment (SSA) form. Meaning each variable is only assigned to once. New variable names are generated for each additional assignment.

In this article we use IR to mean approximately the above. If you look in the compiler of a functional language you may find something utterly different. Also most compilers tend to use a different IR for assembly generation. We are not concerned with that.

IR with state is where we are given a range of possible values for each variable. This means we may also be given multiple versions of the code for each branch which sets a variable differently. Thus resulting in an “exploded” CFG.

If that makes no sense to you, then imagine being able to see multiple parallel universes. With the caveat that each universe could be further split into more exact possibilities.

For many checks, the AST is not an ideal level of abstraction. Even just figuring out if a memory location is modified from the AST can be difficult. There are lots of syntactic constructs which will cause a store instruction to be issued.

The reason compilers have “linearized” IR, is to cut out a bunch of syntactic details. Making it easier to perform optimisations and machine code generation. Sometimes the AST is the best place to do certain types of analysis, but often not.

LTP Requirements

The review of each tool is heavily influenced by the LTP’s requirements. With a different project in mind, one may come to very different conclusions.

The LTP project can not tolerate more barriers to development. We have contributors from many different organisations. All using different Linux distros. There are even some downstream forks of LTP on other operating systems.

The test API is very large, there is a lot to learn for a new contributor. Not to mention that the thing we are testing is exceedingly complicated.

There are many things we want to create checks for. However we have begun by trying to enforce the following rule:

The test author is guaranteed that the test API will not modify the
TST_ERR and TST_RET. This prevents silent errors where the return
value and errno are overwritten before the test has chance to
check them.

These global variables are used by the TEST() macro and similar. These are intended for exclusive use by the test author. However they were being written to by library code.

Possibly there is somme way to define these variables as const in library code, but not in test code. In general though we want an extensible way of performing checks. This seemed as good a place as any to start.

The goal is to enforce these checks and for contributors to run them locally. Both to save reviewer time and to save contributor time. It is much less expensive to correct mistakes before sending a patch.

If checks are not mandatory, then they tend to be forgotten about. Or it becomes one person’s job to run and maintain them. This then means the check have to be robust. We need to avoid false positives and allow checks to be disabled sometimes.

Tool overview

We took a look at the following tools.

• GCC
  • Analyzer
  • Plugins
• Smatch
• Clang
  • Clang Plugins
  • Clang analyzer
  • Clang Tidy
  • Clang LibTooling
  • libclang
• Coccinelle
• Sparse
• Tree-sitter

There are more out there. These are not the only ones we found even. We just have limited time and resources.

We did not investigate any proprietary tools. It is expected that LTP developers can freely download, modify and run any mandatory development tools.

The amount of time and effort assigned to each tool was not equal. They have been listed roughly in the amount of progress made before giving up. With tools such as GCC I quickly abandoned them. This is as much due to the nature of the LTP as the tool in question.

GCC

The GCC compiler is available on practically every Linux distribution and desktop OS. It is the main compiler used to build LTP. We don’t need to worry about parsing problems and non-standard C when using GCC.

Some of the older parts of the LTP are quite disgusting. They are not compliant with any C standard, but GCC has accepted them. We of course want to remove this code, but in the meantime we have to deal with it.

GCC Analyzer

The GCC Analyzer appears to be powerful if inaccurate. It tracks program control and data flow. Including between procedures, which is often absent in analysers of this type. This means you can see an approximation of what values a variable may take at any given point in the program.

At the time we investigated it, there did not seem to be any way to extend it. So we couldn’t use it to develop LTP specific checks without forking GCC.

It did find some general errors. Such as null pointer dereferences. It also found some false positives and missed other errors.

GCC Plugins

GCC has plugins which allow one to interfere with various compilation passes. They provide access to more than one type of intermediate representation used by GCC.

The access is both read and write. So we could also create our own instrumentation, that is, insert runtime checks.

Primarily there are two reasons for discarded this option. Firstly GCC’s code base is nearly opaque. Secondly plugins appear to be version dependent. Possibly GCC’s internal representation does not change much. However even small changes would create issues when the resident compiler “expert” is not around.

This means we would have high up front cost and ongoing maintenance. This is a shame because we can always rely on LTP developers to have GCC.

Smatch

The Smatch analyser is in the same league as the GCC Analyzer and Clang Analyzer. It can do inter-procedural control flow and state tracking. At least if you can figure out how to operate it.

To get the full might of Smatch one needs to generate a database. This required quite some time and fiddling for the LTP. It was never quite clear if this was fully working. On the other hand, it found some general bugs without any false positives.

To extend Smatch we could either fork it or submit LTP specific tests upstream. It is clear how a new check is added to Smatch. It is less obvious how to construct the check logic.

Smatch now uses Sparse to parse the C AST. Otherwise it seems to be its own beast. Possibly the only analyser of this type which can find bugs in the Linux kernel without producing huge amounts of noise.

We discarded it for now because of the high level of friction. In the future we may able to swap our checks from Sparse to Smatch.

Clang

The LLVM C frontend. Clang is essentially part of LLVM. All of the tools below are in the LLVM mono repository. It appears that they all get wrapped up into LLVM releases.

While LLVM and Clang are supported by every major Linux distribution. It is often a much older version on stable releases. We also found that compiling against LLVM on multiple distributions is inconvenient.

LLVM comes with the llvm-config utility to help figure out what compiler flags are needed and such. This itself comes in multiple versions on some distributions. There isn’t necessarily a default version either.

Unsurprisingly building LLVM and Clang from source is quite time consuming. So we can not sidestep distribution package issues by vendoring it in to the LTP.

Clang can output LLVM IR. We could read this and perform checks on it. We did not see an easy way to do this. So it was not properly investigated.

Clang Plugins

Like GCC, Clang has plugins. These appear to be based on the same interface(s) as Clang Tidy and LibTooling described below.

Clang Analyzer

The Clang Analyzer is another powerful analyser capable of tracking state. It is comparable to GCC’s analyser described earlier. Less so to Smatch which is far more self-contained. Out of the analysers, Clang appears to produce the most false positives.

Unlike GCC it has an extention mechanism. It’s not clear how well supported or popular this is. It appears that analyser extensions do not get inter-procedural state information.

It was dismissed primarily for the same reasons as LibTooling and Clang Tidy. Although it provides an exploded graph instead of AST matching. The analyser appears to be accessible through Clang Tidy and LibTooling.

This is a very attractive option for those already invested in the LLVM ecosystem.

Clang Tidy

Checks developed with the clang-tidy command need to be added to the LLVM mono repository. Other project specific checks have been added to upstream. So perhaps LTP specific checks would also be accepted.

The issue for us is the time between a check being accepted into LLVM upstream and the check being available to all LTP contributors. Considering the frequency of LLVM releases and stable distribution releases. It could be years before we can demand test developers run the checker.

Demanding our contributors download and compile LLVM is not reasonable. So we can dismiss the Clang Tidy approach.

Clang LibTooling

Clang has an unstable C++ interface and a stable C interface. LibTooling represents the C++ interface.

As the C++ interface is not stable, any checks written with it will need to be adapted for each LLVM release. Although Clang and LLVM are much less opaque than GCC. We still can’t afford that kind of maintenance.

The LTP is also written in C not C++. This is only a minor point, but it does save some effort to use C throughout.

libclang

This is the stable C interface. It is a wrapper for the C++ interface. The main advantage is that functions are only added, not changed or removed.

It appears that the interface’s primary clients are text editors. Specifically to allow features like auto completion. The primary header is even called Index.h.

It does provide some access to the AST. This is done through a relatively simple and well documented API. Combined with its stability promises, we decided this was enough to give it a serious try.

Below is the code which performs the check in version three of the patch series. Code for printing errors and such has been removed.

#include <clang-c/Index.h>

/* The rules for test, library and tool code are different */
enum ltp_tu_kind {
    LTP_LIB,
    LTP_OTHER,
};

/* Holds information about the TU which we gathered on the first pass */
static struct {
    enum ltp_tu_kind tu_kind;
} tu_info;

static int cursor_cmp_spelling(const char *const spelling, CXCursor cursor)
{
    CXString cursor_spelling = clang_getCursorSpelling(cursor);
    const int ret = strcmp(spelling, clang_getCString(cursor_spelling));

    clang_disposeString(cursor_spelling);

    return ret;
}

static int cursor_type_cmp_spelling(const char *const spelling, CXCursor cursor)
{
    CXType ctype = clang_getCursorType(cursor);
    CXString ctype_spelling = clang_getTypeSpelling(ctype);
    const int ret = strcmp(spelling, clang_getCString(ctype_spelling));

    clang_disposeString(ctype_spelling);

    return ret;
}

/*
 * Check if the TEST() macro is used inside the library.
 *
 * This check takes an AST node which should already be known to be a
 * macro expansion kind.
 *
 * If the TU appears to be a test executable then the test does not
 * apply. So in that case we return.
 *
 * If the macro expansion AST node is spelled TEST, then we emit an
 * error. Otherwise do nothing.
 */
static void check_TEST_macro(CXCursor macro_cursor)
{
    if (tu_info.tu_kind != LTP_LIB)
        return;

    if (!cursor_cmp_spelling("TEST", macro_cursor)) {
        emit_check_error(macro_cursor,
               "TEST() macro should not be used in library");
    }
}

/* Recursively visit each AST node and run checks based on node kind */
static enum CXChildVisitResult check_visitor(CXCursor cursor,
                         attr_unused CXCursor parent,
                         attr_unused CXClientData client_data)
{
    CXSourceLocation loc = clang_getCursorLocation(cursor);

    if (clang_Location_isInSystemHeader(loc))
        return CXChildVisit_Continue;

    switch (clang_getCursorKind(cursor)) {
    case CXCursor_MacroExpansion:
            check_TEST_macro(cursor);
        break;
    default:
        break;
    }

    return CXChildVisit_Recurse;
}

static void collect_info_from_args(const int argc, const char *const *const argv)
{
    int i;

    for (i = 0; i < argc; i++) {
        if (!strcmp("-DLTPLIB", argv[i])) {
            tu_info.tu_kind = LTP_LIB;
        }
    }
}

int main(const int argc, const char *const *const argv)
{
    CXIndex cindex = clang_createIndex(0, 1);
    CXTranslationUnit tu;
    CXCursor tuc;
    enum CXErrorCode ret;

    tu_info.tu_kind = LTP_OTHER;
    collect_info_from_args(argc, argv);

    ret = clang_parseTranslationUnit2(
        cindex,
        /*source_filename=*/NULL,
        argv + 1, argc - 1,
        /*unsaved_files=*/NULL, /*num_unsaved_files=*/0,
        CXTranslationUnit_DetailedPreprocessingRecord,
        &tu);

    if (ret != CXError_Success) {
        emit_error("Failed to parse translation unit!");
        return 1;
    }

    tuc = clang_getTranslationUnitCursor(tu);

    clang_visitChildren(tuc, check_visitor, NULL);

    /* Stop leak sanitizer from complaining */
    clang_disposeTranslationUnit(tu);
    clang_disposeIndex(cindex);

    return error_flag;
}