Inspiration
Low Earth orbit is getting crowded. There are tens of thousands of tracked debris objects and far more that are too small to track. Even the International Space Station has to fire its engines now and then to avoid debris. With satellite constellations launching by the thousands, collision avoidance can't depend on someone on the ground spotting the danger and sending commands in time. The satellite has to see the danger and react by itself.
That raised the questions we built around. What does a flight computer need for its AI to be trusted with the thrusters? How do you let a neural network suggest a manoeuvre without ever letting it crash the spacecraft? That made the QNX challenge a natural fit, because a real-time OS is built for exactly that kind of guarantee.
What it does
DODGE-1 (Debris Observation and Dodge Guidance Engine) turns a Raspberry Pi 5 running QNX 8.0 into a satellite's flight computer:
- A USB webcam on the Pi watches a real black ping-pong ball (the "asteroid") as it's pushed toward the camera.
- An
int8CNN running on the Pi recognises the ball and ignores look-alikes. Its distance is computed from its known 35mm size. - The satellite's virtual orbit slot sits 30cm in front of the camera. When the ball gets within 15cm of it, the Pi's control process plans a minimum-propellant sidestep, fires simulated cold-gas thrusters, lets the asteroid pass, and flies back to its orbit slot.
- A laptop plays "the universe": rigid-body physics, a 3D view, and a dashboard showing the Pi's live camera feed with CNN overlays. Every sensing and steering decision is made on the Pi.
Our design rule: the AI suggests; the QNX control process decides.
It also survives faults:
- If the vision process is killed, it's restarted.
- If the control process freezes, it's killed and restarted.
- If the link drops, all valves close.
- If a thruster sticks open, it's detected and the satellite switches to a backup thruster.
How we built it
Architecture: three isolated QNX processes on the Pi
| Process | QNX Priority | Job |
|---|---|---|
control |
40 | 50Hz guidance, attitude, fault detection, fail-safes |
stereo_capture (C) |
20 | QNX camera library -> POSIX shared memory |
perception |
15 | candidates -> CNN -> depth -> Kalman filter |
A supervisor starts each process with on -p <priority> and watches them with heartbeats:
- A process that crashed is restarted.
- A process that hung gets
SIGKILLand a restart. - A process that keeps failing is restarted with back-off.
Perception and control never share memory; they exchange small UDP messages. Control treats every track as untrusted: it ignores old tracks and rejects malformed or non-finite values, and a detection must be confirmed in two consecutive frames before control is told about it.
Camera on QNX
QNX's sensor framework owns the USB webcam through our own config. A small C program uses the camera library (libcamapi) to receive frames in a callback, and publishes them to /dev/shmem behind a sequence lock, so perception never reads a half-written frame.
Vision
Candidates come from a dark-pixel colour mask plus a NumPy gradient-Hough circle detector. Each candidate goes through a small int8 TFLite CNN (32 x 32 input, about 1ms per candidate), with a stricter shape-and-isolation check as a fallback. The CNN was trained on patches from our own ray-traced scene renderer (hands, balls, posters, boxes, random lighting), run through the same pipeline as the Pi. With one camera, depth comes from the ball's apparent radius r in pixels:
$$ Z = \frac{f \cdot D/2}{r - b} $$
where D = 35mm, f is the focal length (calibrated with one command, holding the ball at 30cm, and b~0.6px corrects a small bias in the edge measurement. A constant-velocity Kalman filter turns positions into position and velocity.
Deciding when and how far to dodge
Real motion is amplified 3x: 1cm of ball travel is 3m in space. Control predicts the closest approach and plans a robust min-max dodge. It must clear both the asteroid's current offset and its predicted miss distance, widened by the measurement noise u. For each side s = 1 of the cross-track axis:
$$ d_s = \max_{a \in {a_{\text{now}},\, a_{\text{pred}}}} (s \cdot a) \;+\; u \;+\; \sqrt{R_{\text{safe}}^2 - a_\perp^2} $$
and it chooses the side with the smaller $d_s$. Because propellant use grows with velocity change,
$$ \dot{m} = \frac{n F}{I_{\text{sp}}\, g_0}, \qquad \Delta m \approx m \frac{\Delta v}{I_{\text{sp}}\, g_0}, $$
the satellite moves only as far and as fast as it needs to ($v = \frac{d}{0.6\,t_{\text{cpa}}}$), and returns slowly.
Physics
The laptop simulator is hardware-in-the-loop:
- Translation uses Newton's laws plus the Clohessy–Wiltshire relative-orbit equations.
- Rotation uses Euler's equations with reaction wheels and torque from an off-centre centre of mass.
- Propellant mass flow follows from the thrusters'.
If no command arrives for 150ms, a watchdog closes all valves.
Fault detection
Commanded and measured acceleration are compared over a 2 window. If the gap stays above 50% of one thruster's force for 0.3, that thruster branch is isolated and the backup branch B takes over. A thruster's last working branch is never isolated.
Challenges we ran into
- Python couldn't drive the QNX camera. On QNX 8.0, Python
ctypescallbacks crashed with a segfault even as root, so we wrote the camera bridge in C and compiled it on the Pi with QNX'sclang. - QNX 8.0 vs. 7.1 documentation mismatches. We worked these out from byte dumps:
- Camera unit numbers are shifted by one (unit
0means "none"). - Our YUYV stream reports frame type
14. - The frame-size fields of
camera_buffer_tsit 24 bytes later than documented.
- Camera unit numbers are shifted by one (unit
The C program now finds the fields by value and checks the pixel pointer safely with a pipe() write, which fails cleanly on a bad address instead of crashing.
- The sensor service. The boot-time instance expects ribbon cameras and serves nothing without them; USB devices showed up as device
2, not1; and-b externalonly works with the ribbon-camera library. - Scheduling surprises.
on -Con QNX 8.0 selects a CPU cluster, not a core, so our core-pinning attempt killed the processes. We rely on priority scheduling instead. - Stereo was harder than expected. Two cheap webcams 7cm apart kept drifting out of calibration, and the usable depth range was narrow. We switched to one camera with depth from known size, which worked better.
- Vision false positives. Dark poster corners and round buttons fooled the shape check, and the CNN can't judge size from a rescaled crop. We added an isolation test, CNN-first ranking, two-frame confirmation and, in stereo mode, a physical size check: small round decoys went from 141 of 150 frames accepted to 3 of 150.
- Physics is unforgiving. A dodge sized from a noisy prediction was sometimes too small. At high time-scale, bang-bang control chattered and drained the tank. A render stall once made fault detection blame a healthy thruster. The fixes:
- A robust min-max dodge plan.
- Deadbands scaled to the velocity change one control tick produces.
- Skipping fault-detection windows when the simulator's clock is unsteady.
Accomplishments that we're proud of
- The whole perception-to-decision chain runs on a Raspberry Pi under QNX, with no cloud: QNX sensor service, a C camera bridge, an
int8CNN, depth, a Kalman filter, guidance, attitude control, fault detection, and a supervisor. - Measured on the Pi:
- 19.9 perception
- 1.2 CNN per candidate
- ~ 20ms latency
- Camera capture at 15fps with 0 bad frames
- 12 of 12 dodge passes clear in full-system runs on the laptop, each followed by an automatic reset.
- 0 collisions in closed-loop tests at hand speeds up to 0.2m/s.
- Faults handled end to end in testing: a killed perception process was restarted, a frozen control process was killed and restarted, and a stuck-open thruster was isolated, with no false alarms.
- The CNN scored 99.97% on held-out test patches.
What we learned
- Real-time means guarantees, not speed. The CNN can be slow sometimes; control can never be late. QNX's priority-preemptive scheduler and process isolation let us design for that instead of hoping for it.
- Treat AI output as untrusted sensor data: confirm it, sanity-check it against physics, keep it fresh, and let a simple deterministic controller own the actuators.
- Every failure needs a defined safe state, and you only find them all by injecting faults on purpose.
- Reading byte dumps and system logs (
slog2info,pidin) beats trusting documentation written for a different version. - Simpler hardware (one camera plus known object size) can beat "better" hardware (stereo) when calibration is the bottleneck.
What's next for DODGE-1
- Measure QNX timing under load (control-loop jitter and overruns, with and without CPU hogs) and publish the numbers.
- Port the control loop to C with
SCHED_FIFO,clock_nanosleep, and QNX native messaging (MsgSend/MsgReceive), and use QNX adaptive partitioning to guarantee CPU budgets for safety-critical work. - Retrain the CNN on real camera patches and support several pieces of debris at once.
- Drive real actuators: GPIO LEDs or solenoid valves per thruster, then an air-bearing or 2D test rig.
- Add a planner that weighs propellant against collision probability, and coordination between satellites in a constellation.
Built With
- c
- edge-ai
- embedded
- numpy
- opencv
- panda3d
- posix
- python
- qnx
- raspberry-pi
- rtos
- stereo-vision
- tensorflow
- tflite
- udp