Bitmapping a Battleship: Forward-Deployed Engineering at Gecko

By: Nicholas Parkes

I found myself in a US Navy shipyard in Japan wearing fire-resistant flannel, jeans, and a hard hat in 100 degree weather. Sitting on the edge of a Pelican case, I was hunched over a laptop with four terminal windows open. The hull of the destroyer towered over me, along with my team and the critical gaze of the shipyard’s senior leadership. Gecko had been contracted to perform a non-destructive inspection of the destroyer's hull. The only problem? Our robot had seized. It sat motionless, affixed to the steel of the ship's hull while the growing crowd shifted their glare between the robot and me, furiously typing away. The window of opportunity was short and closing fast; Gecko had a massive square footage scope to hit. I had 90 minutes max to get this robot moving again or else we might not get the job done.

We had brought some new hardware along for this inspection, and the ROS middleware for the system had been written on a tight deadline before our deployment across the globe. But I hadn't written the middleware. I had never even touched ROS before, and the team that had was asleep halfway across the world. So with the scrutinizing eyes of the shipyard execs bearing down on me, the clock ticking, and my body sweltering under the flannel, I decided there was no better time to start learning the fundamentals of ROS.

What does it mean to be a Forward Deployed Engineer at Gecko? Much ink has been spilled trying to answer this, but here is my best attempt: Gecko FDEs build scrappy systems under tight constraints to solve critical problems for our customers.

The Navy needed a digital twin of their destroyer sitting in dry dock, and they needed it soon. Maintenance overruns have been endemic to the Navy for decades, and a platform for targeted repairs would enable ships to get back out to sea on time and on budget. But with such a tight deadline to build a whole new product from scratch, it seemed impossible. Here’s the story of how I brought that destroyer from the world of atoms into the world of bits.

Day 0

Alongside our Komodo ship deck solution, Gecko had already built the Toka for ultrasonic data collection and Cantilever for data visualization. How hard could it be to use these existing systems to collect ultrasonic data on a ship hull?

There are two great prior posts in this Notes from the Lab series that describe the capabilities of our tech stack as this project began:

1. The Localization Revolution tells how Gecko evolved from a company using one-dimensional wheel encoders to a company using state-of-the-art LiDAR and SLAM systems for precise, robust 3D robot localization.
2. AssetViz talks about the design behind Gecko's Cantilever frontend and the methods used to overlay heat maps of scan data onto asset models. As robust as this capability was, it was built for primitive geometries: cylindrical tank shells, flat boiler walls, cones, and domes. Not Navy battleships.

Ship hulls are complex shapes. They are a patchwork of plates, coatings in various states of disintegration, and obstacles like anchors. I knew this geometry would challenge our localization tech to the limit. Whereas our supported asset classes were localized by wheel encoder, ship hulls would require a fundamentally new approach for their curved and obstacle-ridden surfaces.

Although the robotics platform team was well underway on a LiDAR-driven localization payload, it would not be ready in time for my trip to the shipyard on the other side of the planet. I couldn't wait, and neither could the Navy. I had to get the job done with the tools of today and feed field insights back into Gecko’s roadmap.

Building the Field System 

A good robotics company is always tempted to build whatever it needs in-house. A great robotics company will know when to resist that temptation - especially in the time-constrained world of forward-deployed engineering.

Gecko had a best-in-class ultrasonic sensor array delivered by the TOKA robot but did not yet have a localization system that would work for ships. It is not enough to know there is damage; port engineers must also know where to make repairs.

Novel environments bring novel challenges but can also relax previous constraints. Gecko is used to working in cramped boilers or crowded tank farms. In comparison, a shipyard is a remarkably spacious environment. So spacious that we could bring in a line-of-sight laser rangefinder; you’ve almost certainly seen a surveyor peering through one on a street corner.

By attaching a small mirrored prism to our robot, we could now track the position of the TOKA to millimeter-level precision. The robot had 24 sensors, each at a known, mechanically static distance away from the location of the prism. Using the robot's onboard inertial measurement unit, we could derive the TOKA's orientation and calculate the transform from the prism’s location to the location of each of our sensors. Now we had precise localization measurements for each of our ultrasonic readings.

By using an off-the-shelf tracker, we circumvented most of the complexity of building our own localization system, but we still needed to integrate it with the rest of our field software stack. This meant writing some middleware to take the coordinate readings from the laser and fuse them with sensor readings from the robot.

The first version of field software for localization was a quagmire of Docker containers, ROS middleware, and Linux shell scripts that fused the data in real time. We had a laser, a robot, two Windows laptops, and a Linux machine all wired together with a mess of serial connectors and ethernet cables. This is the system that I found myself debugging in the sweltering heat of that shipyard in Asia. 

After the shipyard, I tore it all down and started over. I realized that the decision to fuse data in real time had presented an unnecessary constraint.

The second version of field software for localization was a Python script that pulled coordinates from our laser tracker via serial port and saved them to timestamped CSV files on a USB stick. We then combined that set of coordinates with sensor readings manually using NumPy and Pandas back in the office. Elegant? No. Did it localize our robot just as accurately as a fully integrated sensor fusion stack would have? Absolutely.

When working toward MVPs, complexity is the enemy. Only build what you must.

A Little Bit of Linear Algebra

We now had a laser tracking the TOKA on the side of a hull, and each ultrasonic reading was paired with a corresponding coordinate in space. But the laser could not see the entire hull from a single spot. It would need to be moved from time to time in order to cover the full area, and since each position the laser reports is relative to where it is sitting, move the laser and your coordinate system moves with it. Our field software needed to be able to account for this repositioning if we wanted our resulting data to be consistent.

So we developed this piece of hardware to help with calibration. I call it the Batarang.

The Batarang has four magnetic sockets, each designed to hold a precision reflector. By mounting it on a known location on the hull, we can capture four fixed reference points. These points will be in the same physical locations every time, no matter where the laser tracker is sitting.

By designating one known point on the hull as the origin of our scan data coordinate frame, we were able to use these points to derive a rigid transform that could be applied to each new reading from our laser.

With each point P from the laser represented in homogenous coordinates, P = [x, y, z, 1]ᵀ, the rigid transformation from one coordinate frame to another is defined in 3D as a combination of rotation and translation components, each with three degrees of freedom:

After capturing four points with the Batarang, we had each point expressed in both the laser's local frame B and the unified hull frame A. Using the Kabsch algorithm, we could derive the transformation T_AB and apply it to all future laser readings to store them in our global frame with a simple matrix multiplication, P_A = T_AB·P_B. Move the laser again? Capture the points and generate a new transformation. At the end of our scan, all captured data existed in one unified point cloud with an origin at a known, referenceable location on the ship's hull.

We had our data. Now we needed somewhere to put it.

Building the Digital Twin

We could have just handed the Navy the raw point cloud. But the one place where we wouldn't compromise for this MVP was in the usefulness of our delivered product. Gecko’s digital twins wrangle all that complexity from the field into something intuitive and actionable.

Remember what our frontend visualization software could handle at this point: (x,y) coordinates on procedurally generated primitive geometries. The data was stored as a texture on this generated mesh where each pixel corresponded to a particular bin of data.

To date, Gecko had primarily inspected tanks (cylinders) and boilers (flat walls).

We can model a cylinder as a rectangle and wrap it around a known radius. Procedurally generating the hull in a similar way didn't seem possible. Would we be able to do any better than our point cloud? We debated breaking the data out by individual steel plates, diving headfirst into the development of a high-fidelity photogrammetric model, or just tasking our mechanical design team with creating a CAD model from ship drawings directly in SolidWorks.

It was a tough problem. So, naturally, I decided to procrastinate. 

I launched one of my favorite games - World of Warplanes - and saw an ad in the corner of my screen for the studio’s other franchise: World of Warships. I figured it might give me some inspiration, so I downloaded it and started playing. And there it was in front of me: a model of a warship, floating around on my screen. If game designers had models this good, maybe I could find one too. After a little research, I quickly found a thriving ecosystem of meticulously crafted, high-fidelity, ship-schematic-accurate CAD models for navy ships licensed for commercial use by third-party designers. These were typically made with games like World of Warships or War Thunder in mind, but they would work for my needs just fine. 

Only build what you must.

Now that I had my CAD model, I needed to find a way to encode the scan data into the model's texture. For a cylinder, the mapping from texture space to 3D is a simple trig function. For a hull, there's no closed-form relationship; the texture is a disjoint, arbitrary flattening of a complex surface.

At this stage, I was faced with quite a few problems:

1. How do we unwrap our 3D model into texture space in the first place?
2. How do we make each texture pixel correspond to exactly the same area in physical space if the model gets warped as it is projected from 3D to 2D?
3. How do we associate each point from our scan data to the 2D texture space?

While puzzling over these questions, I thought back to World of Warships. Perhaps there was more to be gleaned from the wisdom of the game development industry. Compared to the cutting edge of gaming and VFX, what I was doing seemed trivial.

So I found myself searching online for tutorials once more - this time on the fundamentals of Blender. Blender seemed intimidating at first until I discovered its scripting environment. From then on, it became nothing more than a Python package with a fancy GUI, and I felt right at home.

I slowly found answers to each of my problems:

1. Unwrapping was easy. Built-in Blender functions handle this automatically.
2. The right combination of Blender tools and open source add-ons allowed for massive mitigation of warping. Averaging the scale of each disjoint island yielded the biggest gains, and the uv test grid texture was particularly helpful in showing precisely where my ship still had issues. In those areas of high curvature, increasing vertex density helped to reduce the warping until the texture showed even squares across the whole of my ship's surface.
3. The following algorithm could run in a Blender script to map scan data to texture space, and allowed our millions of data readings to be projected onto the texture in a single pass:

1. Take each world-space sample (x, y, z)

2. Snap that point to the nearest point on the mesh surface

3. Identify which triangle face that nearest point lies on

4. Compute barycentric coordinates of the snapped point inside that triangle

5. Use those barycentric weights to interpolate the triangle’s per-corner 2D texture coordinates        

With a textured model in hand, I uploaded it into Cantilever alongside our other digital twins. Because it lived on the same platform, the Navy immediately got everything they'd get for any other asset - repair planning, predictive corrosion mapping, lifecycle management - without us building a single additional feature.

After all the tiny challenges along the way - every docker and linux bug in the field, every misaligned point cloud that had me scratching my head inverting matrices on an office whiteboard, every time the ship model flickered in and out of the viewport like the Flying Dutchman - it was all worth it the first time I saw the model live in production.

Epilogue

In the time since completing this MVP, we productized our hull offering. Our 3D models now have photogrammetry layers. Our robots have LiDAR. Our field software no longer runs on a USB stick and a prayer.

But there will always be a new MVP to build. A new product with seemingly insurmountable requirements. Haters will say it can’t be done. As I said at the beginning, Gecko FDEs build scrappy systems under tight constraints to solve critical problems for our customers. The job description, put another way: Proving the haters wrong.

And you bet I was happy when the ask came in: "Scanning hulls in dry dock is great. Think we can do that underwater…?"