encelo.github.io

nCine Dev Update 23

Sat, 27 Jun 2026 00:00:00 +0000

Welcome back to another development update for the nCine, covering what has been accomplished in the last quarter of 2025 and the first half of 2026.

Before diving into the technical part of the article, I should probably mention that a few days ago marked the 15th anniversary of the first commit of the project. 🎂

Let’s also cheer for the new Hugo site and the availability of signed commits 🔑, two important infrastructure updates from last year.

GrAIL

Probably the biggest development news is the work in progress toward a Render Hardware Interface (RHI) that will allow modern graphics APIs to be employed. In the nCine the RHI is called Graphics API Integration Layer, or GrAIL.

The first iteration only supports Vulkan, but implements the underlying architecture that will make it possible to support other backends. From the HypeHype rendering architecture presentation by Sebastian Aaltonen, I borrowed the concept of using an opaque handle (made of an index and a generation, like in my job system) for all the common resources, like textures, pipelines, and buffers. I also copied the concept of immutable bind groups, an abstraction over descriptors, from WebGPU.

The entry point of GrAIL, similarly to other modern APIs, is the device object. The device is not a traditional C++ base class, but a common header with different implementation files chosen at compile time by CMake. I chose to sacrifice run-time API selection to remove a layer of virtual call dispatch.

The plan is to use Vulkan on Linux and Windows (maybe I will add D3D12 in the future), to add WebGPU for Emscripten, and to test KosmicKrisp and MoltenVK on macOS before eventually adding a Metal backend.

Another important missing piece is the actual link between GrAIL and the scenegraph system, which will be handled by a new renderer class. The renderer will make it possible to remove rendering code from the nodes and will represent a chance to reorganize rendering data in a more data-oriented fashion. For the time being I will also keep the legacy OpenGL code around, which means I will need to port the old system to this new renderer approach.

At the moment, to isolate the work on this feature, I’m taking advantage of the new NCINE_WITH_SCENEGRAPH CMake variable that allows the nCine to be compiled without any scenegraph-related classes, like the nodes, the render commands, or viewports.

There is still a lot of work before you can use GrAIL instead of OpenGL, but you can already try the grail branch on GitHub and play with the particle simulation test.

Current GrAIL apptest, with compute simulated particles and a rotating nCine logo

Batching with instances

While GrAIL represents the future, it does not mean that the OpenGL path would be deprecated any time soon. As a matter of fact, in June I have worked on optimizing regular sprite blitting.

As you might remember, the RenderBatcher class is in charge of automatically creating draw batches for every type of rendering commands that it encounters while navigating the sorted queues.

I have added another command collection path that works with regular sprites (so no TextNode or MeshSprite nodes) and uses a completely different approach. Instead of using a Uniform Buffer Object (UBO) to gather the information about each sprite and expand their 4 vertices triangle strip into two triangles and 6 vertices, it uses instancing and vertex attributes.

This introduces two nice properties: with instances we can still use the 4 vertices triangle strip primitives for each sprite, and with the attributes we can gather the data in a Vertex Buffer Object (VBO). VBOs don’t have the size restriction that UBOs have (usually 64 kb), meaning that it is now possible to batch a lot more sprites together in the same draw call.

The trick that enables vertex attributes to be used per instance and not per vertex is a call to glVertexAttribDivisor() with a divisor value of 1.

There is also another optimization that allows for bigger batches, and it also works when using UBOs: packing the data needed to render nodes.

For example, let’s have a look at the InstanceBlock structure in the sprite_vs.glsl shader:

// 104 bytes, rounded to 112 for alignment purposes
layout (std140) uniform InstanceBlock
{
 mat4 modelMatrix;
 vec4 color;
 vec4 texRect;
 vec2 spriteSize;
};

// 48 bytes in total, no padding needed
layout (std140) uniform InstanceBlock
{
 vec4 transform;
 vec4 translation;
 uint color;
 uint spriteSize;
 uint uvEndpointsU;
 uint uvEndpointsV;
};

The new structure needs less than half the memory, allowing for more than double the number of sprites in the same batch.

On the C++ code side, the framework will try to first update the new uniforms, but it will fall back to the old fields if the process fails. On the shader side, new functions will unpack the data into floats, trading the space gains for some additional low-effort per vertex computation.

Benchmarks

I have tested the instancing on my slowest laptop, a Xiaomi Mi Notebook Pro with an i7-8550U, and further limited the CPU and GPU frequencies to the minimum to stabilize the results and to highlight any performance benefit on slow devices. I ran the updated apptest_bunnymark with 5000 bunnies and no V-Sync, then I toggled instancing on and off.

xychart
title "Five seconds average runs"
x-axis ["Run 1", "Run 2", "Run 3", "Run 4", "Run 5"]
y-axis "Frametime (ms)" 0 --> 35
bar "A" [26.201, 27.067, 25.969, 26.232, 26.187]
bar "B" [19.890, 19.366, 18.398, 19.654, 19.780]

As you can see from the chart, enabling instancing cut the frametime to 74% of the baseline, from an average of 26.3312 ms to 19.4176 ms.

Faster directory traversing on Windows

I am certainly not an expert in Windows API, but I remember reading an article about how to make directory traversal faster by using FindFirstFileExA() and its additional parameters, like FIND_FIRST_EX_LARGE_FETCH. I then went to have a look in the FileSystem namespace and I experimented with the new approach. I removed some unneeded string copies, and I changed the API call from a simple FindFirstFile() to:

hFindFile_ = FindFirstFileExA(buffer, FindExInfoBasic, &findFileData, FindExSearchNameMatch, nullptr, FIND_FIRST_EX_LARGE_FETCH);

xychart
title "Scanning C:/Windows/System32 (4786 files)"
x-axis ["Run 1", "Run 2", "Run 3", "Run 4", "Run 5"]
y-axis "Time (ms)" 0 --> 4
bar [3.4946, 2.8509, 2.9092, 2.8022, 3.4254]
bar [3.2654, 2.3800, 2.3263, 2.3020, 2.4079]

In this first case, using the new API call lowered the average from 3.09646 ms to 2.53632 ms, or a new time that is 82% of the old one.

xychart
title "Scanning C:/msys64/usr/bin (604 files)"
x-axis ["Run 1", "Run 2", "Run 3", "Run 4", "Run 5"]
y-axis "Time (ms)" 0 --> 1
bar [0.7142, 0.5205, 0.5406, 0.6141, 0.6587]
bar [0.5092, 0.5084, 0.5168, 0.4214, 0.4306]

In the second case, the new API call reduced the average from 0.60962 ms to 0.47728 ms, or a new time that is 78% of the old one.

nCTL updates

Optional class

While working on GrAIL, I needed a way to control when to construct objects that are fields of other objects. I usually do this with nctl::UniquePtr, by choosing the right time to call nctl::makeUnique(), but in this case I wanted to reserve the space for the object beforehand, and avoid allocating memory.

I ended up replicating some functionalities from std::optional, a class that is usually used to return an object or a value that can be interpreted as null (when the optional class is not “engaged”) when a function returns.

The class is really just a buffer as big as its template argument, and objects are created there with a placement new call. These features satisfy my requirements and that’s how nctl::Optional was born, of course with its accompanying set of unit tests, as is tradition in nCTL.

Iterators refactoring

All iterators have been refactored so that they obey the same invariants, and behave similarly to STL.

For example, a reverse iterator at the beginning of a container is now always constructed from a regular iterator at the end of the same container. The same is valid for the symmetric situation, a reverse iterator at the end of a container.

 /// Returns a reverse iterator to the beginning
 inline ReverseIterator rBegin() { return ReverseIterator(end()); }
 /// Returns a reverse iterator to the end
 inline ReverseIterator rEnd() { return ReverseIterator(begin()); }

I now explicitly check for those equivalences in the unit tests, together with other invariant checks, like checking that when I increment then decrement a begin() iterator, or do the opposite with an end() one, I don’t alter it in a way that makes it no longer equivalent to begin() or end() themselves.

Last but not least, I rewrote the operator!= in all iterators to be consistent and just always negate the result from operator==.

Pair class

Just like nctl::Optional, the template library keeps extending with some less used but sometimes useful classes, for example a reimplementation of std::pair.

It does not replace the custom pair implementation used by nctl::UniquePtr to store its deleter, as it is a class intended for general use cases.

While the optional class needed some new type traits to remove both the const and the volatile qualifiers from types, the pair class needs type decay when creating objects with makePair().

Hash containers refactoring

Hash containers have also been refactored. Hashmaps, for example, now accept a nctl::Pair when inserting a key and a value.

Most importantly, I have refactored the HashMap/StaticHashMap and HashSet/StaticHashSet classes so they don’t duplicate any code when traversing the data structure.

The results of probing into them are now grouped in a structure:

 /// The returning structure after probing for a key
 struct ProbeResult
 {
 /// The ideal bucket index for the hash
 unsigned int ideal;
 /// Only valid if the found flag is true
 unsigned int found;
 /// First empty node in a chain
 unsigned int empty;
 /// Previous node in a chain (for delta patching)
 unsigned int prev;
 /// True if the node contains a value
 bool foundFlag;
 };

For HashMapList and HashSetList classes, the total number of elements stored is now saved in a variable, instead of calculated from the buckets.

Initializer list support

I have added yet another feature that makes nCTL containers more powerful and compatible with STL ones, the ability to use initializer lists to specify the initial content at construct time.

Take a look at this snippet from a nctl::Array unit test, for example:

nctl::Array<int> newArray({ 0, 1, 2, 3, 4 }, Capacity);

The array is constructed with a copy of the objects passed in the initializer list, similarly to what you would do with a std::vector.

For this to work I had to add #include <initializer_list>, as there is no way to reimplement this header, the implementation is too tightly integrated with the compiler. But it does not bring in any STL dependency, the include is minimal and only serves the purpose of supporting the initialization through the curly braces.

Initializer lists can now be used with nearly all containers (Array, HashMap, HashSet, List, SparseSet) and unit tests have been updated to use them.

Other additions

I removed the fixed capacity option from the nctl::String class, and extended the size for the small buffer from 16 to 24 bytes.
I have added a new nctl::StringView class that doesn’t own memory and allows you to format C-style arrays of characters, similarly to std::string_view.
The new nctl::Span class does something similar to nctl::StringView, but for arrays that are not characters.
Surprisingly I wasn’t checking for self-assignment in StaticHashMap and StaticHashSet classes, this has been fixed.
I rewrote the functions in the nctl::PointerMath namespace, now all the alignment parameters have been extended from uint8_t to size_t.

Hashing functions

This is another change that was needed for GrAIL, where I heavily use hashing for caching purposes.

The fasthash64() function now works more reliably, as it does not require padding to 64 bits anymore. I have also made it the default function for hashing keys in all HashMap and HashSet containers. It is a lot faster than FNV1a and statistically stronger.

To reach this conclusion I added a micro-benchmark and some unit tests that compare the various hashing functions on different inputs.

I have also added a couple of functions that, given a 64-bit hash, can create a proper 32-bit version of it without losing too much information. This is a task that is often needed in GrAIL, where 64-bit Vulkan opaque handles are converted to 32-bit keys for containers.

Benchmarks

Hashing a short string repeatedly with the various functions returns the following results on my Asus laptop in Performance mode.

xychart
title "Hashing a 256 bytes string"
x-axis ["Sax", "Jenkins", "FNV1a", "FastHash64"]
y-axis "Bandwidth (GiB/s)" 0 --> 10
bar [1.08797, 0.871923, 1.11025, 7.83879]

The FastHash function is nearly 8x faster than all the others, achieving a bandwidth close to 8 GiB/s on a single core. 🚀

Application configuration structures

Yet another change that spawned from the work done on GrAIL. This was made to ease future extensions of the AppConfiguration settings when GrAIL will be merged.

Instead of having all the flags and values together as fields of the class, they are now organized in structures. There is a Logging structure, a Window one, a Graphics structure, an Audio one, and so on.

void MyEventHandler::onPreInit(nc::AppConfiguration &config)
{
 config.consoleLogLevel = nc::ILogger::LogLevel::OFF;
 config.resolution.set(1920, 1080);
 config.deferShaderQueries = false;
}

void MyEventHandler::onPreInit(nc::AppConfiguration &config)
{
 config.logging.consoleLevel = nc::ILogger::LogLevel::OFF;
 config.window.resolution.set(1920, 1080);
 config.graphics.opengl.deferShaderQueries = false;
}

This gives a lot of room for future changes, and it has also been ported to the environment variables that you can specify when running an nCine executable.

Minor changes

The Vector4f and Matrix4x4f classes now align at 16 bytes, to help the compiler’s auto-vectorization.
On desktop, it is now possible to set the resizable flag of the window at run-time. The feature was already available in all backends and there was no reason not to expose it.
By using RUNPATH for Linux executables and by copying MinGW dependency libraries when creating a package, those two platforms should now correctly support completely portable installations, without relying on system-wide installed libraries.
GitHub Actions workflows now write a manifest JSON file with information about the packages they assemble. This in turn is used by the website to always offer the latest binaries for download.
Texture loader classes are now just image loaders and are not coupled with OpenGL at all, facilitating their use with GrAIL.
I have split some long CMake scripts in smaller chunks based on the platform, and moved them in subdirectories.
Now that Lua 5.5 is out, I have added proper support for it.
For consistency I renamed all occurrences of FullScreen to Fullscreen, fullScreen to fullscreen, and full_screen to fullscreen in code and documentation, treating it as a single word.
I fixed some JNI version checks before calling newer Android APIs, and now nCine runs again on my 2016 Xiaomi Mi 5 running Android 8! 🤖

An Intro to Wave Intrinsics

Fri, 27 Mar 2026 00:00:00 +0000

Most graphics programmers know about the concept of a warp (or a wave in AMD parlance). It is a group of GPU threads (or lanes), typically 32, though some architectures use more, that execute the same instructions in lockstep.

Maybe not all of them know about wave intrinsics, though. They are a group of special GPU instructions exposed in HLSL (and similarly in other shading languages) that a shader programmer can use to orchestrate work across the threads in the same warp.

Enforcing Uniform Control Flow

Let’s start with a basic example, using the WaveAllTrue(expression) intrinsic. This instruction returns true only if all threads evaluate the expression as true.

We said previously that all waves share the same instructions, but they operate on per-lane values of the same variable, stored in vector registers. It’s like running the same function with different parameters, the return value won’t be the same. Wave threads are similar: they have an execution context determined by a set of vector registers, and each thread sees its own version of the same variable.

That’s why they can evaluate the same condition in different ways and why knowing whether all lanes produce a uniform result with WaveAllTrue can be so useful.

Let’s suppose that we have a condition called fastPath that tells us if a shader computation can use a less general but faster approach.

In this case we could write a snippet like this:

bool useFastPath = WaveAllTrue(fastPath);

if (useFastPath)
{
 // All threads go through the fast path
}
else
{
 // All threads go through the general path
}

What does this achieve? We have reduced divergence on the fast path. Now either all lanes remain active for the fast path or none of them do.

Why is this beneficial? Why not just write if (fastPath)?

Because in this case the GPU evaluates fastPath per lane and uses an execution mask to track which lanes are active. This mask enables or disables lanes for subsequent instructions. Lanes where the condition evaluates to true remain active, while the others are masked off. The hardware then executes the if body with that mask, flips it, and executes the else path for the remaining lanes.

From the programmer’s perspective this looks like a branch, but in practice both paths are typically executed sequentially under different masks.

Note

The execution mask persists across control flow, and wave intrinsics operate over the currently active lanes rather than the full wave.

Delegating Work to a Single Lane

Other times we want to spare most of the threads from doing an expensive operation, especially an atomic one. An atomic operation would often be serialized between threads, so it makes sense to reduce its use.

We can do exactly this with an intrinsic like WaveIsFirstLane() and the help of another one called WaveReadLaneFirst(expression).

Let’s have a look at the following snippet first.

uint oldValue;
if (WaveIsFirstLane())
{
 Buffer.InterlockedAdd(bufferOffset, addend, oldValue);
}
oldValue = WaveReadLaneFirst(oldValue);

In this example the expensive InterlockAdd operation is guarded by a wave intrinsic. This pattern ensures that only the first active lane in the execution mask will execute the if statement. After performing the operation on the first thread, the value is broadcast to the remaining lanes by using the WaveReadLaneFirst() function.

Reserving Space in a Buffer

The previous example can be further expanded to show how it is possible to reserve some space in a buffer for the whole warp to operate on it without stomping on other waves’ space.

Suppose that each thread has a different number of items to write to a buffer. How can we know how many items the whole warp needs to reserve? And how can we be sure that each thread writes its items in a sub-slice of the allocated space? Yet again, wave intrinsics come to our aid.

// The number of items that this lane has to write to the buffer
const uint laneNumItems = ...;

// The offset in the reserved space where this lane can write its items
const uint laneOffset = WavePrefixSum(laneNumItems);

// The starting offset for the space reserved by the whole warp
uint absoluteOffset;

// The following statement will be executed only by the last lane in the warp
if (WaveGetLaneIndex() == WaveGetLaneCount() - 1)
{
 uint totalNumItems = laneOffset + laneNumItems;
 CounterBuffer.InterlockedAdd(0, totalNumItems , absoluteOffset);
}
absoluteOffset = WaveReadLaneAt(absoluteOffset, WaveGetLaneCount() - 1);

// Each lane writes its items at the designated offset
for (uint i = 0; i < laneNumItems; i++)
{
 DataBuffer[absoluteOffset + laneOffset + i] = value;
}

Let’s analyze the code in more detail. First of all, each lane has its own number of items to store. Then we encounter a new intrinsic: WavePrefixSum(). This instruction computes a prefix sum of laneNumItems. Each lane receives the sum of all previous lanes, excluding its own value. So if the first active lane wants to write 2 elements, the second 3, and the third 1, the prefix sum would be 0 at the first lane, 0+2 at the second, and 0+2+3 at the third. This gives us the offset at which a lane needs to write its items.

The next if statement executes only on the last lane in the warp, using WaveGetLaneIndex() to get the index of the lane executing it and WaveGetLaneCount() to know the total number of lanes in the warp. Inside the statement, only the last lane will perform the atomic add to reserve space for the total amount of items to store. To determine the size, it adds its number of items to its prefix sum.

Now every lane knows both the absolute and local offsets to correctly append their items to the buffer without affecting the others.

Conclusions

We have seen different ways in which wave intrinsics can help orchestrate the work between the threads composing a warp. Used wisely, they can improve performance in more complex patterns.

Nevertheless, always profile your baseline before implementing them in your code.

A New Site

Thu, 25 Sep 2025 00:00:00 +0000

After eight years of serving me well, I decided to retire my old Jekyll-based site, which used the Beautiful Jekyll theme, a popular Jekyll template created by Dean Attali. All content has now been migrated to Hugo, powered by the Blowfish theme by Nuno Coração.

I needed more flexibility and more speed. Hugo feels like the perfect choice to handle my site for at least the next eight years! 💪

I’m also looking forward to experimenting with what Hugo and Blowfish make possible, so expect more improvements and updates over time.

nCine Compilation Benchmark 3

Mon, 22 Sep 2025 00:00:00 +0000

Compilation speed directly affects iteration time when developing an engine. Since nCine continues to grow, I thought it would be interesting to measure how long a full build takes today compared to 2018, when I wrote the first article, and 2022, when I wrote the second one.

I was thinking about writing this since the end of last year, when I bought both the second hand Mac and the small 14" Intel laptop.

Just like in the past, let’s start with the hardware and the software I’ve used for my tests.

Hardware and software

Asus ROG Zephyrus G15 (2022)

Hardware

AMD Ryzen 7 6800HS
16GB RAM, DDR5 4800 MHz, dual channel
Western Digital SN735 NVMe SSD, 1TB
NVIDIA GeForce RTX 3060, 6GB GDDR6

OS

Microsoft Windows 11 Home (x64)
Arch Linux (x86_64), Linux 6.16.8

Alurin Flex Advance

Hardware

Intel Core i5-1235U
16GB RAM, LPDDR4X 4266 MHz, single channel
Western Digital Blue SN580 NVMe SSD, 512GB

OS

Microsoft Windows 11 Professional (x64)
Arch Linux (x86_64), Linux 6.16.8

Mac Mini M1 (2020)

Hardware

Apple Silicon M1
8GB RAM, LPDDR4X 4266 MHz, single channel
NVMe SSD, 256GB

OS

macOS Tahoe 26

Compilers and tools

Arch Linux

GCC 15.2.1
CMake 4.1.1
Ninja 1.12.1

macOS

AppleClang 17
CMake 4.1.1
Ninja 1.13.1

Results

The nCine version is the master branch as of today, compiled with CMAKE_BUILD_TYPE=Release, but left all the rest untouched, which means it was compiled with the GLFW backend, with NCINE_BUILD_TESTS set to ON, and NCINE_BUILD_UNIT_TESTS and NCINE_BUILD_ANDROID both set to OFF. The tests have been conducted by running the compilation process multiple times and recording the best timings.

This time I’ve only timed the build phase, using ninja on all devices. For the laptops, I’ve repeated the test on both the power saving and the performance profile (the former limits CPU frequency and power draw to extend battery life, while the latter allows higher sustained clocks and power consumption).

Tables and charts

Arch Linux

	Asus	Alurin	Mini
Power Save	22.359s	47.213s
Performance	14.379s	17.725s	37.33s

Compilation Chart 2025

Conclusions

It’s quite strange to see the Ryzen 6800HS and the Intel i5-1235U so close in Performance mode, maybe compiling C++ code is not the most parallelizable process in the world. 😅 With other benchmarks, like Cinebench, this Ryzen CPU dominates the smaller i5 on the multi-core test, with around double the performance.

It seems that power profiles are tuned quite differently between the two laptops, seeing how much the Intel one loses in Power Save.

I was a bit disappointed by the M1, on paper it should be a bit faster than the Intel on the CPU side with Cinebench, but it loses in my tests. Maybe it’s just that I’m comparing apples and oranges, with a different OS, environment, and compiler.

I hope you have found this second benchmark post at least as interesting as the first one. 😉

nCine Dev Update 22

Sun, 21 Sep 2025 00:00:00 +0000

Welcome back to another development update for the nCine, covering what has been accomplished in the first part of this year.

Introspective sort

Back in December 2023, the author of Jazz² Resurrection reported a crash when sorting a render queue with more than 3000 commands. The sequence was unbalanced and quicksort recursion went too deep, overflowing the stack.

He suggested switching to introspective sort (a hybrid of quicksort, heapsort, and insertion sort) which is exactly what I ended up implementing, using the same thresholds as many standard implementations.

The algorithm sets a maximum quicksort depth at twice the base-2 logarithm of the number of elements. Up to that depth, recursive quicksort partitioning is used, which is safe for slightly unbalanced trees. If the depth is exceeded, the algorithm falls back to an iterative heapsort, avoiding stack growth. Finally, for partitions smaller than 16 elements, insertion sort takes over, as it’s faster on small ranges.

Global Game Jam 2025

I took part in the Global Game Jam 2025 in my city, where we built a game using nCine. After the jam I polished it and released it on GitHub as Wet Paper.

Compared to the jam version, I added several features: a complete menu system, player statistics, TOML-based configuration, custom blur and refraction shaders, crossfading for music, a low-pass filter when pausing, and joystick vibration.

The jam also gave me a chance to improve nCine itself with new features and quality-of-life tweaks. For example, the engine now keeps track of the previous frame state for keyboard and joystick input, making it trivial to check exactly when a button was pressed or released.

I also made the vector classes return a zero vector when the length is too short (like many other engines), renamed interval() to frameTime() and apptest_scene to apptest_gui for clarity, and fixed broken alpha getters in the SceneNode class. A color setter override that accepted a float parameter has been renamed, and a method was added to set a TimeStamp to the current time without allocating a new object.

Working on custom shaders for Wet Paper also led me to fix viewport clearing logic and repair OpenGL debug groups, which hadn’t been working properly for viewports.

One of the last additions was joystick vibration. SDL2’s rumble API (SDL_JoystickRumble()) is quite barebones: you set the intensity for the two motors and a duration in milliseconds, but each call cancels the previous effect. That’s why I started working on a JoyVibrator class to interpolate motor intensities independently. The work isn’t finished yet and lives in a local joy_vibrator branch. 😅

Presentation at /dev/games

Presenting at /dev/games 2025

On June 5th I was invited to Rome to present my 14-year journey with nCine at the /dev/games conference.

It was both a chance to catch up with friends and an opportunity to show developers and students the craft, the struggles, and the technical insights involved in sustaining a long-term project like this.

You can browse the presentation online. I made it with Slidev and published the Markdown source on GitHub. The talk was recorded and should appear on the official YouTube channel later this year.

RenderDoc integration update

While preparing fresh screenshots for the talk, I revisited the RenderDoc integration and updated it to the latest 1.6.0 API. This allowed me to add features such as capture titles, automatically opening the RenderDoc UI after a capture, and enabling API validation and callstack capturing by default when using an OpenGL debug context.

Run-time environment variables

Here’s a small but useful quality-of-life change. You can now override AppConfiguration values at runtime through environment variables instead of recompiling your application.

This makes it easy to test your game under varying conditions: from sound frequencies and shader cache usage to window resolutions, render command pool size, or log levels.

For example:

NCINE_APPCFG_CONSOLE_LOG_LEVEL=0 NCINE_APPCFG_FILE_LOG_LEVEL=3 NCINE_APPCFG_LOG_FILE="game.txt ./my-ncine-game

This command disables console logging while redirecting it to a file of your choice. Combine this with scripts and you can quickly test multiple configurations. More details are available in the wiki article.

Crashpad integration

More than three years ago I began integrating Google Crashpad, a modern replacement for the now unsupported CrashRpt that I had used in ncParticleEditor. Crashpad is cross-platform, actively maintained by Google (they use it in Chrome), and runs as an out-of-process component, a perfect fit for my requirements.

The tricky part was Android, where distributing the crashpad_handler executable can trigger security restrictions depending on OS version. The solution was to ship it disguised as a JNI library and use nativeLibraryDir() to retrieve its location.

Another addition is the ability to extract debug info files to a user-specified directory without enabling Crashpad:

cmake -S nCine -B nCine-build -D NCINE_DEBUGINFO=EXTRACT -D DEBUGINFO_DIR=${CMAKE_BINARY_DIR}/symbols

This way you can upload debug symbols to platforms like Sentry, which itself uses Crashpad internally.

Array improvements

Even the well-proven Array class had room for improvement.

First, I fixed a subtle bug with insertions and removals in the middle: the old implementation overwrote elements without destroying them first. Thanks to W4RH4WK on Discord for reporting it! There’s now a unit test checking construction, destruction, and assignment counts for these operations.

Second, I reworked the type-trait helpers in utility.h. They now distinguish between trivially copyable, movable and copyable, movable-only, copyable-only, and fully non-movable/non-copyable objects. Two new setCapacity() implementations now use tag dispatching to select the right behavior based on the Array class’s template type. Thanks to this, you can create Array instances containing non-copyable and non-movable objects, with the restriction that arrays must be empty to resize and new elements can only be added at the back via emplaceBack(). This wasn’t even compiling before. 😮

Multi-threaded job system

Tracy capture of apptest_jobsystem

The star of this update is the new job system. After months of iteration, it has finally stabilized in structure and API.

Jobs now use opaque JobId handles that encode both a pool index and a generation number, which makes detecting stale IDs easy. I spent a long time experimenting with a fully lock-free job pool but couldn’t get correct behavior. The final design uses per-thread job caches and a global pool protected by a mutex.

Worker synchronization now uses semaphores instead of a mutex + condition variable pair to wake threads when new jobs are available. This avoids the extra lock/unlock overhead and makes the wakeup path more direct. On each platform the fastest available primitive is used: on Linux a futex-based userspace semaphore, on Windows WaitOnAddress() with atomics instead of a kernel semaphore object, and on macOS a Grand Central Dispatch semaphore instead of a pthreads implementation.

To tame the many threading issues during development, I added jobsystem_debug.h, which lets you enable Tracy zones and plots, statistical counters, additional logs, and job state tracking via compile-time flags. These debug tools have been lifesavers.

There’s also a serial job system: if you set a single-thread mode in AppConfiguration you’ll get the same API without synchronization, perfect for debugging or measuring scalability.

A recent commit also added a handle class with an object-oriented API over job IDs, new job state flags to prevent double submissions and support cancellation, and a new submit call to allow for multiple jobs to be submitted at once.

CPU topology

CPU Topologies Diagram

I completely reworked how the thread pool is created and how affinity is set, making the system topology-aware.

The idea is to sort cores by speed, to leave the main thread unpinned, and to start pinning worker threads from the second fastest physical core.

I have tested it on some of my devices:

On a Ryzen 9 6900HS (8C/16T), it creates 7 workers pinned to physical cores, leaving the main thread free.
On an Intel i5-1235U (2P + 8E, 10C/12T), it creates 9 workers: one on the second performance core and eight on the efficiency cores, with the main thread unpinned.
On a Mac Mini M1 (4P + 4E), it pins 7 workers, leaving the main thread unpinned and the first performance core free.
On a Snapdragon 8 Gen 3 (1Pr + 3P + 2P + 2E), it uses sysfs cpu_capacity values to pin 7 workers across the three weaker clusters, leaving the main thread unpinned and the Prime core free.

This could eventually allow more advanced scheduling in the future, with certain job tags mapped to faster or slower cores depending on priority.

Conclusions

The best part: the job system is fully exposed to applications. You can try it right now in the new apptest_jobsystem test. 🙌

The job_system branch is already live and up to date, though it needs more testing before merging. Meanwhile, I’ve begun work on a data-oriented ECS, an ideal testing ground for the job system. More on that in the next update.

Minor changes

Added conversion functions between Vector*i and Vector*f, and between Recti and Rectf classes
Updated README with documentation links and screenshots 📷
Added new dirty bits to nodes (thanks to Jugilus) to avoid redundant transformations of culled nodes
Fixed a long-standing bug with string capacity changes when using custom allocators
Updated GitHub Actions runners: added macOS 15 and Ubuntu 24.04, retired VS2019
Vector, matrix, and quaternion classes now have inequality operators 😄

nCine Dev Update 21

Tue, 14 Jan 2025 00:00:00 +0000

In this article, we’ll go over the progress of the nCine throughout 2024.

OpenAL EFX

The biggest change this year has been support for the OpenAL EFX extension. You can now apply effects to any audio player, like reverb, echo, flanger, and more, and use low and high-pass filters.

This was also a chance to improve the OpenAL code by adding new features and fixing some old bugs.

For example, you can now adjust the OpenAL context attributes to change the number of audio sources or adjust the frequency. Sources now support more properties, like velocity, direction, and cone angles. With the EFX extension, even more properties become available.

There’s now a pool of OpenAL sources that are assigned to players as needed. This lets you have more players than hardware sources, as only those actively playing need a source. You can also lock sources to frequently used players to avoid the system from reassigning properties. And if there’s an OGG audio dropout (like an OV_HOLE error), decoding continues without stopping.

The new effects and filters API is also available in Lua, and the apptest_audio application has been updated to test both it and the source pool.

Multi-threaded job system

Another major feature in progress is a multi-threaded job system, though it’s not finished yet.

I’ve been thinking about addressing CPU bottlenecks in the nCine by implementing a data-oriented ECS with a multi-threaded job system. I started working on the molecular matters job system, which includes a lock-free work-stealing queue, parent-child jobs, continuations, and parallel for loops. Their articles provide a simple implementation of a solid system that seems to fit my needs well.

While making the logger thread-safe using a queue of messages, I paused work on the job system to focus on something else.

Lua development improvements

While working on the job system, I came across the Lua Language Server and decided to add support for it. Improving IDE support for the nCine with Lua has always been on my mind, and LuaLS was the right tool for the job.

It took a few weeks to collect all the Lua API functions into LuaCATS definition files, but it was worth it.

As shown in the video, using the Lua extension for Visual Studio Code now gives you autocomplete, type checking, and inline API documentation. This makes the nCine experience similar to using frameworks like LÖVE 2D or Solar2D. The LuaCATS work also made it easy to create a new LDoc documentation that’s available online.

Reviewing the Lua API also allowed me to fix missing function exports, rename inconsistent methods, and write documentation that filled in missing Doxygen comments for the C++ API.

Lua Language Server

Improvements to `apptest_lua`

The apptest_lua application, which becomes the ncinelua executable in the Lua Distribution version, is the tool users use to write their games in Lua.

It already supported features like script hot-reloading and basic on-screen error display, but not all errors were shown, and users often had to rely on the console log.

Now, errors from application callbacks like on_init() or on_frame_start() are displayed on-screen, just like in the console. I also added support for tab characters in the TextNode class, so Lua call stacks are properly indented. 😄

The executable can now be run with -h/--help and -v/--version parameters, making it more like a standard command-line tool. To support this, I added a way to quit the framework from the onPreInit() callback, so no initialization happens, and no window briefly appears. You can also disable logging from this callback to completely silence the framework. 🤐

Minor changes

Joystick axes mapped as buttons are now supported
Added new constructors to the MemoryFile class to take ownership of externally allocated buffers
Fixed a typo in a preprocessor conditional that prevented joystick hats from working in GLFW
Moved the Particle class header to public headers to fix a forward declaration warning in ParticleSystem
macOS builds now use two GitHub Actions runners, with one compiling nCine natively for Apple Silicon

nCine Dev Update 20

Tue, 12 Dec 2023 00:00:00 +0000

Lately, the development rate of the nCine slowed down a bit. I think it is normal for a project that spans so many years, and developed by a single person, to see some oscillations. This is why this article covers such a long period, a period in which there have been maybe a few new features, but important ones.

Binary shaders

For sure the most interesting, and time-consuming, feature added since the last article is the support for binary shaders.

When it is enabled, OpenGL shaders are compiled only the first time an application runs. They are then cached on disk as binary blobs and loaded on subsequent runs. Loading a binary file is, of course, a lot faster than loading and compiling sources.

The feature was developed mainly for ANGLE, as its OpenGL to DirectX shader translation and compilation layer is quite slow. ANGLE is used on Windows inside browsers, but WebGL disables loading and saving binary shaders for security reasons. The main interest in making ANGLE shaders load faster resided in Jazz² Resurrection support for Universal Windows Platform and thus Xbox. 🎮

The feature is enabled by default on Android as it can help slow devices to start an application faster.

Hashing

A fundamental part of this new feature is hashing shader sources. If shader sources have not changed, we can continue to load the same binary file, otherwise, we need to compile and save a new one.

I was initially hashing all the sources used by a shader, every time I was loading it. This leads to correct results all the time, of course, but it can be slow and superfluous, especially for the default shaders that come with the nCine.

Default shader sources can be embedded in the nCine library or loaded from files, in both cases hashing the sources at run-time is not needed. In the first case, I use CMake to calculate an MD5 hash of source contents and embed it in the library. In the second case, I calculate a custom hash value from the name, size, and modification date of the file.

The hash is then compared with the value in a shader info file, a text file that contains a list of all successfully compiled shaders.

While the hash calculated by sources is a very important value to identify a particular shader, there is also another hash value that is combined to create the binary shader filename: the platform hash. It is calculated from the GL_RENDERER and the GL_VERSION strings and it ensures that, if the OpenGL driver is updated, shaders will be recompiled.

The hashing methods for strings and files are part of a new Hash64 class that internally uses fasthash64 and provides a Lua API.

Maximum batch size

Not related to binary shaders, but implemented while revamping the shader compilation pipeline, is the double compilation of batched shaders. Under certain circumstances, for example when the maximum supported size for Uniform Buffer Objects is less than 64 Kb, a batched shader is compiled a first time with a batch size of one so that its size requirements can be queried.

It is then compiled a second time with the batch array perfectly sized to match the available UBO maximum size. This feature improves the compatibility on some Android devices that do not support the classic 64 Kb UBO size and failed the compilation. In all other cases, the batch size value in shader sources is adjusted for a 64 Kb UBO and the shader does not need to be compiled two times.

While testing this feature I also encountered a bug in the RenderBatcher class that could have caused issues with small UBOs and big batches: basically, the condition that forced a split in a batch upon finishing UBO space was wrong. 😅

HiDPI support

Another quite important feature added last year was the support for HiDPI displays.

The first step towards this goal was extracting as much information as possible from the backends. This is why multiple monitors are now supported.

Based on the backend, it is now possible to know in real-time if a monitor has been connected or disconnected and to move a window from one monitor to another. Even on Android, using some JNI glue code, it should be possible to handle monitor connection and disconnection events.

I have also added the possibility to specify a particular refresh rate on init, for when the application starts in fullscreen. The user can also specify a window position so that the application can start on the specified monitor and at the specified position.

But supporting HiDPI also means adapting the window size and its contents to the scaling factor applied by the operating system. This feature is demonstrated by the new apptest_scaling application. It uses the new onChangeScalingFactor() callback to detect if the user explicitly changed the scaling factor of the monitor or dragged the application window to a different monitor.

Unfortunately, even after a long period of testing, the feature does not seem to be working perfectly, especially on Emscripten. I hope things will get better with new versions of the backends.

Thread class move fixes

The Thread class was developed years ago and tested only marginally, it is only normal it had some hidden bugs.

One of these was discovered by the Jazz² Resurrection author and happened when a thread object was moved into another one. Sometimes the source object went out of scope before the OS scheduled a new thread, and the pointer passed to the OS API function became invalid, causing a crash.

To fix this, I wrote custom move special member functions and a swap method, and I moved the threading information structure inside a UniquePtr so that it is allocated on the heap and moved correctly.

I also used the opportunity to rename some internal variables, change some method signatures to return a success boolean flag, and add a new detach() method.

Bunnymark

Bunnymark is a famous 2D benchmark that has been implemented in many different engines and frameworks to showcase the rendering and game logic performance.

I have implemented a nCine version, apptest_bunnymark, using the same sprite and logic from the original one. I have also improved the frame timer class and it is now more powerful and flexible.

Unfortunately, the performance isn’t yet where I would like them to be, but the benchmark will be an additional tool to help make the nCine even faster! 🚀

Improvements to particle systems and affectors

There have been some minor improvements to the particle systems and affectors. For example, affectors can now add steps even if specified out of order, instead of rejecting them. They also come with a new Lua API, so that complex systems can be built with scripting.

I have also resolved a minor issue when initializing a new particle system with random parameters: they had to respect certain numeric constraints or the system would not work as expected (or even assert and exit 💣). Now the constraints are not only checked but enforced by clamping or reordering values.

To have more flexibility with complex systems, there is now the option to disable a specific affector or to completely disable the particle update.

Minor changes

Long error textnodes in apptest_lua are now scaled down to be readable in their entirety
There is a new method to add multiple rectangles to an animation at once
Two new CMake presets have been added to support unit tests and micro-benchmarks use cases
The apptest_lua application will detect any changes to the script.lua file and hot-reload it automatically
The audio, font, and particles apptests have now an ImGui debug interface
The Android JNI code should now be more stable by using global references
All input backends have now the support for a “drop files” event
The application will never exit when a file cannot be opened
You can now choose to render TextNode objects by using the fragment shaders of regular sprites
Android gamepad code has been revamped by supporting a fallback system mapping, more button strings, and mapping buttons as axes

nCine Compilation Benchmark 2

Thu, 06 Oct 2022 00:00:00 +0000

I just got a new laptop, an Asus ROG Zephyrus G15 GA503RM (2022), and of course, I’m timing the nCine compilation to see how much time it will make me save. 😉 Should you be interested in the first compilation benchmark article, it is available here.

As always, let’s start with the hardware and software details.

Hardware and software

Xiaomi Mi Notebook Pro (2017)

Intel Core i7-8550U
16GB RAM, DDR4 2400 MHz, dual channel
Samsung 970 EVO Plus NVMe SSD, 512GB
NVIDIA GeForce MX150, 2GB GDDR5

Asus Zephyrus G15 (2022)

AMD Ryzen 7 6800HS
16GB RAM, DDR5 4800MHz, dual channel
Western Digital PC SN735 NVMe SSD, 1TB
NVIDIA GeForce RTX 3060, 6GB GDDR6

OS, Compilers, and Tools

Arch Linux 5.19.13 x86_64
GCC 12.2.0
CMake 3.24.2
Ninja 1.11.1

Results

The nCine version is 2020.05.r115.g001bdce (or r422.001bdce) from the master branch, compiled with the default options:

NCINE_PREFERRED_BACKEND=GLFW
NCINE_BUILD_TESTS=ON
NCINE_BUILD_UNIT_TESTS=OFF
NCINE_BUILD_ANDROID=OFF

As usual, the tests have been conducted by running the compilation process multiple times and recording the best times.

In the Configure phase, CMake is invoked to configure the project and generate Ninja files.

time cmake -S nCine -B nCine-build -G Ninja &> /dev/null

In the Build phase, ninja is invoked to build the project with all or just one core.

time ninja &> /dev/null
time ninja -j1 &> /dev/null

In both phases, I redirected all the output to /dev/null to save the console printing time.

Tables and charts

	Mi Notebook	Zephyrus G15	Ratio
Configure Ninja	1.715 s	1.330 s	1.289x
Build Ninja	12.774 s	4.683 s	2.727x
Build Ninja `-j1`	48.141 s	31.731 s	1.517x

Ninja multi-core chart

Ninja single-core chart

Conclusions

The Ryzen 6800HS, with its eight Zen 3+ cores, smashes the old Mi Notebook Pro, with its four Kaby Lake R cores, compiling the nCine in one-third of the time! 💪 I’m sure that the faster SSD and RAM also help in this test.

I was expecting a bit more from the single-core compilation results, but completing the compilation in roughly 65% of the time is not bad.

These results are very close to what we can expect by running Cinebench R23, for example.

The smaller difference can be found in the configuration phase: the new laptop still needs 3/4 of the time of the old one. CMake is surely using just one core and maybe hitting an I/O bottleneck, but a small improvement can still be found.

I hope you enjoyed this benchmarking article as much as I am enjoying my new machine. 😉

nCine Dev Update 19

Wed, 07 Sep 2022 00:00:00 +0000

Yet another update coming after a very long time since the previous, apologies for that. Well, at least it comes packed with a lot of enhancements from the last months. 💪

Custom shaders

Probably the biggest feature of 2022, and the culmination of work that started with viewports a year ago, is the support for custom shaders.

You can now write your own vertex and fragment shaders and assign them to a node. All of that is possible while also retaining the usual automatic batching. 😲 In combination with viewports, it is now possible to create complex scenes, featuring post-processing and modern effects.

Take for example the new apptest_shaders, it shows regular and mesh sprites, all of which use custom shaders, custom vertex attributes, and they preserve automatic batching. You will notice that some sprites are rendered with normal and specular mapping and that the light position can be moved in real-time! 💡

That’s not all, with the ImGui interface or with the keyboard, you can enable a full screen gaussian blur post-processing or a bloom effect. The latter is achieved with a quite complex viewports setup, with off-screen rendering, down-sampling, blurring, up-sampling, and compositing.

apptest_shaders

But the road to achieving all that was quite long and required many months. It all started by checking out a very old branch in which I was planning out how to approach the matter in question.

ShaderState class

In the first iteration the ShaderState class, the one handling shader uniforms, was embedded inside a DrawableNode. The user could then set a lambda function through the ShaderState object that the parent node would call each time it was going to be rendered.

In the second one, I moved the ShaderState class outside, so that nodes using the default shaders would know nothing about it. To change uniforms you could query the culling state of a node in the onPostUpdate() callback and update their values.

OpenGL layer

The OpenGL classes have seen upgrades too, for example in the way shaders and shader programs were created and loaded. Shaders are now just another resource, they can be loaded from files or strings multiple times, and if they fail to compile this would only affect the nodes that use them for rendering.

Two other important upgrades were fundamental to support the normal mapping and bloom effects in apptest_shaders: the first is multi-texturing and the other is multiple render targets. The former allows more than one texture to be used as input for a shader, while the latter allows a shader to write on more than one texture.

The GLShaderAttributes class has been deleted and its functionality has been moved inside the GLShaderProgram class. Missing the flexibility of changing the attribute setup per node is not going to affect the rendering at all, while the change simplifies the code of many classes with a possible uplift in performance.

If you run your game with a debug context inside a graphics debugger like RenderDoc, you should see additional information thanks to more glObjectLabel() calls.

Viewports

Viewports have been upgraded as well. First of all, setting up a chain of viewports for rendering is a lot easier now as you can directly access the Viewport::chain() array instead of using multiple obscure and error-prone calls to setNextViewport(). It is also possible to easily set up cycles for multi-pass rendering techniques.

The second important change, one that simplifies the internals while adding flexibility, is the removal of the texture inside the Viewport class. A Texture object should now be passed to a viewport from outside, a solution that opens a range of possibilities, from sharing a texture between multiple viewports to using the output of one as the input of another.

Those two enhancements are at the base of the blur and bloom setups in apptest_shaders.

More changes

There is now a new onResizeWindow() callback that the game can use to know when the window resolution changes. It comes in handy to recreate viewport textures if you have a post-processing setup in place.

The example Lua script has been updated as well to show how you can use viewports and shaders with scripts.

Static string class

Like many other items on my to-do list, this was something I have been thinking about for some time. Having a string class that does not allocate space for characters should remove the need for a plain old char array while retaining all the useful formatting, appending, and comparing methods of the standard String class.

I have begun to use it in new code and tried to backport in some old cases where it made the most sense. As usual, it comes with a battery of GTest unit tests that should ensure its correct functioning.

Fixes to viewports

Apart from the big viewports API overhaul made in the custom_shaders branch, there were other small issues that I discovered after merging the corresponding branch.

For example, if multiple viewports were rendering the same scene node, its update() method would have been called multiple times. 😅 This is now fixed by using a counter that stores the last frame a node has been updated by any viewport.

I have inverted the position and rotation values used by a camera so that it feels like an object that you can move in the scene. It leaves back the old concept of simulating movement by moving the rest of the world in the opposite direction.

When using the Qt5 backend the viewports were not rendering correctly as Qt5 uses its own framebuffer object to render a QOpenGLWidget off-screen and composite it later.

Rendering order based on node visit

This change brings the nCine closer to the behavior of other frameworks like Godot, Defold, or Cocos.

The user can still affect the drawing order of nodes with layers, but by default, it is dictated by the visiting order of the node in the scenegraph. This change makes the order more reliable, intuitive, and similar to what users are already used to.

Safer Lua pointers dereferencing

This is a very important change for scripters, before this change they might have encountered a crash too frequently while working with user data pointers.

Let’s say you created a nCine object inside a Lua script, like a sprite. The engine would then create a Sprite object and a UserDataWrapper object. The latter would be set to contain the pointer and type of the sprite, added to an array of wrappers, then its pointer returned to the user inside a light userdata.

You could have now used this variable to change the properties of the original sprite that you just created. But if you passed a non-valid wrapper, like one containing the pointer of a recently deleted object, the application would just crash.

This should not be the case anymore: every Lua function that operates on objects like sprites, textures, fonts, particle systems, and so on, will now check if the variable used is valid. There is no wrapper object anymore, the light userdata contains the pointer to the native object and it is checked by accessing a hashmap where the key is the pointer itself and the value is its type.

No more arrays to iterate, if the hashmap does not return a value or if the type is not the expected one, the operation is not carried out and nothing happens!

Updates to the Android building process

It has been quite some time before I had a look at the Android Developers documentation, the Android Gradle plugin, and the rest of the SDK tools.

The build.gradle script has been rewritten to support the latest plugin version. While doing so, all source files have been moved inside an app directory that now represents a separate building module.

A minor Android change is the support of the description attribute in the manifest.

Minor changes

The vector, quaternion, and matrix classes now initialize their values. A small change that will make happy some users without affecting performance.
Thanks to the support for glDebugMessageInsert(), the OpenGL debug context should produce some new debugging messages.
Hashmaps and hashsets could not perform a rehash if they contained non-copyable objects.
The window resizable option has been fixed when using the Qt5 backend.
The Font class can now use an external Texture object, enabling texture sharing among multiple nodes and modifications while the texture is in use.
The color classes use a group of variables instead of a StaticArray for channels. It should help with understanding the color value in a debugger.
Starting with macOS 10.12, clock_gettime_nsec_np() is the preferred way to query for a timer and makes the old mach_absolute_time() unsafe and deprecated.
An audio player will now check if it’s possible to register itself in the audio device before setting its state to PLAYING.
The Lua API can now manipulate vectors with four elements.

I hope you enjoyed this development update installment and some of the background work around custom shaders.

nCine Dev Update 18

Sun, 30 Jan 2022 00:00:00 +0000

After a very long time without an update, here comes a new one. It is filled with all the work done in the last six months. 💪

Template project files alongside the engine

This was a very important change that I had in mind for quite a while.

By moving the template CMake scripts inside the engine repository, I can now update them once and reap the benefits in all nCine projects.

Often an engine update needs a change in the way projects are built, and now a single commit can logically capture that.

As the template project demonstrates, you now only need a simple CMakeLists.txt file, and your project is good to go!

Lua developer distribution

I have added a new distribution option presets of the engine in the nCine-artifacts repository. It should make it easier to use nCine with Lua, in a fashion similar to LÖVE.

This version only has one executable: apptest_lua. You can pass it an argument on the command line for the script you want to load and, thanks to a TextNode, it will show you on screen a message should any errors occur.

Yes, this also means you can now pass command line arguments to any of your nCine applications. 😉

Raspberry Pi support

I bought a nice Raspberry Pi 4 Model B with 8GB of RAM, an Argon ONE M.2 case, and a 240GB Kingston A400 SATA SSD. The main objective was to have SpookyGhost running on it. 👻

Raspberry Pi 4B

To achieve that I had to tinker a lot with the CMake scripts. Even if OpenGL ES was already supported to enable the Android port, I had to resolve additional issues that only came to the surface when having OpenGL ES on Linux.

I had to add a CMake script to find and check the compilation results of the atomic library on 32bit systems on ARM (Raspberry OS is a 32bit distribution), add a new WITH_OPENGLES preprocessor definition, and check the CMake version before using file(ARCHIVE_EXTRACT) (Raspberry OS comes with an older CMake version).

In the end, the result was rewarding and recognized by both reddit and Tom’s Hardware. 😊

Viewports and cameras

The biggest achievement of this update is represented by the two months’ work on viewports and cameras.

It is now possible to have multiple viewports, each one with its size, its render queue, an optional offscreen render target, an optional camera, and a particular order in the rendering chain.

The use cases are various. With the upcoming work on custom shaders you can use a viewport texture to perform full screen post-processing like blur (even with a separable filter, if you setup the viewports order chain accordingly), or you can have multiple viewports and cameras and implement a split screen!

apptest_viewports

When you use a camera instead of a common camera node you can potentially improve performance as the matrix multiplications to transform nodes would be performed in a shader and not on the CPU. This can be seen in action in apptest_camera by pressing V to switch the new viewport method on and off.

There is even a greater performance gain margin when you pause sprites animations by pressing P. In this case, a new dirty flag system will completely skip node transformations, AABB calculations, culling rectangle intersections, and render command updates!

The dirty flags are wrapped inside a new nCTL BitSet container class similar to the standard bitset one.

The new camera system has been also ported to bigger projects like ncTiledViewer, ncJump, and JugiMap based ones. The following tests have been performed on my Mi Notebook Pro (2017) by forcing the CPU speed to 800Mhz.

Project Name	Frames (20s)	FPS	Frame Time	Percentage
ncJugiMapFrameWorkDemo (old)	6387	319.35	3.131ms	100%
ncJugiMapFrameWorkDemo (new)	8830	441.5	2.265ms	72%
ncJugiMapParallaxScrollingDemo (old)	7168	358.4	2.79ms	100%
ncJugiMapParallaxScrollingDemo (new)	8682	434.1	2.30ms	82.5%
ncJugiMapSpriteTimelineAnimation (old)	3537	176.85	5.65ms	100%
ncJugiMapSpriteTimelineAnimation (new)	5058	252.9	3.95m	69.9%
ncJugiMapGuiDemo (old)	7913	395.65	2.52ms	100%
ncJugiMapGuiDemo (new)	10691	534.55	1.87ms	74.0%
ncJump (old)	4879	243.95	4.10ms	100%
ncJump (new)	6047	302.35	3.30ms	80.7%

Continuous integration fixes

For a long time, I have had a list of problems in issue #11 related to the GitHub Actions continuous integration.

macOS libraries

On macOS, the Vorbis library could not dynamically link to the OGG one because, at compile-time, the system installed one was at a different path than the one on my machine and install_name_tool could not perform the @rpath substitution. This was recently fixed in nCine-libraries by modifying the Vorbis CMake building script to run otool to discover the embedded OGG path:

COMMAND sh -c "install_name_tool -change $(otool -L ${FRAMEWORK_DIR_VORBIS}/${TARGET_VORBIS} | grep ${DYLIBNAME_OGG} | cut -f2 | cut -d \" \" -f1) \"@rpath/${TARGET_OGG}.framework/${TARGET_OGG}\" ${FRAMEWORK_DIR_VORBIS}/${TARGET_VORBIS}"

Thanks to Fahien and Coda for testing out the new libraries on the latest version of macOS. 🙏

Clang strip on MinGW

On MinGW with recent versions of Clang, the stripping of binaries fails with an obscure unexpected associative section index message by llvm-strip. I disabled it when running on this platform and opened an issue to report it.

GCC optimization bug

With newer versions of GCC, both on Linux and MinGW, some of the tests in the gtest_list_movable unit test fail in release mode. Upon further investigation, I discovered that turning down optimizations from O3 to O2 in List.h with #pragma GCC optimize ("O2") fixed the issue. This made me think it is a compiler optimization bug, you know, one of those that goes away when you put printf() calls to understand what’s happening. 😅

It seems that in this particular test the front() method of the List class returns the wrong address the first time it is called, and the correct one after that. I was going to “fix” the issue by reordering the instructions in my test but then I discovered that the workaround worked on newer versions of GCC but made the error appear on older ones. 😩

While I was creating an isolated repro I also discovered that -O3 -funsafe-math-optimizations was working and that not linking the GoogleTest libraries and putting the test code in a main() function was also working. Changing the version of GoogleTest from the latest v1.11.0 to v1.10.0 or to the main branch didn’t affect the result. 😞

In the end, I decided to disable the gtest_list_movable unit test altogether when compiling in release with GCC. Better safe than sorry!

Minor changes

The user has now access to the ITextureSaver interface to save textures in PNG or WebP formats.
SDL 2.0.16 added a method to flash the window to request the user’s attention. As this functionality was already present in the other two desktop backends, it was possible to add a generic method to the API for this to work regardless of the preferred backend used.
The user can now use the NCINE_WITH_SCRIPTING_API CMake variable to enable or disable the Lua API, even when Lua utilities functions and the state manager are available.
The standard vector and list classes are known to have undefined behavior when trying to retrieve a front or back element from an empty container. But I decided to change the nCTL versions to fatal assert in this case. It has already proven useful to fix an issue in a unit test.

I hope you enjoyed the return of development updates with this packed installment. 😉

nCine Dev Update 17

Mon, 28 Jun 2021 00:00:00 +0000

Quite some time has passed since the previous development update but I’m here again to talk about the latest nCine progress. By the way, in case you missed the latest article, the project has recently reached its tenth anniversary. 😉

New nCine projects

Many of the following fixes and features have been driven by the work on some new nCine projects that have seen the light of day during those months:

ncTiledViewer is a loader and viewer of Tiled maps that you can easily integrate into your game.
ncJump is a platform game from Fahien. He has integrated Box2D physics to create one of the most complex nCine projects to date! 🙏
SpookyGhost, my procedural sprite animation tool, has recently become free and open source. I took advantage of this opportunity to add many new features.

Default constructors for nodes

As I was anticipating in the last update, I pushed the resource system even further by having nodes that can initially be constructed without any resource assigned to them.

Think for example of a bunch of sprites in an array that are constructed via their default constructor. They will have no texture or parent but they can still have a color. Solid color sprites will use a specific shader that doesn’t sample any textures at all. They will be batched together and render faster than regular textured sprites.

But it does not end here: animated and mesh sprites can both be default constructed and not be associated with a texture. Text nodes can be default constructed too and particle systems can have texture-less particles.

This functionality has been extended to audio buffer players as well, they can be default constructed as well.

You can of course assign a resource to a default constructed node at a later time.

Colored console messages

This feature was requested by Fahien while he was working on ncJump and I couldn’t help but satisfy the user working on the most ambitious project. 😉

So not only I added this feature but I fixed the lack of a console window on Windows, an issue plaguing this platform since I changed the default subsystem.

Now we can have the windows subsystem and the WinMain function, and open a console if the user wants it or if we are running a debug build. This is possible by using the AttachConsole API function.

As a plus, I have ported from ncline the code to set the ENABLE_VIRTUAL_TERMINAL_PROCESSING and have color output in the Windows command prompt. 💪

Colored Console Messages

Fault-tolerant Lua scripts execution

The most important new feature of this update is probably the new way of running Lua scripts. In the past, an error in a Lua script caused an assertion that killed the application. This is, of course, not acceptable. Users should be able to safely play with scripts, either in games or tools. For example, writing scripted animations in SpookyGhost.

The main change was to replace each lua_call() with lua_pcall(). The latter works in protected mode and returns an error in case the script is malfunctioning.

The error message is optionally returned to the application with additional information extracted from a lua_Debug structure. The user can use it to show where the error is in the script source.

SpookyGhost Script Error

I have also removed all occurrences of luaL_argerror() and replaced them with simple warning messages to avoid any exit caused by this kind of error.

To give the user even more flexibility I made it possible to load a script without running it immediately: loading a Lua “chunk” and calling it are now two separate actions.

As a bonus, I also fixed the parameter order of all calls to lua_createtable(). 😅

New application and input events

Text input event

Reacting only to keypresses is not enough in the world of Unicode and UTF8, as text characters are not the same as pressed keys. For this reason, I have exposed to the user a new input event that should make it easier to process text: onTextInput().

SDL2 already came with an SDL_TEXTINPUT event and GLFW with a CharCallback. They work slightly differently but wrapping them was easy enough.

For Qt5 I check the length of the string returned by QKeyEvent::text() and generate the event if it’s longer than zero. It means that this particular key event is the last in a Unicode codepoint sequence.

The Android implementation is probably the weaker one as KeyEvent::getUnicodeChar() does not seem to support codepoint sequences of more than one key.

Quit request

I have added an onQuitRequest() input event that makes it possible to override a quit request coming from the system.

Think about pressing the window close button or using the contextual menu from the taskbar to perform the same action. In those cases, the new event gets called and the user has a chance to save the game or to display a dialog window. The latter is what happens with SpookyGhost now.

SpookyGhost Quit Confirmation

Post update callback

The new onPostUpdate() application event solves an old problem: being able to know the absolute transformation of a node before it gets rendered.

It gives a new meaning to query methods like SceneNode::absPosition() as it can now be called after all nodes have been transformed so you don’t get stale data from the last frame. I have also added a SceneNode::setWorldMatrix() method to further modify the transformation before rendering.

This change, together with a fix that caused an additional frame delay when rendering ImGui or Nuklear in presence of a scenegraph, made the overlay in ncTiledViewer perfectly synchronized with the movements of the underlying map.

ncTiledViewer Imgui Overlay

Automatic capacity extension for strings

String capacity is another quite annoying issue that I have been willing to address for a long time. In the early days of nCine, I opted for fixed strings, you decided the capacity at construction time and the object allocated a buffer only once.

Great for performance, right? Possibly, but also a great source for bugs all around the place, with truncations happening when you least expect them.

I decided to change the default behavior: now strings reallocate if they need more space. You can keep appending text and the string object will make sure a truncation never happens. Of course, the user can still choose the old fixed behavior if needed.

This feature will make the Savefile Size configuration entry in SpookyGhost obsolete. The program will not need to preallocate a big enough string to contain the project file.

Fahien’s suggestions and feedback

I fixed an issue he discovered while working on ncJump in the behavior of Array::setSize(). Now it creates objects when extending the size, same as StaticArray and as you would expect. 🤦‍♂️
He requested the ability to move-construct and move-assign scenegraph nodes and to add a clone() method. This change will hopefully make both Rust and modern C++ users more comfortable. 🏠
To make it easier for ncJump to use the docking ImGui branch I exported the version tag of integrated software as CMake variables. It is now also possible to automatically download and extract a GitHub release instead of using a repository.
Thanks to a smart suggestion I created a specialized version of the hash functions for strings that simplified a lot the declaration of hashmaps with a string key. They now work as expected with both string objects and array of characters.
Additional feedback led to a new method to delete all children of a node and to a method to retrieve the current rectangle animation of an animated sprite.

Minor changes

I have added the option to mark a color as the chroma key when loading an RGB texture. I can now load some Tiled example tilesets that use magenta for transparency.
Another functionality dictated by Tiled capabilities is the custom frame durations for rectangle animations: every animation frame can have a unique duration different from others.
With the support of ImGui v1.80 comes the new Tables API which far surpasses the old Columns API.
No more documentation is distributed with the nCine. It is now pushed online at every commit by GitHub Actions.
For SpookyGhost, I needed separate blending functions for RGB and alpha channels so I now store additional states in the nCine wrapper class for OpenGL blending.
I exposed yet another window property from the three desktop backends: its position. The user can now move the window around with code or query its current position.

I hope the article was worth your wait and you enjoyed the new features. 😉

Ten years of nCine

Mon, 21 Jun 2021 00:00:00 +0000

A bit more than ten years have passed since that first commit.

The presence of a .hgignore file reveals that I was using Mercurial at the time, an easier transition to DCVS for someone like me used to Subversion.

Some things were already there and stayed the same until now: like the Doxygen comments or the CMake building process (my first CMake real project after years of SCons). Many other things have changed instead, as can be expected from a project with such a long life span.

But what made me persevere for a decade on the same project?

nCine logotype

The early years

I have been publishing open-source software for more than 20 years. 💾

The very first was MiniStat, published on Aminet in 2000, a very simple program to perform some statistical calculations that I wrote to practice what I learnt after reading my first C course. At that point, I had been an Amiga user for ten years and even if I was extremely fascinated by games and the demo scene, I was also quite attracted to system programming.

So when I became a Linux user shortly after, I oddly decided to work on a CGI script. I had recently finished reading the GaPiL book about Linux programming and those were the days of slow always-on ADSL connections and home servers. That’s how Sonda (2002-2003) was born, a CGI script for user polls written in C and hosted on a friend’s home server. 😄 The project was my first contact with SourceForge, with licenses, with CVS, and later with SVN.

Then Doom 3 came out and my interest switched back to games and shiny graphics. First I joined my good friend Vivaladav to work on Mars, Land of No Mercy, a turn-based strategy game with isometric graphics, and then I started experimenting with OpenGL demos.

With GL O.B.S. (2006-2007), my next big project, the focus stayed more or less the same: graphics, performance, and tools. Globs was a benchmarking solution based on a PyGTK interface, a PHP script to gather user results, and my OpenGL demos as individual benchmarks.

The tool was even used briefly by Phoronix long before OpenBenchmarking.org was a reality. 😱

Later I had some time to deviate a bit from graphics with PacStats (2007-2010): a little PyGTK project that extracted information from the Arch Linux pacman log file, stored it in an SQLite database, and visualized it with Matplotlib.

Finally, in July 2009, I graduated with a thesis on computer graphics and I was ready to join the game industry! 💪

The beginning

I looked for a job for a year, doing phone and on-site interviews in the UK with no luck. I remember I was considering going indie with a twin-stick shooter I was programming with XNA. 😆

But then the opportunity presented itself and I started working in a small indie studio in Italy.

While I was daydreaming about AAA games, rendering, and engine programming on next-gen consoles, the reality was a lot different: I was stuck with GUI programming with no escape routes.

But that is the precise moment the nCine was born: at work, I was coding boring user interfaces but at home, I could be whoever I want, and I wanted to be an engine and graphics programmer! ⚙️

My first two working machines were two very small and slow netbooks: a Medion E1222 first and a Lenovo IdeaPad S205 a bit later. 😫 But they had all I needed: Arch Linux, Qt Creator, GCC, CMake, and Doxygen. Those were my main tools then as they are today.

When I planned my work I deliberately decided to start with simple rendering: just 2D sprites and no shaders. My main focus at the time was the engine architecture, the data structures, the sound system, the abstractions… all things that I neglected before.

Of course, I carried my obsession with “perfect” commits to this new project. My policy dictates that I rebase a commit if I later discover an issue as if every commit was a micro release itself. This constant rebasing is the reason why there are very few commits and also makes a collaborator’s life harder. Fortunately, I have none. 😅

The consolidation

Over the years I changed jobs several times to come closer to low-level graphics and engine programming. This didn’t stop me from working on the project.

My two jobs in the UK were centered around mobile and Android development, and Android became an important platform for me.

At the time it represented the closest thing to a console accessible by indie developers: those were the days of OUYA, GameStick, Gamepop, MOJO, and Android TV set-top boxes like the Shield or the Fire TV.

When I began working on the nCine I knew I wanted to release it as open-source one day, but I believed it was too rough and I always delayed the date.

Both my employers in the UK allowed me to release the source code whenever I wanted so I thought it was not going to be a problem with my new company in Sweden. Unfortunately, I quickly learnt that their legal team was very strict on this point: I could never release the source while working with them.

So I continued to work behind closed doors but I started to open up a bit to the world. I wrote development updates on this blog, distributed binaries to some friend developers, and opened a Discord server.

Around this time I began exploring the idea of creating a real game with the nCine. I put together a list of 2D games made with custom engines by small indie teams to motivate me and keep the dream alive. 😄

The present day

Being unable to make my work public was one of the reasons I quit my job. I moved thousands of kilometers to the south, in Spain, and for two years straight I worked day and night on my projects alone.

I started by working on a new game: a turn-based isometric prototype code-named ncIsometric but after just a couple of months I got stuck with the utility AI code and put it on hold. 😅

The second, more important effort, was keeping the promise with myself and release the engine as free and open-source software. The 30th of May 2019 was the big day: the nCine source code was released on GitHub with an MIT license! :godmode:

It was covered by both Phoronix and Game From Scratch, and this initial press coverage that quickly raised the number of GitHub stargazers. The stars showed the appreciation of the people and made me happy, but what I needed was battle-testing it on the ground: could others use it to create beautiful games?

This was a very important point because before even feeling the excitement of creating a game, I was longing for the excitement of creating a tool that someone else could use to create one. I believe this weird indirect feeling defined me as an engine, graphics, and tools programmer: the satisfaction of making people happy and grateful for my support and efforts.

At this point, I created a template project to make it easier for potential users to develop games and a command-line tool to automate the download and compilation process.

Then the unexpected happened: Jugilus asked for help to support the nCine in JugiMap, his 2D level and map editor. The tool lets you create levels, animations, collisions, and some logic, then a runtime library loads the data and uses the underlying engine to render everything.

JugiMap’s runtime was a great way to stress-test the nCine and reveal a lot of bugs and limitations that, once addressed, made it a lot more capable and solid! 💪

Then it came the time for me to create my tool: SpookyGhost. By now you already know the joy I feel when a content creator uses one of my tools and I have wished to create one for pixel artists for a long time. Both because I love pixel art and because I always wanted to create a game with this style.

I planned to sell it on Itch.io, get rich and start an indie company about game technology, tools, and game development. Naturally, nothing of that ever happened, I sold a handful of copies and I had to return being an employee in the game industry. 😅

While a traditional job means less spare time, it also means more economic resources. Those resources might be invested in the project one day: they could translate into a marketing campaign or in a game jam with prizes. 🏆

Speaking of economic resources, at the beginning of this year I was awarded the Icculus MicroGrant 2020, a small help but a great recognition for my ten years of hard work.

Besides, now that I don’t need the money so bad I might as well make SpookyGhost an open-source software and hope more artists will notice it. 🤔 And that’s precisely what I did: I recently released it on GitHub under an MIT license hoping more users will try it.

The future

After all those words, where are we today and where are we going tomorrow?

Ten years have passed but the user base is still very sparse and small. The majority of people I come in contact with will just test it a bit and disappear in the end. I can’t blame them, the project is not well documented or marketed and there are no real games made with it, so why should they risk and be the first?

For this reason, I often think I should maybe be the one creating something: a simple but polished game that shows the capabilities and the potential. It would also satisfy my ancestral dream of creating a game completely from scratch. The main issue is that every time I start with a game project I end up adding a missing feature or fixing a bug in the engine. 😄

That’s our curse, engine programmers, always thinking at the foundations and never at the finished product.

A question I get sometimes is how I managed to be focused all this time on the same project. Well, the nCine initiative is a collection of different projects, I never get bored and I can always start a new one.

What keeps me going is the dream of seeing a successful indie game made with my technology. I work on my project a bit every day, with both small actionable tasks and a long-term roadmap, treating it as a real product but never missing out on the opportunity to have some fun. 😉

I hope you enjoyed the journey so far and I hope to have the strength to keep going for at least another ten years. 💪

nCine Dev Update 16

Sat, 28 Nov 2020 00:00:00 +0000

If you follow the project on GitHub you might have noticed a big development slowdown during the summer. I blame it on a combination of excessive heat and fatigue that led to a general lack of motivation and perseverance. ☀️

Fortunately, this does not mean that development didn’t resume at its normal pace or that there are no things to talk about in this article. 😉

Loading resources from memory

Let’s start with a feature that was requested by a user on Discord: the ability to load a resource from a memory buffer.

To minimize changes and reuse interfaces I just implemented a MemoryFile class. You can open, close, seek, read and write to it just as if it were a normal file on disk.

This way a texture, a font, an audio buffer… everything can be loaded from memory, giving much more flexibility to the user.

Fault-tolerant loading and reloading of resources

But loading from memory was not enough, I had more ideas to enhance the resource loading API and grant even more flexibility. One of the main limitations of the system was apparent if the user tried to load a non-existent file: the whole application would just suddenly exit.

This behavior might be acceptable if you are debugging a game but it is incompatible with any kind of tool workflow, where the user might erroneously try to load a sound file in place of a texture.

The first idea I had to solve this was to provide additional classes that could attempt to load a specific data type and, if successful, be consumed by the corresponding resource.

Of course, those new classes alone would have not been enough, I also needed to remove all exit() calls from loader classes that take care of specific file format, like Png or Ogg Vorbis.

The new system was working as expected: should the user construct a Texture from a non-existent file the engine would straight out exit just as before. But now it was also possible to explicitly check if the loading process was successful by using one of the new data classes and then pass it to the resource object. In this example, the user would use a TextureData instance to load an image from the disk, check if the loading was successful, and then pass it to the Texture constructor.

While this solution was working sufficiently well I didn’t like the fact that the user had to rely on those new classes to perform a safe loading and I scraped them altogether.

I took one step forward and made all resources class completely dynamic when it comes to loading data. Take again the Texture class, for example, you can now create an object with a default constructor, which might come useful when you need a pool of textures. But what is even more interesting is that you can load data in a texture (or an audio buffer, or a font) multiple times. You can assign an empty texture to a sprite and it will initially render nothing. Later on, you can load an image and have it applied to your sprite, and even load a second one later on.

Anytime there is a loading error nothing will happen to the application nor to the texture object: it will retain whatever data had before the last attempt. I was inspired by the SFML Texture class, with its default constructor and multiple load methods. In the future, I might even push further and have a Sprite that can be created with a default constructor and have a texture assigned at a later time.

I have also added a brand new feature: it is now possible to load uncompressed texels in a texture and PCM samples in an audio buffer, allowing for CPU generated procedural data. 🎉

GitHub Actions

As I have written in a news on the nCine site I now use GitHub Actions instead of Azure Pipelines for continuous integration.

I have been following the development of Actions for some time and I think it has now all the features I needed so I spent some days converting the building scripts and testing the new workflows, as they are now called.

The nice thing is the integration: the code and the C.I. are very close together, on the same page. Besides, the configuration is easier as I don’t have to visit a different site for setting the C.I. up.

nCine page for GitHub Actions

As I was taking advantage of an advanced feature of GitHub I thought I could honor an old note I wrote a long time ago: make use of its project management features by bringing my Trello tickets to Issues and having a roadmap with Projects.

There are no milestones or project boards yet but I started tracking some tasks with issues. 🗒️

I have also decided to simplify the development process by getting rid of the develop branch and just rely on feature branches, like in the GitHub flow model.

Decoding UTF-8 strings

While Jugilus was working on the Gui Demo he asked me about supporting UTF-8 strings. At the time I had very little knowledge about the topic except for the absolute minimum every developer must know.

As many times with the nCine, my coding from scratch efforts is an excuse to research more about something, and the same happened this time. In a short amount of time, I knew a lot more about Unicode, code points, decoding UTF-8, testing it, or how to interpret some of the information from a table like this.

apptest_font rendering Unicode glyphs

Android soft keyboard

I always wanted to enable the use of the Android virtual keyboard to allow users to enter text in as it is expected from a mobile device, that is without plugging in a physical keyboard. 😅

I just didn’t have the will to dig into the Android API and JNI once more to support the feature. In the end, I was able to gather my strength and implement it.

One downside is that toggling visibility seems to be more reliable than querying and settings the visibility state, as per this StackOverflow question about showSoftInput vs toggleSoftInput. 😞

Tracy multiple memory pools tracking

I remember back at the end of March when I asked Tracy author on Discord if he was considering adding a way to tag allocations so that it would be possible to tell which custom allocator was responsible for it.

I didn’t even have custom allocators at the time, but I was starting working on them. And now, after many months, we don’t only have custom allocators in the nCine, but string tags in Tracy v0.7.3 allocation macros! 💪

Multiple Memory Pools in Tracy v0.7.3

Minor changes

After a very long time since the last release, Lua 5.4 is finally out. It broke a couple of things, from CMake older than version 3.18 not being able to find it to a compilation error due to a missing constant.
The FileSystem API was updated to support Android assets: it’s now possible to perform queries on asset files and traverse asset directories.
A user made me notice that the root node was not being transformed, breaking the assumption that parent nodes affect children’s transformations. I fixed it by slightly changing the nodes tree traversal.

Hopefully, this update was enjoyable enough that you are coming back for the next one. Stay tuned! 😉

nCine Dev Update 15

Tue, 14 Jul 2020 00:00:00 +0000

I have spent nearly two months on a big task this spring: custom memory allocators. They can be useful in different scenarios to alleviate the performance cost of allocating and deallocating memory.

But before diving into that I had to be sure that the containers were ready.

Split allocation and construction

Just like with STL ones, there has always been a difference between capacity and size in nCine containers.

Size represents the number of elements currently stored in the container and it has an initial value of zero. The capacity defines the maximum number of elements that can be stored in a container.

Some containers like the Array class are dynamic, similarly to an STL vector, it can grow to accommodate more elements by allocating a bigger chunk of memory and copying over the old ones. If you already know how many elements the container is going to store during its lifetime, you can reserve an initial capacity to avoid reallocations and copies.

Now, the big difference between an old nCine Array and an STL vector is in their construction. The first will allocate its memory like this: array_ = new T[capacity_].

What is wrong with a perfectly safe statement like that? The problem is that the class is both allocating memory and constructing elements up to its maximum capacity.

This behavior has several implications:

Time is spent constructing objects even if no one asked for that
The T class needs a default constructor
New elements can only be copy-assigned or move-assigned
It is not possible to emplace new elements
Popping elements does not destruct them

The solution is to split the allocation phase from the elements construction. The Array allocation then becomes: array_ = static_cast<T *>(::operator new(capacity_ * sizeof(T))).

When we want to add new elements they will be copy-constructed (new (extendOne()) T(element)) or move-constructed (new (extendOne()) T(nctl::move(element))) in the array using a placement new operator. Something similar will happen when we emplace back (new (extendOne()) T(nctl::forward<Args>(args)...)) an element. Popping elements will destruct them as you would expect from an STL vector class.

There is also an additional important optimization based on type traits: if an element is trivially destructible then its destructor will not be called upon deallocation.

All nCine containers and the relative unit tests and benchmarks were modified and updated to support the new behaviors and functionalities.

Custom memory allocators

Now that the containers have a clear and separated allocation phase it is time to work on the allocators themselves.

They are always initialized with a pointer to the beginning of a memory arena and its size. This makes the allocators very flexible as the memory can be allocated in different ways, it can be a region accessible by the CPU or mapped by the GPU. It can even be a subset of a region managed by another custom allocator!

The memory arena is usually allocated only once by the operating system then control is passed to custom allocators. They are generally faster to allocate and deallocate memory and they can achieve even more performance within the use cases they were designed for.

One of my main references has been an article on Gamedev.net called C++: Custom memory allocation. It shows a basic interface for allocators plus four different implementations.

The engine implements the same four types of allocators:

The Linear allocator is the simplest one and allocates in constant time. It doesn’t have any kind of memory overhead beyond the alignment requirements but it doesn’t do any allocation bookkeeping and thus it cannot deallocate. It supports a clear operation to deallocate all memory at once though, a useful feature that can be used to implement a fast per-frame scratchpad memory.
The Stack allocator introduces a header at the start of each allocation to keep track of the address adjustment made to satisfy the alignment requirements. This makes it possible to deallocate the last allocation made.
The Pool allocator is a very classic allocator type for game development. It allows very fast allocations and out-of-order deallocations when the allocations have all the same size and the same alignment requirement.
The Free List allocator is capable of serving allocations of different sizes and alignments and deallocating out-of-order.

There are also some differences between the article and my implementation:

The allocator interface has no virtual methods but relies on function pointers.
The allocator interface supports reallocation. It can happen in place or by allocating a bigger chunk and copying the old elements, depending on the situation. Reallocation is very important to properly support Lua garbage collector and it is also used by Nuklear memory functions.
The Free List allocator performs fast compaction on deallocation to keep external fragmentation low. It has also three different fit strategies for allocations: Best, Worst, and First.

Allocation manager

The entry point for allocators setup is the AllocManager class. It is responsible for creating allocators before the first allocation is ever made and for providing those allocators to the application so that memory can be acquired and released.

Ensuring that the allocation manager initializes the allocators before even static variables have a chance to allocate memory is the tricky part. The first solution I investigated was the Nifty Counter, but it is a dirty hack that can have performance implications.

I then changed my implementation to use specific compiler features such as the init_priority attribute of GCC and Clang, and the init_seg pragma of MSVC.

There are a couple of optional defines in the AllocManager class that can be used to tweak some aspects.

The first is OVERRIDE_NEW, by defining it the class will redefine the global new and delete operators so that every allocation and deallocation will use custom allocators transparently.

The second one is RECORD_ALLOCATIONS, by defining it every allocator will record an entry for each allocation and deallocation made, with information about the number of bytes, the alignment, the timestamp, and so on. You can then use some included functions to print all entries or to query the list and know which allocation has never been freed.

Apptest_allocators

Last but not least, a new application test has been added to show how allocators work: apptest_allocators. It uses some custom ImGui drawing code to display a memory map similar to a disk partition editor.

Minor changes

There is now a node inspector in the debug overlay. It is useful for understanding what’s going on in the scene and for changing node properties.
The FileSystem API has been extended to support Android assets files and directories.

I hope you found this article interesting and useful. Ah, and don’t forget to check the 2020.05 release! 😉

nCine Intel Mesa 20 Driver Benchmark

Mon, 30 Mar 2020 00:00:00 +0000

Today I upgraded my Arch Linux workstation with pacman as I usually do every day and a little surprise was waiting for me. After a long time in [testing], Mesa 20 came out of the [extra] repository, ready to be installed.

This release brings a ton of fixes and new features, alongside the new iris driver for modern Intel integrated GPUs based on Gallium3D.

I have conducted some benchmarks on my Intel Core i7-8550U with Intel HD Graphics 620 (GT2), an Intel Gen 9.5 GPU.

Setup

I have compiled the nCine in release and disabled V-Sync, then I ran a script to minimize CPU throttling and frequency variations:

echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo 800000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
echo 800000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
[repeat for remaining cores, cpu1 to cpu7]

The performance governor should keep the CPU running at the same frequency all the time while the minimum frequency of 800 Mhz is set as both the minimum and maximum overall frequency. Those steps should both prevent the CPU from overheating and possibly highlight the reduced CPU load of iris.

To choose between the two drivers I used the MESA_LOADER_DRIVER_OVERRIDE environment variable. I set it to i965 for the legacy one or unset the variable for the new one, as it is now the default.

To confirm which driver was in use during a test I set the LIBGL_DEBUG environment variable to verbose.

Tests

Let’s see how many frames can be rendered over a time of five seconds.

Test	i965	iris	Ratio
`apptest_camera` (800 Mhz)	1665	1659	0.99x
`apptest_camera`	4532	4646	1.02x
`apptest_meshsprites` (800 Mhz)	1281	1294	1.01x
`apptest_meshsprites`	3451	3515	1.01x
`apptest_particles_100x` (800 Mhz)	385	387	1.00x
`apptest_particles_100x`	509	508	1.00x

Conclusions

It seems that for the nCine kind of workload, 2D sprites and low vertex count, there is practically no difference between the two drivers on my machine. The results do not change when the CPU frequency is capped at its minimum.

This is not necessarily a bad thing. I would have been glad to see some performance uplift with those 2D tests but I’m glad to see that iris does not seem to suffer at all from being a lot less mature than i965.

nCine Dev Update 14

Mon, 23 Mar 2020 00:00:00 +0000

Welcome to another nCine development update! As usual, there are a lot of new things to cover.

ANGLE

To extend the support to more devices and platforms I have ported the nCine to ANGLE. It was an easy task as I just needed to tell GLFW and SDL2 to use EGL and open an OpenGL ES 3.0 context on Windows.

With ANGLE the application would use Direct3D 11, overcoming any possible bug in old OpenGL drivers on old GPUs. It would also enable a future porting to UWP.

You have to sacrifice some performance, unfortunately. On my Intel i7-8850u with UHD 620 GT2 the apptest_camera test is capable of rendering nearly 1.5x frames in 5 seconds when using the OpenGL driver instead of ANGLE.

Version	Batches size	Frames	Ratio
OpenGL 3.3	unlimited	6263	1.461
OpenGL 3.3	10	6006	1.401
ANGLE	10	4286	1.000
OpenGL 3.3	disabled	4819	1.432
ANGLE	disabled	3365	1.000

When buffer mapping is enabled the performance takes a huge hit. 😣

Version	Batches size	Frames	Ratio
OpenGL 3.3	unlimited	5559	2.871
OpenGL 3.3	10	5302	2.738
ANGLE	10	1936	1.000
OpenGL 3.3	disabled	4069	1.432
ANGLE	disabled	974	4.177

Qt5 backend

I have added Qt5 as a new desktop backend.

This will open more possibilities for tool developers, as they can now create a Qt5 interface and embed the nCine as a custom QOpenGLWidget.

Using Qt5 for the interface does not mean you can’t continue to use ImGui and Nuklear too. You can even use all three together if you so want, as the updated apptest_scene demonstrates. 😉

If the Qt Gamepad library is found then gamepad events will be supported. I also decided to expose touch events to desktop applications, as both SDL2 and Qt5 support them.

FileSystem API

Another big addition is the new file system API. It allows an application to manipulate paths, query file and directory attributes, copy, rename or delete files, create directories and inspect their contents and so on.

It works on all supported platforms as it features both a POSIX and a Windows API implementation.

It comes with a unit test that should assure everything works as expected and with a new apptest. apptest_filebowser displays a file selection window made with ImGui that makes it easy to browse the file system.

apptest_filebrowser

Emscripten

I fixed the compilation with Emscripten version 1.39.5 that makes DISABLE_DEPRECATED_FIND_EVENT_TARGET_BEHAVIOR the default.

It means it is not possible anymore to pass a nullptr as a target in the HTML5 API and expect it will automatically choose a reasonable element. You have to be explicit and pass things like "canvas" or EMSCRIPTEN_EVENT_TARGET_WINDOW.

Minor changes

As it is sometimes useful, it is now possible to multiply a vector by a matrix with the matrix on the right side of the multiplication.
C-style strings can now correctly be used as hashmap keys. Previously the key comparison code was comparing memory pointers instead of characters, making this type of strings useless with hashmap containers.
If you compile the engine statically you will have access to a couple of classes that save textures as PNG or WebP images.

It is all for now. See you for the next update! 😉

nCine Dev Update 13

Tue, 24 Dec 2019 00:00:00 +0000

A lot of work has been put into the project as usual during those last months of the year.

Plenty of new and important features have been added to the engine, many of them are related to extending the capabilities of sprite rendering.

apptest_anchor

JugiMap

I have been recently contacted by Jugilus, JugiMap’s author, for a collaboration. He was interested in the nCine as a sprite rendering backend, alongside Cocos2d-x, AppGameKit, and SFML, to test the tool integration API and preview in real-time exported maps and logic.

He made a list of requirements for features that were not in the engine at the time but that every other backend had. To be honest the features he mentioned were, in a way or another, already in my to-do list so I just had to change my priorities to accommodate the need of the project. 😉

The features I have implemented from his list are the following:

Custom anchor points

It allows a node to be anchored to a point different than its center, like one of the four corners, and be transformed relative to it. The feature needed a lot of refinements to work in every case and with every type of node.
Non-uniform scaling

Nodes can now be scaled independently along the horizontal or the vertical axis. This feature was already requested by a tester of ncParticleEditor a long time ago. 😅
Texture flipping with a boolean flag

This change makes a lot of sense as you can now query and check if a sprite is flipped along one of the axes. Previously flipping was just an action to perform and there was no way to tell whether the sprite was flipped or not. This also means that, just like the custom anchor point, if you change something in the sprite, like the texture, the engine needs to automatically reapply any flipping.
Custom blending factors

Another feature requested by a tester of ncParticleEditor a long time ago. 😄 The user can now specify a custom source or destination factor for blending or use one of the presets, allowing for premultiplied alpha textures or additive effects.

I was very happy to have the API demo test as a big stress test for the sprite rendering, it highlighted serious issues like:

Frame latencies with parent/child transformations
Drawable nodes culling not working with negative scaling
One frame delay to update a text node cached boundaries

You can find the project repositories on GitHub: ncJugiMapAPIDemo and ncJugiMapAPIDemo-data. You can also test the Emscripten web test.

ncJugiMapAPIDemo

Emscripten enhancements

The Emscripten port has seen some improvements and fixes too. First of all, it can now build with version 1.39.0 or newer, which made the LLVM WebAssembly backend the default.

Another important fix is the correct handling of browser window resizing: an Emscripten application should now be correctly displayed regardless of browser window size or fullscreen state. Continuous resizing should now also work for desktop native versions where the perspective matrix was previously not updated. 😅

Deploying a nCine based tool to Emscripten makes a lot more sense as it is now possible to load and save files locally. It means you can load a file from your computer into a web build or save it from there to your computer. You can test this feature in the updated ncParticleEditor web test. You will also notice the support for ImGui custom font loading in the form of FontAwesome icons. 😉

Last but not least, after a long tribulation, Emscripten builds can now use automatic render commands batching! 💪 There is just one small catch, the batch size is fixed. 😩

Sure, you can configure it on initialization, but it is fixed, there is no minimum or maximum, the application will collect a certain amount of commands per batch. It will split a batch if it has reached the predetermined size or it will render single commands unbatched if there aren’t enough. You should also keep this number quite small to prevent the D3D shader compiler in ANGLE from taking a lot of time. 😄

Nuklear integration

Nuklear is an immediate UI similar in concept to Dear ImGui but skinnable. I have added this integration as I expect a game to use Nuklear and customize the graphics while a tool would use the superior flexibility of Dear ImGui.

Nuklear integration

Additional improvements

There are a lot of smaller things that have been added during this period:

There are two new application events you can subscribe to, onSuspend and onResume. Frame and profile timers will not take into account the time while the application has been suspended.
There is now a ColorHdr class that allows for unclamped floating-point color values which can be used for supporting HDR in tools or demos.
Sorting of render commands is now stable, there should be no more random popping of a sprite in front of another. Two commands with the same material sort key will be sorted according to their creation time.
The 4x4 matrix class supports in place transformations. It means that less memory will be used and fewer multiplications will be performed when translating, rotating or scaling in place.
The deletion of children scene nodes upon parent destruction is now optional.
As usual, the integrations with ImGui and Tracy have been updated to support the latest versions at the time of writing.

Enjoy the new nCine features and this festive season, whether you celebrate Christmas or not. ⛄

nCine Dev Update 12

Sat, 21 Sep 2019 00:00:00 +0000

The last two months of work on the nCine were mostly dedicated to the quality of life improvements for users.

First of all, I decided to get rid of the legacy debug overlay. It was a very old and problematic code that didn’t have any reason to be today. With the ImGui and Tracy integrations in place, the nCine is more than covered in that aspect. 💪

In an attempt to make the use of nctl::string less confusing for users of std::string I refactored and renamed many copy methods to assign.

FNT parser

While I was explaining to a user how to make the font in his game bigger by creating a new texture, I suggested fontbuilder, a multi-platform tool I found on GitHub. I then tried the tool myself and when I used the generated texture the text was not properly rendered in the nCine. 😅

I knew the FNT format has different ways to encode glyph data but I was supporting only a couple of them. I took some time to rewrite my FNT parser with the specifications in mind, extracting all the available data allowing me to add some additional encoding formats. Most importantly it provides the engine with the reasons why a specific encoding is not supported, leading to more meaningful error messages.

Timer refactoring

I refactored the Timer class to be just an interface to a stopwatch. The platform specific code to access the monotonic clock and retrieve the elapsed ticks is now inside a Clock class.

Besides using the timer, the user can now also use the TimeStamp class to record or accumulate counter values, with the possibility to convert them to various units of measure in a way similar to std::chrono::duration.

Tracy v0.5

With the release of Tracy v0.5 last August it was time to update its integration inside the engine and make it easier for external projects to use it.

Additionally, you can now rename a thread to easily spot it in the main profiler window and access all engine log entries in the messages section. Good news for macOS users, memory profiling is now working and has been re-enabled. This new version of Tracy should also be able to compile on MinGW.

Windows changes

The Windows port was given some care, with the removal of every Windows.h inclusion and the minimization of the headers used. The atomic implementations were moved from headers to sources, removing the need for including a Windows API header in <nctl/Atomic.h>.

Two more Windows improvements are the use of the Windows subsystem instead of console for apptest executables and the installation of a desktop link by the NSIS generated installers.

Last but not least the engine and the game projects now generate a VERSIONINFO resource file. Thanks to it you can open the properties window of files like ncine.dll or ncpong.exe and retrieve product and version information.

VERSIONINFO resource

Granular library dependencies

Some users didn’t like Lua at all and asked me how to remove the bindings integration. 😄 I told them this was totally possible as the nCine was modular since its inception: you could already take out library dependencies at the expense of functionality. The problem was that the feature was not easily accessible to users and properly tested among all combinations.

Now it is and you can turn off the NCINE_WITH_LUA CMake option to disable the integration. The engine will not be linked with the Lua library and not packaged with it in installers or archives. There are similar options to disable threads, audio or the integration with decoding libraries such as libpng, libwebp or libvorbis.

Additional projects

A lot of work was devoted to adding or improving accompanying projects too, in an attempt to further ease the life of nCine users.

For example, a new ncTemplate repository was pushed to GitHub. It is supposed to be the starting point for creating a new project with the engine. By using it you ensure that all the CMake logic to support all platforms is in place. Check the README.md file for additional information.

There is a second repository that has been created to make working with the nCine easier: ncline, the nCine command line tool. It all started a couple years ago when I thought a Python script could automate some of the tasks related to calling CMake on different platforms.

That’s when ncDo.py was born, an internal tool I used to speed up the compilation and testing of the nCine when switching from an operating system to the other. Today the continuous integration helps a lot in that regard but an automation tool is still very valuable for me and for users.

I decided to rewrite it in C++ as it was growing more complex and I wasn’t comfortable with Python anymore. Thanks to clipp and cpptoml I managed to keep all the features that were once carried out by argparse and configparseer. The tool has now more functionalities than ever before, being able to download source code or artifacts using Git. Just like earlier, have a look at the README.md file for more information.

Last but not least, I made a first GitHub release and started keeping track of differences between releases and the develop branch. Yet another small quality of life improvement, especially for users migrating to a newer version.

I hope those changes will make the nCine easier to use to create awesome games! 😉

nCine Dev Update 11

Tue, 16 Jul 2019 00:00:00 +0000

Exciting news for this development update: a new supported platform! 🍾

Emscripten

I remember playing with the idea of porting the nCine to Emscripten years ago. After all I had every requirement in place: I used OpenGL ES for Android, GLFW and SDL2 as input backends, OpenAL and Vorbis for audio, libpng for images and already supported a POSIX API.

Unfortunately there was always a showstopper preventing me to achieve some progress and gain more motivation. I remember managing to compile unit tests and run them from the console with Node.js but that was it, no apptest was ever working or sometimes even compiling.

This time was different, I studied the documentation in detail, put a lot more energy and dedication but here we are, it works!

Of course there were a lot of little issues that ended up eating plenty of time. 😩

Rendering

I encountered the first issue when I tried running apptests on Windows, the browser just froze. Further investigations led me to the culprit: shader compilation was super slow when using ANGLE.

I did some research on the internet and I stumbled upon the Asynchronous Shader Compilation demo. The article explains a way to give browsers a chance to compile shaders asynchronously while performing other loading related tasks.

I followed the advices: compile and link shaders but defer any error checking or introspection to the first use. It didn’t fix the issue at all but made loading time faster on every platform. 😅

Just have a look at the following traces and compare how much initCommon() takes with immediate ("This trace") versus deferred ("External trace") queries.

Tracy - deferShaderQueries

I then consulted a friend about ANGLE and he concluded that the long arrays I used for collecting batching instances information were the cause, and that the Direct3D HLSL compiler is well-known for having a hard time with them. After redeclaring those as single element arrays the compilation speed was reasonable again.

If the shader declares a single element array then it should only be able to access that element, even if you bind a bigger buffer, right? According to my tests that’s not true and every GPU on every OS out there will perform an out of bounds access and render every instance. Well, every combination except Catalyst drivers on Windows, therefore restricting these shaders modifications to Emscripten only.

But batching was broken no matter what, with or without single element arrays there was always some instance not being drawn. I suspected an issue with Uniform Buffer Objects alignment as apptest_camera with culling enabled rendered flickering sprites, very similarly to what happens when alignment is incorrect. I tried for days with different alignments and buffer sizes but I didn’t have any luck and I decided to disable the feature on Emscripten for the time being. 😢

When I was lucky rendering fixes came from extending Android OpenGL ES specific #define conditions to include Emscripten, like going from #ifndef __ANDROID__ to #if !defined(__ANDROID__) && !defined(__EMSCRIPTEN__).

Buffer mapping might have been another big issue as it is not supported on WebGL 2. Fortunately it gets emulated when you pass -s FULL_ES3=1 to emcc and it’s an optional engine feature that can be disabled. This way I got the code to compile with no modifications while avoiding a possibly slower emulated path.

Input backends

Rendering was not the only area in need for a change. On SDL2 auto-suspension on focus lost just hangs the application indefinitely. That is caused by SDL_WaitEvent() and it appears the only fix is not to call this function.

Emscripten also doesn’t seem to play well with GLFW windows at the moment, it needs a little help in the form of a glfwWindowHint(GLFW_FOCUSED, 1) when you open one. I also had to exclude some GLFW 3.3 code I added recently, but nothing that couldn’t be fixed by a #if GLFW_VERSION_MAJOR == 3 && GLFW_VERSION_MINOR >= 3. 😉

At least joystick mapping worked without much hassle, I just had to return the special “default” GUID for every connected joystick in order to match the Emscripten entry in the SDL2 game controller database.

Additional changes

After guarding death tests with #ifndef __EMSCRIPTEN__, unit tests were also good to go. Thanks to the Emscripten CMake platform script, running ctest invokes node as the CMAKE_CROSSCOMPILING_EMULATOR allowing unit tests to run from the console. 💪

Apptests were also mostly working, with the exception of apptest_simdbench. I had to refactor it in order to use a lot less memory and allocate it from the heap.

As you might have already read on Emscripten website, accessing the file system is different too. The C API is emulated, and that’s very useful, but you still have to provide data to your web application in a special way.

I decided to go with preloading instead of embedding to always have a separation between data and code. For apptests I directly call the file packager tool within CMake, I generate the data once and share the file between them.

The nCine-libraries project also gained Emscripten support and it can now compile Lua and WebP for the new platform.

OpenAL fixes

While working on the port I had some people try my web tests and a tester reported a strange issue with music and sound effects in apptest_audio. In the beginning I thought it was something related with the new platform but the issue could be found on all of them. 😱

The OpenAL implementation was probably one of the oldest pieces of code, it had not been touched in a very long time and it showed. 😅

First of all the source id recycling was pretty much wrong and paused or stopped players never relinquished their ids. Also, they could never transition from a paused to a stopped state. 😩

Even if the number of OpenAL sources is fixed and there is only one source per active player and vice versa, I was managing currently playing players with a list. The optimization was as easy as to swap nctl::List<IAudioPlayer *> players_ with nctl::StaticArray<IAudioPlayer *, MaxSources> players_ and change some accessing code. 😉

While I was there I took some time to sprinkle a couple of alGetError() calls around source and buffer generations, just as suggested by the OpenAL Programmer’s Guide.

I have also added querying methods for buffers, streams and players that allowed for the creation of a new section in the ImGui debug overlay, one showing information about active audio players.

ImGui debug overlay - AudioPlayers

Monitor video modes

Last but not least, there is now a way for games to query and change monitor video modes on PC. The implementation uses functions like glfwGetVideoMode() and glfwSetWindowMonitor() on GLFW and SDL_GetDisplayMode() and SDL_SetWindowDisplayMode() on SDL.

The ImGui debug overlay has been updated to take advantage of this new feature. The user can select a full screen video mode from a drop-down list of supported resolutions and refresh rates combinations.

ImGui debug overlay - WindowSettings

That’s all for now and I hope you are excited about porting your nCine projects to the web. 🕸️

nCine Dev Update 10

Wed, 03 Jul 2019 00:00:00 +0000

I’m sure many of you have heard it already: the nCine source code has been released on GitHub!

This means that lately most of the time was dedicated to publication related tasks, for example updates to the site like the addition of a “why nCine?” page and a gallery.

But definitely one of the most complex task has been continuous integration, which has a new page on the site too. 😉

Continuous Integration

A couple of years ago I began experimenting with Travis and AppVeyor and managed to have them build the libraries, the engine and ncPong for all supported platforms: Linux, Windows (both MSVC and MinGW), macOS and Android.

In an effort to simplify this setup I decided to migrate everything to Azure Pipelines. It supports all major desktop systems meaning I now need only one script to support everything. It also features a more flexible YAML language that organizes the whole process as a sequence of jobs and steps with conditions, as opposed to specific steps like after_test or before_install.

Now that the source code is available online, let’s have a look at those scripts.

Building the libraries

First thing to do is to build the dependencies, both for desktop and for Android. As you can notice libraries are built on Linux with both GCC and clang and on Windows with both VS2019 and VS2017. On Android libraries are built for the three supported architectures but only on Linux.

Next is when things become interesting, an idea that I kept from my original tests in 2017. In order to have artifacts available regardless of the C.I. platform and available for users to download I created some repositories on GitHub with the sole purpose of storing them.

For libraries the repository is named nCine-libraries-artifacts and is made of multiple branches, each one for a specific platform.

Building the engine

Next is the engine itself, it pulls the library artifacts and engine data and push installers and portable archives. The script gets more complicated by a test matrix that now includes build types, like Debug and Release, together with the DevDist preset used for installers.

The first steps prepare the environment, they download Doxygen and Graphviz to build the documentation and the NDK to compile for Android. On Windows and Linux they also download the RenderDoc API header because CMake is invoked with -D NCINE_WITH_RENDERDOC=ON. Additional steps will build unit tests and run them through CTest. Benchmarks will also be built but not executed.

At the end packages are created via the package CMake target and files pushed in the nCine-artifacts repository. It will again have multiple branches depending on platforms but this time they will also also depend on engine source branches.

Building the projects

Now that the engine has been built and its artifacts pushed on GitHub it is the turn of the accompanying projects. Both the pong example game and the particle editor have very similar C.I. scripts.

They download the libraries, the engine and the project data and they build for all supported platforms, including Android APKs using Gradle. They push artifacts in the ncPong-artifacts and ncParticleEditor-artifacts GitHub repositories.

Some notes

On Linux CMake is not very recent and a special step has to download and install an updated version from the official site.
On Windows all steps use PowerShell. Life is too short to mess with the Command Prompt. 😅
Often enough git commands write their output to stderr and make scripts fail. That’s the reason for $env:GIT_REDIRECT_STDERR = '2>&1' in PowerShell steps.
On PowerShell curl is an alias to an internal Invoke-WebRequest command which does not understand all curl options. The solution is Remove-item alias:curl.
Sometimes there is a need for git commands to invoke a fallback when they fail, it’s achieved this way:
- git fetch --unshallow || true on Bash
- git fetch --unshallow; if (-not $?) { return } on PowerShell
MSYS steps always set the CHERE_INVOKING environment variable for Bash to use the current working directory.

More changes

Continuos integration has been a good way to spot some tricky issues occurring only with specific combinations:

Tracy memory profiling not working on macOS
Wrong atomics version used on MinGW
Unit tests and benchmarks not compiling on MSVC
Wrong version of OpenGL headers included on macOS
Static library support breaking in some situations

Fortunately in the end it wasn’t all about C.I. and I had some time for a small new feature: a configurable frame limiter. It might come in handy especially on mobile to limit FPS below VSync. 💪

nCine Dev Update 9

Tue, 14 May 2019 00:00:00 +0000

It has been a month and a half of small but useful updates for the nCine.

LibPNG

The PNG image loader has been modified to support more color types, by copying some code from the libpng example. It means that any nCine game is now able to properly load PNG images with palette or with gray-alpha channels and to expand or strip bit depths that are different than the standard 8 bits.

Lua fixes

I have also sucessfully converted the ncPong test game to a Lua script and fixed a lot of Lua bindings bugs in the making. 🐛 Things like missing methods or constants that were never exported to the Lua API.

There are also some Lua utils additions and it is now possible for games to set and retrieve fields directly or to set and retrieve globals.

Headers

Another small change that improves the quality of life of a game developer using the nCine is the moving of headers inside an ncine subdirectory. It is now very clear when you are using your game headers or the engine ones, as you would now see something in the likes of:

#include <nctl/Array.h>
#include <ncine/Sprite.h>
#include "mygame.h"

Using the angle brackets makes it also easier to spot them, and they made their way into the included examples as well.

Symbol stripping

Next is symbol stripping from binaries, a way to reduce the size of Android libraries on all supported platforms. It is also a way to strip the base libraries on ArchLinux as the makepkg stripping option is not cross-compiler aware and it ends up stripping Android ARM libraries with the x86_64 strip command, ruining their content. 😅

Uniforms update

I have decided to finally spend some time identifying and squashing a long standing bug introduced with the latest renderer revamp that would cause a one frame delay when committing uniform variables. The result was very noticeable when disabling, moving and later re-enabling drawable nodes: they would be rendered in the old position for only one frame causing a very annoying graphical glitch.

At the same time I also fixed another small graphics bug that would treat 3 channels textures as gray and render them with the wrong shader. It was evident in the texture formats example test, as it would render RGB textures in gray shades. 😞

Joystick hats

The next change is all dedicated to joystick hats, the work was in stand-by as I was waiting for the release of GLFW 3.3. Now hats are completely separated from buttons (on GLFW with glfwInitHint(GLFW_JOYSTICK_HAT_BUTTONS, GLFW_FALSE)) and queried on their own on all three backends: GLFW, SDL2 and Android. When talking about joysticks one should expect the usual magic involving bit masks when working on Android, in this case to make both AINPUT_EVENT_TYPE_KEY and AINPUT_EVENT_TYPE_MOTION events update the hat state, as some joystick report one or the other type of event (see also the official Android developer article about processing directional pad input). 😄

This change made it also possible to add the directional pad as a set of buttons when using the gamepad mapping, finally bringing it on par with the original functionality in SDL2. 💪

Application suspension

After some feedback by one tester I decided to rename the application pause related methods to suspension, a word that makes it clear that in-game pause has nothing to do with global event suspension at the application level. ☺️

As a bonus I have also introduced a flag that the user can set to disable the automatic suspension that happens when the focus is lost.

Application configuration

Another small quality of life change is the removal of all the getter and setter methods from the application configuration class. Now the user can just set a value in the configuration structure, similarly to what was already happening when changing the configuration in Lua.

config.withVSync = false; // instead of config.enableVSync(false);
config.windowTitle = "My Game"; // instead of config.setWindowTitle("My Game");

clang-format

I finally decided to commit myself to a code beautifier and stick with it. After years of playing with Artistic Style and Uncrustify without ever fully liking the results, I decided to put together a .clang-format configuration file.

Maybe I’m getting old and tired or maybe I’m just less scared by the stylistic compromises but I should admit I’m pretty satisfied with it. The only exception being the IndentPPDirectives option: I would really like to be able to use BeforeHash instead of manually edit the results given by AfterHash, but it seems I will have to wait for the Clang 9 release. 😄

nCTL containers

The nCine Template Library has been given some care too, with the addition of some containers and many new unit tests (now more than 1000! 😱) and microbenchmarks. Most containers have seen the addition of an emplace method that uses placement new.

I have also added set containers, some are based on the hashmap ones stripped of the value associated to the key but there is also a new sparse set container that should be very fast when storing numbers and when the space-time tradeoff is advantageous.

RenderDoc integration

Next we are back with graphics, with the in-application integration of RenderDoc. This means that the user can now trigger a capture, launch the Qt interface when running under renderdoccmd, disable the overlay, provide comments for a capture, set the file path template for saving and more.

RenderDoc in-application integration

Buffer mapping

The last change I’m going to cover in this update is another long standing graphics issue that seems to only manifest itself on Windows 10 and Intel GPUs. There are some reproducible cases when the driver would end up displaying the first frame and doing nothing more, while the application continues to run and process events. The error does not occur when running the tests under apitrace, RenderDoc or Intel GPA, making it harder to examine.

I have discovered that the problem is related to mapping and unmapping of uniform buffer objects and my solution so far has been to disable the feature altogether while providing an option for the user to enable it again. 😩 When mapping is not available the renderer falls back to using a single per-frame call to glBufferSubData() followed by buffer orphaning, resulting in comparable performances in many situations.

And that’s the end of another pretty long and detailed update, I hope you enjoyed it! 😉

nCine Dev Update 8

Mon, 01 Apr 2019 00:00:00 +0000

I have spent some very intense weeks this March to completely overhaul and refactor my CMake scripts. We are talking about more than two thousand lines of code! 😱

Android

I started by changing the way I cross-compile on Android. I dropped the use of CMake integrated NDK support introduced with version 3.7 and went back using the toolchain file provided by the NDK itself. It seems the way to go in order to provide full control and flexibility to each NDK update.

I have also used this opportunity to add support for NDK r19 and its embedded stand-alone toolchain, the new 3.x Gradle plugin and LLVM’s libc++_shared.

Target properties and expressions

I then started to broaden the use of more target related CMake commands, as target_sources or target_link_options. This last command raised the minimum CMake version to 3.13.

Taking full advantage of CMake targets allowed me to completely remove any set command involving variables like CMAKE_CXX_FLAGS or CMAKE_SHARED_LINKER_FLAGS. Besides that I also started to rely more deeply on generator expressions for things like $<TARGET_FILE:tgt>, $<BUILD_INTERFACE:...> or $<CONFIG:cfg>. This last expression got rid of every if(CMAKE_BUILD_TYPE MATCHES "Debug") conditional check. 😎

Imported targets

But I didn’t stop there and I went on to convert my scripts to use targets everywhere. I now use imported targets for every external dependency library. In the end it’s a better and more flexible way of handling the problem but it put a lot of stress on testing as not all find_package scripts correctly set targets and some platforms need additional tweaking too.

For example MSYS/MinGW needed a custom CMake function to find the DLL and set the IMPORTED_LOCATION target property as the corresponding find_package scripts only discovered the import library.
As another example on macOS I needed a different custom CMake function to split the link options set by find_package scripts into the ones representing framework directories and the ones being -framework link directives. There is also a need to append the library symlink name to the framework directory as that is what the IMPORTED_LOCATION target property expects when dealing with macOS frameworks (someone has even opened an issue ticket about that).

Discoverability and exported targets

In an attempt to make nCine discoverability more robust for external projects I have changed a lot of code based on cascaded conditions to a bunch of a lot cleaner find_path calls.

I now also always use exported targets and this has two very important benefits:

It is now possible to link to the ncine target and automatically have access to interface properties like for example INTERFACE_INCLUDE_DIRECTORIES.
By using the nCine_DIR CMake variable when configuring a project it is possible to import the targets from a build directory.

The outcome of this last change is that there is no more NCINE_HOME custom variable, all the discovery use cases are handled by the CMake way of doing things: using a <package>_DIR variable that points to where the nCineConfig.cmake script is. This will in turn include the nCineTargets.cmake script to bring all the imported nCine targets into the namespace of an external package CMakeLists.txt file.

Bonus

A couple of additional changes are:

I now build a new ncine_main static library and export it as a target, instead of distributing a main.cpp file that the user never really had the need to modify.
I have added an interface library pseudo target for the code coverage compiler and linker options as a way, as the CMake manual states, “to employ an entirely target-focussed design for usage requirements”. 💪

That’s all for this update, I hope you enjoyed this in-depth review of CMake changes.

nCine Dev Update 7

Tue, 08 Jan 2019 00:00:00 +0000

Just a few weeks after the last update here I’m again to write the next one in which I’m going to show you the performance of my new hash table implementation.

It is based on the Leapfrog Probing article by Jeff Preshing, where the author explains a new probing algorithm for collision resolution when using open addressing.

My version is not multi-threaded but in return it is able to remove elements, a task that turned out to be quite more challenging than what I expected in the beginning. 😅 I also took the opportunity to change the default hash function to FNV-1a.

My previous implementation was based on separate chaining with list head cells, a pretty common technique that allows for unlimited number of stored elements in the face of many potential cache misses when the load factor increases.

The following results are a courtesy of the Google Benchmark library, the latest third-party software integrated in the nCine. They have been recorded on an Intel i7-8550U, running Arch Linux with the performance scaling governor and a fixed frequency of 800 MHz.

The tests have been performed on different implementations, they all have in common an unsigned int for the key and the value and 1024 buckets:

StaticHashMap is the new open addressing hash map in a version that takes the number of buckets as a template argument, similarly to std::array
HashMap is the new open addressing hash map in a more classic version that allocates on the heap
HashMapList is the old hash map with separate chaining
unordered_map is the standard hash table implementation from the STL

Some tests have a number after their name representing either the number of elements already in the hash table, like for Copy, Retrieve or Clear, or the number of times the operation is repeated, like for Insert, Remove or ReverseRemove.

Benchmark	StaticHashMap	HashMap	HashMapList	unordered_map
Creation	443 ns	34106 ns	86589 ns	13143 ns
Copy/256	2859 ns	31501 ns	78531 ns	1656125 ns
Copy/512	2912 ns	31004 ns	745993 ns	3600840 ns
Copy/768	2895 ns	34312 ns	1075128 ns	4931709 ns
Insert/256	4725 ns	9670 ns	9313 ns	1581495 ns
Insert/512	8423 ns	19566 ns	674695 ns	3169524 ns
Insert/768	13896 ns	33050 ns	1007072 ns	6504032 ns
Retrieve/256	66 ns	67 ns	75 ns	72 ns
Retrieve/512	71 ns	74 ns	81 ns	72 ns
Retrieve/768	76 ns	77 ns	83 ns	73 ns
Clear/256	1733 ns	22215 ns	42527 ns	1541229 ns
Clear/512	1733 ns	21789 ns	701556 ns	3078237 ns
Clear/768	1734 ns	22076 ns	1025812 ns	4609838 ns
Remove/256	5083 ns	25895 ns	35680 ns	1565479 ns
Remove/512	9459 ns	38793 ns	703407 ns	3136167 ns
Remove/768	14828 ns	53607 ns	1040247 ns	4662149 ns
ReverseRemove/256	4906 ns	25564 ns	35849 ns	1560456 ns
ReverseRemove/512	9620 ns	38567 ns	700163 ns	3113046 ns
ReverseRemove/768	14527 ns	51159 ns	1031670 ns	4669154 ns

Last but not least, I have replaced all the occurrences of the classic rand() function with a new PCG32 random number generator based on the official minimal C implementation.

It has a higher statistical quality while also being a lot faster. In my microbenchmarks, with the same CPU settings as the hash map ones, it is able to generate 1024 integers in around 14203 ns, as opposed to 74303 ns when using simple rand() calls.

nCine Dev Update 6

Sun, 16 Dec 2018 00:00:00 +0000

After all the work carried out during the last months and culminated in the previous update, I took some time to experiment with some different things. I wanted to leave the rendering side for a bit and take a look at how to optimize other parts of the engine.

I got my hands dirty straightaway with low-level aspects by playing with SIMD intrinsics, both Intel SSE and ARM NEON. The project involved quite a lot of different tasks, such as:

Looking online for example code (like SSE Quaternion Multiplication or SSE 4x4 Matrix Inversion)
Implementing functions for both SSE and NEON
Unit testing my code for correctness and parity against non SIMD one
Dealing with compilers details (always_inline, Vector Extensions, alignas)
Analyzing generated assembly on godbolt.org
Measuring performance on both PC and Android

At the end the performances were too close to what the compiler alone could achieve with auto-vectorization and instead of pursuing more optimizations I put the task on hold. It was still a fun and learning experience and it allows me to show you my SIMD benchmark utility. 😄

It is powered by ImGUI, it can load and save data from test runs using Lua and then compare them.

apptest_simdbench

After archiving the simd branch on my repository I jumped on another big project: trying to come up with an ECS implementation for the nCine. It would flatten the hierarchy of game elements and allow for a faster and possibly multi-threaded approach at updating them.

I have already put together some early tests but before going on I took some days to have a look at some profiling tools. Embedding custom timers and graphs into the engine can be fun, as can be writing ad hoc benchmarks, but for this task I needed something more powerful.

I stumbled upon Tracy, a very capable frame profiler that shows CPU and GPU zones, has Lua instrumentation, annotates allocations and locks and displays call stacks. Instrumentation of user code is easy, capturing data has a very low overhead and works locally or remotely on Linux, Windows, macOS and Android.

The profiler that collects and visualize the data has a super rich ImGui interface with line plots, histograms, colored sections and the ability to compare two captures side by side.

Tracy profiler

With the integration in place I can now be sure that I will be able to assess any performance change related with big refactorings or optimizations. 💪

nCine Dev Update 5

Sat, 29 Sep 2018 00:00:00 +0000

During those months two very important features appeared in the nCine.

The first one is the integration of Lua for scripting, a language which is very easy to integrate and runs very fast. With Lua the user can quickly prototype ideas or actually write the entire game with just scripts, in a way similar to other engines.

There is also a second way of interacting with the language, as the engine has now a bunch of useful methods to exchange data with it. You can load data from Lua tables to use a data-driven approach, or you can script the behavior of specific game objects. Scripts can also be reloaded while the application is running, for a quicker iteration time and easier tuning of the game.

The second feature is the integration of ImGui, a widely used immediate mode GUI that allows the engine to sports more meaningful and detailed performance data in a brand new debug overlay. The old on-screen HUD has been rewritten and plenty new statistical plots have been added, including a new section dedicated to Lua showing things like the amount of memory used or the number of operations per second. While integrating ImGui, the rendering system was enhanced in order to accomodate the scissor test and more flexible options for indexed drawing and vertex formats.

I have also worked for a month on ncParticleEditor, an editor for particle systems that use both new features: Lua for loading and saving and ImGui for the interface. As the editor is really just an nCine application, it can run with no problems on Windows, Linux, macOS and of course Android! The particle system was overhauled to support additional features like new particle affectors, a new particle random initializer structure and grayscale sprite rendering.

I have to thank my friend Helba for his infinite patience while testing the program and for allowing me to show some of his projects in the video. To help with testing I have integrated CrashRpt in my debug Windows builds to ease the creation of Minidump files.

I hope you have enjoyed both the videos and the new features. 😉

nCine Dev Update 4

Mon, 06 Aug 2018 00:00:00 +0000

Plenty of work has been done during these few last months as I decided once again to update the engine renderer. This time it went from using OpenGL 2 and OpenGL ES 2 to 3.3 and 3.0 respectively.

I started by rewriting all the shaders to support the new in and out keywords in place of the old attribute and varying ones. For the sprite shader I also changed it in a way that it doesn’t need any vertex data anymore, it generates a unit square by itself. I also took the chance to add support for Uniform Buffer Objects, one of the important new features of this OpenGL version.

As I wanted to share an UBO between multiple rendering entities I have written an OpenGL buffer manager so that a rendering command, say for example a command used to render a sprite, can ask for a certain amount of bytes without worrying about alignment or creating a new buffer when there is no more free space left. The manager is also responsible for mapping and unmapping memory once per frame.

For mapping I usually use a combination of GL_MAP_INVALIDATE_BUFFER_BIT and GL_MAP_FLUSH_EXPLICIT_BIT, as suggested on the Buffer Object Streaming wiki page, in order to maximize performances. I don’t use persistent mapping as the ARB_buffer_storage extension is only available with newer versions of the API. I have extended the manager to also handle VBO memory and thus enabling the use of a single common buffer for all the vertices that don’t live in a custom VBO (which is a VBO that is only used by a single render command).

Another important addition is the support for the KHR_debug extension in the form of debug groups and object labels. This feature is really important with tools like apitrace and RenderDoc that offer support for them.

apitrace

Handling vertex formats has been renewed as well and there is now a pool of Vertex Array Objects that minimize the number of calls when changing formats.

One feature I was willing to add for a long time is the ability to render sprites with a custom mesh. It can be used to split the transparent outline from the opaque part of a sprite, like it is described in this article, or to trim a particle down to the area with non completely transparent pixels, or to animate with shape deformations. It also allows for the definition of a set of indices in order to reuse vertices, a feature that was not available in the engine before and that required some effort and testing. As per vertex data, indices are handled by the memory manager and collected each frame in a set of common buffer objects.

The new OpenGL ES 3.0 Android renderer allows the use of the ETC2 compressed format and immutable textures. For this second feature to work on desktop I need to check the presence of the ARB_texture_storage estension as OpenGL 3.3 doesn’t offer it out of the box. Viceversa there is one thing I use on desktop OpenGL that is not supported by OpenGL ES 3.0: glDrawElementsBaseVertex. On Android I have to simulate this drawing command variation by using an additional offset.

For sure one of the most complicate new feature, as it has many corner cases that needed extensive debugging, is the automatic batcher. It allows to reduce the total number of draw calls issued by aggregating uniforms, vertices and indices from multiple commands into a single one. It can batch text nodes, regular sprites and custom mesh sprites.

For entities with external vertex data, like text nodes and custom meshes, it has to supply an additional vertex attribute to act as a mesh id, while for custom meshes it has to add artificial indices to sprites that do not use them if they are going to be batched together with sprites that have user supplied ones.

It’s degenerate vertices and patched indices all the way down. 😉

RenderDoc

Some of those new features have a runtime setting, like switching automatic batching on and off:

/// True if batching is enabled
bool batchingEnabled;
/// True if using indices for vertex batching
bool batchingWithIndices;
/// True if node culling is enabled
bool cullingEnabled;
/// Minimum size for a batch to be collected
unsigned int minBatchSize;
/// Maximum size for a batch before a forced split
unsigned int maxBatchSize;

Some others have an initialization configuration, like the dimension of each common VBO or the size of the VAO pool:

/// Sets the maximum size in bytes for each VBO collecting geometry data
void setVboSize(unsigned long vboSize);
/// Sets the maximum size in bytes for each IBO collecting index data
void setIboSize(unsigned long iboSize);
/// Sets the maximum size for the pool of VAOs
void setVaoPoolSize(unsigned int vaoPoolSize);
/// Enables OpenGL debug context
void enableGlDebug(bool shouldEnable);

There is also a bunch of new rendering statistics that are aggregated per frame and displayed on screen, like the amount of video memory used by textures and buffer objects, or the number of VAO pool recycles.

apptest_sinescroller

Rendering should now be more efficient on all platforms, with fewer draw calls and more sprites on screen! 💪

nCine Compilation Benchmark

Sun, 04 Mar 2018 00:00:00 +0000

Are you curious about the time I spend to compile the nCine? 😄 In this post I’m going to show how much time is needed to compile the engine on different platforms.

Let’s start with the hardware and the software I’ve used for my tests.

Hardware and software

Xiaomi Mi Notebook Pro

Hardware

Intel Core i7-8550U
16GB RAM, DDR4 2400 MHz, dual channel
Samsung PM961 NVMe SSD, 256GB
NVIDIA GeForce MX150, 2GB GDDR5

OS

Microsoft Windows 10 Professional (x64), Build 16299.248
Arch Linux x86_64, Linux 4.15.3

Xiaomi Mi Notebook Air 13 2016

Hardware

Intel Core i5-6200U
8GB RAM, DDR4 2133 MHz, single channel
Samsung PM951 NVMe SSD, 256GB
NVIDIA GeForce 940MX, 2GB GDDR5

OS

Microsoft Windows 10 Professional (x64), Build 16299.248
Arch Linux x86_64, Linux 4.15.3

Chuwi Hi 10

Hardware

Intel Atom x5-Z8300
4G RAM, DDR3 1600 MHz, single channel
Hynix HCG8e eMMC 5.1, 64GB

OS

Microsoft Windows 10 Home (x64), Build 16299.248

Compilers and tools

Windows

VS2017 15.5.6 (MSVC 19.12.25835)
CMake 3.10.2 64bit

MSYS2 and Arch Linux

GCC 7.3.0
CMake 3.10.2
Ninja 1.8.2

Results

The nCine version is 2018.02.r183-2d21790, compiled with GLFW, NCINE_BUILD_TESTS set to ON, NCINE_BUILD_UNIT_TESTS and NCINE_BUILD_ANDROID both set to OFF. The tests have been conducted by running the compilation process multiple times and recording the best timings.

In the Configure phase CMake is invoked in order to configure the project and generate either a Visual Studio solutions, or Makefiles, or Ninja files. In the Build phase CMake is invoked with the --build option in order to build the project. It takes advantage of multiple cores to compile only if using Ninja or MSVC with the /MP option. When using standard makefiles I have additionally provided the timings of invoking make -jN for the multicore compilation.

Tables and charts

MSVC

	Mi Pro	Mi Air
Configure	14.20s	12.02s
Build	22.23s	32.93s
Build with `/MP`	14.87s	26.48s

MSVC chart

MSYS2

	Mi Pro	Mi Air	Chuwi
Configure Ninja	8.13s	12.33s	40.90s
Build Ninja	7.13s	18.84s	55.71s
Configure Make	14.14s	19.75s	61.13s
Build Make	46.80s	66.12s	226.53s
Build Make `-j`	12.65s	30.59s	84.90s

MSYS2 chart

Arch Linux

	Mi Pro	Mi Air
Configure Ninja	1.03s	1.31s
Build Ninja	3.68s	7.85s
Configure Make	1.25s	1.58s
Build Make	13.56s	17.31s
Build Make `-j`	4.35s	9.17s

Arch Linux chart

Conclusions

The Mi Notebook Pro, with its Kaby Lake R featuring 4C/8T, achieves more than double the performance of the smaller Mi Notebook Air 13 when compiling with Ninja or make -j8 on all platforms. On Windows the speed-up is slightly less, but still very noticeable. The Mi Notebook Pro comes with a faster SSD and dual channel RAM which surely have affected timings. The little and fanless Chuwi Hi 10 comes last when talking about performances, but it can still configure and build the project in under 100 seconds with Ninja. 😄

Speaking about the configure phase, it strangely takes a bit more time on the Mi Notebook Pro compared to the slower Mi Notebook Air on Windows. It is also interesting to note that the Ninja generator in CMake is faster than the Makefiles one on all platforms, even if the difference on Linux is barely noticeable. Speaking about Linux, I wasn’t prepared for such a big margin when compared with Windows. Can it be a matter of Ext4 vs NTFS? Or maybe a slowdown when invoking commands through the MSYS2 console? I’m not sure of the reasons but lucky me for I usually develop on Linux and then test on the supported platforms. 😅

I hope you have found this post interesting even if it wasn’t about a development update. They will come back soon so stay tuned! 😉

nCine Dev Update 3

Sat, 10 Feb 2018 00:00:00 +0000

A lot of work has been carried out during the last three and a half months.

First of all I have added some macros to allow asserts in the code, think about checks like this: ASSERT_MSG_X(index < size_, "Index %u is out of bounds (size: %u)", index, size_); The macro also invokes a breakpoint if a debugger is connected, allowing to inspect the context around the failure. It was a long needed feature that I finally had the time to implement. 😄

I have also found the time to configure Uncrustify so that I could use it alongside Artistic Style in order to further check consistency with my code conventions.

But the big change is the support of many features of C++11, a leap forward for the whole codebase!

I was wise enough to stop for a moment and think about the plethora of changes that the template library would require. It is then that I realized that I would need to be disciplined and write unit tests first. I adopted Google Test, which is well supported by Qt Creator, and I started to write tests, plenty of them. The compiler helped with code coverage data and Gcovr with HTML reports. After a month of work I had more than 500 tests and discovered many subtle bugs. 😅

Now I was ready to work on supporting C++11. I setup a feature branch and started small by collecting low-hanging fruits first. The very first change has been the replacement of all NULL occurrencies with nullptr, a thing that alone needed set(CMAKE_CXX_STANDARD 11). 😄 Then more simple changes came, using =delete for special member functions that were previously private, the override specifier, delegating construtors instead of init functions, enum classes, alias declarations instead of typedefs and then I was ready for some bigger changes…

I moved the template library in its own namespace, nctl, standing for nCine Template Library. 😉 I added a reverse iterator adapter for templated iterators, and added a sentinel to the hashmap iterator and to the list class in order to enable an easy and flexible way to iterate in both directions. But the code couldn’t be modern without some range-based for loops and some lambda functions all around. 😉

nctl::Array<int> array;
for (int i : array_)
 printf("%d ", i);

forEach(players_.begin(), players_.end(), [](IAudioPlayer *player){ player->pause(); });

Then it was time for the biggest features, move semantics and smart pointers. In order to support the first one I have added move constructors and move assignment operators to all the containers, but also new insertion methods to allow the insertion of movable only objects, for example:

inline void pushBack(const T &element) { operator[](size_) = element; }
inline void pushBack(T &&element) { operator[](size_) = nctl::move(element); }

Time for smart pointers, enter nctl::UniquePtr and nctl::SharedPtr! For the shared one to work reliably with multiple threads I have added atomics on all supported platforms. I have replaced most of the raw pointers with unique ones and all factories now return a unique pointer to clearly show ownership transfer.

nctl::UniquePtr<ITextureLoader> texLoader = ITextureLoader::createFromFile(filename);
load(*texLoader.get());

It has been a very long but also very satisfactory task and I’m proud of how the new codebase has turned out. Now it’s time again to think about something entirely new and exciting!

nCine Dev Update 2

Sun, 22 Oct 2017 00:00:00 +0000

During those three months I have been working on two big features.

The first one has been the support of SDL2 GameController mapping format. Initially my plan was to build a layer on top of my joystick input functions and leave the mapping code outside, as helper functions in a file distributed along the source of my tests and only linked by them. Later on I decided to refactor everything and bring the code inside the engine, in an effort to make it easier for an application to transparently use it.

The mapping logic is still just a layer on top of the original joystick functions, but now the state of a mapped joystick can be queried easily from the input manager, just as you can do with the non mapped one. It is also possible for an application to listen to some new mapped events I have introduced and completely disregard the old ones. So, for example, instead of being notified of button 1 being pressed, the application would be notified of button “B” being pressed. This also works for axis, of course, so that an application can perform its logic on named axis, like X direction on the left stick, instead of an anonymous axis 5.

In order for the mapping to work, the joystick must be recognized. SDL2 uses a custom code per each platform in order to derive a GUID to identify the connected joystick. When compiled against SDL2 the engine can just use SDL_JoystickGetGUID() and SDL_JoystickGetGUIDString(), but on GLFW things are different. It seems like next version, GLFW 3.3, is going to have full SDL2 GameController suppport and provide a glfwGetJoystickGUID() function, which I’m already using inside a #if GLFW_VERSION_MAJOR == 3 && GLFW_VERSION_MINOR >= 3 block.

For previous versions of GLFW there is no GUID to return and the joystick mapping code falls back on comparing the name of the connected joystick with the one in the database, it is not completely reliable but it still mostly works. When running on Android, SDL 2.0.6 uses the first 16 characters of the joystick name as its GUID, a less than optimal way that just resembles closely my GLFW 3.2 fall back method. 😄 That is why I tried to do things better, especially because it is not so hard, when on API level 19, to just call the getVendorId() and getProductId() of the InputDevice class and create an unique GUID with those.

I have mentioned the mappings database before, it is assembled by combining all strings from SDL_gamecontrollerdb.h, all strings from gamecontrollerdb.txt plus some more that are useful when running on GLFW 3.2 or on Android, given their particular GUID code.

The second big task has been the complete transition from ndk-build to CMake, as starting from version 3.7 the developers have added the support for Android as a platform through the NDK or a standalone toolchain. You start by defining CMAKE_SYSTEM_NAME=Android and some more additional CMake variables to specify the API level, the NDK location, the CPU architecture or the STL type. I have decided not to use the modified CMake 3.6 that is distributed with the Android SDK, even if it is natively supported by the Android Plugin for Gradle, but the upstream latest version. This way I can support and test a single version of CMake to build on every platform.

See you again soon with new and exciting updates from the nCine development! 😉

nCine Dev Update 1

Mon, 18 Sep 2017 00:00:00 +0000

During June and July 2017 I have been working as usual, in my spare time, on the project. 😉

The first big June addition has been the automatic screen culling of sprites, a very important feature needed in order to support games extending on multiple screens. The culling works on sprites of any kind (regular ones, particles, text nodes) and regardless of their scaling or rotation parameters. If those sprites are completely outside of the screen they will just not be rendered, saving draw calls from being issued.

The next feature has been the complete support for mouse and keyboard on Android, a task which was made more interesting by some hacks I had to perform. For example, I had to implement a combination of bitmask operations just to support the right mouse button, a trick needed because most Android devices (but not my Shield TV) map the right mouse button to the special back button, which is handled as a key. 😫 It means that the AINPUT_SOURCE_MOUSE can generate AINPUT_EVENT_TYPE_KEY, just as if it was a AINPUT_SOURCE_KEYBOARD. In this case if the keycode is AKEYCODE_BACK I have to simulate a press from AMOTION_EVENT_BUTTON_SECONDARY. 😅

Those changes together made possible the creation of a new test example where many different sprites wave randomly across the screen while the user can move the view as if controlling a top-down camera. As usual in my nCine tests the user can use multiple input methods like keyboard, mouse, joystick or the touch screen on Android.

The last change is the full support for SDL2 that I added in July. It was required because the old SDL1 back-end was diverging too much in terms of functionalities when compared with the GLFW one. Now both back-ends support events for the connection and disconnection of a joystick, events for the mouse wheel scrolling (which on SDL1 I could only simulate) and, above all, the possibility to pass parameters during the OpenGL context creation. This last feature will be vital for a future support of newer versions of OpenGL and OpenGL ES.

That’s all for now, see you soon in a next installment of the nCine Dev Updates! 😉

Get to know the nCine

Sun, 20 Aug 2017 00:00:00 +0000

I am writing this to present my latest and biggest project to date, the nCine. I was originally going to publish an article the day that I was going to release the source code but for various reasons you will have to wait a bit more. 😢

I am nevertheless going to describe today some of the technical aspects behind it hoping that when it’s ready for release there would already be someone interested in using it. 😉

The nCine banner

nCine is a multi-platform 2D game engine that works both on PC (meaning Linux, Windows and OS X) and on Android. Yes, the name is a portmanteau of my nickname “Encelo” and the word “engine”. 😅

I started working on it in June 2011, at the time I was using a Mercurial repository to keep track of changes but later on I converted it to a Git one. Some of the developing tools I use all the time are Qt Creator as my IDE of choice (plus the Community Edition of Visual Studio on Windows), CMake for everything related with the building and packaging phases, Doxygen and Graphviz for the documentation, Cppcheck for the static analysis, Valgrind for additional memory debug and Artistic Style for the automatic formatting.

The project has a wide support in compilers as well, GCC and LLVM on Linux, MSVC and GCC (via MSYS2/MinGW-w64) on Windows, LLVM on OS X and GCC and LLVM on Android with the NDK. At the moment there are more than one and a half thousand lines of CMake scripts and more than twenty five thousand of C++ code, excluding comments and blank lines. 💪

The nCine is a framework of classes and can be built as a static or dynamic library.

This work has always been intended as a way for me to learn new things and develop my skills as an engine programmer, this is the reason why I have implemented many things from scratch instead of relying on external libraries.

For example I don’t use the STL but I have my own implementation of template based containers (list, array, hashmap, string, …), algorithms and iterators. It was a very good way to overcome the fear of templates and learn about type traits, concepts, tag dispatching or SFINAE. 😄

The project also gave me the opportunity to learn low-level OS APIs such as monotonic timers, threads and their synchronization and affinity. All those primitives are handled differently by the Linux, Windows and Mach (OS X) kernels.

There are still, of course, some external dependencies: for the rendering I am relying on OpenGL, capable of reaching all the OSes I’m targeting, while for the sound I have used OpenAL Soft. On top of the latter I use Ogg and Vorbis to enable music streaming and playing compressed sound samples (with uncompressed Wave being another supported option). For interfacing with the window and the input system I use both GLFW 3 and SDL 2, depending on which one is available, while on Android I use EGL and the Android API, sometimes through the NDK API and sometimes, like when accessing the joystick, directly through JNI calls.

For texture loading I use libpng and WebP, but the engine is also perfectly capable of loading GPU compressed formats such as DXT, ETC1, ATC, PVR and ASTC, embedded in DDS, KTX or PVR texture container formats.

When it comes to font rendering I have written a parser for AngelCode’s BMFont FNT format so that the engine can render text strings with support for kerning pairs and customizable horizontal alignment between multiple lines.

The dependencies that I have just mentioned are compiled directly from upstream sources for all supported platforms thanks to a set of custom CMake scripts. I am able to create the .so shared libraries on Linux, the DLLs on Windows, the frameworks on OS X and cross-compile for all the supported Android architectures (armeabi-v7a, arm64-v8a and x86_64). The scripts are also responsible for putting together the NSI installer and the ZIP portable archive on Windows, the application bundle on OS X and the TAR.GZ portable archive on Linux. For Arch Linux and MSYS2, which are both based on Pacman, I have written the corresponding PKGBUILD scripts to create a compressed package.

I will soon release the engine on GitHub under a MIT license. At the moment you can have a look at the website, which contains the initial API documentation, and you can follow the project on Twitter.

I hope to come back often in the future to write about new features and very soon to announce a full release!

About me

Mon, 01 Jan 0001 00:00:00 +0000

I started my journey in computing on the Amiga, first with a stock A500 in 1991, and later with a towered A1200 upgraded with a BlizzardPPC and BVision. Those years shaped my fascination with graphics, games, programming, and the culture around making machines do more than anyone thought possible.

In March 2000 I had my first real contact with the *nix world, running NetBSD 1.4.2 on my 68030 A1200, followed soon after by LinuxPPC 2000. That opened the door to free software and the joy (and pain) of tinkering with low-level systems.

Since February 2005 I’ve been a daily Arch Linux user. It fits the way I like to work: simple, transparent, and close to the metal.

I’ve always loved the demoscene, chiptunes, sinus scrollers, and roto-zoomers: the playful side of pushing hardware limits. At the same time, I follow modern breakthroughs in real-time graphics with the same excitement.

This site is a place to share my work, notes, and projects. All opinions expressed here are mine alone.

Credits

Mon, 01 Jan 0001 00:00:00 +0000

This site is built with Hugo using the Blowfish theme by Nuno Coração.

The background image is by Philip Oroni on Unsplash.

This site is licensed under the Creative Commons BY-NC-SA 4.0 International license.

GnuPG Public Key

Mon, 01 Jan 0001 00:00:00 +0000

This is my personal GnuPG public key, used to sign my Git commits.

The same key is also published on ncine.github.io, GitHub, keys.openpgp.org, and keyserver.ubuntu.com.

Fingerprint: 387C DEFE 8A87 AEB3 2897 8C80 7289 85A1 8FEE 4962

Support

Mon, 01 Jan 0001 00:00:00 +0000

My work on open source projects like nCine and SpookyGhost has always been a labor of love. I build and maintain these tools in my free time, and I share them so anyone can learn, create, or just enjoy tinkering with code and graphics.

If you’ve found value in my work, or simply want to help me keep developing and documenting these projects, you can support me through one of the platforms below. Every contribution, big or small, makes a real difference and helps me dedicate more time to open source.

GitHub Sponsors— best for ongoing monthly support
Patreon — another way to support regularly
Ko-Fi — quick one-time tips (or coffees! ☕)
Buy Me a Coffee — same idea, just another friendly platform
LiberaPay — focused on free software support
PayPal — direct donations

Thank you for helping me keep these projects alive and evolving. 🙏

encelo.github.io

nCine Dev Update 23

GrAIL

Batching with instances

Benchmarks

Faster directory traversing on Windows

nCTL updates

Optional class

Iterators refactoring

Pair class

Hash containers refactoring

Initializer list support

Other additions

Hashing functions

Benchmarks

Application configuration structures

Minor changes

An Intro to Wave Intrinsics

Enforcing Uniform Control Flow

Delegating Work to a Single Lane

Reserving Space in a Buffer

Conclusions

A New Site

nCine Compilation Benchmark 3

Hardware and software

Asus ROG Zephyrus G15 (2022)

Hardware

OS

Alurin Flex Advance

Hardware

OS

Mac Mini M1 (2020)

Hardware

OS

Compilers and tools

Arch Linux

macOS

Results

Tables and charts

Arch Linux

Conclusions

nCine Dev Update 22

Introspective sort

Global Game Jam 2025

Presentation at /dev/games

RenderDoc integration update

Run-time environment variables

Crashpad integration

Array improvements

Multi-threaded job system

CPU topology

Conclusions

Minor changes

nCine Dev Update 21

OpenAL EFX

Multi-threaded job system

Lua development improvements

Improvements to apptest_lua

Minor changes

nCine Dev Update 20

Binary shaders

Hashing

Maximum batch size

HiDPI support

Thread class move fixes

Bunnymark

Improvements to particle systems and affectors

Minor changes

nCine Compilation Benchmark 2

Hardware and software

Xiaomi Mi Notebook Pro (2017)

Asus Zephyrus G15 (2022)

OS, Compilers, and Tools

Results

Tables and charts

Conclusions

nCine Dev Update 19

Custom shaders

ShaderState class

OpenGL layer

Improvements to `apptest_lua`