SCons an Introduction

SCons: an introduction

Dean Giberson <dean@deangiberson.com>

Dec 17, 2008

If you've tried putting a build system together for a modern game you know
that you are facing a monumental, never ending task. Once you do get a basic
pipeline working, modern games require source assets that are in the range of
100 Gb of raw data. Tracking this amount of data takes time and resources.
SCons (http://www.scons.org) is a Python based make replacement; with
it you can tame your dependencies and data sizes. What follows is quick art
centric introduction.

1 The Canonical Example
The SCons documentation1 would give an example like this:

env = Environment()

env.Program( 'HelloWorld', ['HelloWorld.cpp'])

follow this and provide a compilable HelloWorld.cpp and you would have a
program ready to go. But how would you extend this to use your own pipeline
tools?

2 Adding Compressed Textures
Lets assume that you want to compress textures for use in a game. NVidia
Textures2 Tools can be used on your TGA/PNG/JPG/DDS images. For exam-
ple you want to convert this image3 into a DXT5 compressed image for use on
target:

nvcompress -color -bc3 gunmap.png gunmap.dds

1 http://www.scons.org/doc/0.97/HTML/scons-user/book1.html
2 http://developer.nvidia.com/object/texture_tools.html
3 Images provide by G3D Engine Data Collection http://g3d-cpp.sourceforge.net/.

1

Once we understand how to do this from the command line adding it to
SCons is quite easy.

import SCons

env = Environment(ENV = os.environ)

env['NVCOMPRESS'] = 'nvcompress'
env['NVCOMPRESSTYPE'] = SCons.Util.CLVar('-color')
env['NVCOMPRESSFLAGS'] = SCons.Util.CLVar('-bc3')
env['NVCOMPRESSCOM'] = '$NVCOMPRESS $NVCOMPRESSFLAGS $SOURCE $TARGET'

NvCompressAction = Action( '$NVCOMPRESSCOM')
NvCompressBuilder = Builder( action=NvCompressAction,
suffix='.dds')

env['BUILDERS']['NvCompress'] = NvCompressBuilder

env.NvCompress( 'image', 'image.png')

2

3 A Quick walk-through to understand what is
happening
import SCons

We import the SCons python module into our script. Normally you don't
have to do this, but I'm using the 'SCons.Util.CLVar' class provided by SCons
so I import the module to gain access to it.


Construct a SCons Environment object. SCons does not copy the system
environment by default, this is by design as a build environment should be as
explicit as possible. For now I'll just copy the system environment, but please
note that this is bad practice.


Now begin populating the Environment with data. Environment objects
are class instances that behave like dictionaries. We can add to them through
assignment, and query the contents using normal Python dictionary functions.
In our case I'm lling four slots with:
• the name of the executable ('NVCOMPRESS'),
• the default type ag ('NVCOMPRESSTYPE'),
• the default compression ag('NVCOMPRESSFLAGS'),

• a template command line ('NVCOMPRESSCOM').
SCons uses late binding of variables found within environment strings,
any sub-string that starts with a '$' character is interpreted at the calling
site for the current value within the current environment. This acts like a
controlled dynamic scoping system.
I'll come back to what this means in practical terms in a moment, but
for now accept that our default command line evaluates to: 'nvcompress
-color -bc3 $SOURCE $TARGET' .

3

suffix='.dds')

These three lines are the core of our SCons extension; an Action object is
created, and the command line string is set. The evaluation of this command
follows the same rules as for environment variables set earlier.
Then a Builder is constructed using the Action object just created. We also
set the default extension for all targets.


The Environment is then extended with this builder, and a name is given
for calls to it.

env.NvCompress( 'image', 'image.png')

The only thing left is to construct a Node in the dependency tree for the
target le.
SCons is a three stage process:
1. Create or gather a collection of tools and set the environment,
2. Create a tree of dependencies with sources and targets,
3. The dependencies are scanned, then for any out of date data, the action
is called.
Each of these steps is discrete, and must be occur in this order. Step
1 happens during the setup phase of SCons (default tools are scanned
for and basic environments are constructed) and to a lesser extent within
the running SConstruct script itself (as I've just shown). Step 2 happens
within the SConstruct script and continues until the end of the script.
After control returns to the SCons system the Step 3 begins and the
dependency tree is scanned and the actions are triggered. Always in this
order.
It's for this reason that I say that a dependency node is constructed from
the last line of the SConstruct. It doesn't actually run 'nvcompress' at
this point. Only an object, representing the potential to run 'nvcompress',
is constructed and added to the system. Work is done later by an internal
SCons class, 'Taskmaster.

4

4 More les, more controls
One of the great things about SCons is that it's embedded with Python. A
consequence of this choice is that you, the tool writer, have access to all of
Pythons functions.
I've shown you how to add one Builder and access that Builder for a single
target texture. Games are not made from one texture, you're going to want
access to many textures and that means several targets. Glob
to the rescue.

from glob import glob

for tex in glob( r'./*/*.png'):
target = tex.replace('.png','.dds')
env.NvCompress( target, tex)

This will nd all of the source textures in subdirectories, and add them to
the dependency graph. Any textures that are added are found automatically.
The same strategy can be applied to other types of data as well. Python
has great libraries for XML, SQL, even direct in memory structure access or
peeking in compressed les. You will not have to drop out of Python for very
many reasons. Check the module documentation4 for details.

5 Other compression types
The previous method of adding dependencies will add each texture into the
pipeline with the same compression options. Normally you will want a range
of compression methods for textures. I want to focus on the clearest method,
changing the values in the Environment.
I mentioned earlier that the evaluation of strings set for Builders and Actions
acts like dynamic scoping for variables. This feature allows us to change the
functionality of a call by changing values when the dependency node is built.

env.NvCompress( 'image2', 'image.png', NVCOMPRESSFLAGS='-bc2')

Which will result in this command: 'nvcompress -color -bc2 image.png
image2.dds'.
Adding a method to match lename patterns in a database (or text le)
gives us a simple way to control the compression of individual textures.

# Global texture compression options
4 http://docs.python.org/modindex.html

5

# format 'glob,opts'
.envmaps*.png,-bc1 # Does not include cubemaps
.nmaps*.png,-bc3n

This simple text le has a line for each le pattern, a comma (',') and
the compression option for the command line. Comments start with a hash
character ('#') and continue to the end of the line. A parser for this format is
easy to write.

from fnmatch import fnmatch

gCompressionOptions = []

f = open('texture_options.txt','rt')
try:
for line in f:
line = line.split('#')[0]
if line != '':
(pattern,options) = line.split(',')
gCompressionOptions.append( (pattern,options))
finally:
f.close()

for tex in glob( r'.**.png'):
hasCustomPattern = False
for pat,opt in gCompressionOptions:
if fnmatch(tex,pat):
opt = opt.strip()
env.NvCompress( target, tex, NVCOMPRESSFLAGS=opt)
hasCustomPattern = True
if not hasCustomPattern:

Once we have the patterns into an array it's simple to check if any les found
by the glob matches a given pattern. If there is a match, set the compression
options for that texture. If not the default is used.

6 Exploring dependencies
A core strength of SCons is it's dependency system. This system is not normally
based on time stamps but on a one way hash of the le contents (MD5). Using
hash values allows for stronger connections between assets.

6

In order to follow what SCons is doing with dependency checking you can
use the 'debug=explain' command line option. This option will print out
information about dependency checks and commands being run.
The interesting thing about using hash values for dependency check is that
you can't use touch to force a recompile of an asset, you must change the
contents of the le or force SCons to rebuild an asset.
On the ip side of this, you get control of the derived le from both the
contents of the source les and the contents of the command line used to build
the derived le. SCons combines all of these values into a string that represents
the derived target asset, if any sources have changed that targets action is
invoked.
To get a view of the dependency tree use the tree=derived option.

scons: Reading SConscript files ...
special .envmapsgunmap.dds .envmapsgunmap.png -bc1
special .nmapsgunmap.dds .nmapsgunmap.png -bc3n
normal .texgunmap.dds .texgunmap.png
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
+-.
+-envmaps
| +-envmapsgunmap.dds
+-nmaps
| +-nmapsgunmap.dds
+-tex
+-texgunmap.dds
scons: done building targets.

7 Sharing the results
Strong dependency systems are great for personal development, you can be sure
that you only need to build the minimum following a change. This doesn't help
on a large team; if an artist makes a change then once submitted every person
on the team needs to build that same asset. In my case building a compressed
texture takes 9.5 seconds, if you have 20 other people on your team, you team
will spend 190 seconds for each texture change. The odd part of this result is
that every person is trying to get the same result from the same source.
SCons took the strong hash key info and took it to the next logical level. If
everyone calculates the hash the same way, then it's possible to store copies of
the result in a shared location under that name. Then when you go to build, do

7

a quick check rst. If the asset exists under that name in this cache then just
copy it; if not then build the asset and place it into the cache under it's hash
name.
The result is a distributed build cache. And you can take advantage of it
out of the box. Create a location with a lot of available disk space, available for
everyone on your team, some server. Next place this line into you SConstruct
le.

env.CacheDir(r'x:Locationofcachedir')

Now your derived les are shared, and only one person, normally the origi-
nator of the source art, needs to build the nal result. In most cases, this saves
hours of cumulative time for a large team of people. You results will be better
if you have a continuous integration server for art.

8 The script we have
Taking all of these changes into account we get the following script.

import SCons


env.CacheDir(r'x:Locationofcachedir')


suffix='.dds')


from fnmatch import fnmatch

gCompressionOptions = []

f = open('texture_options.txt','rt')
try:

8

for line in f:
line = line.split('#')[0]
if line != '':
(pattern,options) = line.split(',')
gCompressionOptions.append( (pattern,options))
finally:
f.close()

for tex in glob( r'.**.png'):
hasCustomPattern = False
for pat,opt in gCompressionOptions:
if fnmatch(tex,pat):
opt = opt.strip()
env.NvCompress( target, tex, NVCOMPRESSFLAGS=opt)
hasCustomPattern = True
if not hasCustomPattern:

This will build all of your textures found in sub directories, using the com-
pression options found in the texture_options.txt le, and share the results
with colleagues using a shared drive.
I hope you see how easy it is to build a strong build system for art given the
right tools.
SCons is not just for code, but can be used for art builds as well. Coupled
with a continuous integration server, having a quick, robust pipeline is within
your grasp.

9

SCons an Introduction

More Related Content

What's hot (9)

Viewers also liked (18)

Similar to SCons an Introduction (20)

More from slantsixgames (11)

Recently uploaded (20)

SCons an Introduction