Problems and Solutions in Game Audio

Bits, Bytes and Beats: Problems and Solutions in Video Game Audio Karen Collins [email_address]

Outline History of Game Audio Innovations Three Fundamental Open Problems: Mixing Variation Adaptability Directions of my research Summary/Conclusions

2. A Brief History of Game Audio Innovations (What’s so different about game audio??)

A Brief Historical Outline 2.1 8-Bit (1970-1990) 2.2 16-Bit (1985-1995) 2.3 64-Bit (1995-2000) 2.4 128-Bit (2000-2005) 2.5 Mobile/Handheld 2.6 Today

Computer Space! (Nutting Associate 1971) First game to have sound. 2.1 8-BIT

Pong (Atari 1972) (Al Alcorn) 2.1 8-BIT

Space Invaders (Midway 1978) First use of continuous “background” music. 2.1 8-BIT

Rally X (Namco/Midway 1980) The birth of the loop as response to memory constraints. 2.1 8-BIT

Making a “beep” in assembly ( Time & memory constraints ) Beep PROC USES AX BX CX IN AL, 61h ; Save state PUSH AX MOV BX, 6818 ; 1193180/175 MOV AL, 6Bh ; Select Channel 2, write LSB/BSB mode 3 OUT 43h, AL MOV AX, BX OUT 24h, AL ; Send the LSB MOV AL, AH OUT 42h, AL ; Send the MSB IN AL, 61h ; Get the 8255 Port Contence OR AL, 3h OUT 61h, AL ; End able speaker and use clock channel 2 for input MOV CX, 03h ; High order wait value MOV DX 0D04h ; Low order wait value MOV AX, 86h ; Wait service INT 15h POP AX ; restore Speaker state OUT 61h, AL RET BEEP ENDP From Using Assembly Language by Allen L Wyatt 2.1 8-BIT

Technological Constraints Up ‘N Down (Sega 1983) 2.1 8-BIT

Atari VCS (2600) Up ‘N Down (Sega 1984) 2.1 8-BIT

Captain Comic (Color Dreams 1988) 2.1 8-BIT

Captain Comic ’s songs: Borrowing from classical music. ( Skill constraints ) 2.1 8-BIT

Working With Constraints: Nintendo NES Metroid (Nintendo 1987) (Hip Tanaka) vibrato (pitch modulation), tremolo (volume modulation), slides, portamento, echo effects 2.1 8-BIT

Ballblazer (LucasArts 1984) (Peter Langston) Algorithmic generation “ Riffology” method (Optimized randomness) by Peter Langston 32 eight-note melody fragments Algorithm chooses how fast, how loud, when to omit notes, when to insert rhythmic break Developed based on “lazy guitarist” 2.8-BIT

Working With Constraints: Commodore 64 Track 1 (e.g. “Level_One”) Instrument 1 (envelope, waveform, effect filters, etc.) Instrument 2 (envelope, waveform, effect filters, etc.) Rob Hubbard’s Module Format 2.1 8-BIT Pattern 1 (sequence of notes) Pattern 2 (sequence of notes) Pattern 3 (sequence of notes)

; track format: ; high address of pattern to execute ; low address of pattern to execute ; max index in pattern to execute (FF=max, can be terminate by instruction) ; number of times to repeat the given pattern track1: .byte >pat00, <pat00, $3D, $02 .byte >pat00, <pat00, $51, $00 .byte >pat01, <pat01, $0A, $04 .byte >pat02, <pat02, $16, $08 .byte >pat03, <pat03, $10, $16 .byte >pat03, <pat03, $FF, $00 .byte >pat0b, <pat0b, $FF, $00 .byte >pat0c, <pat0c, $FF, $00 .byte >pat0d, <pat0d, $FF, $00 .byte >pat0e, <pat0e, $FF, $00 .byte >pat0f, <pat0f, $FF, $00 .byte >pat10, <pat10, $FF, $00 .byte >pat11, <pat11, $FF, $00 .byte >pat12, <pat12, $FF, $00 .byte >pat13, <pat13, $FF, $00 .byte >pat14, <pat14, $FF, $00 .byte >pat15, <pat15, $FF, $00 .byte >pat16, <pat16, $FF, $00 .byte >pat17, <pat17, $FF, $00 .byte >pat18, <pat18, $FF, $00 .byte >pat26, <pat26, $FF, $00 .byte >pat37, <pat37, $FF, $00 .byte >pat27, <pat27, $FF, $00 .byte $00, $00, $00, $00 ;C900 pat38: .byte $86, $0D, $00 ; set instrument .byte $23, $D5 .byte $9B, $05, $23 ; play note .byte $86, $0E, $00 ; set instrument .byte $9B, $05, $5A ; play note .byte $D6 .byte $9B, $05, $5A ; play note .byte $86, $0D, $00 ; set instrument .byte $9B, $05, $23 .byte $D5 .byte $9B, $05, $23 ; play note .byte $86, $0E, $00 ; set instrument .byte $9B, $05, $5A ; play note .byte $D6 .byte $9B, $05, $5A ; play note .byte $86, $0F, $00 ; set instrument .byte $00, $D0 .byte $9B, $03 ; restore state .byte $00 .byte $00, $00 2.1 8-BIT

Shadow of the Beast 2 (Psygnosis 1989) (David Whittaker) 2.2 16-BIT MOD/Tracker on Amiga

Combining modules (in MIDI) with control statements MIDI and the Creation of iMUSE 2.2 16-BIT Land, Michael Z. and Peter N. McConnell. Method and Apparatus for Dynamically Composing Music and Sound Effect Using a Computer Entertainment System . US Patent No. 5,315,057. 24 May, 1994.

Super Mario World (Nintendo 1991) (Koji Kondo) 2.2 16-BIT Musical layering technique Mario jumps on Yoshi & gets extra layer of music (SNES).

Legend of Zelda: Ocarina of Time (Nintendo 1999) (Koji Kondo) (N64) Proximity-based algorithms control cross-fades 2.3 64-BIT

The Sims (Maxis 2000) Player-input/selectable music 2.4 128-BIT

Music driving gameplay elements. New Super Mario Bros (Nintendo DS 2006) (Koji Kondo) 2.5 Mobile

State-of-the-Art Today 7.1 to 8.1 surround sound Combination of synth with orchestra, choir At least 512 channels of sound God of War (Gerard Marino, Sony 2006) Bioshock (Gary Schymann, 2K 2007) 2.6 TODAY

3. Three Fundamental Open Problems

Fundamental Problems 3.1 Mixing 3.2 Repetition. Repetition. Repetition (variability!) 3.3 Adaptability Pathology : turning off sound/music, cognitive dissonance (failure of music to respond) > reduces immersiveness

3.1 Mixing Who needs mixing? Chicken Shift (Bally 1984)

Problem: Mixing: Unpredictability, Variability 3.1 Mixing

Problem: Mixing: current state of dynamic range … in a popular film … in a popular game Graphics adapted from those supplied by Rob Bridgett of Swordfish Studios . 3.1 Mixing

Solutions: Real-time Weighted Mixing Weighted permutations Predict which sounds can recur without making obvious. Example: Dialogue, Sound FX A. Sound FX B, player sounds, music, ambience If dialogue = “run!”, set parameter to 1 If gunshot is coming towards us, set parameter to 2 If no action, fade out music and raise ambience REQUIREMENT : “intelligent” Engine to predict and set weighting 3.1 Mixing

Solution: Location-Based Run-Time Mixing Real-time DSP to adjust sound E.g. bottle drop on hard floor of kitchen or in next carpeted room Factor in 5.1 surround to adjust real-time panning REQUIREMENT : audio engine to pass parameters from game and from player back and forth to engine. 3.1 Mixing

3.2 Variability Problem : Users get bored with hearing same sounds BUT sound designers can’t possibly record enough variations of sounds (time, budget) Problem : Users need a new experience every time they play the game (promised by LucasArts’ Euphoria technology) Problem : audio not responding to physics

Solution: Granular Synthesis 3.2 Variability

“ Granular” Synthesis “ acoustical quanta ” (Dennis Gabor: 1947 "Acoustical Quanta and the Theory of Hearing." Nature 159 (1044):591-594.) “ sonic quanta ” (Abraham Moles 1968 ”Information Theory and Esthetic Perception”. Urbana: University of Illinois Press.) “ particle audio ” (Parker and Behm 2007 ”Generating Audio Textures by Example”, Journal of Game Development, 2007) 3.2 Variability

Granular synthesis: Graphic Equivalent 3.2 Variability Input Sample Synthesized Result "Texture Synthesis from Multiple Sources", by Li-Yi Wei. In SIGGRAPH 2003 Applications and Sketches.

Making a Sound Granular 3.2 Variability Parker and Behm 2007 ”Generating Audio Textures by Example”, Journal of Game Development, 2007

Granular Synthesis Examples Crowd Tennis Speech 3.2 Variability Crowd and speech examples borrowed from Leonard Paul at Vancouver Film School

Granular: Remaining Open Questions What elements in a sound effect can be varied while still maintaining the “ meaning ” of the sound? How can we create AI systems that are aware of these potential meanings, and make real-time adjustments to sounds in a game? How to develop an “ audio physics engine ”: e.g. footsteps change based on how much player is carrying, etc. 3.2 Variability

3.3 Adaptability Problem : Games are non-linear, unpredictable and very long! A to B: 16 units A to all: 376 units 30 rooms: 11280 units 10 levels: 100K+ units Transitional Units

Solution: Game Audio Algorithms By varying existing individual parameters, we can create algorithms to: Write transitions Vary compositions Create new compositions Allow user- generate d content 3.3 Adaptability

Variable Musical Parameters Variable tempo Variable pitch Variable rhythm/metre Variable volume/dynamics Variable DSP/timbres Variable harmony (chordal arrangements, key or mode) Variable mixing : from the speaker placement of certain sounds to run-time adjustments of orchestration mix 3.3 Adaptability

8. Variable form ( open form ) random structure 9. Variable form ( branching parameter-based music) 10. Variable melodies : algorithmic generation … in what follows we will focus on these last three 3.3 Adaptability

#8 Variable (Open) Form Random structure Songs are segmented into components whose order can be changed Used in “hyrule field” of Legend of Zelda: Ocarina of Time : player spends a lot of time, and the same sequence in the same order would get monotonous 3.3 Adaptability

#8 Example: Variable (Open) Form 3.3 Adaptability http://www.home.cs.utwente.nl/~zsofi/mozart/ Variations: 11 14 x 2 2 = 1 518 999 334 332 964

#8 Variable Form: Non-linear Sequencing Musical control structures (repetitions, jumps, procedure calls) and grammars modelled on existing characteristics Music is to some extent already hierarchical (notes > phrases > sections > movements> pieces)  how do we teach/learn to composer in this manner? “ Grammars as Representations for Music” C. Roads; Paul Wieneke, Computer Music Journal , Vol. 3, No. 1. (Mar., 1979), pp. 48-55. How can we create sequencing software to better prepare composers to write this type of music? 3.3 Adaptability

#9 Parameter Based Music: Parameters Number/action of non-playing characters Number/action of playing characters Actions Locations (place, time of day, etc.) Scripted or unscripted events Player health or enemy health Difficulty Timing Player properties (skills, endurance) Bonus objects Movement (speed, direction, rhythm) “ Camera” angle The transition matrix approach and the creation of transitional units 3.3 Adaptability

#9 Example: Parameter-Based Music No One Lives Forever (Guy Whitmore 2000) Six standard music states are based on number of NPC enemies: Silence Super ambient Ambient Suspense/sneak Action/combat 1 Action/combat 2 3.3 Adaptability

#9 Example: No One Lives Forever Earth Orbit: Ambush theme starts in music state 5 (combat 1), transitions to music state 2 (ambient: in elevator) then transitions to music state 6 (combat 2) 3.3 Adaptability

#10. Algorithmic Variations (ongoing research focus) Problems: How do we create emotionally effective algorithmic adaptive audio? What aspects of audio carry meaning ? How do these work individually and together? What universals (within the Western world) are there that can be codified? How generalized/simplified can/do the rules / grammar need to be? 3.3 Adaptability

Semiotics Sound/music as a symbolic language What (and how) does music/sound communicate? How can we study and break these down into a grammar to generate algorithms? What combinations are effective? What variations/substitutions can be made with and without changing meanings? (For more info, see the work of Philip Tagg, Eero Tarasti, Jean-Jacques Nattiez, and Raymond Monelle; especially Phiip Tagg’s “Ten Little Title Tunes”, Mass Media Music Scholars’ Press 2002) 3.3 Adaptability

Semiotics of sound: Why is it important? An example… 3.3 Adaptability

Defining a Sound Semiotics Grammar Problem : Can we codify a semiotic grammar of sound? How? How do we gather enough data? One solution : distributed classification, or crowd sourcing 3.3 Adaptability

Distributed Classification Examples 3.3 Adaptability

What Does the User Get? Contribution to knowledge Feeling of being part of community Believe it or not -- fun! See Luis von Ahn, “Games with a Purpose” IEEE Computer Magazine or “ Why do tagging systems work?” Conference on Human Factors in Computing Systems CHI 06 3.3 Adaptability

ESP Game 3.3 Adaptability Player 1 Player 2 GUESSING: KID GUESSING: BOY GUESSING: CAR GUESSING: HAT GUESSING: CAR SUCCESS! Consensus on: CAR Input:

Games for Audio Tagging: Interactively Building an Online Database Three games under development: Game like ESP game but for audio (PHP and Flash front end with MySQL backend) Audio-visual game in which users select image to audio Audio-visual based game where users select appropriate audio content for visual image 3.3 Adaptability

Adapting the Algorithms For MIR MIR = Music Information Retrieval Retrieval based on bpm, harmonic content, melodic intervals, timbre, etc. How can we use MIR techniques to make better game audio? User-generated playlists + new algorithms = appropriate and new user-generated audio content 3.3 Adaptability

Unified Architecture Routing, allocation and scheduling (includes system clocks) Input: Game Data Parameters Detection (Beat tracking, phrase matching, pitch matching, harmony and key matching). Prediction (Neural nets, fuzzy logic.) Wave banks Audio Data (MIDI) Algorithmic composition/modelling Samplers, synths, tone generators Intelligent mixing engine AI Audio Engine

Why CS needs Arts (and vice versa)

Thank-you to … Further information: [email_address] www.GamesSound.com (my web site) www.algorithmic.net www.granularsynthesis.com www.audiokinetic.com www.iasig.org

Problems and Solutions in Game Audio

More Related Content

Similar to Problems and Solutions in Game Audio (20)

Recently uploaded (20)

Problems and Solutions in Game Audio

Editor's Notes