Harlan beverly gaming levels up networking ieee 10 2009

Video Games “Level Up” NetworkingVideo Games “Level Up” Networking
By: Harlan Tytus BeverlyBy: Harlan Tytus Beverly
Co-Founder of Bigfoot NetworksCo-Founder of Bigfoot Networks
Networking and Computing
Requirements for Video
Games: Trends & Solutions.

About HarlanAbout Harlan
Harlan Tytus Beverly
Founder of Bigfoot Networks
IEEE Member since 1995
Hardcore Gamer
Programmer (Networking)
Chip Designer/Architect
Favorite Games:
Lord of the Rings Online,
Duels.com, Plants vs.
Zombies, CS: S, BF Heroes
Beverly
• Founded Bigfoot Networks &
Designed “Killer NIC.” Raised over
$13MM in funding.
• Filed over 30 patents, including 16
about Online Game Design.
• Designed “World's first” 10-Gig
Ethernet Adapter at Intel.
• Architected & helped write Bigfoot's
LLR2 Network Protocol (x-platform
networking SDK)
• Wrote several games including
“Online” Othello for Linux

“It’s boom time for an industry now even bigger than Hollywood” – San
Francisco Chronicle, 2004
Gaming is Bigger than Hollywood

Gaming PCs,
$5B
PC SW + Sub,
$6B
Midware/Hosting,
$2B
Gaming
Accessory, $10B
Discrete Graphics,
$21B
Mobile
$6B
Console HW,
$12B
Console SW,
$11B
Gaming Market: $72B+ in 2008
Source: DFC Intelligence +
Online Gaming Explosion
Total MMOG Active Subscriptions
R
2
= 0.9791
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
14,000,000
1999 2000 2001 2002 2003 2004 2005 2006
0%
20%
40%
60%
80%
100%
2002 2003 2004 2005 2006
Online Games (% Total)
•2.7M Gaming PCs/Yr
• 39% Y/Y Growth
•Big OEMs Entering Market:
• Dell, HP, Acer, Lenovo

What is fueling gaming?
Answer:Technology & Culture.
Technology: Leap-frogs in technology are leading to ever-
more immersive (and addictive) games. Graphics are leading
the charge, networking right behind driven by broadband
penetration.
Culture: Gamers are coming of age. Nintendo generation
are in their 30s… spending power has shifted to gamers… so
has business.

An Industry of Leapfrogs
PC Shooters have advanced state of the art Graphics for 25
years.
Online games have advanced networking for 15 years.

Holy Grail of Online Games
The Holy Grail of online games is defined as:
Unlimited number of players (infinitely scalable).
With real-time & rich interaction (twitch, voice, video).
And zero Lag.

Lag is Waiting.Lag is Waiting.
Ever been stuck in a traffic jam?
Lag is defined as the feeling of frustration
while waiting for something online.
 “Over 40% of gamers have quit an
MMO because of Lag!” -2005 MW
Research
 If 40% of WoW users continued their
subscription an extra 12 months that
would be $700,000,000 more
revenue.

Lag Example VideoLag Example Video
http://www.youtube.com/watch?v=rgsxZbv0rrM

The 3 Causes of LagThe 3 Causes of Lag
PAGE
• 35%*
Game Servers (Overloaded, Costly)
• 35%*
Game Clients (Un-trusted, Multitasking)
• 30%*
The Network (Latency, Bandwidth)
*
Source: 10,000
LagMeter Submissions

Client Network Server
• Bursty Graphics (CPU/GPU
Interaction)
• Bursty Application
(CPU/Memory)
• Hard-Drive accesses
• Memory
Management/Swapping
• Background apps/tasks
• Client Bandwidth (Voice/etc.)
• Long Routes (hops)
• High Latency (variety)
• Packet Loss
• Faulty Equipment
• ISP Congestion
• Backbone congestion
• Bursty datagrams
• Per-User Calculations
• Per-Object Scoping + Message
Duplication (Exploding/Output
Processing)
• Validation of User Actions (Input
processing)
• Application: simulation, DB, etc
• Server Network Bandwidth
• Backend Interaction (Database)
• Congestion Recovery (bursts)
35% (+/-15%) 30% (+/-10%) 35% (+/-25%)
LagMete
r
• LagMeter: Free “viral” tool to measure Lags-per-
Minute (end-to-end responsiveness)
• 2007 Study: 10,000 Unique LagMeter Data
Submissions

Latency vs. ThroughputLatency vs. Throughput
(Seriously)(Seriously)
• Pipes are a ‘better analogy’ than
tubes.
• Pipes have “Diameter” and carry
water from A to B at a given “rate”.
• A 10-Gig Internet Pipe carries 10-
Gb per second of data.
• A DSL Pipe carries 2Mbps per
second of data.
• Which has “lower latency?”
PAGE

Latency vs. ThroughputLatency vs. Throughput
(Seriously) Answer(Seriously) Answer
• Which is lower latency?
• DEPENDS on the latency &
packet size (e.g. flow rate of
pipe, not diameter)!
• What if the 10Gig only
updates once per second???
(“rate”)?
• Latency = 1000ms!!!!
• What if the DSL updates at
2,000,000/second?
• Latency = 0.5ns
PAGE
.
Its like a Train w/TONS
of cargo, vs a Plane
with a smaller cargo…

Latency in GamesLatency in Games
• For Games, Latency *USUALLY* Matters MUCH more than
throughput.
• Most games use LESS than 64kbps of throughput.
• HOWEVER, games are interactive, so latency is very important,
since you want to react ASAP!
• FPS games can’t usually tolerate more than 100ms.
• MMO games can’t usually tolerate more than 500ms.
• Exception to the rule: Bandwidth Waste:
• Constrained bandwidth can CAUSE backup
buffer over-run, & packet-loss…
• Leading to Excessive Latencies due to
retransmission & deep packet queues.
PAGE

The Fight = Innovation
The Fight against Lag has lead to dozens of innovations in
networking.
Dealing with Dial-up bandwidth.
Applicable today with cellular bandwidth.
Dealing with Scalability.
Today called “the cloud”.
Dealing with crowding.
Applicable today with social networks & still a problem for games.
Dealing with “time-synchronization”
Applicable today for dozens of industries.

Dial-up Bandwidth
LAN gaming was born first. (Doom)
Dealing with Dial-up was a major
challenge for games, blocking the road to the holy grail by
limiting the number of users and making a‘simulation’
required.
Solving Dial-up meant having “imperfect” simulation on the
client… only tiny state updates could be sent.
John Carmack’s solution has become an industry norm:
Delta systems.
Ultima Online introduced a new concept:
Interest systems.

Delta Systems
*Source: Book of Hook
• With Quake3, John Carmack dropped the notion of
a reliable packet altogether … the server sends
sequenced game state updates that are delta
compressed from the last acknowledged game
state the client received.
• The client's receive logic boils down to:
PAGE
if ( newState.sequence < lastState.sequence )
{ //discard packet }
else if ( newState.sequence > lastState.sequence )
{ lastState = deltaUncompress( lastState, newState );
ackServer( lastState.sequence ); }

Interest Systems
An interest system managesWHAT relevant state data you
are sent.
This reduces bandwidth needs.
And processing time byAl.
Things “close” to Al… sent.
Things “not close” to Al… not sent.
Al
Joe
Al
Bob
Joe
“Bob + Joe interested in Al.”
“Bob moved.”

Scalability
MMOG – Massively Multiplayer Online Game
But how massive?
Build too few: epic disaster as people “wait in line” to play.
Build too many: way too expensive capital overhead/user.
Gaming Solutions:
Latency Hiding
Sharding
Virtualization
Task Based Clusters

Latency & Sharding: Solving
the Scalability Problem.
• WoW: Hide the Latency! Animations, Can or Can't
Move While Attacking, “Recharge” Rates.
– Allows ‘variable’ latencies & heavily loaded servers to
tolerate (hide) up to 500ms+ of latency!
• Ultima Online, WoW, nearly every MMO ever….
• Sharding. Create multiple copies of the “game
world” and get people to try to average themselves
out with wait queues.
PAGE

Modern Approaches to Scalability
Run it in the cloud! *or virtualization*
Can improve bandwidth & server processing problems, but…
Still problems with the persistent database.
Many game companies looking at 10-Gig & infiniband for solutions to
database & server interconnect.
Task-based game server design.
Don’t run servers based on users or location or AI, etc.
Run them based on a distributed task system (clustering).
Still problems with the persistent database & interconnect.
10-Gig/Iband.
Complexity problem exponential.
Use Instancing (see next section)

PAGE
Ultima Online: Localization
Problem (Crowding)
• One of the VERY first to
have 'localization' or
'crowding' problem.
(gamers grouping at the
bank).
Causes server processing
and bandwidth get
overloaded…

Ultima Online: Localization
Solution
• Game Design Solutions for
“Localization”/”Crowding” Problem:
– Increased locations for most common grouping
issues (e.g. more Dragons, more Banks, etc.)
– Sharding: create more shards and therefor, fewer
users will be ‘able’ to crowd, and encourage people
to use those other shards.
– Instancing: where crowding might happen, simply
make it an instance and limit the instance to a fixed
size. (see Guild Wars)
PAGE

Time Synchronization
Synchronizing the “views” of millions of clients is a daunting
challenge.
Similar to the challenge of‘data sync’ between two real-time
systems.
Games have evolved very unique solutions:
Server Authority
Latency Compensation
Interpolation
Dead Reckoning

Server Authority
Very early on it became obvious… the simplest way to
maintain data synchronization was to keep it all in one place.
One authority: the server.
The problem: latency varies by user. (see Latency
Compensation).
Other problems: (see scalability)

PAGE
CS & CS: Source:
“Twitch” Internet Gaming
• “Internet” Latencies VARY
wildly between players
(Some on dial-up, some on
LAN, some on Broadband)!
• Game Design Solution(s):
•Player Hosted Servers
•Internet Server Browser, sort
by latency!
•Moderated Rules “kicking”
players above certain Ping
(Latency)

Latency Compensation
*Source: Valve Software Wiki
• Basic Idea: Use Latency Information
about each Client & Position & Velocity
to compensate for Lag & Latency
PAGE
Example
showing “Hit
Box” of 200ms
Lagged
Shooter.

Latency Compensation
*Source: Valve Software Wiki
1. The server estimates at what time the command was
created
• Command Execution Time = Current Server
Time - Packet Round Trip-Time - Client View
Interpolation
• Note: Interpolation is Defined Next…(think view
latency caused by 'smoothing')
2. Then the server moves all other players back to
where they were at the command execution time.
3. The user command is executed and the hit is
detected correctly.
PAGE

Interpolation
*Sources: Valve & OpenTNL.org
• Basics: Interpolation is used to smoothly move an
object from where the client thinks it is to where
the server declares it to be over some short period
of time.
• Uses history of states to smooth out the
Animations on the Client…
• E.g. without it, EVERYTHING IS JERKY!!! (LAGGY!)
PAGE

Interpolation Advanced
*Sources: Valve & OpenTNL.org
• Interpolation exacerbates the time difference
between the client and the server, because the
client is spending even more time than the one-way
message time to move the object from its current
position to the known server position.
• E.g. Interpolation Delay adds to Total Latency.
• Extrapolation – Guessing of new state by velocity
(for REALLY bad Latency)
• Client-Side Prediction – Better guessing of OWN
state, based on ow user input. (e.g. doesn’t wait for
server ACK)
PAGE

Dead Reckoning
*Source: Gamasutra.com
• Interpolation, Extrapolation, Client-Prediction,
Delta System ALL TOGETHER… within ‘limits’
or Threshold of ‘rightness’. Broken Limits =
New ‘Full-State’ Message..
PAGE

New Approaches to
“the holy grail” of Lag-Free gaming.
Peer-to-Peer (security & time-synch problems abound).
Asynchronous games.
Massively single player.

WarRock – No Servers? Free-
to-Play FPS?
• Genre: “Free FPS”
• Problem: If it is free,
who pays for servers?
• Solution: P2P Game
Design, Lobbies do
Matchmaking.
• (sidenote: Cheating is
a HUGE problem in
WarRock)
PAGE
.

Duels (Challenge Games) – Fight
Other Players…asynchronously?
• Genre: Web Based
Strategy & Collectible
Persistent Game
• Problem: Challenging
Players that aren’t online?
• Solution: Asynchronous
Challenge System
(Message Center)
– Player-to-Player Latency
irrelevant.
– ‘Slow’ Web Content, just fine
since Asynchronous.
PAGE
.

Spore – User Content for Single
Player Online?
• Genre: Strategy Game,
Massively Single Player Online
• Problem: Players at different
‘stages’ of game interacting.
Bragging/Showing your User
Generated Content (UGC) big
part of fun.
• Solution: Loads Dynamic UGC
at startup (if available), and
auto-uploads UGC to
Sporopedia (Spore Wiki) for
bragging rights.
PAGE

Trackmania – International
Competition Online?
• Genre: Online Racing
Simulation
• Massively Single-player
• Problem: International
Competition in “Twitch”
Racing Game?
• Solution: Cars do NOT
collide! No/Limited User-
User Interaction. Client
based Simulation.
PAGE

Game Software Development
Networking developers are RARE and in great need in the
gaming industry.
Skills coding in Sockets and knowledge of networking &
network infrastructure are in high demand.
Tools of the trade?
Network libraries
Server software architecture
Network hardware & configuration

PAGE
Network Libraries
•FPS Style
– Licensed Engine (UT3,
Gamebryo, Source)
– Free/Semi-Free Engine (MS
XNA, Torque, others)
– Truly Free (and very good) Net
Code (Rakknet, HawkNL)
•MMO Style
– Engine: (Emergent, Hero,
Multiverse, Sun Darkstar)
•RTS or Casual Style
– MS .net, SmartFox, Unity, XNA
• GameDev.net and
Gamasutra.com are great
resources.
• Also see
http://developer.valvesoftware.com/wiki

Testing an MMO
Online game testing is 2x as hard
as offline.
What is different?
Testing for different latencies &
packet loss.
Testing different bandwidths.
Tools?
Monkey clients (fake clients doing
automated stuff).
Massive open and closed alpha &
beta programs.
Post-launch patches & bug-fixes
GALORE.

Backup
Email me or connect on Linked-in.
HarlanT. Beverly
beverly@ieee.org

Harlan beverly gaming levels up networking ieee 10 2009

More Related Content

Viewers also liked (19)

Similar to Harlan beverly gaming levels up networking ieee 10 2009 (20)

More from Harlan Beverly (7)

Recently uploaded (20)

Harlan beverly gaming levels up networking ieee 10 2009

Editor's Notes