Cloud Hosting and Dedicated Servers - Multiplayer Game Programing

Multiplayer Game Programming

1. Multiplayer Game Programming
Chapter 13
Cloud Hosting
Dedicated Servers

2. Chapter 13
Objectives
Hosting Pros and Cons
–Is hosting the right choice for your game?
Tools for backend development
–REST, JSON, Node.JS
Terminology
–Game Instances, Processes and Virtual Machines
Managing Everything
–Local Server Manager
–Virtual Machine Manager

3. To Host or Not To Host
Should you host dedicated servers?
–Originally, only option for AAA studio with big
upfront budgets
Cloud removes upfront hardware costs to hosting
dedicated servers
Just because you can, should you?

4. To Host or Not To Host, Cont’d
Cons
–Complexity
•must manage fleet of servers!
–Reliance on third party
•Cloud hosts go down sometimes- even Amazon,
Google and Microsoft
–Unexpected hardware changes
•Hosts must meet min spec, but there can be lots of
variety above that spec
–Loss of player ownership
•Evangelist players take pride in running servers

5. To Host or Not To Host, Cont’d
Pros:
–Reliable, Scalable, High bandwidth servers
•Not dependent on players to host games
•Cloud host IT makes sure the data keeps flowing
–Cheat prevention
•Players have no access to authoritative code
–Reasonable copy protection
•No need for intrusive DRM on client

6. Tools
C++ and custom protocols good for fast, low level
client and server, but not for high level server
management
–No high level memory management
–No built in async request handling
–No built in high level request exchange protocol
–No built in human readable, debuggable data format

7. REST
 Representational State Transfer
–Self contained request format
–Text Base
•Human readable
•Easily Debuggable
– Built on top of HTTP
•GET
–Retrieve a resource
•POST
–Create / Edit a new resource
•DELETE
–Delete a resource
•PUT
–Put a resource directly at a URI

8. JSON
JavaScript Object Notation
–Text based, human readable
–Compilable JavaScript
–Objects are hashes of primitive data types
•string
•number
•bool
•array
•object
•undefined
•null

9. JSON
{
“x”: 3.14,
“name”: “McK”,
“isReady”: true,
“favoriteNumbers”: [ 1, 1, 2, 3, 5 ],
“visitationCounts”:
{
“Los Angeles”: 33,
“New York”: 12
}
}

10. Node.JS
Open Source JavaScript platform
Runs Googles V8 Engine
–Now semi-ES6 compatible
Single threaded asynchronous processing model
–Non-preemptive multitasking
•Disk I/O, Net requests, etc.
–Event driven
•Ticks, like a game!
Express.JS
–Popular Web / REST Server

11. What Does “Server” Mean?
Overloaded!
–Server Game Instance
–Game Server Process
–Game Server Machine

12. Server Game Instance
Simulates a game world shared by its players
–A 16 player FPS battle
–A 5v5 LoL game in Summoner’s Rift
–A 4 player Battle Mode game in Mario Kart

13. Game Server Process
Your game, as far as the OS is concerned
–Hosts one or more server game instances
–Multiple games instances per process?
•Pros
–Share large immutable resources
»Navigation grid, collision geometry, etc
–Fine grain division of CPU time
•Cons
–No mutable singletons
–No protected memory
»Crash in one instance can corrupt / bring down all in
process
–Harder to test

14. Game Server Machine
Each machine has one Operating System
–But one or more game server processes!
–Multiple processes per machine?
•Pros
–Processes live in protected memory- Safe!
–Fewer OS resources required per Server Game
Instance
•Cons
–Coarse control over multithreading
»One process could potentially hog resources

15. Hardware
In the cloud, machines are typically Virtual Machines
–Virtual Machine Image (VMI) encapsulates OS and
game process executable
–Cloud host can move virtual machine to different
physical machines at will
•Rapidly handle hardware failure
•Rapidly spin up new servers as requested
•Utilize hardware to fullest extent
–Multiple VMs per physical machine

16. Local Server Process Manager
Need to administer multiple game processes per server
machine
–Start up new process when necessary
•With desired configuration ( Team Deathmatch! )
–Monitor process health
•Sometimes processes crash or freeze
–Report back to overall platform
Maybe could use tools built into OS
–Not cross platform
–Not fully customizable
Custom solution built in Node.JS!
–Cross platform and full customizable!

17. Local Server Process Manager
REST
REST Endpoints
–GET /processes
•Gets a list of currently running processes, with unique ids
–POST /processes
•If not running maximum process
–spawn new game process using
childProcess.spawn
–Save unique id of process in running list
»When process closes, automatically remove from
list
»Read configuration data from body of request and
pass to process

18. Local Server Process Manager
REST Cont’d
REST Endpoints
–POST /processes/:processUUID/kill
•Attempt to kill the process with given id
–POST /processes/:processUUID/heartbeat
–Register that given process is still alive
»Not frozen
–What good is that?
»Monitoring!

19. Process Monitoring
Periodically run through the process list and check the
last time a heartbeat was received
–If not received within threshold, kill process
•Why important?
•Make room to run another process!
•Also can notify devops to investigate
Game server process should hit heartbeat endpoint
periodically to let LSPM know it’s alive
–Microsoft Open Source C++ REST SDK
–Or just use TCP socket ( SDK usually preferable )

20. Process Monitoring Sample Code
function checkHeartbeats()
{
var processesToKill = [], processUUID, process, heartbeatAge;
var time = getUTCSecondsSince1970();
for( processUUID in gProcesses ) {
process = gProcesses[ processUUID ];
heartbeatAge = time - process.lastHeartbeat;
if( heartbeatAge > gMaxStartingHeartbeatAge ||
( heartbeatAge > gMaxRunningHeartbeatAge
&& process.state !== 'starting' ) ) {
console.log( "Process " + processUUID + " timeout!" );
processesToKill.push( process.child );
}
}
processesToKill.forEach( function(toKill) {toKill.kill();} );
}
setInterval( checkHeartbeats, gHeartbeatCheckPeriod );

21. Virtual Machine Manager
Need to administer multiple machines!
–Find a VM with a space for a new process
–If no VM with space for a process, ask cloud host to
spin up a new VM
–Monitor VM and LSPM health!
•Don’t want to leak VMs!
–LSPM could crash / freeze and you might lost
track of processes that exit unexpectedly
–Spin down VMs when not in use
•Save money!

22. Virtual Machine Manager REST
Endpoints
–POST /processes
•Find an LSPM with room for a new process and
spin spawn it
–Posts to LSPM’s /processes endpoint
–If no LSPM with space available
»Provision VM from cloud provider
»LSPM starts at boot
»Then request process
•Tracks any new process or LSPM in global lists
•Beware of race conditions from multiple incoming
requests at once!

23. Provisioning a VM
Different for each cloud provider
Typically, Node.JS package for cloud provider
–VMI must be stored with cloud provider already
–Can store multiple VMIs
•Request VMI and min system specs through
cloud provider API

24. Virtual Machine Manager REST,
Cont’d
Endpoints
–POST /vms/:vmUUID/heartbeat
•Register VM and associated LSPM is still alive
•LSPM must call periodically
–After checking for heartbeats from processes?
–Can also send update on number of processes running
–Can send heartbeat whenever number of processes
change
»Crash, game over, spin up request complete
•Heartbeat indicating zero processes running on a VM can
trigger VMM to request cloud provider to spin down VM
–Should wait a little to make sure no new process
request incoming

25. Virtual Machine Monitoring
Check for heartbeat every period ( 60 seconds? )
–Similar to LSPM
–Beware race conditions now!
•VM might already be shutting down
–Starting up
–Starting a new process
•Need to track every in progress request

26. Matchmaking and Virtual
Machine Management
Keep them separate!
–Virtual Machine Manager is only responsible for
spinning up machines
–Matchmaking is only responsible for finding
matches
•If Matchmaking service cannot find an available
match, it asks VMM to spin up a server
•Separation of Concerns!
•Allows integration with any third party service
–Steam, PSN, Xbox Live, Game Center, etc.