System Design: Online Game Server

Requirements

Functional Requirements:

• Host real-time multiplayer game sessions with 2-64 players per session
• Authoritative server-side game loop running at configurable tick rates (20-64 Hz)
• State synchronization: clients receive world state updates; server processes and validates player inputs
• Session lifecycle management: create, join, spectate, pause, resume, and end game sessions
• Lag compensation: server rewinds game state to reconcile late-arriving player inputs
• Spectator mode: observers receive delayed (5-second) world state stream without influencing the game

Non-Functional Requirements:

• Maximum 50ms server-to-client update latency for players in the same region
• Support 100,000 concurrent game sessions with average 10 players per session (1M concurrent players)
• Game server crash must not lose more than 1 game tick of state (100ms at 10 Hz)
• Server tick rate stable under load — no tick rate degradation above 80% CPU utilization
• Anti-cheat: server validates all game state transitions; no client-authoritative state

Scale Estimation

100k concurrent sessions × 10 players × game state updates at 20 Hz = 20M state broadcast messages/second. Each state packet is ~500 bytes (compressed delta). That's 10 Gbps of outbound game data. At 64 players/session and 64 Hz (AAA FPS game), a single session generates 64 × 64 Hz × 500 bytes = 2 MB/second of broadcast traffic. A game server node running 100 such sessions uses 200 MB/second network — requiring dedicated 10 Gbps NICs. Input packets: 1M players × 64 Hz = 64M input packets/second inbound, each ~50 bytes = 3.2 Gbps inbound traffic.

High-Level Architecture

The infrastructure separates the matchmaking/lobby plane (stateless, horizontally scaled web services) from the game server plane (stateful, session-bound dedicated server processes). The game server plane uses a fleet of bare-metal or dedicated VM instances (game servers need predictable CPU without virtualization overhead for consistent tick rates). A game server allocation service (Agones on Kubernetes, or a custom allocator) manages the fleet: provisioning idle server processes, matching players to allocated sessions, and reclaiming completed sessions.

Each game session runs as an independent OS process (or a lightweight VM on a shared host) with a fixed CPU affinity to avoid scheduler jitter. The game loop runs at a fixed tick rate using a high-resolution timer. Per tick: (1) dequeue all player inputs received since last tick, (2) apply lag-compensated input validation, (3) advance game simulation by one tick, (4) compute state delta versus last broadcast state, (5) broadcast compressed delta to all players via UDP (or WebSocket for browser-based games). State persistence: at each tick, the authoritative state snapshot is written to a local RocksDB instance (for crash recovery) and asynchronously replicated to a Redis cluster (for cross-node failover).

A session gateway sits in front of the game server fleet, acting as a stateful connection proxy. Players connect to the gateway via WebSocket/UDP; the gateway routes packets to the correct game server process based on session routing table stored in Redis. This indirection allows game server processes to be replaced (for updates or crash recovery) without dropping player connections.

Core Components

Authoritative Game Loop

The game loop is the core of server-side game logic. It runs on a dedicated OS thread with real-time scheduling priority (SCHED_FIFO on Linux) to prevent preemption during tick processing. The loop measures actual tick duration and emits a tick_overrun metric if processing exceeds the tick budget (50ms at 20 Hz). Input processing uses a per-player input queue sorted by client timestamp; inputs are applied in order up to the current server tick time. The simulation step advances physics, AI, collision detection, and game rule checks. Only the state delta (changed entities vs. last broadcast) is serialized and broadcast, reducing bandwidth by 80% versus full-state broadcast for most ticks.

Lag Compensation System

Lag compensation allows the server to validate player actions (e.g., a shot) against the game state as the shooting player experienced it, accounting for their network latency. The server maintains a circular buffer of the last 1 second of world state snapshots (20 snapshots at 20 Hz, ~1 MB per session). When a player fires a shot, their input packet includes a client timestamp. The server computes the player's estimated RTT (from ping measurements), rewinds the world state to the tick corresponding to (current_tick - player_RTT), performs hit detection in that rewound state, and applies the result to the current state. This prevents players on high-latency connections from having a severe disadvantage but caps rewind depth at 500ms to prevent abuse.

Session Recovery Service

Game server crashes are rare but must be handled gracefully. Each tick's authoritative state is written asynchronously to Redis with a 5-tick trailing write (buffered for throughput). On crash, the session recovery service detects the dead process via missed heartbeat, reads the latest state snapshot from Redis, spins up a replacement game server process on the same or a different host, restores state, and broadcasts a "reconnect" signal to all players. Players are pre-configured with the reconnect flow — the game client automatically reconnects to the session gateway within 2 seconds of detecting a connection drop. Maximum state loss: 5 ticks × 50ms = 250ms of gameplay.

Database Design

Redis: session:{session_id}:state (binary blob of latest game state snapshot), session:{session_id}:players (set of player_ids with connection metadata), session:{session_id}:tick (current tick number), server:{server_id}:health (TTL-based heartbeat key — absence indicates crash). PostgreSQL (transactional data): game_sessions (session_id, game_mode, created_at, ended_at, result_json, player_ids[]), player_stats_aggregate (player_id, game_mode, total_sessions, wins, losses, avg_score). ClickHouse: match_events (session_id, tick, player_id, event_type, payload_json) — detailed match history for anti-cheat analysis and replay generation.

API Design

• POST /sessions/allocate — body: {game_mode, player_ids[], region}, allocates a game server, returns {session_id, server_endpoint, token}; called by matchmaking service
• GET /sessions/{session_id}/status — returns session state: active/waiting/ended, current player list, tick count
• WebSocket /game/{session_id}?token={t} — bidirectional WebSocket for player connection; client sends input packets, server sends state delta packets
• POST /sessions/{session_id}/spectate — body: {observer_id}, adds observer to delayed state stream (5-second buffer)

Scaling & Bottlenecks

CPU is the primary bottleneck for game server scaling — physics simulation and collision detection are CPU-intensive and cannot easily be distributed across cores for a single session. Optimize the simulation loop with spatial indexing (BVH trees for collision detection) and limit simulation complexity per session (entity count caps, LOD for distant objects). Vertical scaling (large CPU instances) is more effective than horizontal scaling for individual session performance. Fleet scaling is horizontal: more sessions on more server instances.

Network bandwidth is a secondary bottleneck. At 2 MB/second per high-intensity session, a 100-session server node needs 200 MB/second (1.6 Gbps) outbound. Use kernel bypass networking (DPDK or io_uring) for game server processes that send UDP packets at high rates — standard socket calls at 64 packets/second/player saturate a single core with syscall overhead at 64 players. State delta compression (using a domain-specific binary protocol, not JSON) reduces bandwidth by 5-10× compared to text serialization.

Key Trade-offs

• UDP vs. WebSocket: UDP provides lower latency and native packet loss handling for game inputs but is not available in browsers; WebSocket over TCP adds head-of-line blocking for game state updates. For browser games, use WebTransport (QUIC-based, no HOL blocking) where available, falling back to WebSocket.
• Authoritative server vs. client-side prediction: Client-side prediction (client runs game logic locally and reconciles with server) dramatically improves perceived responsiveness but complicates anti-cheat (clients can modify local simulation). Full server authority with lag compensation is the standard for competitive games.
• Dedicated server processes vs. actor-based multi-tenancy: Dedicated OS processes per session provide isolation (a crash doesn't affect other sessions) but have higher overhead; running all sessions in one process with actor-based concurrency is more efficient but a bug in one session can corrupt others.
• Bare metal vs. VMs for game servers: VMs introduce hypervisor jitter (~1ms periodic pauses) that causes tick rate inconsistency; bare metal eliminates this but slows provisioning time from seconds to minutes — use a warm standby pool of pre-provisioned bare metal instances.