Apache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization

Apache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization | Databricks Blog Skip to main content

Summary

Explore how Apache Spark™ Real-Time Mode enables real-time gaming sessionization for millions of active device sessions

Learn how transformWithState timers power proactive, timer-driven heartbeats — generating output on a schedule, independent of incoming data

See how Real-Time Mode paired with transformWithState replaces custom in-house applications and external streaming engines — delivering sub-second precision for both input processing and timer driven output.

In the gaming industry, every millisecond counts. To drive in-game personalization, fuel recommendation engines, and make dynamic content scheduling decisions, platforms must process session data for millions of global players with sub-second latency. Today, meeting these ultra-low latency requirements no longer requires a disjointed architecture with multiple engines. In this blog, we explore a real-world implementation of Apache Spark Real-Time Mode . By leveraging the new transformWithState operator for complex stateful logic, we demonstrate how Spark delivers end-to-end millisecond performance. Discover how your team can accelerate development and build mission-critical operational applications using the familiar Structured Streaming ecosystem. Use Case Overview From Game Start to Game End - Why Session Tracking Matters For gaming platforms, knowing which devices are active and for how long isn't just an infrastructure concern — it drives the business. Real-time session data powers personalized in-game experiences, fuels recommendation engines, informs content scheduling decisions, and provides device health signals across millions of consoles and PCs. Operations teams use it to enforce parental controls and detect abnormal session patterns. Session Event Fundamentals Session events from both consoles and PCs flow into Kafka topics. Each event carries a device ID and a session ID. The device ID identifies the console or PC; the session ID identifies the gaming session. Only one session can be active per device at any time. The pipeline handles four scenarios: Session Start (GameStart): A start event arrives. The pipeline stores the session ID and start time, emits a SessionActive event, and registers a 30-second processing-time timer. If another session was already active for that device, it ends the old one first. Session Heartbeat (Active): The timer fires every 30 seconds. The pipeline calculates now - start_time, emits a SessionActive heartbeat with the current duration, and re-registers the timer. Session End (GameEnd): An end event arrives matching the active session. The pipeline emits a SessionEnd with the final duration and clears the state. Session Timeout (GameSessionTimeout): The timer fires and the calculated duration exceeds a configurable maximum. Instead of emitting a heartbeat, the pipeline emits a SessionEnd with a timeout reason and cleans up the state.

Why Spark with Real-Time Mode is a game changer Spark Structured Streaming in micro-batch mode can handle stateful sessionization, but when the use case demands sub-second precision for both input processing and timer-driven output, micro-batch falls short. In the past, that gap pushed teams toward managing an additional specialized engines or building custom solutions. With Apache Flink: State management and timers can be implemented, but adopting Flink means adopting an entire parallel ecosystem: a separate cluster, state backend, deployment model, monitoring stack, and codebase, all alongside the Databricks Platform. The result is infrastructure fragmentation, operational complexity, and the cost of operating and staffing a second streaming engine. With custom in-house solutions: Some teams build their own sessionization service — for example, an Akka-based actor system where each device gets an actor that manages session state, timers, and heartbeat emission. These carry the same infrastructure and operational overhead as Flink, with an additional challenge: they don't scale. Distributing millions of stateful actors across nodes is something you have to engineer yourself. These systems work initially, but over time end up in maintenance mode — stable enough to run, but not easily extendable. Today, Real-Time Mode closes this gap for customers — delivering sub-second precision with the same Spark APIs teams already use, all in a single unified engine. Real-Time Mode with transformWithState transformWithState is a next-generation operator in Spark Structured Streaming that makes complex stateful processing flexible and scalable. Key features include object-oriented state management, composite data types, timer-driven logic, automatic TTL support, and schema evolution. Combined with Real-Time Mode, it delivers sub-second precision for both input processing and timer-driven output. The gaming sessionization use case demands two things: Reactive processing: handling session starts and ends as they arrive. Proactive output: producing a heartbeat for every active session on a schedule, independent of incoming data

transformWithState delivers both in a single StatefulProcessor class with two dedicated methods. handleInputRows() reacts to incoming Kafka events — processing session starts and session ends, maintaining sessionization state as events arrive. handleExpiredTimer() handles everything that happens in between — firing to produce proactive output like heartbeats and timeouts, independent of whether any new data has arrived. How It Works: Building a Real-Time Gaming Sessionization Pipeline Pipeline Architecture Overview Event Ingestion: Session events (starts and ends) from consoles and PCs arrive on Kafka topics. Each event is parsed, and a deviceId is derived from the device-specific identifier. Stateful Grouping: The stream is grouped by deviceId — ensuring all events for a given device are routed to the same stateful processor instance. Process: transformWithState applies the Sessionization processor, which uses a MapState keyed by session ID to track the active session per device. When a session start arrives, handleInputRows() stores the session state, emits a SessionActive event, and registers the first 30-second timer. From that point on, handleExpiredTimer() takes over — emitting heartbeats every 30 seconds and checking for timeouts. When a session end event arrives, handleInputRows()...