Fundamentals of Event Streams

Event Streams:

The Never-Ending Data River: Time Travel for Your Application

🎯 Challenge 1: The Bank Statement Problem

Imagine this scenario: You need to know your current bank balance, but you also want to understand how you got there.

Traditional Database Approach

Database Table:

Questions you CAN'T answer:

• ❌ How did we get to $1,542?
• ❌ What transactions happened last month?
• ❌ Can we audit the account history?
• ❌ What if we need to recalculate?

Event Stream Approach

Event Log (Immutable):

Current balance calculation:

Questions you CAN answer:

• ✅ Replay all transactions to verify
• ✅ Query any time period
• ✅ Audit trail for compliance
• ✅ Rebuild balance at any point in time
• ✅ Multiple views (by month, by category, etc.)

Pause and think: What if instead of storing the current state, you stored every event that ever happened?

The Answer: Event Streams

Event streams are immutable, append-only logs of events. They're like a video recording versus a snapshot photograph:

• ✅ Never delete events (permanent record)
• ✅ Events are in chronological order (time-ordered)
• ✅ Can replay from any point (time travel!)
• ✅ Multiple consumers read independently (parallel processing)
• ✅ Source of truth for what happened (audit trail)

Key insight: Event streams transform the question from "What is the current state?" into "What happened, in order?"

🎬 Interactive Exercise: Snapshot vs Event Stream

Database Snapshot (Current State)

Users Table:

What you know:

• Alice's current status is Active
• Her current email

What you DON'T know:

• ❌ When did she join?
• ❌ Did she ever change her email?
• ❌ Was she ever inactive?
• ❌ What was her journey?

Event Stream (Complete History)

Event Log:

Current State (computed from events):

• Name: Alice
• Email: alice@example.com (changed 2 times!)
• Status: Active (was suspended for 35 minutes!)

What you KNOW:

• ✅ Complete timeline
• ✅ All state changes
• ✅ Can answer "what happened at 10:45?"
• ✅ Can rebuild state at any timestamp
• ✅ Perfect audit trail

Real-world parallel: A database is like a photograph capturing one moment in time. An event stream is like a video recording that captures the whole story.

🏗️ Core Event Stream Concepts

1. Events (The Facts)

An event is an immutable fact about something that happened in the past.

Event structure:

Characteristics:

2. Stream (The River)

A stream is an ordered sequence of events flowing through time.

Visual representation:

Time flows →

Features:

3. Offset/Position (The Bookmark)

Each event has a position in the stream, like a line number in a book.

Stream positions:

Each consumer tracks their own position independently!

4. Consumers (The Readers)

Multiple independent readers can consume the same stream at different paces.

Multiple consumers diagram:

Each consumer:

📊 Complete Flow Example

Producer writes events:

Events are stored permanently (with configurable retention policies).

Consumers read independently:

Visual flow:

New Consumer can join anytime:

When a Recommendation Engine joins later:

Real-world parallel:

• Event = Transaction on bank statement
• Stream = Bank statement (all transactions)
• Offset = Line number you're reading
• Consumers = Different people reading the statement

🎮 Decision Game: Event Stream vs Traditional Database?

Match each use case to the best approach.

Scenarios:

A. Store user's current profile B. Track all user actions for analytics C. Shopping cart contents D. Financial transaction ledger E. Show real-time stock price F. Audit trail for compliance G. Simple CRUD application H. Event sourcing system

Options:

1. Traditional Database (current state)
2. Event Stream (complete history)

Think about: Do you need the history or just the current state?

Answers:

A. User profile → Database (1) You only need the current state, not the history.

B. User actions → Event Stream (2) Analytics needs the complete history to identify patterns.

C. Shopping cart → Database (1) Current items matter most, not the history of what was added/removed.

D. Financial ledger → Event Stream (2) Audit trails are critical; you can't lose any transactions.

E. Stock price → Database (1) + Stream (2) Store the current price in the database, maintain history in the stream.

F. Audit trail → Event Stream (2) By definition, audit trails need the complete history.

G. CRUD app → Database (1) Simple create/read/update/delete operations work best with traditional databases.

H. Event sourcing → Event Stream (2) Events ARE the source of truth in event sourcing architectures.

🚨 Common Misconception: "Event Streams Are Just Logs... Right?"

You might think:

"Event streams are just application logs with a fancy name."

The Reality:

Event streams are a first-class data source with structured business events!

Application Logs (Different Purpose):

Characteristics:

• Purpose: Debugging, troubleshooting
• Format: Human-readable text
• Structure: Unstructured or semi-structured
• Retention: Days to weeks
• Consumption: Humans, log analysis tools

Event Streams (Business Events):

Characteristics:

• Purpose: Business logic, data processing
• Format: Structured (JSON, Protobuf, Avro)
• Structure: Well-defined schema
• Retention: Months to forever
• Consumption: Services, analytics, ML models

Comparison diagram:

Real-world parallel:

• App logs = Security camera footage (for diagnostics)
• Event streams = Business transaction receipts (business data)

🎯 Key Takeaways

1. Event streams store history, not just state - They preserve the complete story of what happened.

2. Events are immutable facts - Once written, they never change, providing a reliable audit trail.

3. Multiple consumers can read independently - Each service processes events at its own pace without affecting others.

4. You can replay and rebuild - Time travel through your data by replaying events from any point.

5. Different from application logs - Event streams are structured business data, not debugging information.

6. Choose based on your needs - Use traditional databases for current state, event streams for complete history.