# Flink's Dataflow model

***

The good news is that while APIs evolve, the **Dataflow Model** is the mathematical and architectural foundation of Flink. These concepts remain 100% relevant because they define how any modern, distributed stream processor handles "unbounded" data.

***

### The Dataflow Graph

At its simplest, a Flink program is a **Directed Acyclic Graph (DAG)**.

* **Data Streams:** The edges in the graph. These represent the continuous flow of records.
* **Operators:** The nodes in the graph. They perform computations (like `Map`, `Filter`, or `Window`).
* **Logical vs. Physical Graphs:** You write a logical graph (your code), which Flink optimizes into a physical graph that can be distributed across multiple worker nodes.

This represents the logical flow of data from a source through various transformations to a destination.

{% @mermaid/diagram content="graph LR
A\[(Source: Kafka)] --> B(Map: Parse JSON)
B --> C(Filter: Valid Trips)
C --> D(KeyBy: VendorID)
D --> E(Window: 5min Agg)
E --> F\[(Sink: Database)]" %}

### Parallelism and Data Partitioning

Since Flink is a distributed system, it splits streams into **stream partitions** and operators into **operator subtasks**.

* **One-to-one (Forwarding)**: Data stays in the same partition (e.g., a `Map` following a `Source`).
* **Redistributing**: Data is reshuffled based on a key or a random distribution (e.g., `keyBy`). This is where "shuffles" happen, which are expensive because they involve network communication.

This illustrates how tasks are distributed. Notice how `Map` keeps data local (One-to-one), while `KeyBy` forces a Redistribute shuffle across the network.

{% @mermaid/diagram content="graph TD
subgraph Worker\_Node\_1
Source1\[Source Subtask 1] --> Map1\[Map Subtask 1]
end

```
subgraph Worker_Node_2
Source2[Source Subtask 2] --> Map2[Map Subtask 2]
end

Map1 -. Redistribute .-> Agg1[Agg Subtask 1]
Map1 -. Redistribute .-> Agg2[Agg Subtask 2]
Map2 -. Redistribute .-> Agg1
Map2 -. Redistribute .-> Agg2" %}
```

### Windows and Triggers

Since a stream never ends, you can't "sort" or "aggregate" the whole thing. You must slice it into **Windows**.

* **Tumbling Windows:** Fixed-size, non-overlapping (e.g., every 5 minutes).
* **Sliding Windows:** Overlapping windows (e.g., every 5 minutes, starting every 1 minute).
* **Session Windows:** Grouped by periods of activity followed by a gap of silence.

This visualizes how time-based streams are sliced.

{% @mermaid/diagram content="graph TD
subgraph Tumbling\_Windows
T1\[00:00 - 00:05] --- T2\[00:05 - 00:10] --- T3\[00:10 - 00:15]
end

```
subgraph Sliding_Windows
S1[00:00 - 00:10]
S2[00:05 - 00:15]
S1 -. overlap .-> S2
end" %}
```

### Time Semantics

This is arguably the most important terminology in the book:

* **Event Time:** When the event actually happened (based on a timestamp inside the data). This is the most accurate but hardest to handle.
* **Processing Time:** The time on the clock of the machine doing the work. Easy to use, but results are inconsistent if the system slows down.
* **Watermarks:** The "clock" for Event Time. A watermark is a special marker that tells the system, "I'm reasonably sure no more data with a timestamp older than $$X$$ is coming."

This shows the relationship between the stream of events and the Watermark "clock" that follows them to handle lateness.

{% @mermaid/diagram content="sequenceDiagram
participant Stream as Data Source
participant Flink as Flink Operator

```
Note over Stream, Flink: Time moves forward
Stream->>Flink: Event (TS: 10:01)
Stream->>Flink: Event (TS: 10:03)
Stream->>Flink: [Watermark: 10:00]
Note right of Flink: "I've seen everything <br/>before 10:00. Processing..."
Stream->>Flink: Event (TS: 10:05)" %}
```

*(Note: In a real stream, the Watermark tells the operator: "Assume no more data prior to 10:00 will arrive.")*

### State and Checkpoints

Unlike "stateless" functions, Flink remembers things.

* **Keyed State:** Information stored relative to a specific key (e.g., a running total for a specific UserID).
* **Checkpoints:** Periodic, internal "snapshots" of the state. If a server dies, Flink rolls back to the last checkpoint and resumes.
* **Savepoints:** Manually triggered snapshots. These are "stop-motion" frames that allow you to update your code or migrate clusters without losing your progress.

This demonstrates how Flink takes a "snapshot" of the current operator memory and stores it in durable storage (like S3 or HDFS) for fault tolerance.

{% @mermaid/diagram content="graph LR
subgraph Flink\_Cluster
Op\[Operator Task] -- 1. Internal State --> Mem(Memory)
end

```
Mem -- 2. Snapshot/Checkpoint --> DS[(Distributed Storage)]
DS -- 3. Recovery --> Mem" %}
```

***

#### Key Terminology Quick-Ref

| **Term**           | **Definition**                                                                           |
| ------------------ | ---------------------------------------------------------------------------------------- |
| **Backpressure**   | When a downstream operator is slower than an upstream one, causing the flow to throttle. |
| **Lateness**       | Records that arrive after the Watermark has already passed their timestamp.              |
| **Operator State** | State that is bound to a parallel operator instance rather than a specific key.          |

***