Apache NIFI

Data integration tool

---

https://www.youtube.com/results?search_query=Apache+NiFi+101%3A+Introduction+and+Best+Practices+-+Timothy+Spannwww.youtube.com

Steven KoonYouTube

GitHub - jfrazee/awesome-nifi: A list of useful Apache NiFi resources, processor bundles and toolsGitHub

---

---

Where does Apache NiFi's functionality fit in a data pipeline?

Image source

Based on the diagram you provided, Apache NiFi is primarily an Ingestion and "Data Logistics" tool. It doesn't fit neatly into just one box; instead, it spans across the Ingestion and Processing layers, acting as a "universal plumber."

Here is exactly where NiFi fits into that AWS diagram:

1. Primary Role: "Stream Ingestion" (The Collector)

NiFi replaces/augments: Kinesis Agent, AWS SDK, and MSK Connect.

• How it works: NiFi is often placed outside the cloud (on-premise) or at the very edge of the architecture. It connects to the raw sources (IoT Sensors, Enterprise Apps, Logs), collects the data, and reliably pushes it into AWS.

• The Scenario: If you have a legacy on-premise database and weird log files on a factory server, you use NiFi to grab that data and push it into Kinesis Data Streams or S3.

• Why: It is much more powerful than the Kinesis Agent. It can handle backpressure, retry logic, and encryption before the data even touches AWS.

2. Secondary Role: "Stream Processing" (The Router)

NiFi overlaps with: AWS Lambda and simple tasks in Kinesis Data Analytics.

• How it works: NiFi is excellent at "Event-at-a-time" processing.

  • Filtering: "Drop this record if the value is null."

  • Routing: "If type=error, send to S3; if type=transaction, send to Kinesis."

  • Transformation: "Convert this XML to JSON."

• Limitations: NiFi is NOT a heavy stream processor like Apache Flink (Kinesis Data Analytics) or Spark (EMR). It struggles with complex windowing (e.g., "Calculate the average over the last 15 minutes"). For complex math, you still need Flink/Spark.

3. What NiFi does NOT replace

• Stream Storage: NiFi is not a buffer. It does not replace Amazon Kinesis Data Streams or MSK. It feeds them.

• Heavy Compute: It does not replace Amazon EMR or AWS Glue for massive batch transformations.

---

The "NiFi vs. AWS" Mental Model

If you were to draw NiFi on this diagram, it would likely sit on the far left, wrapping around the "Sources" and "Ingestion" columns.

AWS Component (Diagram)	Apache NiFi Equivalent Role
Kinesis Agent / SDK	Core Use Case. NiFi runs on the source servers to collect data.
Simple Lambda Functions	Strong Overlap. NiFi can clean/format data without writing code.
Kinesis Data Firehose	Partial Overlap. NiFi can write directly to S3/Redshift, effectively bypassing Firehose if you want more control.
Kinesis Data Streams	No Overlap. NiFi is the Producer that writes into the Stream.

In summary: In this architecture, you would typically use Apache NiFi to replace the complexity of writing custom Python scripts (SDK) to get data into the pipeline. Once the data is in Kinesis/MSK, the rest of the AWS diagram stays the same.

---