# What is IaC?

***

## **Infrastructure as Code (IaC): A Deep but Practical Overview**

Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure using machine-readable configuration files rather than manual processes or GUI consoles. For modern data engineering and cloud workflows, IaC is the backbone that makes cloud resources predictable, repeatable, and automatable.

***

### **Why IaC Exists: Historical Context**

IaC stems from configuration management ideas from the 1970s–1980s (think **CFEngine → Puppet/Chef → Terraform/CloudFormation**). The evolution is:

1. **Early Configuration Management (CM)**
   * Tools like **RCS**, **CFEngine**, later **Puppet** and **Chef** automated software installs and OS configuration.
   * Still required someone to provision VMs/servers manually.
2. **Cloud Era**
   * Clouds (AWS/GCP/Azure) introduced highly programmable APIs.
   * This shifted the focus from configuring machines → **creating infrastructure itself programmatically**.
3. **Modern IaC**
   * Tools like **Terraform**, **CloudFormation**, **Pulumi**, **Crossplane**, and **CDK** allow teams to create entire systems (VPCs, clusters, IAM, pipelines) via code.

***

### **What IaC Actually Does**

IaC formalizes infrastructure the same way source code formalizes application logic.

**IaC = Desired State of Infrastructure + Automated Engine to Reach That State**

#### It delivers:

1. **Automation**\
   Everything — VPCs, IAM, databases, compute clusters, data pipelines — is created and updated automatically.
2. **Reproducibility**\
   Same config file → same environment.\
   Production and staging no longer drift apart.
3. **Version Control**\
   IaC files go into Git.\
   You get:
   * history
   * diffs
   * pull requests
   * code reviews
   * rollbacks
4. **Idempotence**\
   Applying the same IaC code multiple times yields the same result.
5. **Scalability for Data Engineering**\
   Especially helpful for:
   * spinning up EMR/Dataproc clusters
   * creating S3/GCS buckets
   * creating IAM roles/policies
   * infrastructure for Kafka, Kinesis, Redshift, Snowflake loaders
   * serverless data pipelines

***

### **Tooling Landscape (with quick mental models)**

#### **1) Terraform**

*Your notes align here: the lecture stresses Terraform using HCL.*

* **Cloud-agnostic** (“multi-cloud, multi-provider”)
* Uses **HCL (declarative)**
* Manages lifecycle using **state files** (local or remote)
* Has rich provider ecosystem (AWS, DBT, GitHub, Kafka, Snowflake, Datadog…)
* Best choice for multi-cloud or data engineering platforms needing broad integration

#### **2) AWS CloudFormation**

* **AWS-native** equivalent to Terraform
* Templates written in **YAML/JSON**
* Great integration with AWS — full support before Terraform gets updates
* No state file (AWS manages state internally)

Generally:

* Use CloudFormation if you're **fully AWS**
* Use Terraform if you're **multi-cloud** or want stronger ergonomics

#### **3) Pulumi**

* IaC using general-purpose languages (Python/TS/Go/C#) → **imperative IaC**
* More dynamic than HCL/YAML
* Good for teams that prefer full programming power

#### **4) AWS CDK / CDKTF**

* “IaC as real code” — similar to Pulumi
* CDK for AWS, CDKTF for Terraform providers

#### **5) Crossplane**

* Kubernetes-native IaC
* Infrastructure resources are Kubernetes CRDs
* Great for platform engineering

***

### **Declarative vs Imperative IaC**

| Approach        | Description                                                    | Typical Tools                                   | Pros                                  | Cons                                                      |
| --------------- | -------------------------------------------------------------- | ----------------------------------------------- | ------------------------------------- | --------------------------------------------------------- |
| **Declarative** | You define the desired end state; engine figures out the steps | Terraform, CloudFormation, Kubernetes manifests | Idempotent, concise, predictable      | Harder to express loops, conditionals, programmatic logic |
| **Imperative**  | You write commands telling HOW to get to that state            | Bash, Ansible, CDK (hybrid)                     | Very flexible, good for complex logic | Higher chance of divergent state; less repeatable         |

Why IaC favors **declarative**?

* Cloud APIs can be flaky
* Imperative scripts risk double-creating, overwriting, or skipping resources
* Declarative engines maintain **execution plans** and **state**

Terraform plan example:

```
+ aws_s3_bucket.my_bucket (create)
~ aws_iam_role.pipeline_role (update)
- aws_lambda.function (delete)
```

This makes infrastructure changes auditable.

***

### **IaC Workflow in Real Teams**

Here’s how IaC is used day-to-day:

1. Developer writes or updates Terraform code
2. Push to Git → pull request
3. CI runs:
   * `terraform fmt`
   * `terraform validate`
   * `terraform plan`
   * post plan diff as comment in PR
4. Reviewer approves
5. CI runs `terraform apply`
6. Infrastructure updates automatically
7. Results stored in Terraform state

For CloudFormation, you use Change Sets and CI pipelines similarly.

***

### **IaC for Data Engineering Infrastructure**

IaC becomes critical when building data platforms because pipelines always depend on infrastructure:

* **Storage**
  * S3 buckets, lifecycle rules, encryption
* **Compute**
  * EMR, Dataproc, Databricks clusters
  * Serverless compute like Lambda/Cloud Functions
* **Networking**
  * VPCs, subnets optimized for clusters
  * Private connectivity for databases/pipelines
* **Data Streaming**
  * Kafka topics, schemas, ACLs
  * Kinesis streams
* **Data Warehouses**
  * Redshift clusters
  * BigQuery datasets
  * Snowflake warehouses
* **Automation**
  * EventBridge rules
  * Step Functions/Workflows
* **Secrets / IAM**
  * Principals for ETL jobs
  * KMS keys
  * Vault

Most large teams use Terraform to orchestrate **entire data platforms** end-to-end.

***

### **IaC Anti-Patterns and Pitfalls**

Important things teams learn the hard way:

1. **Storing state locally**\
   → use remote backends (S3+DynamoDB, GCS, Terraform Cloud)
2. **Mixing environments in the same state file**\
   → separate states for `dev`, `prod`, etc.
3. **Letting Terraform manage rarely-changing core infra + rapidly changing pipelines**\
   → split project into modules and workspaces
4. **Inline IAM policies**\
   → use reusable, tested policies or modules
5. **Terraform destroys important resources unintentionally**\
   → use `prevent_destroy` and resource lifecycle policies

***

### Final notes

Think of IaC as **GitOps for your cloud infrastructure**:

* Declare your infrastructure like you declare code
* Store it in Git
* Apply via automated pipelines
* Treat infrastructure changes like code changes
* Achieve consistency and repeatability across environments

IaC is not just about automation — it’s about bringing **software engineering principles** to your cloud environment.

***

### **When NOT to Use IaC**

* Extremely temporary testing resources
* Manual one-off debugging
* Interactive work in early prototyping
* When platform teams already abstract IaC (e.g., a platform portal that creates clusters for you)

***

***