search

Considerations for choosing infrastructure location

hashtag
Part 1: Infrastructure Options

Choosing where to host technology is an existential decision for modern organizations, balancing the risk of being too slow (on-premises) against the risk of catastrophic costs (poor cloud management).

1. On-Premises (The Traditional Model)

  • Definition: The company owns the hardware or leases colocation space. They are responsible for all maintenance, upgrades, and failures.

  • The "Peak Load" Problem: You must buy enough hardware to handle your busiest day (e.g., Black Friday). This leads to paying for idle capacity during the rest of the year.

  • Pros: Established operational practices; total control.

  • Cons: Hard to scale quickly; requires significant upfront capital; operationally heavy.

2. The Cloud (The Agile Model)

  • Definition: Renting hardware and services (AWS, Azure, GCP).

  • Hierarchy of Abstraction:

    • IaaS (Infrastructure as a Service): Renting raw VMs/disks.

    • PaaS (Platform as a Service): Managed services (e.g., BigQuery, Kinesis) that handle the plumbing for you.

    • SaaS (Software as a Service): Fully abstracted tools (e.g., Snowflake, Fivetran).

  • Pros: Dynamic scaling (scale up for peaks, down for lulls); speed to market for startups; shift from CapEx (buying) to OpEx (renting).

  • Cons: Complexity of pricing; potential for "horrifying bills" if unoptimized.

3. Hybrid Cloud (The Pragmatic Middle Ground)

  • Definition: Running some workloads on-premises and others in the cloud.

  • The "Analytics Pattern": A common, efficient pattern is keeping core applications on-premises but pushing event data to the cloud for heavy analytics. This minimizes egress costs because data flows in (free) but rarely flows out (expensive).

4. Multicloud (The "Best of Breed" Model)

  • Definition: Using multiple public clouds simultaneously (e.g., AWS for compute, GCP for BigQuery).

  • Pros: Access to specific best-in-class tools; serving customers near their own cloud regions; redundancy.

  • Cons: "Diabolically complicated" networking; security challenges; talent fragmentation.

hashtag
Part 2: Key Considerations & Warnings

There are several specific concepts that Data Engineers must understand to avoid failure:

1. The "Curse of Familiarity" & Lift-and-Shift.

Many companies migrate to the cloud by simply copying their on-prem setup (Lift-and-Shift). This works for speed but is a financial disaster in the long run. Running a cloud server 24/7 is usually more expensive than owning it. To save money in the cloud, you must use autoscaling (turning things off when not in use) or spot instances.

2. Cloud Economics = Financial Derivatives.

Cloud providers optimize by slicing up hardware risks. For example, "Archival Storage" is cheap because the provider is betting you won't need to read it often. They sell you cheap storage but charge massive fees for retrieval (IOPs). You are trading cost for accessibility.

3. Data Gravity Data has mass.

Once you move massive datasets into a cloud provider, it exerts a "gravitational pull" on applications and services to move there too. Moving data out (Egress) is intentionally priced high by vendors to lock you in.

4. The "Escape Plan".

It is not advised to have complex multicloud setups unless necessary. Stick to one cloud for simplicity, but maintain an "escape plan" (mental flexibility and architectural designs) in case you need to migrate in the future.

hashtag
Part 3: Additions & Analysis (From a Data Engineering Perspective)

The text covers the high-level architecture well. Here are three additional technical considerations you should keep in mind as you learn Data Engineering:

1. Data Sovereignty and Compliance (GDPR/CCPA)

The text mentions location in terms of "Cloud vs. On-Prem," but physical geography is equally critical.

  • The Constraint: Laws like GDPR (Europe) often dictate that user data cannot leave the physical borders of the region.

  • The DE Impact: If you choose AWS, you cannot just pick us-east-1 (Virginia) if your customers are in Germany. You may be forced into a Multi-Region architecture purely for legal compliance, not technical reasons.

2. Latency and "Speed of Light" Limitations

The text mentions "Data Gravity" regarding cost, but it also applies to performance.

  • The Constraint: If your application runs on AWS us-east but your database is on Azure us-west, the network latency will kill your application's performance.

  • The DE Impact: You must colocate compute and storage. If you use Snowflake (SaaS), you must select the specific cloud region that matches your upstream data sources to ensure high-throughput ETL pipelines.

3. Talent Pool Availability

When choosing a cloud provider, you are also choosing a labor market.

  • The Constraint: AWS has the largest market share and the longest history; therefore, it is statistically easier to hire an engineer who knows AWS than one who specializes in Oracle Cloud or Alibaba Cloud.

  • The DE Impact: If you build a complex architecture on a niche cloud provider, you might struggle to find teammates to help you maintain it later.

hashtag
Summary Recommendation

If you are just starting your projects or a startup idea: Keep it simple. Pick one major cloud provider (AWS or GCP are usually the standard for data engineering due to Redshift and BigQuery), use managed services (PaaS/SaaS) to avoid maintenance headaches, and only optimize costs or move to multicloud once you have a working product that is too expensive to run.

PreviousMinIONextDocker, Docker Compose, and Docker Swarm

Last updated