# Types of storage *** ### **The Physical Foundation: Disk Blocks** Beneath all these abstractions lies the physical storage medium, organized into disk blocks (also called sectors or physical blocks). These represent the actual hardware-level units that the storage device can read or write in a single operation. Modern hard drives typically use 4KB physical blocks, while SSDs may use different page sizes (often 4KB, 8KB, or 16KB). These physical blocks represent the atomic unit of I/O at the hardware level—the device cannot read or write anything smaller without retrieving the entire block. **The Abstraction Hierarchy** The relationship between these levels forms a layered architecture: 1. **Physical disk blocks**: The hardware's native addressable units 2. **Block storage**: Logical blocks exposed to the operating system (may align with or abstract physical blocks) 3. **File systems**: Built on block storage to provide file and directory abstractions 4. **Object storage**: Often built on file systems or block storage, adding its own object-oriented layer Each layer introduces overhead but provides valuable functionality. Block storage adds minimal overhead while enabling flexibility. File systems add metadata management and hierarchical organization. Object storage adds scalability, rich metadata, and simplified distribution at the cost of access pattern constraints (primarily read-entire-object operations).
| **Feature** | **Object Storage** | **Block Storage** | **File Storage** | | ----------- | -------------------------------------------- | ------------------------------------- | ------------------------------ | | Structure | Flat pool of objects | Grid of blocks | Hierarchical folders | | Access | API (PUT, GET) | OS/Driver (Read/Write blocks) | OS/Network (Read/Write files) | | Metadata | Rich, customizable | None | Fixed, limited | | Best For | Unstructured data, backup, archive, web apps | Databases, high-performance computing | Shared files, user directories | ### **Block Storage** Block storage operates at the most granular level, presenting storage as a collection of fixed-size blocks that can be individually read or written. Each block has a unique address, and the storage system doesn't impose any structure on how data is organized within those blocks.
**Characteristics:** * Provides raw storage volumes that appear as local disks to the operating system * Requires formatting with a file system (ext4, XFS, NTFS) before use * Supports both read and write operations at the block level * Offers low-latency access since it's typically directly attached or on high-speed networks * Allows incremental updates—you can modify small portions without rewriting entire files **AWS Example: Amazon EBS (Elastic Block Store)** * Attaches to a single EC2 instance at a time (though some volume types support multi-attach) * Persists independently of instance lifecycle * Offers different performance tiers (gp3, io2, st1, sc1) based on IOPS and throughput needs * Supports snapshots for backup and recovery * Volume sizes from 1 GB to 64 TB **Use Cases for Data Engineering:** * Database storage (PostgreSQL, MySQL, MongoDB) where you need consistent low-latency access * Transactional workloads requiring frequent random reads and writes * Running analytics databases like ClickHouse or TimescaleDB on EC2 * Storing Kafka logs or other streaming data that requires fast sequential writes * Running Spark or Hadoop worker nodes where intermediate shuffle data needs fast local storage **Limitations:** * Single availability zone—if the AZ fails, the volume is unavailable until recovery * Typically attached to one instance, limiting concurrent access * Costs scale linearly with provisioned capacity, regardless of actual usage * Manual management of capacity—you must provision and resize volumes explicitly *** ### **File Storage** File storage adds a hierarchical namespace on top of block storage, organizing data into directories and files. It provides shared access through network protocols, allowing multiple clients to read and write simultaneously.
**Characteristics:** * Presents a POSIX-compliant file system interface * Supports concurrent access from multiple compute instances * Handles file locking, permissions, and metadata automatically * Scales elastically—capacity grows and shrinks based on actual usage * Maintains file system semantics (directories, symbolic links, permissions) **AWS Example: Amazon EFS (Elastic File System)** * Automatically scales from gigabytes to petabytes * Accessible from multiple EC2 instances simultaneously across multiple AZs * Supports NFS v4 protocol * Offers different storage classes (Standard, Infrequent Access) * Provides lifecycle management to move files between storage classes **Use Cases for Data Engineering:** * Shared configuration files and scripts across a cluster of processing nodes * Home directories for user workspaces in notebook environments (JupyterHub, Zeppelin) * Shared libraries and dependencies for containerized applications * Content management systems where multiple applications need access to the same files * Machine learning training data that needs to be accessed by multiple training jobs simultaneously * Centralized logging where multiple services write to shared log directories **Limitations:** * Higher latency compared to block storage due to network overhead * More expensive per GB than object storage * Performance can be inconsistent with highly concurrent workloads * Not ideal for extremely high-throughput sequential reads/writes * POSIX semantics can create bottlenecks at scale (directory listing operations, for example) *** ### **Object Storage** Object storage treats data as discrete, self-contained objects rather than files or blocks. Each object consists of data, metadata, and a unique identifier. The architecture is designed for massive scalability and distributed access. Each piece of data you store is called an Object. As shown in the diagram below, an object is made up of three key components: * Data (Payload): The actual file content you are storing (e.g., an image, a document, a log file). * Metadata: Customizable labels and tags that describe the object (e.g., `author=jane`, `project=website`, `retention=1year`). This makes it easy to search and manage your data. * Unique ID (Key): A unique identifier, similar to a URL, that you use to retrieve the object (e.g., `my-bucket/photos/vacation/beach.jpg`). You interact with this storage pool over the internet using simple API commands like `PUT` (to upload), `GET` (to download), and `DELETE`.
**Characteristics:** * Flat namespace—objects are identified by keys rather than hierarchical paths (though keys can simulate paths) * Objects are immutable—modifications require uploading a new version * Stores unlimited amounts of data with virtually unlimited objects * Highly durable (11 nines of durability for S3) * Eventually consistent for overwrites and deletes (though S3 now offers strong consistency) * Rich metadata support—arbitrary key-value pairs per object * Accessible via HTTP/HTTPS APIs rather than file system protocols **AWS Example: Amazon S3 (Simple Storage Service)** * Organizes objects into buckets (top-level containers) * Supports versioning, lifecycle policies, and event notifications * Multiple storage classes (Standard, Intelligent-Tiering, Glacier, Deep Archive) * Integrates with virtually every AWS service * Offers features like S3 Select for querying data without downloading **Use Cases for Data Engineering:** * Data lakes—storing raw data in various formats (CSV, JSON, Parquet, Avro) * Long-term archival and backup storage * Storing results from batch processing jobs (Spark output, ETL results) * Serving as source/destination for data pipelines (Airflow, Glue, Step Functions) * Hosting static datasets for analytics and machine learning * Storing intermediate results between pipeline stages * Log aggregation and retention * Serving as a staging area for data warehouse loading (Redshift, Snowflake) * Immutable audit trails and compliance data **Advanced S3 Features for Data Engineers:** * **S3 Select**: Query CSV/JSON/Parquet files using SQL without downloading entire objects * **Partitioning**: Organize data using key prefixes (e.g., `s3://bucket/year=2025/month=01/day=15/data.parquet`) * **Storage Classes**: Automatically transition data between tiers based on access patterns * **Batch Operations**: Perform operations on billions of objects (copying, tagging, restoring) * **Event Notifications**: Trigger Lambda functions or SQS messages when objects are created/deleted * **Transfer Acceleration**: Speed up uploads using CloudFront edge locations * **Requester Pays**: Let data consumers pay for transfer costs **Limitations:** * Higher latency than block/file storage—typically 100-200ms for first byte * Must read/write entire objects—no partial updates (though multipart upload helps) * LIST operations can be slow with millions of objects * Not suitable for transactional workloads or databases * Limited query capabilities without additional tools (Athena, Presto, Spark) * Network bandwidth becomes a bottleneck for frequent access patterns #### **Object Storage Versioning** One of the most powerful features of object storage is **versioning**. When you enable versioning on a bucket, the system keeps a history of every object. When you update an object (by uploading a new file with the same key), it doesn't overwrite the old data. Instead, it creates a new version and makes it the current one. The old version is preserved, and you can still access it by specifying its unique version ID. This is crucial for data protection. It allows you to recover from accidental deletions or overwrites. The diagram below illustrates this concept, showing a single logical object (`image.jpg`) with a history of three distinct versions.
In this diagram, a `GET` request for `image.jpg` returns the current version (Version 3). However, a request for `image.jpg?versionId=v1` would retrieve the original file (Version 1). The diagram also shows that even after a 'DELETE' operation, the deleted version is still stored and can be recovered if needed. *** ### **Hybrid and Specialized Storage Options** **Amazon FSx Family:** * **FSx for Lustre**: High-performance parallel file system for HPC and ML training workloads. Integrates with S3 for seamless data access. Offers throughput up to hundreds of GB/s. * **FSx for Windows File Server**: Fully managed Windows file servers with SMB protocol support * **FSx for NetApp ONTAP**: Enterprise NAS with advanced data management features * **FSx for OpenZFS**: File storage built on OpenZFS with snapshots and cloning **Use case**: When EFS doesn't provide enough performance, FSx for Lustre is ideal for ML training, genomics processing, or financial modeling requiring massive parallel throughput. **AWS Storage Gateway:** * Hybrid cloud storage connecting on-premises environments to AWS * File Gateway: Presents S3 as NFS/SMB file shares * Volume Gateway: Presents S3 as iSCSI block storage * Tape Gateway: Virtual tape library for backup applications **Use case**: Gradually migrating on-premises data workloads to cloud while maintaining local access patterns. **Amazon S3 on Outposts:** * Run S3 object storage on-premises with the same APIs * Useful for data residency requirements or low-latency local access *** #### **Performance Comparison** Here's the performance comparison table: | Storage Type | Latency | Throughput | IOPS | Concurrency | Cost (relative) | | ------------------- | --------------- | ------------------ | ------------ | ------------------ | --------------- | | **Block (EBS io2)** | <1ms | 1,000+ MB/s | 64,000+ | Single instance | High | | **Block (EBS gp3)** | Single-digit ms | 125-1,000 MB/s | 3,000-16,000 | Single instance | Medium | | **File (EFS)** | Low ms | Bursts to 10+ GB/s | Varies | Multiple instances | Medium-High | | **Object (S3)** | 100-200ms | Parallel scaling | N/A | Unlimited | Low | | **FSx Lustre** | Sub-ms | Hundreds of GB/s | Millions | Multiple instances | High | **Key Notes:** * **Latency**: Time to access first byte of data * **Throughput**: Maximum data transfer rate (sequential operations) * **IOPS**: Input/Output Operations Per Second (random access operations) * **Concurrency**: Number of clients/instances that can access simultaneously * **Cost**: Relative pricing per GB stored and per operation **For Data Engineering Context:** * Use **Block (io2)** for high-performance databases requiring consistent low latency * Use **Block (gp3)** for general-purpose workloads with balanced performance/cost * Use **File (EFS)** when multiple workers need shared access to the same datasets * Use **Object (S3)** for data lakes and long-term storage where cost optimization matters * Use **FSx Lustre** for ML training or HPC workloads requiring extreme parallel throughput *** ### **Choosing the Right Storage** **Use Block Storage (EBS) when:** * Running databases that need low-latency random access * Operating systems and boot volumes * Real-time analytics requiring fast local storage * Single-instance applications with high IOPS requirements **Use File Storage (EFS/FSx) when:** * Multiple instances need shared access to the same data * You need POSIX file system semantics * Running containerized applications with shared configuration * ML training jobs accessing shared datasets **Use Object Storage (S3) when:** * Building data lakes with diverse data types * Long-term retention and archival * Integrating with AWS analytics services (Athena, EMR, Glue) * Storing immutable data like logs, backups, or compliance records * Cost optimization is critical and access patterns allow higher latency **Data Engineering Pattern: The Three-Tier Approach** Many data engineering architectures use all three: 1. **S3** for raw data ingestion, data lake storage, and long-term retention 2. **EBS** for database storage (Postgres, Airflow metadata) and temporary processing 3. **EFS** for shared resources (DAGs, scripts, libraries) across compute clusters For example, a typical Spark pipeline might: * Read source data from **S3** (cost-effective, scalable) * Use **EBS** volumes for local shuffle data during processing (fast random I/O) * Share UDF libraries and configuration via **EFS** (accessible to all workers) * Write results back to **S3** in optimized formats like Parquet This layered approach optimizes for both cost and performance across different stages of the data lifecycle. ***