7. Storage

SQL vs NoSQL

Feature	SQL (Relational)	NoSQL
Data Model	Tables (rows & columns), schemas enforced	Documents, key-value, wide-column, graph
Schema	Fixed, requires migrations	Flexible, schema-on-read
ACID	Full ACID support	BASE (Basically Available, Soft state, Eventual consistency)
Scaling	Vertical (mostly; sharding is complex)	Horizontal by design
Examples	PostgreSQL, MySQL, SQL Server	MongoDB, DynamoDB, Cassandra, Redis
Use Case	Financial systems, structured data, complex joins	Real-time feeds, IoT, user profiles, caching

Sharding Strategies (In Detail)

Hash-Based Sharding

Use a hash function on the shard key: shard = hash(key) % N. Provides even distribution but makes range queries expensive.

Range-Based Sharding

Split data by value ranges: shard 1 = user IDs 1—1M, shard 2 = 1M—2M, etc. Range queries are efficient but can cause hot spots.

Directory-Based Sharding

Maintain a lookup table mapping keys to shards. Flexible but adds a lookup step and single point of failure risk.

Geographic Sharding

Shard by region: US users on one shard, EU on another. Good for data sovereignty and latency but problematic for global users.

Replication

Replication copies data across multiple nodes for fault tolerance and read scalability.

# Leader-Follower Replication
Writes go to Leader → replicates to Followers
Reads can go to any replica (Follower or Leader)
If Leader fails, a Follower is promoted to Leader

# Leaderless Replication (e.g., Cassandra)
Writes go to all replicas (or a quorum)
Reads query multiple replicas and compare versions
No single point of failure; higher availability

Object Storage

Object storage (AWS S3, Azure Blob Storage, Google Cloud Storage) stores data as objects in flat namespaces with unique IDs. Each object contains data, metadata, and a globally unique identifier.

Characteristics: Highly durable (99.999999999%), scales to exabytes, pay-per-use, ideal for static files, backups, media.

Distributed File Systems

Systems like Google File System (GFS) and Hadoop HDFS split files into chunks (e.g., 64MB or 128MB) and replicate chunks across machines. A metadata server tracks chunk locations. Optimized for sequential reads/writes on large files.

Big Data Storage

Columnar storage: Parquet, ORC — efficient for analytics on specific columns.
Data warehouses: Snowflake, Redshift, BigQuery — optimized for analytical queries on large datasets.
Data lakes: Store raw data in native format; schema applied at read time.
Time-series databases: InfluxDB, TimescaleDB — optimized for time-stamped data (metrics, logs).

Exercise: Design a storage layer for a photo-sharing app with 500M users. Each user uploads 10 photos/month (5MB average). Consider: metadata storage (user profiles, albums), photo storage (blob), serving photos quickly (CDN), and handling peak upload traffic (e.g., New Year's Eve).