System design fundamentals

Vertical scaling (scale up) means a bigger machine: more CPU, RAM or disk. It is simple, needs no code changes, but has a hard ceiling, gets expensive quickly and remains a single point of failure. Horizontal scaling (scale out) means more machines behind a load balancer.

It scales much further and survives individual machine failures. It works best when application servers are stateless: user sessions, uploaded files and caches live in shared stores (Redis, a database, object storage) rather than on a particular server's disk or memory. Stateless servers can be added, removed or replaced at any time without users noticing.

A load balancer distributes incoming requests across a pool of servers. Common strategies: round robin (take turns), least connections (send to the least busy server), and hashing on a key such as client IP or user ID (keeps a client on the same server, called stickiness). Health checks probe each server periodically and remove failing ones from rotation, then add them back when they recover.

Layer 4 balancers work with IP addresses and ports and are fast and simple. Layer 7 balancers understand HTTP and can route by path, host, header or cookie, and often terminate TLS.

A cache keeps frequently read data in fast storage, usually memory, to cut latency and reduce load on the database. Cache-aside (lazy loading): the application checks the cache first; on a miss it reads the database and writes the result into the cache for next time. Read-through: the cache itself loads missing data from the database.

Write-through: every write goes to the cache and the database synchronously, so the cache is always fresh but writes are slower. Write-back (write-behind): writes go to the cache first and are flushed to the database later, which is fast but can lose data if the cache node dies before the flush. Entries carry a TTL, and when memory is full an eviction policy such as LRU (least recently used) or LFU (least frequently used) decides what to drop.

Invalidation, keeping the cache consistent with the source of truth after updates, is the hard part.

A content delivery network (CDN) keeps copies of content on edge servers located close to users around the world. It is ideal for static assets such as images, video, JavaScript and CSS, and can cache some API responses too. Users get lower latency because the bytes travel a short distance, and the origin servers receive far less traffic.

Cache headers such as Cache-Control and max-age tell the edge how long to keep a copy, and cache purging removes stale content early. A CDN also absorbs traffic spikes and some denial-of-service attacks before they reach the origin.

Replication keeps copies of the same data on several nodes. In a leader-follower setup, writes go to the leader and are copied to followers, which can serve reads (read replicas) and take over if the leader fails. Replication is usually asynchronous, so a follower can lag behind: a read right after a write may return stale data (the read-your-own-writes problem).

Sharding (partitioning) splits the data across nodes by a key, for example user ID, so each node holds only a slice. Sharding scales writes and storage, but cross-shard queries and transactions become hard, and a badly chosen key creates hot shards that take most of the load. Consistent hashing keeps data movement small when nodes join or leave.

The CAP theorem says that when a network partition separates nodes, a distributed system must choose between consistency (every read returns the latest write, or an error) and availability (every request gets a non-error response, possibly stale). Partitions cannot be prevented, so the real choice is how the system behaves during one. Strong consistency means reads always reflect the latest committed write.

Eventual consistency means replicas converge over time and reads may briefly be stale. Systems often mix both: eventual consistency for like counts or feeds, strong consistency for balances, inventory and payments. Related idea: linearizability is the strictest form of consistency, where operations appear to happen instantly in a single global order.

A message queue (SQS, RabbitMQ, Kafka and similar) decouples producers from consumers: the producer publishes a message and moves on, and consumers process it asynchronously, possibly much later. Queues absorb traffic spikes, push slow work (emails, reports, image processing) out of the request path, and support retries when a consumer fails. Most queues guarantee at-least-once delivery, which means the same message can be delivered more than once, for example after a consumer crashes mid-processing.

Consumers should therefore be idempotent: processing a message twice must have the same effect as processing it once, typically by recording processed message IDs. Exactly-once delivery is expensive and rare, and ordering is usually guaranteed only within a partition or FIFO group.

Rate limiting caps how many requests a client may make in a period, protecting the service from abuse and overload and sharing capacity fairly. Token bucket: tokens are added at a steady rate up to a capacity, and each request spends one; this allows short bursts up to the bucket size while enforcing the long-term average. Leaky bucket smooths output to a constant rate.

Fixed window counts requests per calendar window (per minute, say) but allows double bursts at window edges. Sliding window log or sliding window counter look at the last N seconds and avoid that edge effect. Limited clients receive HTTP 429 Too Many Requests, ideally with a Retry-After header.

Limits are usually keyed by user ID, API key or IP, and the counters live in a shared store such as Redis so every server sees the same numbers.