System design fundamentals

Vertical scaling (scale up) means a bigger machine: more CPU, RAM or disk. It is simple, needs no code changes, but has a hard ceiling, gets expensive quickly and remains a single point of failure. Horizontal scaling (scale out) means more machines behind a load balancer.

It scales much further and survives individual machine failures. It works best when application servers are stateless: user sessions, uploaded files and caches live in shared stores (Redis, a database, object storage) rather than on a particular server's disk or memory. Stateless servers can be added, removed or replaced at any time without users noticing.

A load balancer distributes incoming requests across a pool of servers. Common strategies: round robin (take turns), least connections (send to the least busy server), and hashing on a key such as client IP or user ID (keeps a client on the same server, called stickiness). Health checks probe each server periodically and remove failing ones from rotation, then add them back when they recover.

Layer 4 balancers forward network connections by IP address and port without reading the request, which makes them fast and simple. Layer 7 balancers read the HTTP request, so they can route by URL path, host name, header or cookie: for example, send /api/ requests to the API servers and everything else to the web servers. They often handle HTTPS as well.

TLS is the encryption behind HTTPS, and the balancer decrypts incoming traffic before passing it on (this is called TLS termination), so the servers behind it receive plain HTTP and don't each need their own certificate.

A cache keeps frequently read data in fast storage, usually memory, to cut latency and reduce load on the database. Cache-aside (lazy loading): the application asks the cache first. If the data is there (a cache hit), it uses it straight away.

If not (a cache miss), it reads the data from the database and also saves a copy in the cache, so the next request for the same data is answered from the cache. Read-through: the cache itself loads missing data from the database. Write-through: every write goes to the cache and the database synchronously, so the cache is always fresh but writes are slower.

Write-back (write-behind): writes go to the cache first and are flushed to the database later, which is fast but can lose data if the cache node dies before the flush. Entries carry a TTL (time to live: how long an entry stays valid before it expires), and when memory is full an eviction policy such as LRU (least recently used) or LFU (least frequently used) decides what to drop. Invalidation, keeping the cache consistent with the source of truth after updates, is the hard part.

A content delivery network (CDN) keeps copies of content on edge servers located close to users around the world. It is ideal for static assets such as images, video, JavaScript and CSS, and can cache some API responses too. Users get lower latency because the bytes travel a short distance, and the origin servers receive far less traffic.

Cache headers such as Cache-Control and max-age tell the edge how long to keep a copy, and cache purging removes stale content early. A CDN also absorbs traffic spikes and some denial-of-service attacks before they reach the origin.

Replication means keeping copies of the same database on several machines, called nodes. The usual setup is leader-follower: one node, the leader, accepts all writes. It then sends every change to the other nodes, the followers, so their copies stay up to date.

Followers help in two ways. They can answer read requests, which takes load off the leader; a follower used like this is called a read replica. And if the leader fails, a follower can take its place.

The catch: the copying usually happens in the background, a moment after the write (asynchronous replication). So for a short time a follower can be behind the leader. If a user saves a change and immediately reads from a follower, they may see the old value.

This is replication lag. A common fix is to send a user's own reads to the leader for a few seconds after they write.

Replication copies all of the data to every node, so each node still holds everything and one leader still takes every write. When the data or the writes are too much for one machine, you split the data instead. Sharding (also called partitioning) puts each part of the data on a different node, called a shard.

A shard key decides where each record lives. For example, with user ID as the key, the ID is hashed and the result picks the shard, so users spread evenly across nodes. Each shard is smaller and handles only its share of the writes.

The costs: a query that needs data from many shards is slower and harder to write, and keeping transactions correct across shards is hard. A badly chosen key creates hot shards that take most of the load: shard by country, and one big country can end up on a single overloaded node. Consistent hashing keeps data movement small when nodes join or leave: keys and nodes sit on the same ring and each key belongs to the next node along it, so adding or removing a node only moves the keys next to it.

Real systems usually combine both ideas: the data is sharded, and each shard is replicated.

The CAP theorem says that when a network partition separates nodes, a distributed system must choose between consistency (every read returns the latest write, or an error) and availability (every request gets a non-error response, possibly stale). Partitions cannot be prevented, so the real choice is how the system behaves during one. Strong consistency means reads always reflect the latest committed write.

Eventual consistency means replicas converge over time and reads may briefly be stale. Systems often mix both: eventual consistency for like counts or feeds, strong consistency for balances, inventory and payments. Related idea: linearizability is the strictest form of consistency, where operations appear to happen instantly in a single global order.

A message queue (SQS, RabbitMQ, Kafka and similar) decouples producers from consumers: the producer publishes a message and moves on, and consumers process it asynchronously, possibly much later. Queues absorb traffic spikes, push slow work (emails, reports, image processing) out of the request path, and support retries when a consumer fails. Most queues guarantee at-least-once delivery, which means the same message can be delivered more than once, for example after a consumer crashes mid-processing.

Consumers should therefore be idempotent: processing a message twice must have the same effect as processing it once, typically by recording processed message IDs. Exactly-once delivery is expensive and rare, and ordering is usually guaranteed only within one partition (an ordered slice of the queue) or FIFO group (first in, first out: messages are handled in the order they arrived).

Rate limiting caps how many requests one client can make, so one user or bot can't overload the service. Over the limit, the server replies 429 Too Many Requests.

Token bucket: every client gets a bucket of coins, and each request costs one coin. The bucket holds at most 20 coins, and the server drops one new coin into it every second. A client that has been idle has 20 coins and can send 20 requests in a row. Then the bucket is empty, and each new request has to wait for the next coin. That is the point: short bursts are allowed, the long-run speed is one request per second. The coins are called tokens.

Leaky bucket: requests go through at a fixed pace, so bursts are smoothed out.

Fixed window: the limit is 100 requests per minute, and the counter resets at the start of every minute. The weak spot is the boundary: a client sends 100 requests at 12:00:59 and another 100 at 12:01:00. Both minutes are within the limit, yet the server took 200 requests in two seconds.

Sliding window: instead of calendar minutes, count the requests made in the last 60 seconds from right now. The 200-in-two-seconds trick no longer works.

Counters live in a shared store such as Redis, so all servers see the same numbers.