mini-GFS

Problem

I wanted to understand how GFS separates metadata from data and how it recovers when a storage node dies. Reading the paper only gets you so far, so I built a master, chunkservers and a client gateway that talk over gRPC. I also wanted the storage nodes to hold nothing readable, which meant encryption had to happen before data left the client.

What I built

• Three separate binaries (master, chunkserver, client gateway) with their contracts defined in two .proto files.
• The master tracks file-to-chunk mappings, replica locations and wrapped keys, and writes them to metadata.json so state survives a restart.
• Chunkservers send a heartbeat every 5 seconds. The master checks every 10 seconds and marks a server dead after 15 seconds of silence.
• When a server dies, the master picks a live node without a copy and tells it to pull the chunk straight from a surviving replica, so the data never passes through the master.
• Each upload gets a fresh AES-256-GCM key. That key is wrapped with RSA-OAEP, so chunkservers only see ciphertext and the master only stores wrapped keys.
• Each upload is stored as one chunk of up to 64 MiB. This kept the write path short, at the cost of not striping large files across servers.

Results

• With a replication factor of 2, one chunkserver can go offline without data loss; the master starts re-replicating within 25 seconds of the last heartbeat.
• Tampered chunks fail to decrypt because GCM authenticates the ciphertext.

What I learned

• The master runs the replication call while holding its global lock, so heartbeats wait until recovery finishes. Choosing the target under the lock and copying outside it would fix that.
• Downloads always read from the first listed replica. If that node is down, the download fails until the master repairs the list, so the client should fall back to the next replica.
• A single master is simple to reason about, but it is also the one component whose loss takes the whole system offline.