Board: a stateful app and a lost host

The goal is to show where the data of a volume-backed app physically lives, what Fly.io copies for you, and exactly what is left when the server under a Machine dies.

This page is drawn from https://fly.io/docs/volumes/overview/, https://fly.io/docs/volumes/snapshots/, https://fly.io/docs/apps/trouble-host-unavailable/, https://fly.io/docs/reference/machine-migration/, https://fly.io/docs/blueprints/resilient-apps-multiple-machines/, https://fly.io/docs/about/pricing/ and https://fly.io/blog/machine-migrations/.

Whiteboard outcome

When you are done, the board shows two Machines on two different hosts, each with its own volume, a replication arrow the app owns, daily snapshots going off the host, and one host crossed out in red with a list of what survives.

Before you draw: three questions

1. What is the data: a database, SQLite files, or uploads?
2. How much loss is acceptable if a host dies?
3. Does the app already replicate, or is it one writer?

Draw it in 8 steps

1. Draw one Machine square labeled “app, region ams” on a host box. Say: “Each Machine runs on a single physical host” and “does not automatically start again on another host” if that host fails (https://fly.io/docs/blueprints/resilient-apps-multiple-machines/).
2. Under it, draw a cylinder labeled volume, NVMe, same server. Say: A volume is “a slice of an NVMe drive on the same physical server as the Machine on which it’s mounted”, so it is “not network storage” (https://fly.io/docs/volumes/overview/).
3. Draw a second host with a second Machine and cylinder. Say: fly volumes create defaults to separate hardware zones, and the docs say to “Always provision at least two volumes per app” (https://fly.io/docs/volumes/overview/).
4. Draw a double arrow between the two cylinders labeled “your app replicates”. Say: “Fly.io does not automatically replicate data among the volumes on an app” (https://fly.io/docs/volumes/overview/). Options named in the docs are LiteFS and Managed Postgres (https://fly.io/docs/volumes/overview/).
5. From each cylinder, draw a dashed arrow off the host labeled “daily snapshot”. Say: Fly.io takes daily block-level snapshots, kept 5 days by default and configurable from 1 to 60 days (https://fly.io/docs/volumes/overview/). They “may not have your latest data” (https://fly.io/docs/volumes/overview/).
6. Cross out host 1 in red. Say: “Volumes are pinned to physical hosts, so when there’s a host outage the volume is unreachable” (https://fly.io/docs/apps/trouble-host-unavailable/). Requests can still be routed to a healthy Machine on host 2 if one is running (https://fly.io/docs/reference/fly-proxy/).
7. Draw the recovery path: fly volumes snapshots list, then fly volumes create --snapshot-id, then fly scale count 1 (https://fly.io/docs/apps/trouble-host-unavailable/). The new volume must be equal or greater in size (https://fly.io/docs/volumes/snapshots/).
8. At the bottom, write the rule: A volume is a disk, not a service. Two volumes, one replication plan, one tested restore. (https://fly.io/docs/volumes/overview/, https://fly.io/docs/volumes/snapshots/)

flowchart LR
  %% source: https://fly.io/docs/apps/trouble-host-unavailable/
  subgraph H1[Host 1 lost]
    M1[Machine A] --- V1[(Volume A)]
  end
  subgraph H2[Host 2 healthy]
    M2[Machine B] --- V2[(Volume B)]
  end
  V1 <-.app replication.-> V2
  V1 -.daily snapshot.-> S[(Snapshot store<br/>5 days default)]
  V2 -.daily snapshot.-> S
  S -->|fly volumes create<br/>--snapshot-id| V3[(New volume<br/>new host)]
  V3 --- M3[New Machine]

What survives a host loss

The Fly.io engineering blog explains why this is a design choice: attached NVMe drives keep data “never more than a bus hop away”, at the cost that “a Fly App with an attached Volume is anchored to a particular worker physical” (https://fly.io/blog/machine-migrations/).

Do not mix up: planned migration vs host loss

• Planned drain. When Fly.io moves a Machine off a healthy-enough host, it stops the Machine, forks the volume to the new host, and starts a new Machine with the same Machine ID but a new 6PN address (https://fly.io/docs/reference/machine-migration/).
• Host loss. There is no live source to fork from, so you are on snapshots and replicas. The blog notes a “restore from backup” path loses data (https://fly.io/blog/machine-migrations/).

What they will ask