Board: the request path

The goal is to explain how a browser request reaches a Machine that is stopped at the moment the request arrives, and where fly-replay can move the request somewhere else.

This page is drawn from https://fly.io/docs/reference/fly-proxy/, https://fly.io/docs/reference/fly-proxy-autostop-autostart/, https://fly.io/docs/launch/autostop-autostart/, https://fly.io/docs/networking/dynamic-request-routing/, https://fly.io/docs/reference/load-balancing/, https://fly.io/docs/reference/suspend-resume/ and https://fly.io/docs/networking/services/.

Whiteboard outcome

When you are done, the board shows a browser, the nearest edge running Fly Proxy, a WireGuard hop to a worker, a stopped Machine being started, and a dotted fly-replay arrow to a second region.

Before you draw: three questions

1. Does the app scale to zero (min_machines_running = 0), or does it keep a Machine up?
2. Where are the users, and in which regions do Machines exist?
3. Does any route need to be served from one special place, such as a primary database region?

Draw it in 8 steps

1. On the left, draw a browser with an arrow labeled https://app.fly.dev. Say: Fly.io announces its IP blocks from all datacenters over BGP (Anycast), so the client reaches the nearest server (https://fly.io/docs/networking/services/).
2. Draw a rounded rectangle “edge: Fly Proxy”. Say: The request lands on the closest edge server, where Fly Proxy adds headers, redirects HTTP to HTTPS and routes it to a worker (https://fly.io/docs/reference/fly-proxy/).
3. Draw a thick arrow to a “worker” box labeled “WireGuard backhaul”. Say: the edge routes “through the WireGuard tunnel backhaul to the nearest healthy worker server that hosts your app’s Machines” (https://fly.io/docs/reference/fly-proxy/).
4. Inside the worker, draw a dashed square “Machine: stopped”. Say: A stopped Machine is not billed for CPU or RAM, only for its root file system (https://fly.io/docs/launch/autostop-autostart/).
5. Draw a lightning arrow from Fly Proxy to the Machine labeled auto_start_machines = true. Say: “If all the running Machines are above their soft_limit setting, then the proxy starts a stopped or suspended Machine in the nearest region”, then routes the request to it (https://fly.io/docs/reference/fly-proxy-autostop-autostart/). With nothing running, the docs describe the same start: “Fly Proxy starts a stopped Machine and routes the request to it” (https://fly.io/docs/launch/autostop-autostart/).
6. Write the wake time next to it. A cold start “takes about 2 seconds or more for common apps”, and a suspended Machine resumes in “a few hundred” milliseconds (https://fly.io/docs/launch/autostop-autostart/, https://fly.io/docs/reference/suspend-resume/).
7. Draw the Machine answering with a header fly-replay: region=iad, then a dotted arrow back to Fly Proxy and out to a second dashed region box. Say: “Fly Proxy replays the whole request as instructed” (https://fly.io/docs/reference/fly-proxy/). The request can go to a region, a Machine, or another app in the org (https://fly.io/docs/networking/dynamic-request-routing/).
8. At the bottom, write the rule: Fly Proxy starts and routes. Your app can say: not me, ask them. (https://fly.io/docs/reference/fly-proxy/, https://fly.io/docs/networking/dynamic-request-routing/) Circle “existing Machines only”.

flowchart LR
  B[Browser] -->|HTTPS| E
  subgraph E[Edge: Fly Proxy]
    P[Headers<br/>HTTPS redirect<br/>load balance]
  end
  P -->|WireGuard| W
  subgraph W[Worker, region ams]
    M1[Machine<br/>stopped]
  end
  P -.autostart.-> M1
  M1 -->|fly-replay<br/>region=iad| P
  P -->|replay| W2
  subgraph W2[Worker, region iad]
    M2[Machine<br/>running]
  end

What can go wrong

• Autostart never creates Machines. It “only works on existing Machines and never creates or destroys Machines for you” (https://fly.io/docs/reference/fly-proxy-autostop-autostart/).
• Autostart off means failed requests. If auto_start_machines is explicitly false and nothing is running, “the request will fail” (https://fly.io/docs/reference/fly-proxy-autostop-autostart/).
• Replay limits. Requests larger than 1MB cannot be replayed, and the app must use the http handler (https://fly.io/docs/networking/dynamic-request-routing/). The replay approach “works for any kind of HTTP request, including WebSockets” (https://fly.io/docs/blueprints/connecting-to-user-machines/).
• Replay to a stopped Machine. With fly-replay, “If the machine is not running, the proxy will start it” (https://fly.io/docs/blueprints/connecting-to-user-machines/).
• Overload. Far above hard_limit for a long time, Fly Proxy “might start returning 503 Service Unavailable responses” (https://fly.io/docs/reference/load-balancing/).

What they will ask