Health & scaling

The server is built to run as many stateless replicas behind a load balancer. This page covers the probes you wire up, how to size the native worker pool, and the one important property of the runtime that shapes your resilience strategy.

Health probes#

Two endpoints, both public (no token required):

curl localhost:3000/healthz   # {"status":"ok"}
curl localhost:3000/readyz    # {"status":"ok"}

Point your load balancer or orchestrator at /readyz for traffic decisions and /healthz for restarts. The Docker image includes a built-in HEALTHCHECK, and the Helm chart wires both probes for you.

The worker pool#

PDF rendering is CPU-heavy and runs in a pool of native worker threads, so requests render in parallel without blocking the HTTP loop. Size it with:

CLOUDPDF_WORKER_POOL_SIZE=max   # an integer, or "max" for all cores

• Default: min(2, cpu count) — deliberately conservative.
• max: use every available core. Good when the instance is dedicated to the server.
• An integer: pin an exact number, e.g. leave headroom for other processes.

A useful rule of thumb: give the instance roughly one CPU per worker, then tune from there using your render latency and CPU graphs.

Memory sizing#

Each worker thread carries its own copy of font and CMap data, so memory grows with the pool size, not just with traffic. When you raise CLOUDPDF_WORKER_POOL_SIZE, raise the instance’s memory to match — budget for per-thread font/CMap duplication on top of your documents’ working set. Watch for restarts caused by hitting a memory limit and give the pool more headroom if you see them.

Crash isolation: the engine host#

PDF parsing runs native code on hostile input, so the resilience story starts with one question: what does a native crash cost you?

CLOUDPDF_ENGINE_ISOLATION=host

Host mode also brings two protections that only exist behind a process boundary:

• Credential separation — the engine child is started with a whitelisted environment: it never sees your database URL, JWT secret, license key, or storage credentials.
• Poison-document quarantine — the server journals which documents were in flight when the engine died. A document that repeatedly crashes the engine (two independent sole-suspect crashes on the same content and engine build) is refused with 422 DocumentQuarantined instead of being allowed to crash-loop your fleet. Inspect and release with the quarantine CLI.

/readyz understands the engine: a sub-second respawn never flips readiness, while an engine that stays down past a threshold does — so your load balancer routes around a genuinely broken instance without flapping on routine recovery.

Backpressure: honest 503s instead of hangs#

Engine capacity is guarded by a two-lane scheduler: interactive work (what a user is waiting on) can use every worker; background work (thumbnail warming) is capped so it can never crowd requests out. Past the queues’ bounds the server sheds with 503 + Retry-After and an EngineBusy error code — a fast, retryable answer instead of a 30-second hang.

The @cloudpdf/engine client retries these automatically (they are safe by construction — a shed request never reached the engine), so overload and engine respawns are invisible to end users. Tune with CLOUDPDF_ENGINE_MAX_IN_FLIGHT and CLOUDPDF_ENGINE_BG_MAX_IN_FLIGHT against the queue metrics below.

Memory recycling (opt-in)#

Long-lived native processes slowly accumulate allocator and cache memory. In host mode you can opt into watermark recycling: the server watches the container’s memory working set and gracefully replaces the engine process before the pod hits its limit — a controlled sawtooth instead of an OOMKill, with no crash-journal noise and no dropped requests for parked work.

CLOUDPDF_ENGINE_RECYCLE=1               # opt-in master switch
CLOUDPDF_ENGINE_RECYCLE_SOFT_PCT=70     # graceful replace at 70% of the limit
CLOUDPDF_ENGINE_RECYCLE_HARD_PCT=85     # immediate replace at 85%
CLOUDPDF_ENGINE_MAX_LIFETIME_HOURS=24   # jittered lifetime cap (slow-leak hedge)

Engine shards (blast-radius dial)#

At larger worker counts you can split the engine into K independent child processes with CLOUDPDF_ENGINE_SHARDS (host mode; the worker total must divide evenly). Documents are partitioned across shards, so one crash or recycle costs 1/K of open documents — the others stay warm and never notice. Leave it at the default 1 until your crash/memory telemetry says otherwise; per-shard metrics (cloudpdf_engine_shard_up) expose a flapping shard the moment you turn it up.

Resilience: run more than one replica#

Process-level isolation shrinks the blast radius inside an instance; replicas remove the instance itself as a single point of failure:

• Run at least two replicas (Docker Compose --scale, or Helm replicaCount/HPA) so one instance restarting never means zero capacity.
• Use a PodDisruptionBudget (the Helm chart includes one) so node maintenance can’t drain all replicas at once.
• Let liveness restart crashed instances and readiness keep traffic off them until they’re back.
• With several replicas, consider the Helm chart’s document affinity option so one document’s traffic lands on one warm replica — see Helm / Kubernetes.

Autoscaling#

On Kubernetes, the Helm chart can create a HorizontalPodAutoscaler on CPU (and optionally memory). Because the server is stateless — all state lives in Postgres and the object store — replicas scale up and down freely:

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

A production checklist#

• Postgres, with migrations run as a separate step (Migrations).
• A shared object store (S3 / GCS / Azure Blob).
• At least two replicas, with /readyz and /healthz wired up.
• CLOUDPDF_WORKER_POOL_SIZE and memory tuned together.
• CLOUDPDF_ENGINE_ISOLATION=host so a malformed PDF costs an engine respawn, never an instance restart.
• A pinned image version or digest — never :latest.
• CLOUDPDF_JWT_SECRET injected from a secret manager.

Next steps#