Storage

Hot persists three kinds of bytes outside its database:

• Application files: everything written with ::hot::file, the Files API, and multipart uploads.
• Internal blobs: large values spilled out of runs, events, and calls. Blobs share the file storage backend but never appear in file listings.
• Build artifacts: deployed project bundles (*.hot.zip).

Each installation uses one file backend and one build backend. Both default to local disk.

Public hot releases include both S3 and GCS support. A binary built without a backend's feature fails at startup with a clear error if that backend is selected. Unknown backend names are rejected rather than silently falling back to local storage.

Selecting a Backend

Precedence: an environment variable overrides the corresponding Hot configuration value, which overrides the default. Every Hot service (api, app, worker, task-worker, scheduler) must see the same storage configuration and credentials.

Upgrading: releases before GCS support ignored storage.type in Hot configuration and used only the environment variables. Configuration generated by hot conf generate for the api, app, worker, and task-worker roles in those releases sets both types to "s3" by default. If such a deployment ran on local storage, set HOT_FILE_STORAGE_TYPE=local and HOT_BUILD_STORAGE_TYPE=local (or edit those lines) before upgrading; otherwise startup fails because no S3 bucket is configured.

Google Cloud Storage

Configuration

<role> is file or build. The bucket is a bare name (my-bucket, not gs://my-bucket). Files and builds can use separate buckets, or one bucket with distinct prefixes:

HOT_FILE_STORAGE_TYPE=gcs
HOT_FILE_GCS_BUCKET=acme-hot-prod
HOT_FILE_GCS_PREFIX=files
HOT_BUILD_STORAGE_TYPE=gcs
HOT_BUILD_GCS_BUCKET=acme-hot-prod
HOT_BUILD_GCS_PREFIX=builds

The same settings in hot.hot:

hot.file.storage.type "gcs"
hot.file.gcs.bucket "acme-hot-prod"
hot.file.gcs.prefix "files"
hot.build.storage.type "gcs"
hot.build.gcs.bucket "acme-hot-prod"
hot.build.gcs.prefix "builds"

Hot uses existing buckets. It never creates, configures, or deletes buckets. The concurrency limit covers download bodies as well as request setup. The request timeout covers headers and the entire body, including backpressure from a slow consumer; dropping the reader cancels the download.

Authentication

Hot authenticates with Application Default Credentials. No keys belong in Hot configuration, and Hot never runs gcloud at runtime.

• On Google Cloud (GKE, Cloud Run, Compute Engine): attach a service account to the workload. On GKE, use Workload Identity Federation for GKE.
• CI and deployments outside Google Cloud: configure Workload Identity Federation and point GOOGLE_APPLICATION_CREDENTIALS at the generated credential configuration file (it contains no private key).
• Local development: run gcloud auth application-default login once. This is the only place gcloud is used.

Service account key files work through GOOGLE_APPLICATION_CREDENTIALS but are not required or recommended.

Google's SDK verifies Cloud Storage endpoints against the operating system's certificate store, not the Mozilla roots Hot bundles for its own outbound requests. Container images that run Hot with GCS need a CA bundle, for example the ca-certificates package.

Bucket Setup

Recommended bucket settings:

• Uniform bucket-level access enabled. Hot never sets object ACLs.
• Public access prevention enforced. Hot never serves objects publicly; downloads stream through Hot after authorization.
• Bucket-scoped IAM: grant the Hot service account roles/storage.objectUser on the bucket only, not on the project. Hot needs to create, read, list, compose, and delete objects. Listing is used only to clean up Hot's own multipart staging objects.

gcloud storage buckets create gs://acme-hot-prod \
  --location=us-central1 \
  --uniform-bucket-level-access \
  --public-access-prevention

gcloud storage buckets add-iam-policy-binding gs://acme-hot-prod \
  --member=serviceAccount:hot@acme-prod.iam.gserviceaccount.com \
  --role=roles/storage.objectUser

Private Networks

From a private network, reach Cloud Storage through Private Google Access or a Private Service Connect endpoint. If you use a Private Service Connect endpoint name or the restricted.googleapis.com VIP under VPC Service Controls, set HOT_FILE_GCS_ENDPOINT and HOT_BUILD_GCS_ENDPOINT to that endpoint (for example https://storage-hotpsc.p.googleapis.com).

Object Layout

Under the configured prefix:

Every file write lands at a new object, and the database record is switched to it as the commit point. A failed or losing write never changes what readers see, and concurrent writers never overwrite each other's bytes. The database, not the bucket, is authoritative for which files exist and who may read them. Hot's file ETag is an MD5 of the content. A multipart upload's ETag identifies the exact parts it was composed from (S3 reports its own multipart ETag). File ETags are independent of GCS object generations.

Multipart Uploads

Each uploaded part is stored as its own object, and the part ETag returned by the API identifies that exact object version. Completion composes exactly the parts selected by the manifest, in order, with generation preconditions, so a concurrent re-upload of a part cannot change the bytes being finalized. More than 32 parts are combined in stages, all within the same bucket.

Completion is recoverable. If a completion is interrupted (a crash, a database error, or a lost response), retrying it on any instance reuses the object it assembled, or reports success if the file already references it. Once completed, retries return the saved completion response even if a later write or deletion changed the path. They never reapply the upload. Completion only replaces the file version that existed when it began: if the file was written or deleted in the meantime, nothing is committed, the upload is aborted, and the request fails with 409 upload_conflict.

While a completion is in progress the upload cannot be aborted, and its parts can no longer change. A failed completion returns the upload to pending. Its original file-version guard is preserved across failures and retries. Starting a completion extends the upload's expiry by an hour, so the expiry sweep only aborts a completion that stopped making progress. Aborting removes the staged parts and any assembled object the file does not reference. Retrying a part upload whose response was lost is safe: the same bytes are acknowledged again, and different bytes for an already uploaded part are refused with 409 part_already_uploaded. Before sending any part bytes to storage, the API reserves that part's content hash in the database. Backend ETags remain opaque object tokens. Concurrent requests for different bytes are rejected before they can replace an S3 part; an identical request still in progress returns 409 part_in_progress and can be retried after it finishes. A crashed part attempt is reclaimable after ten minutes, with the same bytes. Each part operation times out after five minutes.

Abort claims also have a one-hour lease. A client retry or the expiry sweep can reclaim an interrupted abort when that lease expires. Cleanup failures release the claim immediately while retaining the abort intent, and retrying an already aborted upload succeeds.

Migration 011 adds durable part reservations and saved completion responses. Completed uploads from before this migration have no saved response and return 409 completion_result_unavailable when completion is retried. Older Local/S3 parts without reservations can still be retried when their ETag matches the body's MD5; opaque legacy part tokens cannot establish content equality.

Lifecycle, Retention, and Costs

Staged parts are deleted after completion or abort. If a deletion fails or a process dies mid-upload, staging objects can remain. Add a lifecycle rule scoped to the staging prefix (adjust for your file prefix) as a backstop:

{
  "rule": [{
    "action": {"type": "Delete"},
    "condition": {"age": 7, "matchesPrefix": ["files/.uploads/"]}
  }]
}

gcloud storage buckets update gs://acme-hot-prod --lifecycle-file=lifecycle.json

Do not add age- or name-based deletion rules outside .uploads/. Objects named *.v-*, *.cas-*, and *.mp-* hold committed file data even though their names look temporary.

Interactions with other bucket features:

• Soft delete (on by default, 7 days) keeps deleted and overwritten objects restorable and billed for the retention window. Every overwrite of a Hot file deletes the previous object, so write-heavy workloads such as ::hot::sqlite accumulate soft-deleted bytes. Lower or disable soft delete if you do not need it.
• Object versioning retains noncurrent copies of every replaced or deleted object. Hot does not need it; if enabled, add a lifecycle rule for noncurrent versions.
• Retention policies and holds block deletion. Hot logs and continues when cleanup of a superseded or staged object fails, so a locked retention policy makes storage grow until the policy allows deletion. File and blob deletes report an error when the object cannot be removed.

Failed deletions of superseded objects leave unreferenced bytes behind. They do not count against Hot's storage quota (which is computed from database records) but are billed by Google Cloud.

Troubleshooting

Hot logs bucket names and operations but keeps credentials out of logs. Object keys contain organization and environment IDs and appear only at debug level.

Switching Backends

Changing storage.type does not move data. Existing file records, blob objects, in-flight uploads, and builds keep pointing at the backend that wrote them, and Hot refuses to read, overwrite, or clean them up through a different backend instead of guessing. Switching an existing installation requires a separate migration that copies the bytes and rewrites persisted references (file.storage_backend/storage_path, blob_object.storage_backend, and build.storage_backend/storage_path). Abandon or finish pending multipart uploads before switching.

Internal blobs are content-addressed, so storing a blob whose content was already stored by the previous backend writes it to the new backend and moves the blob object there; existing references to it become readable again. Blob cleanup only removes objects in the configured backend.

Fresh installations can select GCS from the start with no migration.

Amazon S3

S3 uses the standard AWS credential chain unless access-key-id and secret-access-key are set under hot.file.aws or hot.build.aws.

Local Filesystem