Storage
Hot persists three kinds of bytes outside its database:
- Application files: everything written with
::hot::file, the Files API, and multipart uploads. - Internal blobs: large values spilled out of runs, events, and calls. Blobs share the file storage backend but never appear in file listings.
- Build artifacts: deployed project bundles (
*.hot.zip).
Each installation uses one file backend and one build backend. Both default to local disk.
| Backend | type value | Build feature |
|---|---|---|
| Local filesystem | local (default) | always available |
| Amazon S3 | s3 | s3-storage |
| Google Cloud Storage | gcs | gcs-storage |
Public hot releases include both S3 and GCS support. A binary built without a
backend's feature fails at startup with a clear error if that backend is
selected. Unknown backend names are rejected rather than silently falling back
to local storage.
Selecting a Backend
| Setting | Environment variable | Default |
|---|---|---|
hot.file.storage.type | HOT_FILE_STORAGE_TYPE | local |
hot.build.storage.type | HOT_BUILD_STORAGE_TYPE | local |
Precedence: an environment variable overrides the corresponding Hot
configuration value, which overrides the default. Every Hot service (api,
app, worker, task-worker, scheduler) must see the same storage
configuration and credentials.
Upgrading: releases before GCS support ignored storage.type in Hot
configuration and used only the environment variables. Configuration
generated by hot conf generate for the api, app, worker, and
task-worker roles in those releases sets both types to "s3" by default.
If such a deployment ran on local storage, set HOT_FILE_STORAGE_TYPE=local
and HOT_BUILD_STORAGE_TYPE=local (or edit those lines) before upgrading;
otherwise startup fails because no S3 bucket is configured.
Google Cloud Storage
Configuration
| Setting | Environment variable | Default |
|---|---|---|
hot.file.gcs.bucket | HOT_FILE_GCS_BUCKET | required |
hot.file.gcs.prefix | HOT_FILE_GCS_PREFIX | none |
hot.build.gcs.bucket | HOT_BUILD_GCS_BUCKET | required |
hot.build.gcs.prefix | HOT_BUILD_GCS_PREFIX | none |
hot.<role>.gcs.endpoint | HOT_<ROLE>_GCS_ENDPOINT | Google's global endpoint |
hot.<role>.gcs.max-concurrent-requests | HOT_<ROLE>_GCS_MAX_CONCURRENT_REQUESTS | 32 |
hot.<role>.gcs.request-timeout-secs | HOT_<ROLE>_GCS_REQUEST_TIMEOUT_SECS | 300 |
hot.<role>.gcs.retry-attempts | HOT_<ROLE>_GCS_RETRY_ATTEMPTS | 5 |
hot.<role>.gcs.retry-time-limit-secs | HOT_<ROLE>_GCS_RETRY_TIME_LIMIT_SECS | 120 |
<role> is file or build. The bucket is a bare name (my-bucket, not
gs://my-bucket). Files and builds can use separate buckets, or one bucket with
distinct prefixes:
HOT_FILE_STORAGE_TYPE=gcs
HOT_FILE_GCS_BUCKET=acme-hot-prod
HOT_FILE_GCS_PREFIX=files
HOT_BUILD_STORAGE_TYPE=gcs
HOT_BUILD_GCS_BUCKET=acme-hot-prod
HOT_BUILD_GCS_PREFIX=builds
The same settings in hot.hot:
hot.file.storage.type "gcs"
hot.file.gcs.bucket "acme-hot-prod"
hot.file.gcs.prefix "files"
hot.build.storage.type "gcs"
hot.build.gcs.bucket "acme-hot-prod"
hot.build.gcs.prefix "builds"
Hot uses existing buckets. It never creates, configures, or deletes buckets. The concurrency limit covers download bodies as well as request setup. The request timeout covers headers and the entire body, including backpressure from a slow consumer; dropping the reader cancels the download.
Authentication
Hot authenticates with
Application Default Credentials.
No keys belong in Hot configuration, and Hot never runs gcloud at runtime.
- On Google Cloud (GKE, Cloud Run, Compute Engine): attach a service account to the workload. On GKE, use Workload Identity Federation for GKE.
- CI and deployments outside Google Cloud: configure
Workload Identity Federation
and point
GOOGLE_APPLICATION_CREDENTIALSat the generated credential configuration file (it contains no private key). - Local development: run
gcloud auth application-default loginonce. This is the only placegcloudis used.
Service account key files work through GOOGLE_APPLICATION_CREDENTIALS but are
not required or recommended.
Google's SDK verifies Cloud Storage endpoints against the operating system's
certificate store, not the Mozilla roots Hot bundles for its own outbound
requests. Container images that run Hot with GCS need a CA bundle, for example
the ca-certificates package.
Bucket Setup
Recommended bucket settings:
- Uniform bucket-level access enabled. Hot never sets object ACLs.
- Public access prevention enforced. Hot never serves objects publicly; downloads stream through Hot after authorization.
- Bucket-scoped IAM: grant the Hot service account
roles/storage.objectUseron the bucket only, not on the project. Hot needs to create, read, list, compose, and delete objects. Listing is used only to clean up Hot's own multipart staging objects.
gcloud storage buckets create gs://acme-hot-prod \
--location=us-central1 \
--uniform-bucket-level-access \
--public-access-prevention
gcloud storage buckets add-iam-policy-binding gs://acme-hot-prod \
--member=serviceAccount:hot@acme-prod.iam.gserviceaccount.com \
--role=roles/storage.objectUser
Private Networks
From a private network, reach Cloud Storage through
Private Google Access
or a Private Service Connect
endpoint. If you use a Private Service Connect endpoint name or the
restricted.googleapis.com VIP under VPC Service Controls, set
HOT_FILE_GCS_ENDPOINT and HOT_BUILD_GCS_ENDPOINT to that endpoint
(for example https://storage-hotpsc.p.googleapis.com).
Object Layout
Under the configured prefix:
| Objects | Purpose |
|---|---|
{org}/{env}/{path}.v-{id} | File bytes from a regular write |
{org}/{env}/{path}.cas-{id} | File bytes from a conditional write |
{org}/{env}/{path}.mp-{upload} | A completed multipart upload |
{org}/{env}/__hot/blobs/... | Internal content-addressed blobs |
.uploads/{org}/{env}/{upload}/... | Multipart staging (parts, intermediate compositions) |
{org}/{env}/{build}.hot.zip | Build artifacts (build prefix) |
Every file write lands at a new object, and the database record is switched to it as the commit point. A failed or losing write never changes what readers see, and concurrent writers never overwrite each other's bytes. The database, not the bucket, is authoritative for which files exist and who may read them.
The object an overwrite replaces, or a delete removes, is kept for 15 minutes so reads already in progress can finish. The database records it in the same transaction, along with each object a write is about to commit. A maintenance task runs every 5 minutes and deletes the recorded objects that are past those 15 minutes and that no file version references, so a crash or a failed delete only postpones the deletion to its next run. Hot's file ETag is an MD5 of the content. A multipart upload's ETag identifies the exact parts it was composed from (S3 reports its own multipart ETag). File ETags are independent of GCS object generations.
Multipart Uploads
Each uploaded part is stored as its own object, and the part ETag returned by the API identifies that exact object version. Completion composes exactly the parts selected by the manifest, in order, with generation preconditions, so a concurrent re-upload of a part cannot change the bytes being finalized. More than 32 parts are combined in stages, all within the same bucket.
Completion is recoverable. If a completion is interrupted (a crash, a
database error, or a lost response), retrying it on any instance reuses the
object it assembled, or reports success if the file already references it.
Once completed, retries return the saved completion response even if a later
write or deletion changed the path. They never reapply the upload.
Completion only replaces the file version that existed when it began: if the
file was written or deleted in the meantime, nothing is committed, the upload
is aborted, and the request fails with 409 upload_conflict.
While a completion is in progress the upload cannot be aborted, and its parts
can no longer change. A failed completion returns the upload to pending.
Its original file-version guard is preserved across failures and retries.
Starting a completion extends the upload's expiry by an hour, so the expiry
sweep only aborts a completion that stopped making progress. Aborting removes
the staged parts and any assembled object the file does not reference.
Retrying a part upload whose response was lost is safe: the same bytes are
acknowledged again, and different bytes for an already uploaded part are
refused with 409 part_already_uploaded.
Before sending any part bytes to storage, the API reserves that part's content
hash in the database. Backend ETags remain opaque object tokens. Concurrent
requests for different bytes are rejected before they can replace an S3 part;
an identical request still in progress returns 409 part_in_progress and can
be retried after it finishes. A crashed part attempt is reclaimable after ten
minutes, with the same bytes. Each part operation times out after five minutes.
Abort claims also have a one-hour lease. A client retry or the expiry sweep can reclaim an interrupted abort when that lease expires. Cleanup failures release the claim immediately while retaining the abort intent, and retrying an already aborted upload succeeds.
Migration 011 adds durable part reservations and saved completion responses.
Completed uploads from before this migration have no saved response and return
409 completion_result_unavailable when completion is retried. Older Local/S3
parts without reservations can still be retried when their ETag matches the
body's MD5; opaque legacy part tokens cannot establish content equality.
Lifecycle, Retention, and Costs
Staged parts are deleted after completion or abort. If a deletion fails or a process dies mid-upload, staging objects can remain. Add a lifecycle rule scoped to the staging prefix (adjust for your file prefix) as a backstop:
{
"rule": [{
"action": {"type": "Delete"},
"condition": {"age": 7, "matchesPrefix": ["files/.uploads/"]}
}]
}
gcloud storage buckets update gs://acme-hot-prod --lifecycle-file=lifecycle.json
Do not add age- or name-based deletion rules outside .uploads/. Objects
named *.v-*, *.cas-*, and *.mp-* hold committed file data even though
their names look temporary.
Interactions with other bucket features:
- Soft delete (on by default, 7 days) keeps deleted and overwritten objects
restorable and billed for the retention window. Every overwrite of a Hot file
deletes the previous object once its 15 minutes have passed, so write-heavy
workloads such as
::hot::sqliteaccumulate soft-deleted bytes. Lower or disable soft delete if you do not need it. - Object versioning retains noncurrent copies of every replaced or deleted object. Hot does not need it; if enabled, add a lifecycle rule for noncurrent versions.
- Retention policies and holds block deletion. Collection logs an object it cannot delete and retries it on every run, so a locked retention policy makes storage grow until the policy allows deletion. Blob deletes report an error when the object cannot be removed.
Objects waiting for collection do not count against Hot's storage quota (which is computed from database records) but are billed by Google Cloud.
Troubleshooting
| Error | Cause |
|---|---|
file.gcs.bucket (HOT_FILE_GCS_BUCKET) is required | GCS selected without a bucket. |
failed to initialize GCS client (Application Default Credentials) | The credential configuration is invalid, for example GOOGLE_APPLICATION_CREDENTIALS names a missing or malformed file. |
access denied | No credentials were found (Hot falls back to the metadata server, which only exists on Google Cloud; run gcloud auth application-default login locally), the identity lacks roles/storage.objectUser on the bucket, or VPC Service Controls blocked the request. |
transient error | Throttling, timeouts, or server errors persisted through the retry limit. Raise the timeout or retry settings, or lower concurrency. |
built without the gcs-storage feature | The binary does not include GCS support. Use a public release or build with --features hot/gcs-storage. |
is stored in the 's3' backend, but this installation uses 'gcs' | Data written before a backend switch. See below. |
Hot logs bucket names and operations but keeps credentials out of logs. Object keys contain organization and environment IDs and appear only at debug level.
Switching Backends
Changing storage.type does not move data. Existing file records, blob
objects, in-flight uploads, and builds keep pointing at the backend that wrote
them, and Hot refuses to read, overwrite, or clean them up through a different
backend instead of guessing. To switch an existing installation, stop Hot and
move the data with hot db move-storage, then change the configuration:
hot db move-storage --role file --to s3 --bucket acme-hot --prefix files --region us-east-1 \
--manifest file-move.jsonl
hot db move-storage --role build --to gcs --bucket acme-hot --prefix builds \
--manifest build-move.jsonl
It runs with the installation's current configuration, which names the
source, and moves one role at a time: build, or file together with
internal blobs. The target must be another backend: the command refuses --to
the backend the role already uses, so it cannot move objects to another bucket
or prefix of that backend. For a local target, --path names the directory, and
a relative path is made absolute.
On PostgreSQL, a real move refuses to start while a Hot scheduler is running, and
a scheduler started during the move dispatches nothing. It checks only the
scheduler, so stop the other services too. --dry-run can run at any time.
For each object it does three things:
- copies the bytes under the same key below the organization and environment (so a file version keeps its unique name);
- reads the copy back and checks it against the source (SHA-256, or the blob's BLAKE3 hash), and that a file's size matches its record;
- then rewrites that one record.
Files and blobs stream through in 64 MiB parts, so their size is not bound by
memory. On S3 anything larger than one part is a multipart upload, which needs
s3:PutObject and s3:AbortMultipartUpload. On GCS, parts are staged under
.uploads/ and composed. Builds move whole, one at a time. A file whose bytes
do not match its record's MD5 ETag still moves, with a warning, since an ETag is
not always an MD5. On a local target, a name over 255 bytes is shortened, and the
record keeps the result.
Reports
--manifest FILE appends one JSON line per object as it finishes, so a re-run
adds to the same file. A re-run skips what already moved, so that file stays the
only full record: use one per role. --manifest - writes the lines to standard
output instead, and turns off logging that would go there. Each line names the
object's kind, id, org_id, env_id, from, to, bytes, and digest, a
status of moved, planned, skipped, or failed, and an error or
warning when it has one.
--dry-run copies nothing. Each object is planned, so the summary counts what
would move, and --manifest lists it. The summary goes to standard error, with
counts of moved, planned, skipped, failed, and warnings, the bytes
moved, and retired_left.
After a move
The command exits non-zero if any object fails. An interrupted or partly failed
move can run again: moved records are skipped. To leave behind an object that can
never move, because its bytes are gone, say, list its id from the manifest in the
HOT_MOVE_STORAGE_SKIP environment variable, a comma-separated list of file,
blob object, or build ids. A longer list can go in a file named by
--skip-file <FILE> or HOT_MOVE_STORAGE_SKIP_FILE, with one id per line. The file
adds to HOT_MOVE_STORAGE_SKIP, and the flag wins over the variable. Blank lines and
surrounding whitespace are ignored, an empty file skips nothing, and an empty variable
names no file. An unreadable file, or a line that is not an id, refuses the move
before it connects to the database. --dry-run reads the list too, and shows those
objects as skipped.
Hot can't read what stays behind: a file left there can only be deleted, and a build must be deployed again. A skipped blob returns to use when the same content is stored again.
After a successful real move, the command prints the exact settings to set. They must match, including an empty prefix, because Hot finds internal blobs and builds through these settings, not through their records. Running the same command after the switch names any bucket, prefix, or path that differs, and exits non-zero either way.
The source keeps every moved object, and no worker deletes it on either
configuration. Each manifest line's from names the source object, for removing
copies by hand once the move is confirmed. Objects retired before the move stay
on the source too, counted in the summary's retired_left.
Abandoned multipart uploads on the source are marked aborted by the worker's
upload cleanup once the new backend is configured.
To go back after a partial move, point the configuration at the target and run
the move in reverse, for example --to local --path <old path>. Records still on
the source are not listed, and each moved record is copied back.
On Hot's self-hosted AWS and Google Cloud stacks, a single machine's host agent
runs this move when a role's storage changes from the host's disk to a bucket, or
from S3 to Cloud Storage. The apply starts it, in a redeploy of the release already
running: see
Moving builds and files to a bucket.
Any other change is refused before anything stops: back onto the host, to another
bucket or prefix of one backend, or out of Cloud Storage, since the apply that changes
the setting also removes the host's access to an existing bucket. While a move has
finished and the deploy it belongs to has not, a change to another role is refused
before anything runs. Put that setting back, apply, and run
deploy-now to finish the deploy, then move the other role in a deploy of its own.
When such a move fails, what to leave behind goes in the installation's settings.
A setting alone does not retry the move: it takes an apply, then deploy-now.
- A few objects. List their ids in
settings.HOT_MOVE_STORAGE_SKIP, separated by commas. - Everything the last report counted. Set
settings.HOT_MOVE_STORAGE_SKIP_FAILEDto the count in that report,failed.storage_failed.count, however many that is. An outage or bad credentials can fail every object, so check the cause first. Any other count is refused and leaves nothing behind, and so is one with no report behind it. A value that is not a whole number of 1 or more is refused at plan and stops any deploy.
Objects HOT_MOVE_STORAGE_SKIP_FAILED left behind in earlier attempts stay behind for
as long as it's set, and taking it out tries them all again. The deploy that finishes
the move warns how many objects stayed behind, and the move's manifest lists them as
skipped. Remove the setting once the move has finished, because the next storage
move refuses it until it's gone. Removing it redeploys the same release, as a change
to settings does.
settings can't name a skip file: HOT_MOVE_STORAGE_SKIP_FILE is refused, like any
_FILE name, and --skip-file is for running the command by hand.
A scalable installation does not move storage, so keep its builds and files on
the backend, bucket, and prefix they started on: Hot refuses to read records
through another backend, finds builds and internal blobs through these settings,
and existing records name their bucket. A change of storage also replaces every
host, so on AWS and Google Cloud alike the scalable stack's plan refuses another
bucket or prefix, comparing against what it recorded at its first apply. On Google
Cloud it refuses a change of backend too; a scalable AWS installation stays on S3.
Put the setting back. A trailing slash on a prefix does not count as a change. An
installation that has stored nothing, such as one whose first apply failed, can
start again with tofu apply -replace=terraform_data.storage_locations on AWS or
tofu apply -replace=terraform_data.storage_backends on Google Cloud, which
replaces its location record as well. The plan workflow cannot pass -replace, so run it
yourself: see
Scalable installations.
Internal blobs are content-addressed, so storing a blob whose content was already stored by the previous backend writes it to the new backend and moves the blob object there; existing references to it become readable again. Blob cleanup only removes objects in the configured backend.
A blob is kept for as long as something refers to it:
- A call's arguments, return value and flow: until the call is deleted by call retention or a purge.
- Event data, a task's arguments and result, and a run's result or failure: as long as the event, task or run. Hot doesn't delete those rows, so these blobs stay and their storage grows with your history.
- A store value: until it's overwritten or deleted, or its store is cleared or destroyed.
Once nothing refers to a blob, cleanup deletes it after 24 hours.
Fresh installations can select GCS from the start with no migration.
Amazon S3
| Setting | Description |
|---|---|
hot.file.aws.s3.bucket / hot.build.aws.s3.bucket | Bucket name (required) |
hot.file.aws.s3.prefix / hot.build.aws.s3.prefix | Key prefix |
hot.file.aws.s3.region / hot.build.aws.s3.region | Region override |
S3 uses the standard AWS credential chain unless access-key-id and
secret-access-key are set under hot.file.aws or hot.build.aws
(HOT_FILE_AWS_ACCESS_KEY_ID and so on). Set both or neither. Add
session-token (HOT_FILE_AWS_SESSION_TOKEN / HOT_BUILD_AWS_SESSION_TOKEN)
for temporary credentials.
Local Filesystem
| Setting | Environment variable | Default |
|---|---|---|
hot.file.storage.path | HOT_FILE_STORAGE_PATH | ./.hot/file, or $HOT_HOME/file when HOT_HOME is set |
hot.build.storage.path | HOT_BUILD_STORAGE_PATH | .hot/build |
hot build and hot deploy --local write a bundle under .hot/build in the
current directory, whatever the build storage. hot deploy --local then copies it
to build storage, local or not, where workers read it. To deploy a build you made
earlier, run it from the directory that built it.
One file directory can serve several services. A process running as root, as the
self-hosted bundle's task worker does, gives what it writes under the directory to
the directory's owner, so services running as other users can still use it. A
symlink anywhere under such a directory then fails the write, with could not give it to the storage directory's owner.