Storage

Hot persists three kinds of bytes outside its database:

  • Application files: everything written with ::hot::file, the Files API, and multipart uploads.
  • Internal blobs: large values spilled out of runs, events, and calls. Blobs share the file storage backend but never appear in file listings.
  • Build artifacts: deployed project bundles (*.hot.zip).

Each installation uses one file backend and one build backend. Both default to local disk.

Backendtype valueBuild feature
Local filesystemlocal (default)always available
Amazon S3s3s3-storage
Google Cloud Storagegcsgcs-storage

Public hot releases include both S3 and GCS support. A binary built without a backend's feature fails at startup with a clear error if that backend is selected. Unknown backend names are rejected rather than silently falling back to local storage.

Selecting a Backend

SettingEnvironment variableDefault
hot.file.storage.typeHOT_FILE_STORAGE_TYPElocal
hot.build.storage.typeHOT_BUILD_STORAGE_TYPElocal

Precedence: an environment variable overrides the corresponding Hot configuration value, which overrides the default. Every Hot service (api, app, worker, task-worker, scheduler) must see the same storage configuration and credentials.

Upgrading: releases before GCS support ignored storage.type in Hot configuration and used only the environment variables. Configuration generated by hot conf generate for the api, app, worker, and task-worker roles in those releases sets both types to "s3" by default. If such a deployment ran on local storage, set HOT_FILE_STORAGE_TYPE=local and HOT_BUILD_STORAGE_TYPE=local (or edit those lines) before upgrading; otherwise startup fails because no S3 bucket is configured.

Google Cloud Storage

Configuration

SettingEnvironment variableDefault
hot.file.gcs.bucketHOT_FILE_GCS_BUCKETrequired
hot.file.gcs.prefixHOT_FILE_GCS_PREFIXnone
hot.build.gcs.bucketHOT_BUILD_GCS_BUCKETrequired
hot.build.gcs.prefixHOT_BUILD_GCS_PREFIXnone
hot.<role>.gcs.endpointHOT_<ROLE>_GCS_ENDPOINTGoogle's global endpoint
hot.<role>.gcs.max-concurrent-requestsHOT_<ROLE>_GCS_MAX_CONCURRENT_REQUESTS32
hot.<role>.gcs.request-timeout-secsHOT_<ROLE>_GCS_REQUEST_TIMEOUT_SECS300
hot.<role>.gcs.retry-attemptsHOT_<ROLE>_GCS_RETRY_ATTEMPTS5
hot.<role>.gcs.retry-time-limit-secsHOT_<ROLE>_GCS_RETRY_TIME_LIMIT_SECS120

<role> is file or build. The bucket is a bare name (my-bucket, not gs://my-bucket). Files and builds can use separate buckets, or one bucket with distinct prefixes:

HOT_FILE_STORAGE_TYPE=gcs
HOT_FILE_GCS_BUCKET=acme-hot-prod
HOT_FILE_GCS_PREFIX=files
HOT_BUILD_STORAGE_TYPE=gcs
HOT_BUILD_GCS_BUCKET=acme-hot-prod
HOT_BUILD_GCS_PREFIX=builds

The same settings in hot.hot:

hot.file.storage.type "gcs"
hot.file.gcs.bucket "acme-hot-prod"
hot.file.gcs.prefix "files"
hot.build.storage.type "gcs"
hot.build.gcs.bucket "acme-hot-prod"
hot.build.gcs.prefix "builds"

Hot uses existing buckets. It never creates, configures, or deletes buckets. The concurrency limit covers download bodies as well as request setup. The request timeout covers headers and the entire body, including backpressure from a slow consumer; dropping the reader cancels the download.

Authentication

Hot authenticates with Application Default Credentials. No keys belong in Hot configuration, and Hot never runs gcloud at runtime.

  • On Google Cloud (GKE, Cloud Run, Compute Engine): attach a service account to the workload. On GKE, use Workload Identity Federation for GKE.
  • CI and deployments outside Google Cloud: configure Workload Identity Federation and point GOOGLE_APPLICATION_CREDENTIALS at the generated credential configuration file (it contains no private key).
  • Local development: run gcloud auth application-default login once. This is the only place gcloud is used.

Service account key files work through GOOGLE_APPLICATION_CREDENTIALS but are not required or recommended.

Google's SDK verifies Cloud Storage endpoints against the operating system's certificate store, not the Mozilla roots Hot bundles for its own outbound requests. Container images that run Hot with GCS need a CA bundle, for example the ca-certificates package.

Bucket Setup

Recommended bucket settings:

  • Uniform bucket-level access enabled. Hot never sets object ACLs.
  • Public access prevention enforced. Hot never serves objects publicly; downloads stream through Hot after authorization.
  • Bucket-scoped IAM: grant the Hot service account roles/storage.objectUser on the bucket only, not on the project. Hot needs to create, read, list, compose, and delete objects. Listing is used only to clean up Hot's own multipart staging objects.
gcloud storage buckets create gs://acme-hot-prod \
  --location=us-central1 \
  --uniform-bucket-level-access \
  --public-access-prevention

gcloud storage buckets add-iam-policy-binding gs://acme-hot-prod \
  --member=serviceAccount:hot@acme-prod.iam.gserviceaccount.com \
  --role=roles/storage.objectUser

Private Networks

From a private network, reach Cloud Storage through Private Google Access or a Private Service Connect endpoint. If you use a Private Service Connect endpoint name or the restricted.googleapis.com VIP under VPC Service Controls, set HOT_FILE_GCS_ENDPOINT and HOT_BUILD_GCS_ENDPOINT to that endpoint (for example https://storage-hotpsc.p.googleapis.com).

Object Layout

Under the configured prefix:

ObjectsPurpose
{org}/{env}/{path}.v-{id}File bytes from a regular write
{org}/{env}/{path}.cas-{id}File bytes from a conditional write
{org}/{env}/{path}.mp-{upload}A completed multipart upload
{org}/{env}/__hot/blobs/...Internal content-addressed blobs
.uploads/{org}/{env}/{upload}/...Multipart staging (parts, intermediate compositions)
{org}/{env}/{build}.hot.zipBuild artifacts (build prefix)

Every file write lands at a new object, and the database record is switched to it as the commit point. A failed or losing write never changes what readers see, and concurrent writers never overwrite each other's bytes. The database, not the bucket, is authoritative for which files exist and who may read them.

The object an overwrite replaces, or a delete removes, is kept for 15 minutes so reads already in progress can finish. The database records it in the same transaction, along with each object a write is about to commit. A maintenance task runs every 5 minutes and deletes the recorded objects that are past those 15 minutes and that no file version references, so a crash or a failed delete only postpones the deletion to its next run. Hot's file ETag is an MD5 of the content. A multipart upload's ETag identifies the exact parts it was composed from (S3 reports its own multipart ETag). File ETags are independent of GCS object generations.

Multipart Uploads

Each uploaded part is stored as its own object, and the part ETag returned by the API identifies that exact object version. Completion composes exactly the parts selected by the manifest, in order, with generation preconditions, so a concurrent re-upload of a part cannot change the bytes being finalized. More than 32 parts are combined in stages, all within the same bucket.

Completion is recoverable. If a completion is interrupted (a crash, a database error, or a lost response), retrying it on any instance reuses the object it assembled, or reports success if the file already references it. Once completed, retries return the saved completion response even if a later write or deletion changed the path. They never reapply the upload. Completion only replaces the file version that existed when it began: if the file was written or deleted in the meantime, nothing is committed, the upload is aborted, and the request fails with 409 upload_conflict.

While a completion is in progress the upload cannot be aborted, and its parts can no longer change. A failed completion returns the upload to pending. Its original file-version guard is preserved across failures and retries. Starting a completion extends the upload's expiry by an hour, so the expiry sweep only aborts a completion that stopped making progress. Aborting removes the staged parts and any assembled object the file does not reference. Retrying a part upload whose response was lost is safe: the same bytes are acknowledged again, and different bytes for an already uploaded part are refused with 409 part_already_uploaded. Before sending any part bytes to storage, the API reserves that part's content hash in the database. Backend ETags remain opaque object tokens. Concurrent requests for different bytes are rejected before they can replace an S3 part; an identical request still in progress returns 409 part_in_progress and can be retried after it finishes. A crashed part attempt is reclaimable after ten minutes, with the same bytes. Each part operation times out after five minutes.

Abort claims also have a one-hour lease. A client retry or the expiry sweep can reclaim an interrupted abort when that lease expires. Cleanup failures release the claim immediately while retaining the abort intent, and retrying an already aborted upload succeeds.

Migration 011 adds durable part reservations and saved completion responses. Completed uploads from before this migration have no saved response and return 409 completion_result_unavailable when completion is retried. Older Local/S3 parts without reservations can still be retried when their ETag matches the body's MD5; opaque legacy part tokens cannot establish content equality.

Lifecycle, Retention, and Costs

Staged parts are deleted after completion or abort. If a deletion fails or a process dies mid-upload, staging objects can remain. Add a lifecycle rule scoped to the staging prefix (adjust for your file prefix) as a backstop:

{
  "rule": [{
    "action": {"type": "Delete"},
    "condition": {"age": 7, "matchesPrefix": ["files/.uploads/"]}
  }]
}
gcloud storage buckets update gs://acme-hot-prod --lifecycle-file=lifecycle.json

Do not add age- or name-based deletion rules outside .uploads/. Objects named *.v-*, *.cas-*, and *.mp-* hold committed file data even though their names look temporary.

Interactions with other bucket features:

  • Soft delete (on by default, 7 days) keeps deleted and overwritten objects restorable and billed for the retention window. Every overwrite of a Hot file deletes the previous object once its 15 minutes have passed, so write-heavy workloads such as ::hot::sqlite accumulate soft-deleted bytes. Lower or disable soft delete if you do not need it.
  • Object versioning retains noncurrent copies of every replaced or deleted object. Hot does not need it; if enabled, add a lifecycle rule for noncurrent versions.
  • Retention policies and holds block deletion. Collection logs an object it cannot delete and retries it on every run, so a locked retention policy makes storage grow until the policy allows deletion. Blob deletes report an error when the object cannot be removed.

Objects waiting for collection do not count against Hot's storage quota (which is computed from database records) but are billed by Google Cloud.

Troubleshooting

ErrorCause
file.gcs.bucket (HOT_FILE_GCS_BUCKET) is requiredGCS selected without a bucket.
failed to initialize GCS client (Application Default Credentials)The credential configuration is invalid, for example GOOGLE_APPLICATION_CREDENTIALS names a missing or malformed file.
access deniedNo credentials were found (Hot falls back to the metadata server, which only exists on Google Cloud; run gcloud auth application-default login locally), the identity lacks roles/storage.objectUser on the bucket, or VPC Service Controls blocked the request.
transient errorThrottling, timeouts, or server errors persisted through the retry limit. Raise the timeout or retry settings, or lower concurrency.
built without the gcs-storage featureThe binary does not include GCS support. Use a public release or build with --features hot/gcs-storage.
is stored in the 's3' backend, but this installation uses 'gcs'Data written before a backend switch. See below.

Hot logs bucket names and operations but keeps credentials out of logs. Object keys contain organization and environment IDs and appear only at debug level.

Switching Backends

Changing storage.type does not move data. Existing file records, blob objects, in-flight uploads, and builds keep pointing at the backend that wrote them, and Hot refuses to read, overwrite, or clean them up through a different backend instead of guessing. To switch an existing installation, stop Hot and move the data with hot db move-storage, then change the configuration:

hot db move-storage --role file --to s3 --bucket acme-hot --prefix files --region us-east-1 \
  --manifest file-move.jsonl
hot db move-storage --role build --to gcs --bucket acme-hot --prefix builds \
  --manifest build-move.jsonl

It runs with the installation's current configuration, which names the source, and moves one role at a time: build, or file together with internal blobs. The target must be another backend: the command refuses --to the backend the role already uses, so it cannot move objects to another bucket or prefix of that backend. For a local target, --path names the directory, and a relative path is made absolute.

On PostgreSQL, a real move refuses to start while a Hot scheduler is running, and a scheduler started during the move dispatches nothing. It checks only the scheduler, so stop the other services too. --dry-run can run at any time.

For each object it does three things:

  • copies the bytes under the same key below the organization and environment (so a file version keeps its unique name);
  • reads the copy back and checks it against the source (SHA-256, or the blob's BLAKE3 hash), and that a file's size matches its record;
  • then rewrites that one record.

Files and blobs stream through in 64 MiB parts, so their size is not bound by memory. On S3 anything larger than one part is a multipart upload, which needs s3:PutObject and s3:AbortMultipartUpload. On GCS, parts are staged under .uploads/ and composed. Builds move whole, one at a time. A file whose bytes do not match its record's MD5 ETag still moves, with a warning, since an ETag is not always an MD5. On a local target, a name over 255 bytes is shortened, and the record keeps the result.

Reports

--manifest FILE appends one JSON line per object as it finishes, so a re-run adds to the same file. A re-run skips what already moved, so that file stays the only full record: use one per role. --manifest - writes the lines to standard output instead, and turns off logging that would go there. Each line names the object's kind, id, org_id, env_id, from, to, bytes, and digest, a status of moved, planned, skipped, or failed, and an error or warning when it has one.

--dry-run copies nothing. Each object is planned, so the summary counts what would move, and --manifest lists it. The summary goes to standard error, with counts of moved, planned, skipped, failed, and warnings, the bytes moved, and retired_left.

After a move

The command exits non-zero if any object fails. An interrupted or partly failed move can run again: moved records are skipped. To leave behind an object that can never move, because its bytes are gone, say, list its id from the manifest in the HOT_MOVE_STORAGE_SKIP environment variable, a comma-separated list of file, blob object, or build ids. A longer list can go in a file named by --skip-file <FILE> or HOT_MOVE_STORAGE_SKIP_FILE, with one id per line. The file adds to HOT_MOVE_STORAGE_SKIP, and the flag wins over the variable. Blank lines and surrounding whitespace are ignored, an empty file skips nothing, and an empty variable names no file. An unreadable file, or a line that is not an id, refuses the move before it connects to the database. --dry-run reads the list too, and shows those objects as skipped.

Hot can't read what stays behind: a file left there can only be deleted, and a build must be deployed again. A skipped blob returns to use when the same content is stored again.

After a successful real move, the command prints the exact settings to set. They must match, including an empty prefix, because Hot finds internal blobs and builds through these settings, not through their records. Running the same command after the switch names any bucket, prefix, or path that differs, and exits non-zero either way.

The source keeps every moved object, and no worker deletes it on either configuration. Each manifest line's from names the source object, for removing copies by hand once the move is confirmed. Objects retired before the move stay on the source too, counted in the summary's retired_left. Abandoned multipart uploads on the source are marked aborted by the worker's upload cleanup once the new backend is configured.

To go back after a partial move, point the configuration at the target and run the move in reverse, for example --to local --path <old path>. Records still on the source are not listed, and each moved record is copied back.

On Hot's self-hosted AWS and Google Cloud stacks, a single machine's host agent runs this move when a role's storage changes from the host's disk to a bucket, or from S3 to Cloud Storage. The apply starts it, in a redeploy of the release already running: see Moving builds and files to a bucket. Any other change is refused before anything stops: back onto the host, to another bucket or prefix of one backend, or out of Cloud Storage, since the apply that changes the setting also removes the host's access to an existing bucket. While a move has finished and the deploy it belongs to has not, a change to another role is refused before anything runs. Put that setting back, apply, and run deploy-now to finish the deploy, then move the other role in a deploy of its own.

When such a move fails, what to leave behind goes in the installation's settings. A setting alone does not retry the move: it takes an apply, then deploy-now.

  • A few objects. List their ids in settings.HOT_MOVE_STORAGE_SKIP, separated by commas.
  • Everything the last report counted. Set settings.HOT_MOVE_STORAGE_SKIP_FAILED to the count in that report, failed.storage_failed.count, however many that is. An outage or bad credentials can fail every object, so check the cause first. Any other count is refused and leaves nothing behind, and so is one with no report behind it. A value that is not a whole number of 1 or more is refused at plan and stops any deploy.

Objects HOT_MOVE_STORAGE_SKIP_FAILED left behind in earlier attempts stay behind for as long as it's set, and taking it out tries them all again. The deploy that finishes the move warns how many objects stayed behind, and the move's manifest lists them as skipped. Remove the setting once the move has finished, because the next storage move refuses it until it's gone. Removing it redeploys the same release, as a change to settings does.

settings can't name a skip file: HOT_MOVE_STORAGE_SKIP_FILE is refused, like any _FILE name, and --skip-file is for running the command by hand.

A scalable installation does not move storage, so keep its builds and files on the backend, bucket, and prefix they started on: Hot refuses to read records through another backend, finds builds and internal blobs through these settings, and existing records name their bucket. A change of storage also replaces every host, so on AWS and Google Cloud alike the scalable stack's plan refuses another bucket or prefix, comparing against what it recorded at its first apply. On Google Cloud it refuses a change of backend too; a scalable AWS installation stays on S3. Put the setting back. A trailing slash on a prefix does not count as a change. An installation that has stored nothing, such as one whose first apply failed, can start again with tofu apply -replace=terraform_data.storage_locations on AWS or tofu apply -replace=terraform_data.storage_backends on Google Cloud, which replaces its location record as well. The plan workflow cannot pass -replace, so run it yourself: see Scalable installations.

Internal blobs are content-addressed, so storing a blob whose content was already stored by the previous backend writes it to the new backend and moves the blob object there; existing references to it become readable again. Blob cleanup only removes objects in the configured backend.

A blob is kept for as long as something refers to it:

  • A call's arguments, return value and flow: until the call is deleted by call retention or a purge.
  • Event data, a task's arguments and result, and a run's result or failure: as long as the event, task or run. Hot doesn't delete those rows, so these blobs stay and their storage grows with your history.
  • A store value: until it's overwritten or deleted, or its store is cleared or destroyed.

Once nothing refers to a blob, cleanup deletes it after 24 hours.

Fresh installations can select GCS from the start with no migration.

Amazon S3

SettingDescription
hot.file.aws.s3.bucket / hot.build.aws.s3.bucketBucket name (required)
hot.file.aws.s3.prefix / hot.build.aws.s3.prefixKey prefix
hot.file.aws.s3.region / hot.build.aws.s3.regionRegion override

S3 uses the standard AWS credential chain unless access-key-id and secret-access-key are set under hot.file.aws or hot.build.aws (HOT_FILE_AWS_ACCESS_KEY_ID and so on). Set both or neither. Add session-token (HOT_FILE_AWS_SESSION_TOKEN / HOT_BUILD_AWS_SESSION_TOKEN) for temporary credentials.

Local Filesystem

SettingEnvironment variableDefault
hot.file.storage.pathHOT_FILE_STORAGE_PATH./.hot/file, or $HOT_HOME/file when HOT_HOME is set
hot.build.storage.pathHOT_BUILD_STORAGE_PATH.hot/build

hot build and hot deploy --local write a bundle under .hot/build in the current directory, whatever the build storage. hot deploy --local then copies it to build storage, local or not, where workers read it. To deploy a build you made earlier, run it from the directory that built it.

One file directory can serve several services. A process running as root, as the self-hosted bundle's task worker does, gives what it writes under the directory to the directory's owner, so services running as other users can still use it. A symlink anywhere under such a directory then fails the write, with could not give it to the storage directory's owner.