hot.dev/sanctions / ::sanctions::cache

The list cache in ::hot::store.

Layout: one metadata store (named by sanctions.store, default "sanctions") with a meta:<source> record per list, and one data store per list version, <store>:<source>:<version hash>, holding the index from ::sanctions::index/build: e:<id> records and b:<hh> buckets. A refresh writes the new version's store completely, then switches meta:<source> to it, then destroys the previous version's store, so readers never see a half-written list.

Refreshing: refresh-lists is meant to run from a schedule (daily). Each list is checked cheaply first (OFAC's SHA-256 file listing, the UK ETag, the EU Last-Modified, the UN body hash) and only re-indexed when it changed.

Sizes: about 19,500 OFAC SDN records, 6,400 UK, 6,300 EU, 1,000 UN and a few hundred OFAC Consolidated, roughly 1 KB each, plus 256 buckets per list. Screening never downloads anything.

Values

LIST_NAMES

LIST_NAMES: Map {
  ofac-sdn: "OFAC SDN List",
  ofac-cons: "OFAC Consolidated (non-SDN) List",
  uk: "UK Sanctions List",
  eu: "EU Consolidated Financial Sanctions List",
  un: "UN Security Council Consolidated List"
}

SOURCES

SOURCES: Vec [
  "ofac-sdn",
  "ofac-cons",
  "uk",
  "eu",
  "un"
]

WRITE_BATCH

WRITE_BATCH: Int 500

Functions

check-sources

fn (sources: Any): Vec<Str>

The requested sources (all when null or empty), or an Err naming an unknown one.

data-store-name

fn (store: Str, source: Str, version: Str): Str

The data store name for one version of a list: <store>:<source>:<first 12 hex digits of SHA-256(version)>.

fetch-source

fn (source: Str, known-version: Str?): Map

Download and parse one list: {source, version, entries, unchanged} (see each source namespace's fetch).

is-source

fn (source: Any): Bool

True for a known source key.

meta-of

fn (source: Str): Map?
fn (source: Str, opts: Map): Map?

The cached metadata for one list ({source, list, version, store, entries, keys, fetched-at, fetched-at-ms, checked-at}), or null when it was never refreshed.

meta-store

fn (cfg: Map): Map

now-ms

fn (): Int

open-lists

fn (ms: Map, cfg: Map, wanted: Vec, opts: Map): Map

open-sources

fn (opts: Map): Map

The cached lists to screen: {lists, sources, warnings} where each list is {source, store}. An Err when a requested list was never refreshed (screening it would silently find nothing), unless opts.allow-missing is true.

refresh-lists

fn (): Vec<Map>
fn (opts: Map): Vec<Map>

Refresh every list (or opts.sources), one after another. A list that fails does not stop the others: its row has status: "error" and an error message, and its previous version stays cached. opts.force: true re-indexes even unchanged lists. A failure that halts the run (a store write error, for example) still stops it; the schedule's retry policy covers those.

Meant for a daily schedule; see the README. The EU list needs http.max-response-bytes of at least 32 MiB.

Example

::sanctions::cache/refresh-lists()
::sanctions::cache/refresh-lists({sources: ["ofac-sdn", "un"]})

refresh-source

fn (source: Str): Map
fn (source: Str, opts: Map): Map

Refresh one list: check whether it changed, and if so download, parse, index and store the new version, then drop the old one. Returns {source, status, version, entries, ms} where status is "updated" or "unchanged". Errors (a download failure, an EU file over http.max-response-bytes) are returned as an Err and leave the cached version in place.

refresh-with

fn (source: Str, opts: Map, cfg: Map): Map

stale-warning

fn (m: Map, max-age-days: Int): Str?

status

fn (): Vec<Map>
fn (opts: Map): Vec<Map>

The cache status of every list: a Vec of metadata records, with {source, list, cached: false} for a list never refreshed.

Example

::sanctions::cache/status()

summarize

fn (source: Str, r: Result): Map

write-pairs

fn (store: Map, pairs: Vec): Int