ValidityField and SupersessionProtocol¶

ValidityField gives popoto models a validity axis — a way to say this record stopped being true, since when, and what replaced it — that is orthogonal to everything else in the ORM. DecayingSortedField and ConfidenceField answer "how important is this memory, and how sure am I?"; ValidityField answers the prior question, "is this memory still a member of the corpus at all?" SupersessionProtocol is the write-side vocabulary on top of it: "this new claim replaces whatever was previously believed about (subject, predicate)." Both feed ContextAssembler, which excludes superseded records from default retrieval the moment they close, while filter(validity__as_of=t) keeps them fully queryable for historical replay.

Overview¶

An agent learns "user is on the free plan." Two weeks later it learns "user upgraded to enterprise." Without a validity axis the stale fact keeps its place in every index and only loses ground gradually, through decay — so ContextAssembler can still pack it into context ahead of the correction. ValidityField makes the first fact stop being a member of default retrieval the instant it is closed.

The core split:

Validity decides membership. A record is either a candidate for default retrieval or it isn't — closed, or not yet started.
Decay decides ordering among the valid. Once membership is settled, DecayingSortedField and friends rank what remains.

The two axes compose because neither knows the other's constants: a ValidityField never appears in a decay formula, and a decay rate never gates membership.

from popoto import Model, KeyField, ValidityField, SupersessionProtocol

class Fact(Model):
    fact_id = KeyField()
    validity = ValidityField()

identity = SupersessionProtocol.identity_key("user_42", "subscription_plan")

old = Fact(fact_id="free").save()
SupersessionProtocol.supersede(old, identity_key=identity)      # -> None (first claim, opens only)

new = Fact(fact_id="enterprise").save()
SupersessionProtocol.supersede(new, identity_key=identity)      # -> old's redis_key (closed it)

Fact.query.filter(validity__current=True)        # -> [new]
Fact.query.filter(validity__as_of=two_weeks_ago)  # -> [old]

SupersessionProtocol.chain(new)  # -> [old, new], oldest first

Declaring the field is not enough — exclusion needs a producer

Every SupersessionProtocol call in the example above is load-bearing, including the first one that returns None. A plain .save() only ever opens an interval (valid_from = save time, or a declared future epoch — see below) and never registers the record as the incumbent for any identity. So a model that declares validity = ValidityField() and writes records with .save() alone, with no declared future valid_from, gets two fully populated interval ZSETs and an exclusion set that is permanently empty with respect to the closed-interval (invalid_at) check — no error, no warning, and Redis state that looks like gating is working. The one thing a plain .save() can exclude is itself: a save that declares a future valid_from is excluded until that moment arrives, via the interval-not-yet-started check. Worse, skipping the first supersede() is not a small omission: without it the second claim finds no incumbent, closes nothing, and returns None too.

Closing an interval requires one of: SupersessionProtocol.supersede() / save_and_supersede(), SupersessionProtocol.invalidate() / save_and_invalidate(), ProvenanceJournal, or ObservationProtocol.on_context_used() with the "contradicted" outcome and instance._superseded_by set (see Observation Protocol). All four are explicit application calls. Whether this should stay imperative is issue #693.

Keyspace¶

On Redis, ValidityField owns six keys per model/field, all under the $ValidityF:{Model}:{field} prefix (Postgres keeps the same state on the record's row; see On Postgres). No bytes are written into the model's own hash — chain links live in derived state so an append-only journal can adopt the field unchanged.

Key	Type	Contents
`$ValidityF:{Model}:{field}:valid_from`	ZSET	member = record redis_key, score = valid-from epoch
`$ValidityF:{Model}:{field}:invalid_at`	ZSET	member = record redis_key, score = close epoch, `+inf` when open
`$ValidityF:{Model}:{field}:ingested_at`	ZSET	member = record redis_key, score = ingest (transaction-time) epoch
`$ValidityF:{Model}:{field}:chain:fwd`	HASH	old redis_key → superseding redis_key
`$ValidityF:{Model}:{field}:chain:rev`	HASH	new redis_key → superseded redis_key
`$ValidityF:{Model}:{field}:open:{digest}`	STRING	identity digest → currently-open record's redis_key

+inf is the open-interval sentinel stored as invalid_at's score for any record still believed true. It is native to both Redis and Valkey sorted sets — ZADD stores it, ZSCORE returns the string "inf", ZRANGEBYSCORE "(t" "+inf" includes it, and Lua 5.1's tonumber() parses it via strtod — so no read path needs special-case handling for an open record.

An as-of-t membership test is valid_from <= t AND invalid_at > t: two ZRANGEBYSCOREs intersected client-side, or two ZSCOREs inside Lua.

Export & import¶

ValidityField declares roundtrip_policy = "carry" (see Writing Custom Fields for the protocol, and Export & Import for the user-facing guide). All six derived keys above survive a popoto.transfer round trip:

Carried	How
`valid_from`, `invalid_at`, `ingested_at` scores	exported per record, re-`ZADD`ed after `save()`
`chain:fwd` / `chain:rev` links for the record	exported per record, re-`HSET`
`open:{digest}` pointers naming the record	digests exported per record, re-`SET`

This is not optional bookkeeping. Because no validity byte lives in the model hash, "rebuild" — the Field default — would give every imported record a brand-new open interval, and since all three gating layers are subtractive, a record with no closure is fully retrievable. An export/import would therefore silently resurrect every superseded record. Carrying the open:{digest} pointer matters for the same reason one step later: SupersessionProtocol.supersede(new, identity_key=...) resolves the incumbent only through that pointer, so a round trip that dropped it would leave the identity's next supersession closing nothing while repointing at the newcomer, orphaning the incumbent open forever.

Cost: the digest is opaque and there is no record → identity reverse lookup, so export SCANs {prefix}:open:* once per record to find the pointers aimed at it. Export is an admin-path operation and that cost is accepted deliberately — the same trade on_delete already makes. Save and read paths are untouched.

Restore ordering does not matter: import_state runs after save() and writes plain ZADD/HSET/SET (never the NX forms on_save uses, which would no-op against the fresh interval on_save just seeded), and a chain link whose counterpart has not landed yet reads as a chain end until it does.

`ValidityField` API¶

class Fact(Model):
    fact_id = KeyField()
    validity = ValidityField()

A plain Field, deliberately not a SortedFieldMixin — this is load-bearing, not an oversight. SortedFieldMixin fields can win a query's ordering; validity must never do that, since it decides membership, not priority. As a plain field it also stays out of the reindex/migration loops that iterate sorted fields.

ValidityField.on_save opens an interval automatically at save time using the field's value as valid_from (or save time, if unset). ValidityField.on_delete removes every trace of the record from the six keys — records are normally closed, not deleted, so this only matters for an explicit delete().

Query filters¶

Filter	Semantics
`{field}__current=True`	Records whose interval covers now
`{field}__current=False`	The literal complement of `current=True`: closed records AND records that have not yet started
`{field}__as_of=t`	Records whose interval covers epoch `t`

Fact.query.filter(validity__current=True)
Fact.query.filter(validity__current=False)
Fact.query.filter(validity__as_of=1755000000.0)

A record with no interval is returned by neither current=True nor current=False

{field}__current and {field}__as_of are deliberate, positive queries over valid_from/invalid_at membership. A record that has no entry in either ZSET — because a ValidityField was added to a model after that record was written, and the record has not been re-saved since — makes no claim about its own validity, so it does not satisfy current=True (it isn't provably valid) and it does not satisfy current=False either (that filter's complement is computed over the union of every member with an interval entry, which excludes it too). This is exactly why these filters are deliberate queries and not what gates default retrieval — see "The exclusion rule" below.

Because these are filter() params, using them consumes a filter slot and therefore disables sorted-range limit pushdown on that query. That is expected, and it is precisely why the default retrieval path (decay Lua, composite mask, assembler post-filter) gates server-side instead of by appending a filter kwarg: filter(limit=N, order_by=<sorted field>) pushdown stays active with validity gating enabled, because gating is never a filter param on the default path.

The exclusion rule¶

All gating in this feature is subtractive. Every layer — the decay-Lua gate, the composite mask, and the assembler post-filter — asks "is this record provably closed or provably not yet started?" and drops it only on a yes. A record with no entry in either interval ZSET is unmanaged and stays fully retrievable everywhere.

This is what makes adding a ValidityField to an existing model safe. Every record written before the field existed has no valid_from/invalid_at entry until it is next saved (or explicitly supersedes/is superseded). Under a subtractive rule those records keep showing up in retrieval exactly as before. Under an inclusive rule (a whitelist of provably-valid keys) they would all silently vanish the moment gating turned on — a data-visibility regression dressed up as "stricter" behavior.

ValidityField.resolve_valid_keys is a whitelist — do not use it for gating

ValidityField.resolve_valid_keys(model, field_name, as_of=t) intersects valid_from <= t with invalid_at > t and returns the records that positively claim validity at t. That is the opposite selection from every gating layer, which computes an exclusion set instead. Retained as a public helper for callers that genuinely want "which records claim validity right now" — audit and provenance tooling — never for retrieval gating. Passing its result to a whitelist-style filter would hide every unmanaged record. The gating call sites are QueryBuilder._apply_validity_mask (composite path) and ContextAssembler._resolve_excluded_keys (assembler path); both compute exclusion sets, not whitelists.

Its subtractive counterpart is ValidityField.resolve_excluded_keys(model, field_name, as_of=None), which sits directly beside it and returns the exclusion set: the union of invalid_at <= as_of (already closed) and valid_from > as_of (not yet started), as a set[str] of Redis keys, in two read-only ZRANGEBYSCOREs. A record absent from both interval ZSETs appears in neither result and stays retrievable. When you want gating, this is the method — resolve_valid_keys is not the one to reach for.

`SupersessionProtocol`¶

SupersessionProtocol is a stateless coordinator of @staticmethods — never a mixin, never instantiated — mirroring ObservationProtocol's shape.

Identity normalization¶

SupersessionProtocol.identity_key(subject, predicate)

Casefolds and strips each component, collapses internal whitespace, joins the two with a \x00 separator, and hashes the result with blake2b(digest_size=8) into 16 lowercase hex characters. The \x00 join prevents delimiter-collision false merges (("ab", "c") cannot collide with ("a", "bc")); the digest keeps raw user text out of the Redis keyspace. Deterministic and LLM-free by design — semantic identity normalization ("is plan the same predicate as subscription_tier?") is a downstream, opt-in concern, not core.

Mutations¶

SupersessionProtocol.supersede(new_instance, *, identity_key=..., at=None)
SupersessionProtocol.invalidate(instance, at=None, superseded_by=None)

supersede() closes whichever record is currently open for identity_key, chains it to new_instance, and repoints the identity's open pointer — one atomic EVAL (SUPERSEDE_LUA). The first claim about a new identity simply opens and writes no chain link, returning None. invalidate() is the direct, identity-free form: close one specific record, optionally chaining it to whatever replaced it.

Both route through ValidityField.execute_supersede, the single seam that knows SUPERSEDE_LUA's KEYS/ARGV order. Key mutation properties, all enforced inside the one script:

Closed, never deleted. Superseding a record closes its interval; the record and its chain links survive for provenance and as_of replay.
Atomic. Interval closure, chain-link writes, and open-pointer repoint happen in one EVAL. There is no observable state where a record is interval-closed but still index-visible, or closed but unchained.
Idempotent under retry. A ZSCORE != +inf guard refuses to re-close an already-closed record, so two writers racing the same identity serialize into a two-link chain rather than forking.
Membership is decided inside the script, at the instant of the write. SUPERSEDE_LUA runs an EXISTS against each member the caller named, in the same atomic script invocation that writes. A member that does not exist raises ValidityMemberAbsentError and nothing at all is written — the script's validation phase returns before its first write command, so no partial index state (no valid_from entry, no chain link, no pointer) is possible.

Deciding it client-side, during key resolution, would answer the right question at a moment when the answer cannot stay true: the write it guards happens later, inside the script, at EXEC time. See Same-transaction successor below for the shape that difference makes possible.

Same-transaction successor¶

Append the new claim and close the old claim's validity interval in a single MULTI/EXEC, so no reader ever sees both open or neither present:

pipe = popoto.batch()
new.save(pipeline=pipe)
SupersessionProtocol.invalidate(old, superseded_by=new, pipeline=pipe)
pipe.execute()

This is the recommended spelling, and it works because the membership guard is inside the script. new.save(pipeline=pipe) only queues the successor's HSET; a probe running at call time would see no such key and have to treat the successor as unsaved. The guard runs instead when the script's body executes inside MULTI, by which point the HSET has applied.

For the same shape with the save included, and with a typed error instead of a raw ResponseError, use the combined entry point:

result = SupersessionProtocol.save_and_supersede(
    new, identity_key=("user_42", "plan")
)
result.closed_key  # the superseded record's Redis key, or None

# The identity-free form, naming the incumbent explicitly:
SupersessionProtocol.save_and_invalidate(new, closes=old)

Both own their pipeline by default, so the save and the close are atomic by construction. Pass pipeline= to compose onto a caller pipeline instead; the returned SupersedeResult then reports closed_key=None — meaning unknown until you execute, not "nothing was closed" — plus a close_index you read out of your own execute() results.

Typed errors¶

Every SUPERSEDE_LUA failure has exactly one Python counterpart. All three subclass ValidityError, which subclasses ValueError — so existing except (TypeError, ValueError) handlers and pytest.raises(ValueError) assertions keep their meaning.

Lua reply token	Exception	Meaning
`POPOTO_VALIDITY_MEMBER_ABSENT <role> <key>`	`ValidityMemberAbsentError`	The successor, or an explicitly-named incumbent, did not exist at write time
`POPOTO_VALIDITY_CLOSE_BEFORE_START`	`ValidityCloseBeforeStartError`	The close instant precedes the record's own stored `valid_from`
`POPOTO_VALIDITY_VALID_FROM_CONFLICT <stored> <requested>`	`ValidityValidFromConflictError`	An asserted `valid_from` disagrees with the stored start

An incumbent resolved from the open-claim pointer is a hint, not an assertion: a pointer left naming a hard-deleted record reads as "no incumbent", the same way chain() reads a dangling link. Only an incumbent the caller named explicitly raises. That is what keeps a partial import_state transfer supersedable.

Pipeline-mode caveat (Redis). The ResponseError -> typed-exception remap lives on execute_supersede's non-pipeline branch. On a caller-supplied pipeline, redis-py raises during pipe.execute() result parsing — long after execute_supersede returned — so you get a raw redis.exceptions.ResponseError. save_and_supersede / save_and_invalidate own their execute() and are the supported way to get a typed error in pipeline shape.

SupersedeDeclinedError is the fourth typed error and the one exception to the ValidityError family above: it is a RuntimeError subclass, because it reports a broken invariant inside the write path rather than a malformed argument or a Lua reply token. save_and_supersede / save_and_invalidate raise it when the successor's save() was declined — by the never-record firewall or the write filter, both of which return instead of raising. Two shapes are checked: a falsy return, and the truthy pipeline-mode return with a _never_record_verdict stamped on the instance. The check sits between the save and the close, so nothing is queued behind a record that was never written, and the error carries the content-free verdict for callers that need the reason code. verdict is itself None on a non-firewall decline, so guard the attribute access rather than the value:

try:
    SupersessionProtocol.save_and_invalidate(new, closes=old, pipeline=pipe)
except SupersedeDeclinedError as exc:
    reason = exc.verdict.reason if exc.verdict else None  # e.g. 'payment_card'

Valid-time has one writer¶

The field value at construction is the single authoritative writer of valid-time:

Fact(fact_id="plan-2", validity=event_time).save()

at= on supersede() / invalidate() is a close-time assertion about the incumbent, never a start-time assertion about the successor. Asserting it for the successor would raise on every ordinary supersede, since the successor's stored start is its own save clock.

Two readings of "valid from" exist and they differ legitimately:

instance.validity is the declared value. None means "not declared, defaulted to the save clock".
ValidityField.get_valid_from(Model, "validity", member_key=...) is the effective value — the index score, which is what every as_of query answers against.

A re-save that declares a start disagreeing with the stored one now raises ValidityValidFromConflictError rather than losing silently to the script's ZADD NX. The check runs from a Field.pre_save_validate hook dispatched from a single site in Model.save(), above the partial/full split and above both eager indexed-field phases, so a rejected save writes nothing at all: not the model hash, not a sibling IndexedField's index entry, not an interval. On the external-pipeline path it queues nothing onto the caller's pipeline for that call.

Capture-time valid_from now has a producer. Reference Resolution (M4) computes a valid_from epoch for a candidate whose single relative_time reference resolves to an onset (see its onset rule) and threads it through ProvenanceJournal.append(..., at=resolution.valid_from) as the field value at construction — the ordinary declared-value path above, not a reconciliation call. A capture never conflicts on an interval this way: every ProvenanceJournal.append() mints a fresh JournalEntry keyed by its own AutoKeyField entry_id, so each resolution's valid_from opens its own interval rather than re-asserting one against an existing record. ValidityValidFromConflictError only fires on a re-save of the same key with a disagreeing declared start; M4 never re-saves and never calls get_valid_from to reconcile — it only ever constructs new entries.

Reconciling a record that already diverges¶

There is no data migration — no stored key, score, or hash field changes shape, and no backfill runs. But a record that already carries a hash/index divergence will refuse a full save() until an operator reconciles it. A partial save of an unrelated column is unaffected, because the dispatch is scoped to update_fields.

# Adopt the index -- the value every as_of query already answers against:
effective = ValidityField.get_valid_from(
    Fact, "validity", member_key=obj.db_key.redis_key
)
obj.validity = effective
obj.save()

# Or make the declared value authoritative (Redis) -- plain ZADD, no NX, so
# it overwrites the score ZADD NX refused to update:
vf_key, _ = ValidityField.get_interval_keys(Fact, "validity")
popoto.get_redis().zadd(vf_key, {obj.db_key.redis_key: float(obj.validity)})
obj.save()

Bidirectional chain traversal¶

SupersessionProtocol.superseded_by(instance)  # one hop forward, or None at the head
SupersessionProtocol.supersedes(instance)     # one hop backward, or None at the tail
SupersessionProtocol.chain(instance)          # full chain, oldest first, from any anchor

chain() walks backward to the oldest ancestor and forward to the newest descendant, so it is recoverable from any member, not just an endpoint. Traversal terminates on a cycle (a seen set) and on a dangling link — a chain HASH entry naming a record whose valid_from entry no longer exists, which happens when on_delete has scrubbed that record's own chain fields but a neighbor's link still names it as a value.

chain() returns [] for an unsaved instance. That guard lives in chain() itself, using the same ZSCORE-on-valid_from rule the walk already applies to a dangling link — an anchor and a link are judged by one criterion. It is deliberately not inherited from key resolution: key resolution issues no Redis command at all, so that the mutation paths pay one fewer round trip and the membership decision stays where it belongs, inside the script. Reads never raise; superseded_by() / supersedes() on an unsaved instance still return None.

Three gating layers¶

ContextAssembler never calls top_by_decay — every retrieval call is composite_score or fuse, and the BM25 and graph-propagation arms bypass the filters dict entirely. That single fact is why validity gating is not one mechanism but three, each with a distinct, non-overlapping job:

Layer 1 — decay-Lua gate. DECAY_SCORE_LUA grows KEYS[3] (invalid_at), KEYS[4] (valid_from), and ARGV[7] (as-of). Before the base-score HGET and before any decay math, the script decides inclusion: skip if invalid_at <= as_of (closed) or valid_from > as_of (not yet started). It answers that question one of two ways, chosen at run time — see "What gating costs" below — and the two answers are identical by construction. Every KEYS[n] read is guarded KEYS[n] or '', so a caller that passes a short numkeys (existing hand-eval test call sites included) gets nil → '' → gate disabled, with byte-identical scores to the pre-#580 script. This layer is authoritative for top_by_decay on a plain DecayingSortedField, whose result is the member list itself with no later union — the one path where "skip in the range read" is "excluded from the result." It does not cover top_by_decay on a CyclicDecayField, which runs CYCLIC_DECAY_LUA instead; that script was deliberately left ungated — see "Known limitations".

Layer 2 — the composite_score mask (QueryBuilder._apply_validity_mask). composite_score merges its per-index temp ZSETs with ZUNIONSTORE ... AGGREGATE SUM. A member the decay Lua skips is merely absent from the decay arm — under SUM its decay contribution becomes 0, but it can still surface in the composite result on the strength of any other weighted arm (a ConfidenceField index, co_occurrence_boost, similarity_boost). Skipping is not excluding. After the union, _apply_validity_mask runs four core commands — ZRANGESTORE the closed set, ZRANGESTORE the not-yet-started set, union them into an exclusion set, and ZDIFFSTORE that exclusion set out of the composite key — which is what actually enforces membership on this path, and the only layer that reaches a bare Model.query.composite_score() call outside the assembler.

Layer 3 — the assembler post-filter (ContextAssembler._resolve_excluded_keys / _scope_by_validity). Covers the fuse, BM25, and graph-propagation arms, none of which route through composite_score's ZUNIONSTORE or consult the filters dict at all. Two read-only ZRANGEBYSCOREs per assemble() call produce an exclusion set; _scope_by_validity drops any candidate record whose key is in it. Mirrors the tag-scoping pattern (_scope_by_tags) already established for issue #492. Since #648 those two reads live in ValidityField.resolve_excluded_keys (see "The exclusion rule"); the assembler method keeps only what is assembler policy — whether the model has a validity axis at all, and the kill-switch check.

No layer is load-bearing for a path another layer already covers — Layer 1 is the only mechanism for top_by_decay on a plain DecayingSortedField; Layer 2 is the only one that enforces membership on the composite path; Layer 3 is the only one that reaches fuse/BM25/graph. The one path no layer covers is a direct top_by_decay on a CyclicDecayField — see "Known limitations".

What gating costs¶

Gating a top_by_decay at 20k records measures at 0.94x ungated retrieval — parity, within jitter (Python 3.12.14, redis-py 8.1.0, Redis DB 8, Darwin 25.6.0 / Apple silicon; p50 of 21 samples, ungated 37.70 ms / gated 35.59 ms). tests/test_validity_field.py::TestValidityBenchmark pins the ceiling at 2.0x against a same-process, same-data ungated control.

Read that as a ratio, never as a wall-clock budget. DECAY_SCORE_LUA full-scans its partition to rank it, so the ungated cost at 20k is already tens of milliseconds and dominates whatever the gate adds; an absolute budget would be measuring the scan. The V0 plan originally stated one ("within 1 ms of ungated"), and #585 withdrew it for that reason.

The parity comes from pre-trimming. The naive gate costs up to two ZSCORE round trips per scanned member — 40,000 extra server-side dispatches at 20k. Instead, the script reads the excluded members up front with two ZRANGEBYSCOREs and builds a Lua lookup table, making the per-member decision O(1) with no round trip.

That is not always the cheaper shape, and the script checks before committing to it. The interval ZSETs are model+field scoped while the scan is one partition, so a small hot partition sitting beside a large archive of closed records makes the range read pull far more than the scan touches — measured at 16x ungated before the guard existed. Two ZCOUNTs bound it: pre-trim runs only while

ZCOUNT(closed) + ZCOUNT(not-yet-started) <= ZCARD(partition) * Defaults.VALIDITY_GATE_PRETRIM_MAX_RATIO

VALIDITY_GATE_PRETRIM_MAX_RATIO defaults to 4.0 (measured crossover is ~5x, flat across partition sizes). Past it, the script falls back to the per-member path. Setting it to 0 or less disables pre-trim entirely and restores the pre-#585 per-member path byte-for-byte; like every other switch on this path it is read from Defaults at call time, never captured at import, so it is flippable at deploy time by an adopter who cannot edit model code. It is a latency knob, never a correctness one — both branches return identical replies, asserted by tests/test_validity_field.py::TestValidityPretrim.

Layers 2 and 3 are already sub-scan and unchanged: four core commands per composite_score call and two ZRANGEBYSCOREs per assemble(), neither per-member.

Point-in-time reconstruction¶

result = ContextAssembler(Fact).assemble(query_cues={"topic": "plan"}, as_of=two_weeks_ago)

assemble() gains a keyword-only as_of: float | None = None. The default None means "now" — only currently-valid records. Passing an epoch reconstructs what the agent believed at that instant, superseded records included, applied consistently across all three gating layers (the same instant is threaded through the decay-Lua gate, the composite mask, and the post-filter). composite_score and top_by_decay accept the same keyword-only as_of directly, for callers that bypass the assembler.

A model with no ValidityField — every shipped model today — makes as_of and all three gating layers a pure passthrough; retrieval stays byte-identical to pre-#580 behavior.

Kill switch¶

from popoto.fields.constants import Defaults
Defaults.VALIDITY_GATING_ENABLED = False  # restores byte-identical pre-#580 retrieval

Defaults.VALIDITY_GATING_ENABLED is a deploy-level boolean, default True, read at call time in every gating layer — never captured at import — so it takes effect at runtime for adopters who cannot edit model code. With it off, interval and chain maintenance still runs (the six keys stay correct), but no retrieval path consults them: filter(validity__current=...) and filter(validity__as_of=...) still work, since those are deliberate queries this switch does not govern. The blast radius of leaving it on by default is zero until a model actually declares a ValidityField — no shipped model, including DefaultMemory, does.

Known limitations¶

A direct Model.query.top_by_decay() on a CyclicDecayField is not gated. #580 extended DECAY_SCORE_LUA only; CYCLIC_DECAY_LUA was deliberately left unmodified (an explicit plan No-Go), so it takes no invalid_at/valid_from KEYS and no as-of ARGV. A top_by_decay call that resolves to a CyclicDecayField therefore receives gating from none of the three layers: Layer 1 never reaches the cyclic script, Layer 2 only acts after composite_score's union, and Layer 3 only runs inside ContextAssembler. A superseded record will be returned by such a call.

Scope: this is a docs/direct-caller gap, not a live retrieval bug. ContextAssembler never calls top_by_decay; its push path uses composite_score (Layer 2), and every candidate it assembles is post-filtered by _scope_by_validity (Layer 3), including results from its cyclic decay proxy. composite_score on a CyclicDecayField is likewise covered by Layer 2's mask. Only a caller reaching past the assembler straight to top_by_decay on a cyclic field sees the gap; such a caller should use composite_score, or intersect with filter(validity__current=True). Pinned by tests/test_validity_field.py::TestCyclicDecayGatingGap, which fails loudly if the gate is ever added — update this entry then. The Postgres backend reproduces the gap (#759 M5): its cyclic ranking expression accepts the validity field and ignores it, and the same test runs on both conformance legs.

Where the gap lives changed in #662, though the gap itself did not. top_by_decay no longer branches on field type to pick a script; it passes the gate triple to field.rank_decayed(...) unconditionally, and CyclicDecayField.rank_decayed accepts validity= and drops it. So the omission is now a documented property of the cyclic override rather than of a dispatch branch in the query builder — read it there, and in the comment above the call site in models/query.py. - DECAY_SCORE_LUA full-scans its partition regardless of gating (a pre-existing property, not introduced here) — the cost scales with partition size, not with how many records are actually closed. Since #585 the gate itself is no longer the expensive part: see What gating costs above for the measured 0.94x figure and the pre-trim mechanism that buys it. The fallback path — reached when the exclusion sets are more than VALIDITY_GATE_PRETRIM_MAX_RATIO times the partition, or when pre-trim is disabled — still costs up to two ZSCOREs per scanned member, which is what the pre-#585 ~1.4x measurement (~37ms ungated / ~51ms gated on a 20k-record partition) reflected. - The TTL warning fires on first save, not at model-definition time. ValidityField.warn_if_ttl logs once per (model, field) pair the first time a record on a Meta.ttl-bearing model is saved, not when the class body executes — so the warning is observable in test output and logs, not at import time. - A TTL on a ValidityField-bearing model truncates chains and breaks as_of correctness. Redis expires the record's hash on its own schedule; the record's interval and chain-link entries do not expire with it, so a chain walk or an as_of reconstruction can reference a record that no longer exists. The warning is deliberately advisory, not a raised exception — refusing outright would break adopters who legitimately want bounded history.

On Postgres¶

On a Postgres-bound model the interval and the chain live on the record's own row (<f>__valid_from, <f>__invalid_at, <f>__ingested_at as double precision epoch seconds, 'Infinity' = open; <f>__supersedes / <f>__superseded_by), and the open-claim pointer is a companion table whose member references the record ON DELETE CASCADE. supersede is SUPERSEDE_LUA phase for phase in one transaction, behind a per-(model, field) advisory lock, and raises the same typed errors with the same text. The exclusion rule, the as_of filters, the kill switch and the cyclic-decay gap all behave as on Redis, and every gate is one WHERE term on the row. See Validity and supersession (M3).

What differs (each a row of Records and other behaviour):

save_and_supersede / save_and_invalidate whose close fails roll the whole unit back, so the successor is not saved either; on Redis MULTI/EXEC keeps it.
A pointer cannot name a record that does not exist: import_state for an unstored record raises ValidityMemberAbsentError, and execute_supersede(mode="open") on a member with no record writes nothing.
With the backend's transaction() as pipeline, the closed key is returned and a typed error raised at the call, so the pipeline-mode caveat above does not apply; a Redis pipeline is refused with ValueError.
A NaN valid_from on save is refused before anything is written, with ModelException; a NaN as_of in a range read raises QueryException where Redis raises ResponseError, with the same text.
On a Meta.ttl model the interval and pointer expire with the row: a chain walk stops at an expired record as at a hard delete, rather than reaching entries that outlive it. The TTL warning still fires.

Redis-only machinery does not apply: the pre-trim and its Defaults.VALIDITY_GATE_PRETRIM_MAX_RATIO (read only by the Redis DECAY_SCORE_LUA path), the "What gating costs" figures, and the raw ZADD reconciliation above. To adopt the effective start on Postgres, use the get_valid_from form.