popoto.fields.append_only¶
popoto.fields.append_only
¶
AppendOnlyMixin — write-once records with no in-place mutation (#560).
A model that composes this mixin accepts exactly one write per Redis key and
refuses every delete. Corrections are expressed as new records that point at
the old ones, never as edits. The provenance journal
(:mod:popoto.recipes.provenance_journal) is the first consumer, but nothing
here is journal-specific: any model that wants write-once semantics composes
it::
class Ledger(AppendOnlyMixin, Model):
entry_id = AutoKeyField()
amount = FloatField()
e = Ledger(amount=1.0).save()
e.amount = 2.0
e.save() # AppendOnlyViolation
e.delete() # AppendOnlyViolation
Ledger.delete_all() # AppendOnlyViolation (routes through instance.delete)
What the guard actually checks¶
EXISTS on self.db_key.redis_key, read from POPOTO_REDIS_DB
directly. Two things this is deliberately not:
- It is not
self._db_contentand notself._saved_field_values. Both are empty in cases the guard must catch._db_contentis empty on a query-loaded instance (base.py's_is_createcheck is EventStream-specific and is not a persisted-ness signal), and_saved_field_valuesis empty on a fresh Python object whose key collides with a stored record — which is exactly the shape a retry or a duplicate ingest takes. - It is never read from a caller-supplied pipeline. An
EXISTSqueued on a pipeline returns thePipelineobject, which is always truthy, so a pipelined guard would refuse every save including the first.
Two refusals the EXISTS check cannot express, both unconditional:
save(migrate_key=True). A key migration makesEXISTSon the new key return 0, so the guard would pass — andModel.save()thenDELETEs the old key. That is a destroy through a supported public kwarg.- A set
obsolete_redis_key, which is the same migration reached by mutating aKeyFieldon an already-saved instance.
The boundary, stated rather than papered over¶
Immutability here is an ORM-layer contract, not a storage guarantee. It
holds against every Python write path in models/base.py — save,
create, get_or_create, update_or_create, both bulk-save sites,
delete, and delete_all (which routes through instance.delete() per
instance). It does not hold against a raw Redis client, and the repo's own
migration cookbook (models/migrations.py) teaches a delete + re-hset
recipe that bypasses it by construction. Redis and Valkey have no per-key
write-once mode; SETNX/HSETNX are the only atomic create-if-absent
primitives and neither covers a multi-field HSET plus the index writes a
Popoto model performs.
Two known TOCTOU shapes, neither claimed as closed:
- Cross-process. Two writers save the same key concurrently; both
EXISTScalls return 0 before eitherHSETlands, so the second silently overwrites the first. Structurally narrowed rather than locked: anAutoKeyFieldidentity means two independent appends cannot collide, so the window is only reachable when a caller supplies an explicit colliding key — a programming error the guard still catches in every non-concurrent case. - Intra-pipeline. Two saves of the same key queued onto one pipeline.
The guard's
EXISTSexecutes immediately againstPOPOTO_REDIS_DBand cannot see a command that is queued but not yet executed, so both pass and the second overwrites. This shape needs no concurrency and is deterministically reproducible.
Closing either shape at the storage layer would mean an HSETNX-based write
path that duplicates Model.save()'s index handling. That is a rabbit hole,
not a follow-up: the boundary is documented instead.
Transfer/export¶
roundtrip_policy = "rebuild" — the mixin owns no Redis state of its own.
on_conflict="overwrite" is unsupported on append-only models:
transfer/import_.py calls instance.save(), so every colliding record
raises :class:~popoto.exceptions.AppendOnlyViolation and is classified
ERRORED. "skip" is the supported conflict mode.
Retention escape hatch¶
:meth:AppendOnlyMixin.hard_delete is a named, greppable classmethod rather
than a skip_append_only=True kwarg (which would require changing
Model.save()'s signature). It exists for retention and erasure, not for
test teardown — popoto.pytest_plugin already flushes the test DB before
every test. It sweeps derived state as well as the record hash; a
hard_delete that left index or chain state behind would be worse than no
escape hatch, because it manufactures orphan index members pointing at a
nonexistent hash.
AppendOnlyMixin
¶
Model mixin enforcing write-once records and refusing deletes.
Compose it ahead of Model (and ahead of any other save-gating mixin
whose work should not run for a write that is going to be refused)::
class JournalEntry(AppendOnlyMixin, NeverRecordMixin, Model):
...
Raises:
| Type | Description |
|---|---|
AppendOnlyViolation
|
From :meth: |
Source code in src/popoto/fields/append_only.py
149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 | |
save(pipeline=None, ignore_errors=False, skip_auto_now=False, update_fields=None, migrate_key=False, skip_write_filter=False, **kwargs)
¶
Persist the record, but only if its Redis key is not already taken.
Runs before Model.save()'s own gates (this mixin is first in the
MRO), so a refused write never reaches the never-record scan, the write
filter, pre_save, or any index. The ordering is MRO-determined and
harmless in the other direction too: the violation message carries only
the Redis key, never content, and content blocked by the firewall is
still never written.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pipeline
|
Optional[Pipeline]
|
Optional Redis pipeline, forwarded unchanged. The existence check is never queued onto it -- see the module docstring. |
None
|
ignore_errors
|
bool
|
Forwarded to |
False
|
skip_auto_now
|
bool
|
Forwarded to |
False
|
update_fields
|
Optional[list[str]]
|
Forwarded to |
None
|
migrate_key
|
bool
|
Always refused. A key migration deletes the old key. |
False
|
skip_write_filter
|
bool
|
Forwarded to |
False
|
**kwargs
|
Any
|
Forwarded to |
{}
|
Returns:
| Type | Description |
|---|---|
Union[Pipeline, int, bool]
|
Whatever |
Union[Pipeline, int, bool]
|
supplied, else a truthy result. |
Raises:
| Type | Description |
|---|---|
AppendOnlyViolation
|
If the key exists, if |
Source code in src/popoto/fields/append_only.py
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 | |
delete(pipeline=None, *args, **kwargs)
¶
Always refuse. Append-only records are closed, never removed.
This also covers Model.delete_all() and bulk_delete(), which
call instance.delete(pipeline=...) per instance.
Raises:
| Type | Description |
|---|---|
AppendOnlyViolation
|
Always. |
Source code in src/popoto/fields/append_only.py
hard_delete(instance, **kwargs)
classmethod
¶
Erase a record and every trace of its own derived state.
Retention/admin only. The scope is exact and is not "every trace of the record anywhere in the keyspace" -- see What survives below.
The deliberate hole in the append-only contract, kept explicit and
greppable so an audit can find every call site. It exists for erasure,
not convenience: POPOTO_NEVER_RECORD_DISABLE=1 is a supported
deployment action, and with the firewall off a keyspace that can only
grow has no way to remove a secret that landed in it.
Sweeps, in order:
- The record hash, the class Set, every
$IndexedF:/$TagF:index Set, and every composite index entry -- by runningModel.delete()itself (reached past this mixin's refusing override via the MRO), so fieldon_deletehooks do the work rather than a second, drifting copy of them. - For each
ValidityFieldon the model: the three interval ZSETs, the record's own field in both chain HASHes, and any{prefix}:open:*pointer still naming it. Steps 1 and 2 overlap by design --ValidityField.on_deletealready does most of this -- because the sweep is the contract here, not an optimization. - The value side of both chain HASHes, which step 1 does not cover:
ValidityField.on_deleteremoves the record as a chain field, but a neighbor's link may still name it as a value (fwdholdsold -> erased). Left behind, that is a dangling link into a record that no longer exists. - Not swept: the model's event stream. If the model composes
EventStreamMixin, the mutation wasXADDed tostream:{_stream_name}, and those entries are retained up to_stream_max_lengthregardless of this call.
What survives, named rather than implied¶
Two things outlive a hard_delete and are not reachable from the
erased record's own derived state:
- An index key whose NAME embeds the erased record's Redis key.
Another record that referenced the erased one by key -- a journal
annotation's
target, say -- owns a$IndexF:{Model}:{field}:{escaped erased key}Set. That Set belongs to the referencing record, not the erased one, so nothing here touches it, and the erased key survives, escaped, inside its name. - Event-stream entries. Any
XADDed mutation carrying the erased record's key (and any_stream_metadata_fields) stays in the stream until trimmed.
Neither carries the record's field values, so an erasure motivated by removing content achieves that; an erasure motivated by removing every occurrence of the record's key does not. Erase the referencing records too, or trim the stream, if that is the requirement.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
instance
|
Any
|
The record to erase. Must be saved. |
required |
**kwargs
|
Any
|
Forwarded to |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the record existed and was removed. |
Source code in src/popoto/fields/append_only.py
279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 | |