Skip to content

popoto.backends.types

popoto.backends.types

Backend-neutral types that cross the model-level storage protocol (#759).

Everything here is plain data: no redis, no psycopg, no network. The shapes are the plan's §2 (docs/plans/sdlc-631-v2.md); where an implementation detail had to be pinned that §2 leaves open, the docstring says so and why.

Values cross the protocol decoded -- no msgpack, key strings or index names -- with one deliberate exception on the Redis side, :class:RecordId.native, which carries the caller's own key object so a bytes key reaches the Redis wire as the same token it always did (the #751 str boundary lesson).

FieldKind = str module-attribute

The name of the popoto field class a field is (or, for a user subclass, the nearest popoto class it derives from): "KeyField", "SortedField", "Field"... A closed enum would have to grow with every field module; the backends' capability tables key on these names instead (validate_spec).

Row = Mapping[str, Any] module-attribute

Decoded Python values; always carries "_id" (a :class:RecordId, or None for a Redis projection row, which has never carried its key).

BackendError

Bases: PopotoException

Base class for every error the storage-backend layer raises.

PopotoException logs its message but does not pass it to Exception, so str(exc) would be empty; this class keeps it.

Source code in src/popoto/backends/types.py 
56
57
58
59
60
61
62
63
64
65
class BackendError(PopotoException):
    """Base class for every error the storage-backend layer raises.

    ``PopotoException`` logs its message but does not pass it to
    ``Exception``, so ``str(exc)`` would be empty; this class keeps it.
    """

    def __init__(self, message: Any) -> None:
        super().__init__(message)
        self.args = (message,)

BackendCapabilityError

Bases: BackendError, NotImplementedError

The selected backend cannot do this.

Raised for a protocol method a backend does not implement (on a non-Redis backend, groups D-H until their milestone), for a field the backend's static capability table refuses (:func:popoto.backends.validate_spec), and for a [PG-only] capability called on a Redis-bound model. NotImplementedError is a base so except NotImplementedError keeps catching the "not on this backend" case.

Source code in src/popoto/backends/types.py 
68
69
70
71
72
73
74
75
76
77
class BackendCapabilityError(BackendError, NotImplementedError):
    """The selected backend cannot do this.

    Raised for a protocol method a backend does not implement (on a non-Redis
    backend, groups D-H until their milestone), for a field the backend's
    static capability table refuses (:func:`popoto.backends.validate_spec`),
    and for a ``[PG-only]`` capability called on a Redis-bound model.
    ``NotImplementedError`` is a base so ``except NotImplementedError`` keeps
    catching the "not on this backend" case.
    """

BackendUnavailableError

Bases: BackendError, ConnectionError

The backend cannot be reached, or is not installed or configured.

The popoto-level outage type both backends share (plan §1.1, M1): an unreachable server or a connect/statement timeout on Postgres, and selecting postgres without POPOTO_POSTGRES_URL or the postgres extra.

Source code in src/popoto/backends/types.py 
80
81
82
83
84
85
86
87
class BackendUnavailableError(BackendError, ConnectionError):
    """The backend cannot be reached, or is not installed or configured.

    The popoto-level outage type both backends share (plan §1.1, M1): an
    unreachable server or a connect/statement timeout on Postgres, and
    selecting ``postgres`` without ``POPOTO_POSTGRES_URL`` or the ``postgres``
    extra.
    """

SchemaDriftError

Bases: BackendError

The stored schema and the model disagree in a way popoto will not reconcile automatically (plan §3, Migrations). Raised by bind().

Source code in src/popoto/backends/types.py 
90
91
92
class SchemaDriftError(BackendError):
    """The stored schema and the model disagree in a way popoto will not
    reconcile automatically (plan §3, Migrations). Raised by ``bind()``."""

BackendRetryableError

Bases: BackendError

A deadlock or serialization failure rolled the work back; running it again is safe (plan §6, TD-2), so a caller catches one popoto type instead of a driver error (#759 M2a). Raised from the driver error:

  • inside a caller-owned transaction() (a statement in it, or its COMMIT), at once -- the whole unit was rolled back and only the caller can run its block again, so popoto does not retry it;
  • by a single statement outside a unit of work, and by ObservationProtocol.on_context_used, once their own bounded retries (Defaults.PG_TRANSACTION_RETRIES) are spent.

Before #773's patch a caller-owned transaction saw the raw psycopg DeadlockDetected / SerializationFailure; code that caught those should catch this instead. A statement whose completion is unknown (40003) is not this error: it may have committed (#769).

Source code in src/popoto/backends/types.py 
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
class BackendRetryableError(BackendError):
    """A deadlock or serialization failure rolled the work back; running it
    again is safe (plan §6, TD-2), so a caller catches one popoto type instead
    of a driver error (#759 M2a). Raised ``from`` the driver error:

    - inside a caller-owned ``transaction()`` (a statement in it, or its
      COMMIT), at once -- the whole unit was rolled back and only the caller
      can run its block again, so popoto does not retry it;
    - by a single statement outside a unit of work, and by
      ``ObservationProtocol.on_context_used``, once their own bounded retries
      (``Defaults.PG_TRANSACTION_RETRIES``) are spent.

    Before #773's patch a caller-owned transaction saw the raw psycopg
    ``DeadlockDetected`` / ``SerializationFailure``; code that caught those
    should catch this instead. A statement whose completion is unknown
    (40003) is not this error: it may have committed (#769)."""

RecordId dataclass

The identity of one record, backend-neutral.

canonical is today's DB_key string -- Model.pk on both backends, and the Postgres _pk column (plan §3). values are the KeyField values in DB_key order (KeyField names sorted, as :meth:ModelOptions.get_db_key_index_position numbers them); it is () when the id was built from a key string that was never decomposed, which no backend needs for a lookup by canonical.

native is not part of identity (compare=False): it is the key object the caller held -- str or the bytes a raw Redis reply produced -- and the Redis backend sends it unchanged, so routing a call through the protocol never changes the token on the wire or the type a caller gets back.

Source code in src/popoto/backends/types.py 
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
@dataclass(frozen=True)
class RecordId:
    """The identity of one record, backend-neutral.

    ``canonical`` is today's ``DB_key`` string -- ``Model.pk`` on both backends,
    and the Postgres ``_pk`` column (plan §3). ``values`` are the KeyField
    values in ``DB_key`` order (KeyField names sorted, as
    :meth:`ModelOptions.get_db_key_index_position` numbers them); it is ``()``
    when the id was built from a key string that was never decomposed, which no
    backend needs for a lookup by ``canonical``.

    ``native`` is not part of identity (``compare=False``): it is the key object
    the caller held -- ``str`` or the ``bytes`` a raw Redis reply produced -- and
    the Redis backend sends it unchanged, so routing a call through the protocol
    never changes the token on the wire or the type a caller gets back.
    """

    model: str
    values: tuple[Any, ...]
    canonical: str
    native: Any = field(default=None, compare=False, repr=False)

    @classmethod
    def from_key(cls, model: str, key: Any, values: tuple[Any, ...] = ()) -> RecordId:
        """Build an id from a key the caller holds (``str``, ``bytes`` or a
        ``DB_key``), keeping that object as :attr:`native`."""
        if isinstance(key, bytes):
            canonical = key.decode("utf-8", errors="surrogateescape")
        elif isinstance(key, str):
            canonical = key
        else:  # DB_key and friends render themselves
            canonical = str(getattr(key, "redis_key", key))
        return cls(model=model, values=values, canonical=canonical, native=key)

    @property
    def key(self) -> Any:
        """What the Redis backend sends: the caller's object, else the string."""
        return self.native if self.native is not None else self.canonical

key property

What the Redis backend sends: the caller's object, else the string.

from_key(model, key, values=()) classmethod

Build an id from a key the caller holds (str, bytes or a DB_key), keeping that object as :attr:native.

Source code in src/popoto/backends/types.py 
138
139
140
141
142
143
144
145
146
147
148
@classmethod
def from_key(cls, model: str, key: Any, values: tuple[Any, ...] = ()) -> RecordId:
    """Build an id from a key the caller holds (``str``, ``bytes`` or a
    ``DB_key``), keeping that object as :attr:`native`."""
    if isinstance(key, bytes):
        canonical = key.decode("utf-8", errors="surrogateescape")
    elif isinstance(key, str):
        canonical = key
    else:  # DB_key and friends render themselves
        canonical = str(getattr(key, "redis_key", key))
    return cls(model=model, values=values, canonical=canonical, native=key)

FieldSpec dataclass

One field, as a backend sees it. options carries what a backend may need beyond the type (partition_by, max_length, auto...) and, for a user-defined subclass, custom_class and overrides_hooks.

Source code in src/popoto/backends/types.py 
165
166
167
168
169
170
171
172
173
174
175
@dataclass(frozen=True)
class FieldSpec:
    """One field, as a backend sees it. ``options`` carries what a backend may
    need beyond the type (``partition_by``, ``max_length``, ``auto``...) and,
    for a user-defined subclass, ``custom_class`` and ``overrides_hooks``."""

    name: str
    kind: FieldKind
    py_type: Optional[type]
    null: bool
    options: Mapping[str, Any] = field(default_factory=dict)

ModelSpec dataclass

A model's storage-relevant shape, built from Model._meta.

Built lazily by ModelOptions.spec and rebuilt when a field is added (the _auto_key field is added at first instantiation, after class creation). backend is the explicit Meta.backend or None (the process default applies).

Source code in src/popoto/backends/types.py 
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
@dataclass(frozen=True)
class ModelSpec:
    """A model's storage-relevant shape, built from ``Model._meta``.

    Built lazily by ``ModelOptions.spec`` and rebuilt when a field is added
    (the ``_auto_key`` field is added at first instantiation, after class
    creation). ``backend`` is the explicit ``Meta.backend`` or ``None`` (the
    process default applies).
    """

    name: str
    key_fields: tuple[str, ...]
    fields: Mapping[str, FieldSpec]
    order_by: Optional[str]
    ttl: Optional[int]
    indexes: tuple[tuple[str, ...], ...]
    backend: Optional[str] = None
    abstract: bool = False
    unique_indexes: tuple[tuple[str, ...], ...] = ()
    """The subset of :attr:`indexes` declared unique (``Meta.indexes``'s
    ``is_unique`` flag). Added in M1.1 beside §2's ``indexes``, which carries
    the field names only."""
    mixins: frozenset[str] = frozenset()
    """Names of the popoto model mixins in the model's MRO
    (``AccessTrackerMixin``, ``WriteFilterMixin``...). Added in M2a: a mixin
    that keeps per-record state needs columns of its own on Postgres."""

unique_indexes = () class-attribute instance-attribute

The subset of :attr:indexes declared unique (Meta.indexes's is_unique flag). Added in M1.1 beside §2's indexes, which carries the field names only.

mixins = frozenset() class-attribute instance-attribute

Names of the popoto model mixins in the model's MRO (AccessTrackerMixin, WriteFilterMixin...). Added in M2a: a mixin that keeps per-record state needs columns of its own on Postgres.

Op

Bases: str, Enum

The closed set of lookup operators: the field-lookup suffixes of plan §1 plus valid_at and within. EXACT is a bare field=value.

Source code in src/popoto/backends/types.py 
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
class Op(str, enum.Enum):
    """The closed set of lookup operators: the field-lookup suffixes of plan
    §1 plus ``valid_at`` and ``within``. ``EXACT`` is a bare ``field=value``."""

    EXACT = "exact"
    IN = "in"
    ISNULL = "isnull"
    STARTSWITH = "startswith"
    ENDSWITH = "endswith"
    CONTAINS = "contains"
    GT = "gt"
    GTE = "gte"
    LT = "lt"
    LTE = "lte"
    BETWEEN = "between"
    ANY = "any"
    ALL = "all"
    VALID_AT = "valid_at"
    WITHIN = "within"

OrderTerm dataclass

A field with a direction, or :data:RANDOM (sample_related_keys).

Source code in src/popoto/backends/types.py 
255
256
257
258
259
260
261
@dataclass(frozen=True)
class OrderTerm:
    """A field with a direction, or :data:`RANDOM` (``sample_related_keys``)."""

    field: Optional[str]
    descending: bool = False
    random: bool = False

ComputedCol dataclass

A named backend-computed value (geo distance) carried on Row.

Source code in src/popoto/backends/types.py 
267
268
269
270
271
272
@dataclass(frozen=True)
class ComputedCol:
    """A named backend-computed value (geo distance) carried on ``Row``."""

    name: str
    expr: Any

QueryCall dataclass

The query exactly as the public API received it.

Not in §2's sketch; pinned here because the Redis backend's select and count are today's filter bodies, moved, and those bodies consume the call's own kwargs and Q objects -- re-deriving them from a compiled :data:Predicate would re-order set algebra and change the wire, which the POC proved is exactly how a "move" stops being one (#735, #746). A backend that needs a predicate tree compiles :attr:QueryPlan.where from this (popoto.backends.compile_where); the Redis backend replays it.

query is the model's Query; kind names which public body the call came from ("filter", "count" or "keys"); options holds the private execution flags those bodies take (_no_track, _allow_pushdown).

Source code in src/popoto/backends/types.py 
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
@dataclass(frozen=True)
class QueryCall:
    """The query exactly as the public API received it.

    Not in §2's sketch; pinned here because the Redis backend's ``select`` and
    ``count`` are today's filter bodies, *moved*, and those bodies consume the
    call's own kwargs and ``Q`` objects -- re-deriving them from a compiled
    :data:`Predicate` would re-order set algebra and change the wire, which the
    POC proved is exactly how a "move" stops being one (#735, #746). A backend
    that needs a predicate tree compiles :attr:`QueryPlan.where` from this
    (``popoto.backends.compile_where``); the Redis backend replays it.

    ``query`` is the model's ``Query``; ``kind`` names which public body the
    call came from (``"filter"``, ``"count"`` or ``"keys"``); ``options`` holds
    the private execution flags those bodies take (``_no_track``,
    ``_allow_pushdown``).
    """

    query: Any
    kind: Literal["filter", "count", "keys"]
    kwargs: Mapping[str, Any]
    q_objects: Sequence[Any] = ()
    options: Mapping[str, Any] = field(default_factory=dict)

QueryPlan dataclass

One read, as a backend executes it. project=() asks for id-only rows; None asks for every field.

Source code in src/popoto/backends/types.py 
300
301
302
303
304
305
306
307
308
309
310
311
312
@dataclass(frozen=True)
class QueryPlan:
    """One read, as a backend executes it. ``project=()`` asks for id-only
    rows; ``None`` asks for every field."""

    where: Optional[Predicate] = None
    order_by: tuple[OrderTerm, ...] = ()
    limit: Optional[int] = None
    offset: int = 0
    project: Optional[tuple[str, ...]] = None
    as_of: Optional[float] = None
    compute: tuple[ComputedCol, ...] = ()
    source: Optional[QueryCall] = None

RankTerm dataclass

One arm of rank_composite (plan §2 group E): a per-record score and the weight it carries in the composite.

kind names where the score comes from: "decay" (a DecayingSortedField's decayed score, the rank_decayed expression), "confidence" (a ConfidenceField's stored confidence), "access" (AccessTrackerMixin's confirmed read count; only records read at least once), "sorted" (a SortedField's score), or a caller-supplied scores mapping ("similarity", "co_occurrence"). where is the arm's own domain -- the partition its index covers -- because each arm of today's ZUNIONSTORE is a separate set: a record scores on the arms whose set holds it, and is ranked when any one does.

Source code in src/popoto/backends/types.py 
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
@dataclass(frozen=True)
class RankTerm:
    """One arm of ``rank_composite`` (plan §2 group E): a per-record score
    and the weight it carries in the composite.

    ``kind`` names where the score comes from: ``"decay"`` (a
    ``DecayingSortedField``'s decayed score, the ``rank_decayed`` expression),
    ``"confidence"`` (a ``ConfidenceField``'s stored confidence),
    ``"access"`` (``AccessTrackerMixin``'s confirmed read count; only records
    read at least once), ``"sorted"`` (a ``SortedField``'s score), or a
    caller-supplied ``scores`` mapping (``"similarity"``, ``"co_occurrence"``).
    ``where`` is the arm's own domain -- the partition its index covers --
    because each arm of today's ``ZUNIONSTORE`` is a separate set: a record
    scores on the arms whose set holds it, and is ranked when any one does.
    """

    kind: str
    weight: float
    field: Optional[str] = None
    where: Optional["Predicate"] = None
    scores: Optional[Mapping[str, float]] = None
    options: Mapping[str, Any] = _dataclass_field(default_factory=dict)
    """Per-arm settings: a decay arm's ``confidence_field`` (modulation)."""

options = _dataclass_field(default_factory=dict) class-attribute instance-attribute

Per-arm settings: a decay arm's confidence_field (modulation).

Expiry dataclass

A record expiry: ttl seconds, or expire_at epoch seconds.

Source code in src/popoto/backends/types.py 
353
354
355
356
357
358
@dataclass(frozen=True)
class Expiry:
    """A record expiry: ``ttl`` seconds, or ``expire_at`` epoch seconds."""

    ttl: Optional[int] = None
    expire_at: Optional[float] = None

SaveOutcome dataclass

What save reports. result is what Model.save returns to its caller: on Redis, the unit of work's pipeline when one was given, else the HSET reply (today's return value, kept exactly).

Source code in src/popoto/backends/types.py 
361
362
363
364
365
366
367
368
@dataclass(frozen=True)
class SaveOutcome:
    """What ``save`` reports. ``result`` is what ``Model.save`` returns to its
    caller: on Redis, the unit of work's pipeline when one was given, else the
    ``HSET`` reply (today's return value, kept exactly)."""

    id: Optional[RecordId]
    result: Any = None

Capabilities dataclass

What a bound backend can do for one model (returned by bind).

Source code in src/popoto/backends/types.py 
371
372
373
374
375
376
377
378
379
380
381
382
@dataclass(frozen=True)
class Capabilities:
    """What a bound backend can do for one model (returned by ``bind``)."""

    backend: str
    groups: frozenset[str]
    field_kinds: frozenset[str]
    key_migration: bool = True
    record_ttl: bool = True

    def supports(self, group: str) -> bool:
        return group in self.groups

UnitOfWork

A transaction handle: what every pipeline= kwarg becomes at the seam.

A wrapper, not a duck-typed pipeline (POC decision 1; TD-10). __bool__ is always True, so the 73 pipeline if pipeline sites in the field layer can never mistake an empty unit of work for "no pipeline" -- which is what made the POC's Postgres unit of work, with a __len__ and no __bool__, falsy while empty.

On Redis it holds the caller's pipeline as :attr:pipeline. The Redis backend rebinds :attr:pipeline to whatever the field hooks hand back, so Model.save(pipeline=p) keeps returning exactly the object it always returned.

Source code in src/popoto/backends/types.py 
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
class UnitOfWork:
    """A transaction handle: what every ``pipeline=`` kwarg becomes at the seam.

    A wrapper, not a duck-typed pipeline (POC decision 1; TD-10). ``__bool__``
    is always ``True``, so the 73 ``pipeline if pipeline`` sites in the field
    layer can never mistake an empty unit of work for "no pipeline" -- which is
    what made the POC's Postgres unit of work, with a ``__len__`` and no
    ``__bool__``, falsy while empty.

    On Redis it holds the caller's pipeline as :attr:`pipeline`. The Redis
    backend rebinds :attr:`pipeline` to whatever the field hooks hand back, so
    ``Model.save(pipeline=p)`` keeps returning exactly the object it always
    returned.
    """

    __slots__ = ("pipeline", "backend")

    def __init__(self, pipeline: Any = None, *, backend: str = "redis") -> None:
        self.pipeline = pipeline
        self.backend = backend

    def __bool__(self) -> bool:
        return True

    @property
    def is_redis_pipeline(self) -> bool:
        import redis

        return isinstance(self.pipeline, redis.client.Pipeline)

    def commit(self) -> list[Any]:
        """Execute what was queued. On Redis, ``pipeline.execute()``."""
        if self.pipeline is None:
            return []
        return list(self.pipeline.execute())

    def __enter__(self) -> UnitOfWork:
        return self

    def __exit__(self, exc_type: Any, exc: Any, tb: Any) -> None:
        if exc_type is None:
            self.commit()
        elif self.pipeline is not None and hasattr(self.pipeline, "reset"):
            self.pipeline.reset()

    def __repr__(self) -> str:
        return f"<UnitOfWork {self.backend} {self.pipeline!r}>"

commit()

Execute what was queued. On Redis, pipeline.execute().

Source code in src/popoto/backends/types.py 
418
419
420
421
422
def commit(self) -> list[Any]:
    """Execute what was queued. On Redis, ``pipeline.execute()``."""
    if self.pipeline is None:
        return []
    return list(self.pipeline.execute())