popoto.backends.types¶
popoto.backends.types
¶
Backend-neutral types that cross the model-level storage protocol (#759).
Everything here is plain data: no redis, no psycopg, no network. The
shapes are the plan's §2 (docs/plans/sdlc-631-v2.md); where an
implementation detail had to be pinned that §2 leaves open, the docstring says
so and why.
Values cross the protocol decoded -- no msgpack, key strings or index names
-- with one deliberate exception on the Redis side, :class:RecordId.native,
which carries the caller's own key object so a bytes key reaches the Redis
wire as the same token it always did (the #751 str boundary lesson).
FieldKind = str
module-attribute
¶
The name of the popoto field class a field is (or, for a user subclass, the
nearest popoto class it derives from): "KeyField", "SortedField",
"Field"... A closed enum would have to grow with every field module; the
backends' capability tables key on these names instead (validate_spec).
Row = Mapping[str, Any]
module-attribute
¶
Decoded Python values; always carries "_id" (a :class:RecordId, or
None for a Redis projection row, which has never carried its key).
ASYNC_PROTOCOL_METHODS = ('bind', 'transaction', 'close', 'save', 'load', 'delete', 'exists', 'increment', 'select', 'count', 'field_call')
module-attribute
¶
The protocol methods :class:AsyncBackend mirrors: groups A-C, which
today's async_* callers reach (async_save/async_create/the bulk
twins through save, async_get/async_get_many through load,
async_filter/async_all/async_keys through select,
async_count through count, async_delete/async_delete_all
through delete), plus field_call, which a hydrating read reaches for
AccessTrackerMixin staging. increment has no async_* caller yet
and is mirrored because it is group B.
BackendError
¶
Bases: PopotoException
Base class for every error the storage-backend layer raises.
PopotoException logs its message but does not pass it to
Exception, so str(exc) would be empty; this class keeps it.
Source code in src/popoto/backends/types.py
BackendCapabilityError
¶
Bases: BackendError, NotImplementedError
The selected backend cannot do this.
Raised for a protocol method a backend does not implement (on a non-Redis
backend, groups D-H until their milestone), for a field the backend's
static capability table refuses (:func:popoto.backends.validate_spec),
and for a [PG-only] capability called on a Redis-bound model.
NotImplementedError is a base so except NotImplementedError keeps
catching the "not on this backend" case.
Source code in src/popoto/backends/types.py
BackendUnavailableError
¶
Bases: BackendError, ConnectionError
The backend cannot be reached, or is not installed or configured.
The popoto-level outage type both backends share (plan §1.1, M1): an
unreachable server or a connect/statement timeout on Postgres, and
selecting postgres without POPOTO_POSTGRES_URL or the postgres
extra.
Source code in src/popoto/backends/types.py
SchemaDriftError
¶
Bases: BackendError
The stored schema and the model disagree in a way popoto will not
reconcile automatically (plan §3, Migrations). Raised by bind().
Source code in src/popoto/backends/types.py
BackendRetryableError
¶
Bases: BackendError
A deadlock or serialization failure rolled the work back; running it
again is safe (plan §6, TD-2), so a caller catches one popoto type instead
of a driver error (#759 M2a). Raised from the driver error:
- inside a caller-owned
transaction()(a statement in it, or its COMMIT), at once -- the whole unit was rolled back and only the caller can run its block again, so popoto does not retry it; - by a single statement outside a unit of work, and by
ObservationProtocol.on_context_used, once their own bounded retries (Defaults.PG_TRANSACTION_RETRIES) are spent.
Before #773's patch a caller-owned transaction saw the raw psycopg
DeadlockDetected / SerializationFailure; code that caught those
should catch this instead. A statement whose completion is unknown
(40003) is not this error: it may have committed (#769).
Source code in src/popoto/backends/types.py
BackendBusyError
¶
Bases: BackendRetryableError
No pooled connection became free in time: every connection the backend's pool may open is checked out by other callers (#759 M5).
Raised when a caller waited Defaults.PG_CONNECT_TIMEOUT_SECONDS for a
connection and none was returned -- many concurrent async_* calls or
transaction() blocks on one event loop, or more threads than
Defaults.PG_POOL_MAX_SIZE on the sync pool. Every connection is in
use, so this is contention, not an outage: :class:BackendUnavailableError is not
raised, the backend's health record is not touched and no dropped
write is counted. Nothing was sent, so running the call again is safe,
which is why it is a :class:BackendRetryableError. A connection that
cannot be opened (the server is down or unreachable) is still
:class:BackendUnavailableError, and so is a wait for a slot held by a
caller stuck connecting to an unresponsive server.
Source code in src/popoto/backends/types.py
MaintenanceIncompleteError
¶
Bases: BackendRetryableError
rebuild_indexes() on Postgres repaired what it could, then a
REINDEX TABLE CONCURRENTLY / ANALYZE step did not finish (#759 M5,
788 review): it waited longer than Defaults.PG_MAINTAIN_LOCK_TIMEOUT_MS¶
on another session's lock (an idle-in-transaction session that wrote the
table is the usual one), ran past Defaults.PG_MAINTAIN_STATEMENT_TIMEOUT_MS,
was cancelled, or failed on the server.
This is maintenance that stopped early, not an outage: the backend's
health record is untouched and no dropped write is counted. Every
companion-row repair and orphan removal before the failed step has
already committed, and running rebuild_indexes() again is safe (it is
idempotent, and it first drops the *_ccnew indexes a failed
CONCURRENTLY leaves behind), which is why it is a
:class:BackendRetryableError.
Attributes:
| Name | Type | Description |
|---|---|---|
indexed |
records the side-row pass covered (the int a successful
|
|
diverged_keys |
rows skipped because their key columns no longer
produce their stored |
|
completed |
the steps that finished, in order ( |
|
failed_step |
the step that did not finish. |
|
invalid_indexes |
INVALID indexes the failure left on the model's
tables that could not be dropped at once (the next
|
Source code in src/popoto/backends/types.py
RecordId
dataclass
¶
The identity of one record, backend-neutral.
canonical is today's DB_key string -- Model.pk on both backends,
and the Postgres _pk column (plan §3). values are the KeyField
values in DB_key order (KeyField names sorted, as
:meth:ModelOptions.get_db_key_index_position numbers them); it is ()
when the id was built from a key string that was never decomposed, which no
backend needs for a lookup by canonical.
native is not part of identity (compare=False): it is the key object
the caller held -- str or the bytes a raw Redis reply produced -- and
the Redis backend sends it unchanged, so routing a call through the protocol
never changes the token on the wire or the type a caller gets back.
Source code in src/popoto/backends/types.py
key
property
¶
What the Redis backend sends: the caller's object, else the string.
from_key(model, key, values=())
classmethod
¶
Build an id from a key the caller holds (str, bytes or a
DB_key), keeping that object as :attr:native.
Source code in src/popoto/backends/types.py
FieldSpec
dataclass
¶
One field, as a backend sees it. options carries what a backend may
need beyond the type (partition_by, max_length, auto...) and,
for a user-defined subclass, custom_class and overrides_hooks.
Source code in src/popoto/backends/types.py
ModelSpec
dataclass
¶
A model's storage-relevant shape, built from Model._meta.
Built lazily by ModelOptions.spec and rebuilt when a field is added.
(The implicit _auto_key is registered at class creation, #826, so it
is in the first spec built.) backend is the explicit Meta.backend or None (the
process default applies).
Source code in src/popoto/backends/types.py
unique_indexes = ()
class-attribute
instance-attribute
¶
The subset of :attr:indexes declared unique (Meta.indexes's
is_unique flag). Added in M1.1 beside §2's indexes, which carries
the field names only.
mixins = frozenset()
class-attribute
instance-attribute
¶
Names of the popoto model mixins in the model's MRO
(AccessTrackerMixin, WriteFilterMixin...). Added in M2a: a mixin
that keeps per-record state needs columns of its own on Postgres.
Op
¶
Bases: str, Enum
The closed set of lookup operators: the field-lookup suffixes of plan
§1 plus valid_at and within. EXACT is a bare field=value.
Source code in src/popoto/backends/types.py
OrderTerm
dataclass
¶
A field with a direction, or :data:RANDOM (sample_related_keys).
Source code in src/popoto/backends/types.py
ComputedCol
dataclass
¶
QueryCall
dataclass
¶
The query exactly as the public API received it.
Not in §2's sketch; pinned here because the Redis backend's select and
count are today's filter bodies, moved, and those bodies consume the
call's own kwargs and Q objects -- re-deriving them from a compiled
:data:Predicate would re-order set algebra and change the wire, which the
POC proved is exactly how a "move" stops being one (#735, #746). A backend
that needs a predicate tree compiles :attr:QueryPlan.where from this
(popoto.backends.compile_where); the Redis backend replays it.
query is the model's Query; kind names which public body the
call came from ("filter", "count" or "keys"); options holds
the private execution flags those bodies take (_no_track,
_allow_pushdown).
Source code in src/popoto/backends/types.py
QueryPlan
dataclass
¶
One read, as a backend executes it. project=() asks for id-only
rows; None asks for every field.
Source code in src/popoto/backends/types.py
RankTerm
dataclass
¶
One arm of rank_composite (plan §2 group E): a per-record score
and the weight it carries in the composite.
kind names where the score comes from: "decay" (a
DecayingSortedField's decayed score, the rank_decayed expression),
"confidence" (a ConfidenceField's stored confidence),
"access" (AccessTrackerMixin's confirmed read count; only records
read at least once), "sorted" (a SortedField's score), or a
caller-supplied scores mapping ("similarity", "co_occurrence").
where is the arm's own domain -- the partition its index covers --
because each arm of today's ZUNIONSTORE is a separate set: a record
scores on the arms whose set holds it, and is ranked when any one does.
Source code in src/popoto/backends/types.py
options = _dataclass_field(default_factory=dict)
class-attribute
instance-attribute
¶
Per-arm settings: a decay arm's confidence_field (modulation).
Expiry
dataclass
¶
SaveOutcome
dataclass
¶
What save reports. result is what Model.save returns to its
caller: on Redis, the unit of work's pipeline when one was given, else the
HSET reply (today's return value, kept exactly).
Source code in src/popoto/backends/types.py
Capabilities
dataclass
¶
What a bound backend can do for one model (returned by bind).
Source code in src/popoto/backends/types.py
UnitOfWork
¶
A transaction handle: what every pipeline= kwarg becomes at the seam.
A wrapper, not a duck-typed pipeline (POC decision 1; TD-10). __bool__
is always True, so the 73 pipeline if pipeline sites in the field
layer can never mistake an empty unit of work for "no pipeline" -- which is
what made the POC's Postgres unit of work, with a __len__ and no
__bool__, falsy while empty.
On Redis it holds the caller's pipeline as :attr:pipeline. The Redis
backend rebinds :attr:pipeline to whatever the field hooks hand back, so
Model.save(pipeline=p) keeps returning exactly the object it always
returned.
Source code in src/popoto/backends/types.py
AsyncBackend
¶
Bases: Protocol
The async twin of :class:popoto.backends.Backend (plan §2, TD-6).
Same arguments and results as the sync methods of
:data:ASYNC_PROTOCOL_METHODS, awaited. transaction() is an async
context manager whose unit of work is passed as pipeline= to the
async_* model methods. run(fn, ...) runs sync popoto code (a
Model.save, a Query.get) with its storage I/O on this backend's
async driver: it is how the async_* model methods are routed, so the
model layer above the seam exists once.
Redis has no implementation: a Redis-bound model's async_* methods
keep today's redis.asyncio / worker-thread paths unchanged.
Source code in src/popoto/backends/types.py
566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 | |