Skip to content

popoto.backends.postgres.aio

popoto.backends.postgres.aio

Native async on Postgres: AsyncPostgresBackend (#759 M5, TD-6).

Before M5, every async_* method of a Postgres-bound model ran the sync call in a worker thread (asyncio.to_thread). This module replaces that shim with an AsyncPostgresBackend on psycopg.AsyncConnection -- one connection pool per (DSN, pid, event loop) -- without a second copy of the backend.

How: one implementation, two drivers

The sync :class:~popoto.backends.postgres.PostgresBackend is ~7,000 lines whose statements are interleaved with the Python that decides them: the lazy bind() (version, encoding, extension and DDL checks), the record-key lock order, the deadlock retry and the #769 dead-connection rule, the savepoint a guarded save takes inside a caller's transaction, the search CTEs, the validity refusals, the access-tracker staging after a read. An async copy would duplicate all of it, and the first behaviour the two copies disagreed on would be a parity bug no test of one copy could see.

So the async backend runs the sync code itself inside a greenlet on the event loop thread (the technique SQLAlchemy's asyncio extension uses), and swaps the one thing that does I/O: :func:popoto.backends.postgres._pool_for returns, inside such a greenlet, a sync-shaped facade over this loop's pool of AsyncConnection objects. Every conn.execute the sync code issues becomes await AsyncConnection.execute on the loop -- the greenlet switches back to the coroutine driving it, which awaits the driver call and switches the result back in. No worker thread is involved, and no SQL, lock order, error mapping or retry rule exists twice: the SQL builders are the sync backend's, by construction.

What stays on a thread: an embedding provider call (provider.embed is a sync API) and the backfill's wait on it, through :func:popoto.backends.postgres._blocking; and any Redis I/O a Postgres model's sync path makes (EventStreamMixin's XADD), which runs on the loop thread as a blocking call. See docs/features/postgres-backend.md.

Event loops

The pool is popoto's own, not psycopg_pool.AsyncConnectionPool (see :class:_LoopPool for why: its workers swallow cancellation, so one left open at loop shutdown hangs asyncio.run()). Pools are per loop: an AsyncConnection belongs to the loop that opened it, so a model used from two loops (pytest-asyncio's function-scoped loops, asyncio.run called twice) gets a pool per loop. A loop's pool is closed with the loop: the pool registers an async generator whose finally closes it, and loop.shutdown_asyncgens() -- which asyncio.run, asyncio.Runner and pytest-asyncio all call before closing a loop -- finalizes it. A loop closed without that call is swept on the next pool lookup from any loop (and when it is garbage collected): its connections are closed at the socket, so pg_stat_activity does not grow with the number of loops a process has used.

BridgeMisuseError

Bases: RuntimeError

A sync call reached an async-only Postgres object outside the bridge.

Raised when a unit of work from AsyncPostgresBackend.transaction() is handed to a sync method (obj.save(pipeline=uow) instead of await obj.async_save(pipeline=uow)): its connection belongs to an event loop and can only be driven from a coroutine on that loop.

Source code in src/popoto/backends/postgres/aio.py
class BridgeMisuseError(RuntimeError):
    """A sync call reached an async-only Postgres object outside the bridge.

    Raised when a unit of work from ``AsyncPostgresBackend.transaction()`` is
    handed to a *sync* method (``obj.save(pipeline=uow)`` instead of
    ``await obj.async_save(pipeline=uow)``): its connection belongs to an
    event loop and can only be driven from a coroutine on that loop."""

AsyncPostgresBackend

The async twin of a :class:~popoto.backends.postgres.PostgresBackend (plan ยง2, AsyncBackend): the same dsn, schema, table memo and :attr:health record, with I/O on psycopg.AsyncConnection.

Get it with :func:async_backend, one per sync backend. Every method runs the sync backend's own method in the bridge, so its SQL, lock order, error mapping (BackendRetryableError, BackendUnavailableError with dropped-write accounting, the #769 no-blind-retry rule), savepoint rule and first-use server checks are the sync backend's, not a copy of them.

Source code in src/popoto/backends/postgres/aio.py
class AsyncPostgresBackend:
    """The async twin of a :class:`~popoto.backends.postgres.PostgresBackend`
    (plan ยง2, ``AsyncBackend``): the same ``dsn``, ``schema``, table memo and
    :attr:`health` record, with I/O on ``psycopg.AsyncConnection``.

    Get it with :func:`async_backend`, one per sync backend. Every method
    runs the sync backend's own method in the bridge, so its SQL, lock
    order, error mapping (``BackendRetryableError``,
    ``BackendUnavailableError`` with dropped-write accounting, the #769
    no-blind-retry rule), savepoint rule and first-use server checks are the
    sync backend's, not a copy of them.
    """

    name = "postgres"

    def __init__(self, sync: Any) -> None:
        self.sync = sync

    def __repr__(self) -> str:
        return f"<AsyncPostgresBackend schema={self.sync.schema!r}>"

    @property
    def dsn(self) -> str:
        return str(self.sync.dsn)

    @property
    def schema(self) -> str:
        return str(self.sync.schema)

    @property
    def health(self) -> Any:
        return self.sync.health

    async def run(self, fn: Callable[..., Any], /, *args: Any, **kwargs: Any) -> Any:
        """Run sync popoto code -- ``Model.save``, ``Query.get`` -- with its
        Postgres I/O on this loop's async pool. The routed ``async_*``
        methods are this call around their sync twins.

        Afterwards, in this task's own context, it records the task as an
        enclosing lock scope when it now has a unit of work open, so a child
        task it creates can be refused at once rather than wait on the
        parent's record locks (``_refuse_self_wait``, #784 review)."""
        try:
            return await _spawn(fn, *args, **kwargs)
        finally:
            self.sync._note_open_scope()

    # -- A. lifecycle ----------------------------------------------------------

    async def bind(self, spec: ModelSpec) -> Capabilities:
        """No network (plan decision 5): the checks run at first use."""
        caps: Capabilities = self.sync.bind(spec)
        return caps

    @contextlib.asynccontextmanager
    async def transaction(self) -> AsyncIterator[UnitOfWork]:
        """``async with backend.transaction() as uow:`` -- one ``READ
        COMMITTED`` transaction on one connection of this loop's pool.
        Pass ``uow`` as ``pipeline=`` to the ``async_*`` methods; it commits
        or rolls back with the block, and a deadlock or serialization
        failure inside it raises ``BackendRetryableError`` at once (the sync
        ``transaction()``'s rule: only the caller can rerun its block)."""
        cm = self.sync.transaction()
        uow = await self.run(cm.__enter__)
        try:
            yield uow
        except BaseException:
            if not await self.run(cm.__exit__, *sys.exc_info()):
                raise
        else:
            await self.run(cm.__exit__, None, None, None)

    async def close(self) -> None:
        """Close this loop's pools (the sync pools are ``sync.close()``'s)."""
        loop = asyncio.get_running_loop()
        with _apools_lock:
            mine = [e for e in _apools.values() if e.loop_ref() is loop]
        for entry in mine:
            await entry.aclose()

    # -- B. records ------------------------------------------------------------

    async def save(
        self,
        obj: Any,
        *,
        fields: Optional[Sequence[str]] = None,
        previous_id: Optional[RecordId] = None,
        expiry: Optional[Expiry] = None,
        uow: Optional[UnitOfWork] = None,
        **options: Any,
    ) -> SaveOutcome:
        outcome: SaveOutcome = await self.run(
            self.sync.save,
            obj,
            fields=fields,
            previous_id=previous_id,
            expiry=expiry,
            uow=uow,
            **options,
        )
        return outcome

    async def load(
        self,
        spec: ModelSpec,
        ids: Sequence[RecordId],
        *,
        fields: Optional[Sequence[str]] = None,
        **options: Any,
    ) -> list[Optional[Row]]:
        rows: list[Optional[Row]] = await self.run(
            self.sync.load, spec, ids, fields=fields, **options
        )
        return rows

    async def delete(
        self,
        spec: ModelSpec,
        ids: Sequence[RecordId],
        *,
        uow: Optional[UnitOfWork] = None,
        **options: Any,
    ) -> int:
        count: int = await self.run(self.sync.delete, spec, ids, uow=uow, **options)
        return count

    async def exists(self, spec: ModelSpec, ids: Sequence[RecordId]) -> list[bool]:
        found: list[bool] = await self.run(self.sync.exists, spec, ids)
        return found

    async def increment(
        self,
        spec: ModelSpec,
        id: RecordId,
        field: str,
        delta: Union[int, float],
        *,
        uow: Optional[UnitOfWork] = None,
        **options: Any,
    ) -> Union[int, float]:
        value: Union[int, float] = await self.run(
            self.sync.increment, spec, id, field, delta, uow=uow, **options
        )
        return value

    # -- C. query --------------------------------------------------------------

    async def select(self, spec: ModelSpec, plan: QueryPlan) -> list[Row]:
        rows: list[Row] = await self.run(self.sync.select, spec, plan)
        return rows

    async def count(self, spec: ModelSpec, plan: QueryPlan) -> int:
        n: int = await self.run(self.sync.count, spec, plan)
        return n

    # -- H. the field-adapter registry (read staging after a hydration) --------

    async def field_call(
        self,
        spec: ModelSpec,
        field: str,
        op: str,
        /,
        *args: Any,
        uow: Optional[UnitOfWork] = None,
        **kwargs: Any,
    ) -> Any:
        return await self.run(
            self.sync.field_call, spec, field, op, *args, uow=uow, **kwargs
        )

run(fn, /, *args, **kwargs) async

Run sync popoto code -- Model.save, Query.get -- with its Postgres I/O on this loop's async pool. The routed async_* methods are this call around their sync twins.

Afterwards, in this task's own context, it records the task as an enclosing lock scope when it now has a unit of work open, so a child task it creates can be refused at once rather than wait on the parent's record locks (_refuse_self_wait, #784 review).

Source code in src/popoto/backends/postgres/aio.py
async def run(self, fn: Callable[..., Any], /, *args: Any, **kwargs: Any) -> Any:
    """Run sync popoto code -- ``Model.save``, ``Query.get`` -- with its
    Postgres I/O on this loop's async pool. The routed ``async_*``
    methods are this call around their sync twins.

    Afterwards, in this task's own context, it records the task as an
    enclosing lock scope when it now has a unit of work open, so a child
    task it creates can be refused at once rather than wait on the
    parent's record locks (``_refuse_self_wait``, #784 review)."""
    try:
        return await _spawn(fn, *args, **kwargs)
    finally:
        self.sync._note_open_scope()

bind(spec) async

No network (plan decision 5): the checks run at first use.

Source code in src/popoto/backends/postgres/aio.py
async def bind(self, spec: ModelSpec) -> Capabilities:
    """No network (plan decision 5): the checks run at first use."""
    caps: Capabilities = self.sync.bind(spec)
    return caps

transaction() async

async with backend.transaction() as uow: -- one READ COMMITTED transaction on one connection of this loop's pool. Pass uow as pipeline= to the async_* methods; it commits or rolls back with the block, and a deadlock or serialization failure inside it raises BackendRetryableError at once (the sync transaction()'s rule: only the caller can rerun its block).

Source code in src/popoto/backends/postgres/aio.py
@contextlib.asynccontextmanager
async def transaction(self) -> AsyncIterator[UnitOfWork]:
    """``async with backend.transaction() as uow:`` -- one ``READ
    COMMITTED`` transaction on one connection of this loop's pool.
    Pass ``uow`` as ``pipeline=`` to the ``async_*`` methods; it commits
    or rolls back with the block, and a deadlock or serialization
    failure inside it raises ``BackendRetryableError`` at once (the sync
    ``transaction()``'s rule: only the caller can rerun its block)."""
    cm = self.sync.transaction()
    uow = await self.run(cm.__enter__)
    try:
        yield uow
    except BaseException:
        if not await self.run(cm.__exit__, *sys.exc_info()):
            raise
    else:
        await self.run(cm.__exit__, None, None, None)

close() async

Close this loop's pools (the sync pools are sync.close()'s).

Source code in src/popoto/backends/postgres/aio.py
async def close(self) -> None:
    """Close this loop's pools (the sync pools are ``sync.close()``'s)."""
    loop = asyncio.get_running_loop()
    with _apools_lock:
        mine = [e for e in _apools.values() if e.loop_ref() is loop]
    for entry in mine:
        await entry.aclose()

close_async_pools()

Close every async pool this process opened, at the socket (sync teardown: PostgresBackend.close(), the test harness). A pool whose loop is still running is dropped too; its next use opens a new one.

Source code in src/popoto/backends/postgres/aio.py
def close_async_pools() -> None:
    """Close every async pool this process opened, at the socket (sync
    teardown: ``PostgresBackend.close()``, the test harness). A pool whose
    loop is still running is dropped too; its next use opens a new one."""
    with _apools_lock:
        entries = list(_apools.values())
        _apools.clear()
    for entry in entries:
        entry.hard_close()

async_backend(sync)

The async twin of sync (a PostgresBackend), created once and kept on it; None when greenlet is not installed (then the routed methods keep the thread shim).

Source code in src/popoto/backends/postgres/aio.py
def async_backend(sync: Any) -> Optional[AsyncPostgresBackend]:
    """The async twin of ``sync`` (a ``PostgresBackend``), created once and
    kept on it; ``None`` when greenlet is not installed (then the routed
    methods keep the thread shim)."""
    if _greenlet_mod is None:
        return None
    twin = sync.__dict__.get("_async_twin")
    if twin is not None:
        return twin
    with _twins_lock:
        twin = sync.__dict__.get("_async_twin")
        if twin is None:
            if not isinstance(getattr(sync, "_intent", None), _GreenletLocal):
                sync._intent = _GreenletLocal()
            twin = AsyncPostgresBackend(sync)
            sync.__dict__["_async_twin"] = twin
    return twin

get_async_backend(model=None)

The async backend of model (a class or instance; None means the process default), for async with get_async_backend(M).transaction() as uow. Postgres only: a Redis-bound model's async_* methods talk to redis.asyncio (or a worker thread) directly and have no async twin, so asking for one raises :class:~popoto.backends.BackendCapabilityError.

Source code in src/popoto/backends/postgres/aio.py
def get_async_backend(model: Any = None) -> AsyncPostgresBackend:
    """The async backend of ``model`` (a class or instance; ``None`` means
    the process default), for ``async with get_async_backend(M).transaction()
    as uow``. Postgres only: a Redis-bound model's ``async_*`` methods talk to
    ``redis.asyncio`` (or a worker thread) directly and have no async twin, so
    asking for one raises :class:`~popoto.backends.BackendCapabilityError`."""
    from .. import get_backend
    from ..types import BackendCapabilityError, BackendUnavailableError
    from . import PostgresBackend

    if model is not None and not isinstance(model, type):
        model = type(model)
    backend = get_backend(model)
    if not isinstance(backend, PostgresBackend):
        raise BackendCapabilityError(
            f"no async backend for the {backend.name!r} backend: AsyncBackend is "
            "Postgres-only; Redis-bound models use their async_* methods "
            "directly"
        )
    twin = async_backend(backend)
    if twin is None:
        raise BackendUnavailableError(
            "the async Postgres backend needs greenlet: pip install 'popoto[postgres]'"
        )
    return twin

run_async(backend, fn, /, *args, **kwargs) async

What a routed async_* method of a non-Redis model awaits: the sync twin fn on the async Postgres backend, or -- for another backend, or without greenlet -- in a worker thread, as before M5.

Source code in src/popoto/backends/postgres/aio.py
async def run_async(
    backend: Any, fn: Callable[..., Any], /, *args: Any, **kwargs: Any
) -> Any:
    """What a routed ``async_*`` method of a non-Redis model awaits: the
    sync twin ``fn`` on the async Postgres backend, or -- for another
    backend, or without greenlet -- in a worker thread, as before M5."""
    from . import PostgresBackend

    if isinstance(backend, PostgresBackend):
        twin = async_backend(backend)
        if twin is not None:
            return await twin.run(fn, *args, **kwargs)
        global _warned_no_greenlet
        if not _warned_no_greenlet:
            _warned_no_greenlet = True
            logger.warning(
                "popoto: greenlet is not installed, so async_* on Postgres-bound "
                "models runs the sync call in a worker thread; pip install "
                "'popoto[postgres]' for the native async backend"
            )
    return await asyncio.to_thread(fn, *args, **kwargs)