popoto.backends.postgres.aio¶
popoto.backends.postgres.aio
¶
Native async on Postgres: AsyncPostgresBackend (#759 M5, TD-6).
Before M5, every async_* method of a Postgres-bound model ran the sync
call in a worker thread (asyncio.to_thread). This module replaces that
shim with an AsyncPostgresBackend on psycopg.AsyncConnection -- one
connection pool per (DSN, pid, event loop) -- without a second copy of the
backend.
How: one implementation, two drivers¶
The sync :class:~popoto.backends.postgres.PostgresBackend is ~7,000 lines
whose statements are interleaved with the Python that decides them: the
lazy bind() (version, encoding, extension and DDL checks), the record-key
lock order, the deadlock retry and the #769 dead-connection rule, the
savepoint a guarded save takes inside a caller's transaction, the search
CTEs, the validity refusals, the access-tracker staging after a read. An
async copy would duplicate all of it, and the first behaviour the two copies
disagreed on would be a parity bug no test of one copy could see.
So the async backend runs the sync code itself inside a greenlet on the
event loop thread (the technique SQLAlchemy's asyncio extension uses), and
swaps the one thing that does I/O: :func:popoto.backends.postgres._pool_for
returns, inside such a greenlet, a sync-shaped facade over this loop's
pool of AsyncConnection objects. Every conn.execute the sync code issues becomes
await AsyncConnection.execute on the loop -- the greenlet switches back to
the coroutine driving it, which awaits the driver call and switches the
result back in. No worker thread is involved, and no SQL, lock order, error
mapping or retry rule exists twice: the SQL builders are the sync
backend's, by construction.
What stays on a thread: an embedding provider call (provider.embed is a
sync API) and the backfill's wait on it, through
:func:popoto.backends.postgres._blocking; and any Redis I/O a Postgres
model's sync path makes (EventStreamMixin's XADD), which runs on the
loop thread as a blocking call. See docs/features/postgres-backend.md.
Event loops¶
The pool is popoto's own, not psycopg_pool.AsyncConnectionPool (see
:class:_LoopPool for why: its workers swallow cancellation, so one left open
at loop shutdown hangs asyncio.run()). Pools are per loop: an AsyncConnection belongs to the loop that opened
it, so a model used from two loops (pytest-asyncio's function-scoped loops,
asyncio.run called twice) gets a pool per loop. A loop's pool is closed
with the loop: the pool registers an async generator whose finally closes
it, and loop.shutdown_asyncgens() -- which asyncio.run,
asyncio.Runner and pytest-asyncio all call before closing a loop --
finalizes it. A loop closed without that call is swept on the next pool
lookup from any loop (and when it is garbage collected): its connections are
closed at the socket, so pg_stat_activity does not grow with the number
of loops a process has used.
BridgeMisuseError
¶
Bases: RuntimeError
A sync call reached an async-only Postgres object outside the bridge.
Raised when a unit of work from AsyncPostgresBackend.transaction() is
handed to a sync method (obj.save(pipeline=uow) instead of
await obj.async_save(pipeline=uow)): its connection belongs to an
event loop and can only be driven from a coroutine on that loop.
Source code in src/popoto/backends/postgres/aio.py
AsyncPostgresBackend
¶
The async twin of a :class:~popoto.backends.postgres.PostgresBackend
(plan ยง2, AsyncBackend): the same dsn, schema, table memo and
:attr:health record, with I/O on psycopg.AsyncConnection.
Get it with :func:async_backend, one per sync backend. Every method
runs the sync backend's own method in the bridge, so its SQL, lock
order, error mapping (BackendRetryableError,
BackendUnavailableError with dropped-write accounting, the #769
no-blind-retry rule), savepoint rule and first-use server checks are the
sync backend's, not a copy of them.
Source code in src/popoto/backends/postgres/aio.py
744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 | |
run(fn, /, *args, **kwargs)
async
¶
Run sync popoto code -- Model.save, Query.get -- with its
Postgres I/O on this loop's async pool. The routed async_*
methods are this call around their sync twins.
Afterwards, in this task's own context, it records the task as an
enclosing lock scope when it now has a unit of work open, so a child
task it creates can be refused at once rather than wait on the
parent's record locks (_refuse_self_wait, #784 review).
Source code in src/popoto/backends/postgres/aio.py
bind(spec)
async
¶
transaction()
async
¶
async with backend.transaction() as uow: -- one READ
COMMITTED transaction on one connection of this loop's pool.
Pass uow as pipeline= to the async_* methods; it commits
or rolls back with the block, and a deadlock or serialization
failure inside it raises BackendRetryableError at once (the sync
transaction()'s rule: only the caller can rerun its block).
Source code in src/popoto/backends/postgres/aio.py
close()
async
¶
Close this loop's pools (the sync pools are sync.close()'s).
Source code in src/popoto/backends/postgres/aio.py
close_async_pools()
¶
Close every async pool this process opened, at the socket (sync
teardown: PostgresBackend.close(), the test harness). A pool whose
loop is still running is dropped too; its next use opens a new one.
Source code in src/popoto/backends/postgres/aio.py
async_backend(sync)
¶
The async twin of sync (a PostgresBackend), created once and
kept on it; None when greenlet is not installed (then the routed
methods keep the thread shim).
Source code in src/popoto/backends/postgres/aio.py
get_async_backend(model=None)
¶
The async backend of model (a class or instance; None means
the process default), for async with get_async_backend(M).transaction()
as uow. Postgres only: a Redis-bound model's async_* methods talk to
redis.asyncio (or a worker thread) directly and have no async twin, so
asking for one raises :class:~popoto.backends.BackendCapabilityError.
Source code in src/popoto/backends/postgres/aio.py
run_async(backend, fn, /, *args, **kwargs)
async
¶
What a routed async_* method of a non-Redis model awaits: the
sync twin fn on the async Postgres backend, or -- for another
backend, or without greenlet -- in a worker thread, as before M5.