One transaction, opened without reaching for the client (#630, #759 M5).
batch() is the supported way for a recipe -- or a caller -- to open a
Popoto transaction. It returns a real redis.client.Pipeline: commands
queue until you call execute(), and everything queued applies in one
MULTI/EXEC.
The return type is deliberately the redis-py pipeline itself and not a
wrapper. Popoto's field layer decides whether a write joins the caller's
transaction with isinstance(pipeline, redis.client.Pipeline) at twenty
sites, several of them shaped pipeline if isinstance(...) else
POPOTO_REDIS_DB. A wrapper object would fail those checks silently, fall
back to the shared client, and execute immediately -- voiding the atomicity
the batch was opened for, with no error raised anywhere.
Because it is an ordinary pipeline, it is also an ordinary context manager:
with releases the connection on exit but does not execute. Call
execute() yourself.
import popoto
pipe = popoto.batch()
JournalEntry.append(agent_id="a", statement="...", pipeline=pipe)
JournalEntry.append(agent_id="a", statement="...", pipeline=pipe)
pipe.execute()
Postgres-bound models (#759 M5, plan TD-5). The pipeline is a
:class:Batch, a subclass of popoto's guarded pipeline, so everything above
holds byte for byte on Redis. A write from a model bound to Postgres that is
handed the batch joins it: the batch opens one Postgres transaction()
on that model's backend at the first such write, the write runs inside it,
and execute() commits it. It is atomic: if a statement fails inside the
transaction, nothing in the batch is written (execute() then raises),
where a Redis MULTI/EXEC applies the other queued commands. A save
refused before it sends anything (pre_save's unique check against a
committed record, a validation error) raises at the call and leaves the
batch healthy, as on Redis. reset(), leaving a
with block, or dropping the batch without execute() rolls it back,
as a Redis pipeline that is never executed sends nothing. Until execute()
other connections do not see the batch's writes, as with queued commands.
One batch, one backend. A batch that has queued Redis commands refuses a
Postgres write, and one holding a Postgres transaction refuses a Redis
command, with :class:~popoto.backends.BackendCapabilityError, before either
is sent. Two stores cannot commit atomically together, and a batch that
looked atomic but was not would be worse than a refusal. Open one batch per
backend.
From async code. When the first Postgres write to join a batch is an
async_* call, the batch's transaction lives on the event loop's async
pool: commit it with await pipe.async_execute() (execute() raises
BridgeMisuseError) and roll it back with await pipe.async_reset().
A batch is driven by whichever side opened it: a sync write to an
async-opened batch, and an async_* write to a batch a sync write opened
(whose blocking connection would stall the loop), both raise
BridgeMisuseError before anything is sent, and leave the batch usable.
Batch
Bases: GuardedPipeline
The pipeline batch() returns: popoto's guarded Redis pipeline, plus
the Postgres transaction a Postgres-bound model's writes join (module
docstring). Redis commands queue and execute exactly as on any pipeline;
nothing here sends a byte to Redis that the plain pipeline would not.
Source code in src/popoto/batch.py
| class Batch(GuardedPipeline):
"""The pipeline ``batch()`` returns: popoto's guarded Redis pipeline, plus
the Postgres transaction a Postgres-bound model's writes join (module
docstring). Redis commands queue and execute exactly as on any pipeline;
nothing here sends a byte to Redis that the plain pipeline would not."""
_pg: Optional[tuple[Any, contextlib.ExitStack, Any]] = None
# -- the Postgres side -------------------------------------------------
def _popoto_join(self, backend: Any) -> Any:
"""The unit of work a write to ``backend`` (a non-Redis backend)
runs in: this batch's transaction, opened on first use."""
from .backends.types import BackendCapabilityError
pg = self._pg
if pg is not None:
if pg[0] is not backend:
raise BackendCapabilityError(
"popoto.batch() holds a transaction on another Postgres "
"backend; one batch commits to one database"
)
if _in_bridge() and not hasattr(pg[2].conn, "async_connection"):
from .backends.postgres.aio import BridgeMisuseError
raise BridgeMisuseError(
"this popoto.batch() holds a Postgres transaction a sync "
"call opened, on a blocking connection: an async_* write "
"joining it would block the event loop for every statement. "
"Write to it with the sync methods (in a worker thread from "
"async code), or open the batch with an async_* call"
)
return pg[2]
if self.command_stack:
raise BackendCapabilityError(_MIXED)
stack = contextlib.ExitStack()
uow = stack.enter_context(backend.transaction())
self._pg = (backend, stack, uow)
return uow
def _popoto_refuse_redis(self) -> None:
if self._pg is not None:
from .backends.types import BackendCapabilityError
raise BackendCapabilityError(_MIXED)
def _popoto_discard(self) -> None:
"""Roll the Postgres transaction back, if one is open."""
pg = self._pg
if pg is None:
return
self._pg = None
_backend, stack, uow = pg
# Nothing committed, so nothing to reap and no Redis side effect to
# send: psycopg swallows the Rollback signal below, so the
# transaction() context runs its after-commit step on this path too.
uow.reap.clear()
uow._after_commit.clear()
# The stream appends (#759 M5) run only on the commit path, which a
# rollback never reaches; cleared anyway so none can outlive it.
uow._before_commit.clear()
uow._stream_appends.clear()
import psycopg
# psycopg's transaction block swallows its own Rollback signal, so
# this rolls back and returns the connection without raising.
stack.__exit__(psycopg.Rollback, psycopg.Rollback(), None)
# -- the pipeline API --------------------------------------------------
def _popoto_async_unit(self) -> bool:
"""Whether this batch's Postgres transaction was opened by an
``async_*`` call: its connection belongs to an event loop, so only
the async bridge can commit or roll it back."""
pg = self._pg
return pg is not None and hasattr(pg[2].conn, "async_connection")
def _popoto_discard_later(self) -> None:
"""``reset()`` of an async-opened batch from sync code (a ``with``
block's exit, ``reset()`` on the loop): roll the transaction back on
its loop, as a task. Without a running loop nothing can drive the
connection; the loop's pool closes it at the socket when the loop
goes, and the server rolls it back."""
pg = self._pg
if pg is None:
return
self._pg = None
backend, stack, uow = pg
uow.reap.clear()
uow._after_commit.clear()
uow._before_commit.clear()
uow._stream_appends.clear()
try:
loop = asyncio.get_running_loop()
except RuntimeError:
return
import psycopg
from .backends.postgres.aio import async_backend
twin = async_backend(backend)
if twin is None: # pragma: no cover - an async unit implies greenlet
return
task = loop.create_task(
twin.run(stack.__exit__, psycopg.Rollback, psycopg.Rollback(), None)
)
_pending.add(task)
task.add_done_callback(_rollback_done)
async def async_execute(self, raise_on_error: bool = True) -> Any:
"""``await pipe.async_execute()``: :meth:`execute` from a coroutine.
A batch whose Postgres transaction an ``async_*`` call opened
(``await obj.async_save(pipeline=pipe)``) commits on the event loop,
through the async backend; :meth:`execute` cannot drive that
connection and raises ``BridgeMisuseError``. Any other batch -- Redis
commands, or a transaction a sync call opened -- runs :meth:`execute`
in a worker thread, so the loop never blocks on it."""
pg = self._pg
if pg is not None and self._popoto_async_unit():
from .backends.postgres.aio import async_backend
twin = async_backend(pg[0])
assert twin is not None # an async unit implies greenlet
return await twin.run(self.execute, raise_on_error)
return await asyncio.to_thread(self.execute, raise_on_error)
async def async_reset(self) -> None:
"""``await pipe.async_reset()``: :meth:`reset` from a coroutine,
rolling an async-opened Postgres transaction back before it
returns."""
pg = self._pg
if pg is not None and self._popoto_async_unit():
from .backends.postgres.aio import async_backend
twin = async_backend(pg[0])
assert twin is not None
await twin.run(self.reset)
return
self.reset()
def pipeline_execute_command(self, *args: Any, **options: Any) -> Any:
self._popoto_refuse_redis()
return super().pipeline_execute_command(*args, **options)
def execute(self, raise_on_error: bool = True) -> Any:
"""Execute the queued Redis commands -- or commit the Postgres
transaction (returning ``[]``). A Postgres batch in which a statement
failed is rolled back whole, and this raises
:class:`~popoto.backends.BackendError`."""
pg = self._pg
if pg is None:
return super().execute(raise_on_error)
_backend, stack, uow = pg
if self._popoto_async_unit() and not _in_bridge():
from .backends.postgres.aio import BridgeMisuseError
raise BridgeMisuseError(
"this popoto.batch() holds a Postgres transaction an async_* call "
"opened, whose connection belongs to the event loop: commit it "
"with `await pipe.async_execute()` (or roll it back with "
"`await pipe.async_reset()`)"
)
from psycopg.pq import TransactionStatus
if uow.conn.info.transaction_status == TransactionStatus.INERROR:
self._popoto_discard()
from .backends.types import BackendError
raise BackendError(
"popoto.batch(): a statement in this batch failed, so the whole "
"Postgres transaction was rolled back and nothing in the batch "
"was written"
)
self._pg = None
stack.close() # COMMIT; then the TTL reaper for the tables it wrote
return []
def reset(self) -> None: # type: ignore[override]
if self._popoto_async_unit() and not _in_bridge():
self._popoto_discard_later()
else:
self._popoto_discard()
super().reset()
|
async_execute(raise_on_error=True)
async
await pipe.async_execute(): :meth:execute from a coroutine.
A batch whose Postgres transaction an async_* call opened
(await obj.async_save(pipeline=pipe)) commits on the event loop,
through the async backend; :meth:execute cannot drive that
connection and raises BridgeMisuseError. Any other batch -- Redis
commands, or a transaction a sync call opened -- runs :meth:execute
in a worker thread, so the loop never blocks on it.
Source code in src/popoto/batch.py
| async def async_execute(self, raise_on_error: bool = True) -> Any:
"""``await pipe.async_execute()``: :meth:`execute` from a coroutine.
A batch whose Postgres transaction an ``async_*`` call opened
(``await obj.async_save(pipeline=pipe)``) commits on the event loop,
through the async backend; :meth:`execute` cannot drive that
connection and raises ``BridgeMisuseError``. Any other batch -- Redis
commands, or a transaction a sync call opened -- runs :meth:`execute`
in a worker thread, so the loop never blocks on it."""
pg = self._pg
if pg is not None and self._popoto_async_unit():
from .backends.postgres.aio import async_backend
twin = async_backend(pg[0])
assert twin is not None # an async unit implies greenlet
return await twin.run(self.execute, raise_on_error)
return await asyncio.to_thread(self.execute, raise_on_error)
|
async_reset()
async
await pipe.async_reset(): :meth:reset from a coroutine,
rolling an async-opened Postgres transaction back before it
returns.
Source code in src/popoto/batch.py
| async def async_reset(self) -> None:
"""``await pipe.async_reset()``: :meth:`reset` from a coroutine,
rolling an async-opened Postgres transaction back before it
returns."""
pg = self._pg
if pg is not None and self._popoto_async_unit():
from .backends.postgres.aio import async_backend
twin = async_backend(pg[0])
assert twin is not None
await twin.run(self.reset)
return
self.reset()
|
execute(raise_on_error=True)
Execute the queued Redis commands -- or commit the Postgres
transaction (returning []). A Postgres batch in which a statement
failed is rolled back whole, and this raises
:class:~popoto.backends.BackendError.
Source code in src/popoto/batch.py
| def execute(self, raise_on_error: bool = True) -> Any:
"""Execute the queued Redis commands -- or commit the Postgres
transaction (returning ``[]``). A Postgres batch in which a statement
failed is rolled back whole, and this raises
:class:`~popoto.backends.BackendError`."""
pg = self._pg
if pg is None:
return super().execute(raise_on_error)
_backend, stack, uow = pg
if self._popoto_async_unit() and not _in_bridge():
from .backends.postgres.aio import BridgeMisuseError
raise BridgeMisuseError(
"this popoto.batch() holds a Postgres transaction an async_* call "
"opened, whose connection belongs to the event loop: commit it "
"with `await pipe.async_execute()` (or roll it back with "
"`await pipe.async_reset()`)"
)
from psycopg.pq import TransactionStatus
if uow.conn.info.transaction_status == TransactionStatus.INERROR:
self._popoto_discard()
from .backends.types import BackendError
raise BackendError(
"popoto.batch(): a statement in this batch failed, so the whole "
"Postgres transaction was rolled back and nothing in the batch "
"was written"
)
self._pg = None
stack.close() # COMMIT; then the TTL reaper for the tables it wrote
return []
|
batch(transaction=True)
Open a batch of queued commands against the shared connection.
Parameters:
| Name |
Type |
Description |
Default |
transaction
|
bool
|
Wrap the queued commands in MULTI/EXEC so they
apply atomically. Defaults to True. False gives plain
command pipelining with no atomicity -- Popoto's own
annotate-and-close paths refuse such a pipeline. Postgres writes
joined to the batch run in one transaction either way.
|
True
|
Returns:
| Type |
Description |
'Pipeline'
|
A redis.client.Pipeline (a :class:Batch). Queue commands on it
|
'Pipeline'
|
(directly, or by passing it as pipeline= to Popoto model and
|
'Pipeline'
|
recipe calls), then call execute().
|
Source code in src/popoto/batch.py
| def batch(transaction: bool = True) -> "Pipeline":
"""Open a batch of queued commands against the shared connection.
Args:
transaction: Wrap the queued commands in ``MULTI``/``EXEC`` so they
apply atomically. Defaults to ``True``. ``False`` gives plain
command pipelining with no atomicity -- Popoto's own
annotate-and-close paths refuse such a pipeline. Postgres writes
joined to the batch run in one transaction either way.
Returns:
A ``redis.client.Pipeline`` (a :class:`Batch`). Queue commands on it
(directly, or by passing it as ``pipeline=`` to Popoto model and
recipe calls), then call ``execute()``.
"""
pipe = get_REDIS_DB().pipeline(transaction=transaction)
# The client builds the guarded pipeline; reassigning the class (as
# GuardedRedis.pipeline does) adds the Postgres side without touching the
# constructor, whose signature moves between redis-py versions.
pipe.__class__ = Batch
return pipe
|
join_unit(pipeline, backend)
What a pipeline= argument becomes for a write to backend.
A :class:Batch handed to a non-Redis backend's write returns the
batch's Postgres unit of work (opening it); handed to a Redis write while
it holds a Postgres transaction, it raises
:class:~popoto.backends.BackendCapabilityError before anything is
queued. Everything else -- a plain pipeline, a unit of work, None, a
batch for a Redis write -- returns None: the caller keeps its own
handling, unchanged.
Source code in src/popoto/batch.py
| def join_unit(pipeline: Any, backend: Any) -> Any:
"""What a ``pipeline=`` argument becomes for a write to ``backend``.
A :class:`Batch` handed to a non-Redis backend's write returns the
batch's Postgres unit of work (opening it); handed to a Redis write while
it holds a Postgres transaction, it raises
:class:`~popoto.backends.BackendCapabilityError` before anything is
queued. Everything else -- a plain pipeline, a unit of work, ``None``, a
batch for a Redis write -- returns ``None``: the caller keeps its own
handling, unchanged."""
if not isinstance(pipeline, Batch):
return None
if getattr(backend, "name", "redis") == "redis":
pipeline._popoto_refuse_redis()
return None
return pipeline._popoto_join(backend)
|
unit_of(pipeline, backend)
The unit of work a non-Redis write runs in: pipeline itself when
it is one (a transaction()), the batch's when it is a
:class:Batch, else None (a plain Redis pipeline cannot carry a
Postgres write).
Source code in src/popoto/batch.py
| def unit_of(pipeline: Any, backend: Any) -> Any:
"""The unit of work a non-Redis write runs in: ``pipeline`` itself when
it is one (a ``transaction()``), the batch's when it is a
:class:`Batch`, else ``None`` (a plain Redis pipeline cannot carry a
Postgres write)."""
from .backends.types import UnitOfWork
if isinstance(pipeline, UnitOfWork):
return pipeline
return join_unit(pipeline, backend)
|
open_unit_of(pipeline, backend)
The unit of work a read issued for a write to backend runs on
(#776): pipeline when it is a non-Redis unit of work (a
transaction()), the transaction a :class:Batch already holds on
backend, else None -- an autocommit read. Never opens a
transaction: a batch with no Postgres write yet holds no connection, so
a read before its first write has no unit to join.
Source code in src/popoto/batch.py
| def open_unit_of(pipeline: Any, backend: Any) -> Any:
"""The unit of work a *read* issued for a write to ``backend`` runs on
(#776): ``pipeline`` when it is a non-Redis unit of work (a
``transaction()``), the transaction a :class:`Batch` already holds on
``backend``, else ``None`` -- an autocommit read. Never opens a
transaction: a batch with no Postgres write yet holds no connection, so
a read before its first write has no unit to join."""
from .backends.types import UnitOfWork
if isinstance(pipeline, UnitOfWork):
return pipeline if pipeline.backend != "redis" else None
if isinstance(pipeline, Batch):
pg = pipeline._pg
if pg is not None and pg[0] is backend:
return pg[2]
return None
|