popoto.transfer.cli

popoto.transfer.cli

popoto-transfer -- a CLI front-end for :mod:popoto.transfer.

Two subcommands:

export Wraps :func:popoto.transfer.export_records. Writes JSON Lines to --out (a file, or - for stdout) and a human summary to stderr. import Wraps :func:popoto.transfer.import_records. Reads JSON Lines from --in (a file, or - for stdin) and prints the reconciliation report to stderr.

Both subcommands refuse to run against Redis database 0 unless --allow-db0 is passed. Database 0 is, on many machines running this ORM, a live store rather than a test database, and an import writes to it. The guard reads the database off the live connection pool -- not an environment variable -- so it catches the unset-REDIS_URL fallback as well as an explicit …/0 URL, and it runs before the operator's --model module is imported and before any Redis command is issued.

The human-readable summary always goes to stderr, never stdout, so that --out - can stream JSON Lines on stdout without the summary corrupting it: popoto-transfer export --model pkg.mod:Model --out - | gzip > b.gz still shows the operator a summary on their terminal. --json claims stdout for a machine-readable summary instead, and is refused together with --out - since both would write to stdout.

Submodule imports (redis, the transfer drivers, the model registry) are kept out of module scope and inside the functions that need them, so argument parsing and the database-0 guard run before any of them is touched. This does not make --help cheap: the console-script entry point imports popoto.transfer.cli, which imports the popoto package, so the ORM is already resolved by the time :func:main is called.

CLIError

Bases: Exception

A diagnosed, user-facing CLI failure.

Raised by :func:resolve_model and the flag-validation helpers. Callers catch it, print str(exc) to stderr, and return exit code 1 -- never a traceback.

Source code in src/popoto/transfer/cli.py

class CLIError(Exception):
    """A diagnosed, user-facing CLI failure.

    Raised by :func:`resolve_model` and the flag-validation helpers. Callers
    catch it, print ``str(exc)`` to stderr, and return exit code 1 -- never a
    traceback.
    """

build_parser()

Construct the argument parser.

Returns:

Type	Description
ArgumentParser	The configured :class:argparse.ArgumentParser. Subparser names are
ArgumentParser	the strings "export" and "import"; import is a Python
ArgumentParser	keyword, so dispatch in :func:main reads args.command rather
ArgumentParser	than an attribute named import.

Source code in src/popoto/transfer/cli.py

def build_parser() -> argparse.ArgumentParser:
    """Construct the argument parser.

    Returns:
        The configured :class:`argparse.ArgumentParser`. Subparser names are
        the strings ``"export"`` and ``"import"``; ``import`` is a Python
        keyword, so dispatch in :func:`main` reads ``args.command`` rather
        than an attribute named ``import``.
    """
    from .export import DEFAULT_CHUNK_SIZE

    parser = argparse.ArgumentParser(
        prog="popoto-transfer",
        description=(
            "Move one Popoto model's records between Redis/Valkey "
            "instances, with a reconciliation report and an exit code a "
            "script can act on."
        ),
        epilog=USAGE_EPILOG,
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    sub = parser.add_subparsers(dest="command")

    export = sub.add_parser(
        "export",
        help="export a model's records to JSON Lines",
        description=(
            "Exports a model's records as JSON Lines: one manifest line "
            "followed by one line per record."
        ),
    )
    _add_shared_arguments(export)
    export.add_argument(
        "--out",
        default="-",
        help="destination file, or '-' for stdout (default: -)",
    )
    export.add_argument(
        "--filter",
        action="append",
        default=None,
        metavar="KEY=VALUE",
        help=(
            "equality filter, repeatable; the value is parsed as JSON "
            "first (so 0.5, true, null work), falling back to a raw "
            "string. Q objects and lookup operators are not expressible "
            "here; use the Python API for those."
        ),
    )
    export.add_argument(
        "--chunk-size",
        type=int,
        default=DEFAULT_CHUNK_SIZE,
        help=f"keys hydrated per round trip (default: {DEFAULT_CHUNK_SIZE})",
    )

    imp = sub.add_parser(
        "import",
        help="import a model's records from JSON Lines",
        description=(
            "Imports records from a JSON Lines export produced by "
            "'popoto-transfer export'. Keys are always preserved, so a "
            "re-run with --on-conflict overwrite converges rather than "
            "duplicating. Import is not atomic across records: if it is "
            "interrupted, re-run with --on-conflict overwrite to finish."
        ),
    )
    _add_shared_arguments(imp)
    imp.add_argument(
        "--in",
        dest="in_path",
        default="-",
        help="source file, or '-' for stdin (default: -)",
    )
    imp.add_argument(
        "--on-conflict",
        choices=["error", "skip", "overwrite"],
        default="error",
        help=(
            "what to do when the destination already holds a key "
            "(default: error, which refuses and cannot clobber)"
        ),
    )
    imp.add_argument(
        "--on-write-gate",
        choices=["reject", "bypass"],
        default="reject",
        help=(
            "honor the destination model's write gate " "(default: reject) or bypass it"
        ),
    )
    imp.add_argument(
        "--regenerate-keys",
        action="store_true",
        help=(
            "mint a new key for every record instead of preserving the "
            "exported one, remapping Relationship references onto the new "
            "keys. NOT idempotent -- a second run creates a second copy of "
            "every record. Only fields that declare a reference are remapped; "
            "a key stored in a plain string field is never rewritten and will "
            "dangle. Requires the destination model's key to be exactly one "
            "AutoKeyField"
        ),
    )
    imp.add_argument(
        "--on-embedding-mismatch",
        choices=["error", "carry", "regenerate"],
        default="error",
        help=(
            "what to do when an exported embedding's provider fingerprint "
            "differs from the destination's (default: error)"
        ),
    )
    return parser

resolve_model(spec)

Resolve a "module.path:ClassName" spec into a Model subclass.

Prepends the current working directory to sys.path first, since a console script does not get the CWD on sys.path the way python -m does, and the single most likely first invocation is from the operator's own project root.

Parameters:

Name	Type	Description	Default
spec	str	A colon-separated model spec, e.g. "myapp.models:Memory".	required

Returns:

Type	Description
Any	The resolved :class:popoto.Model subclass.

Raises:

Type	Description
CLIError	If spec does not have exactly one colon, either half is empty, the module cannot be imported, the module has no such attribute, or the attribute is not a Model subclass. Each failure carries a distinct message naming what went wrong.

Source code in src/popoto/transfer/cli.py

def resolve_model(spec: str) -> Any:
    """Resolve a ``"module.path:ClassName"`` spec into a Model subclass.

    Prepends the current working directory to ``sys.path`` first, since a
    console script does not get the CWD on ``sys.path`` the way
    ``python -m`` does, and the single most likely first invocation is from
    the operator's own project root.

    Args:
        spec: A colon-separated model spec, e.g. ``"myapp.models:Memory"``.

    Returns:
        The resolved :class:`popoto.Model` subclass.

    Raises:
        CLIError: If ``spec`` does not have exactly one colon, either half
            is empty, the module cannot be imported, the module has no such
            attribute, or the attribute is not a ``Model`` subclass. Each
            failure carries a distinct message naming what went wrong.
    """
    import importlib

    from popoto import Model

    if spec.count(":") != 1:
        raise CLIError(
            f"--model {spec!r} must be 'module.path:ClassName' (exactly " "one colon)"
        )
    module_path, _, class_name = spec.partition(":")
    if not module_path or not class_name:
        raise CLIError(
            f"--model {spec!r} must name both a module and a class, e.g. "
            "'myapp.models:Memory'"
        )

    cwd = os.getcwd()
    if cwd not in sys.path:
        sys.path.insert(0, cwd)

    try:
        module = importlib.import_module(module_path)
    except ImportError as exc:
        raise CLIError(
            f"--model: could not import module {module_path!r}: {exc}"
        ) from exc

    try:
        obj = getattr(module, class_name)
    except AttributeError:
        raise CLIError(
            f"--model: module {module_path!r} has no attribute " f"{class_name!r}"
        ) from None

    if not isinstance(obj, type) or not issubclass(obj, Model):
        raise CLIError(
            f"--model: {module_path}:{class_name} is not a Popoto Model " "subclass"
        )
    return obj

main(argv=None)

Entry point for the popoto-transfer console script.

Parameters:

Name	Type	Description	Default
argv	Optional[List[str]]	Argument list, defaulting to sys.argv[1:].	None

Returns:

Type	Description
int	A process exit code: 0 on a clean run, 1 on an operational
int	failure (bad --model, the database-0 refusal, an unreadable
int	file, a manifest mismatch, a query error, a connection error, or an
int	on_conflict="error" collision -- which may have written earlier
int	records before raising), 2 on an argparse usage error (argparse's
int	own convention), or 3 when the run completed but at least one
int	record did not land (any rejected/errored/partial
int	import outcome, or any export error; a skipped import outcome is
int	clean and does not trigger this).

Source code in src/popoto/transfer/cli.py

def main(argv: Optional[List[str]] = None) -> int:
    """Entry point for the ``popoto-transfer`` console script.

    Args:
        argv: Argument list, defaulting to ``sys.argv[1:]``.

    Returns:
        A process exit code: ``0`` on a clean run, ``1`` on an operational
        failure (bad ``--model``, the database-0 refusal, an unreadable
        file, a manifest mismatch, a query error, a connection error, or an
        ``on_conflict="error"`` collision -- which may have written earlier
        records before raising), ``2`` on an argparse usage error (argparse's
        own convention), or ``3`` when the run completed but at least one
        record did not land (any ``rejected``/``errored``/``partial``
        import outcome, or any export error; a ``skipped`` import outcome is
        clean and does not trigger this).
    """
    parser = build_parser()
    args = parser.parse_args(argv)

    if args.command is None:
        parser.print_help()
        return 0
    if args.command == "export":
        return _cmd_export(args)
    if args.command == "import":
        return _cmd_import(args)
    parser.print_help()
    return 0