`popoto.fields.relationship`¶

`popoto.fields.relationship` ¶

Relationship field for creating foreign-key-like references between Popoto models.

This module implements the relationship system that allows Popoto models to reference other models, similar to Django's ForeignKey. The design philosophy centers on using Redis Sets to maintain bidirectional indexes, enabling efficient queries in both directions (e.g., "find all Memberships for a Person" or "find the Person for a Membership").

Key Design Decisions

Set-Based Indexing: Rather than storing just the foreign key value, Popoto maintains Redis Sets that index the relationship. When you save a Membership with person=alice, Popoto adds the Membership's db_key to a Set keyed by Alice's db_key. This enables O(1) lookups of "all objects pointing to Alice".
Lazy Loading: Related model instances are stored as db_key strings and loaded on demand, avoiding expensive eager loading of entire object graphs.
Django-Style Query Syntax: Supports double-underscore traversal for filtering through relationships (e.g., Membership.query.filter(person__name="Alice")).
Explicit Model Declaration: Unlike Django's string-based lazy references, Popoto requires the related model class to be passed directly. This simplifies the implementation but requires careful import ordering.

Example

class Person(Model): name = KeyField()

class Membership(Model): person = Relationship(model=Person) role = Field(type=str)

alice = Person.create(name="Alice") m = Membership.create(person=alice, role="admin")

Query through relationship¶

memberships = Membership.query.filter(person=alice) memberships = Membership.query.filter(person__name="Alice")

`Relationship` ¶

Bases: Field

A field that stores references to other model instances, analogous to a ForeignKey.

The first positional argument is the related Model class. Internally the relationship is stored as the related instance's redis_key string, which is lazily loaded back into a full model instance on access. This prevents infinite recursion with circular references.

The Relationship field enables models to reference other models while maintaining queryable indexes. When a model instance with a Relationship is saved, Popoto automatically maintains a Redis Set that tracks which instances point to each related object.

This bidirectional indexing is the key innovation: traditional key-value stores only allow lookup by key, but Popoto's relationship Sets enable efficient reverse lookups ("find all X that reference Y") without scanning all records.

A field value can be one of three types at runtime:

Model instance - fully loaded relationship.
str - a redis_key (lazy-loaded, not yet resolved).
None - no relationship set.

Attributes:

Name	Type	Description
`model`	`Model`	The related Model class that this field references. Must be a concrete Model subclass, not a string reference.
`many`	`bool`	Reserved for future many-to-many support. Currently not fully implemented.
`null`	`bool`	Whether the relationship can be None. Defaults to True, allowing optional relationships.

Redis Data Structure

For each (Model, field_name, related_instance) combination, Popoto maintains a Set at key: $RelationshipF:ModelClass:field_name:related_db_key This Set contains the db_keys of all instances that reference the related object.

Example

class Author(Model): name = KeyField()

class Book(Model): title = KeyField() author = Relationship(model=Author)

tolkien = Author.create(name="Tolkien") Book.create(title="The Hobbit", author=tolkien) Book.create(title="LOTR", author=tolkien)

Efficient lookup: "all books by Tolkien"¶

books = Book.query.filter(author=tolkien) # Returns both books

Note

The type attribute is set to Model (the base class), not the specific related model. This is because type checking happens at the Field base class level and needs to accept any Model subclass.

Source code in src/popoto/fields/relationship.py

class Relationship(Field):
    """
    A field that stores references to other model instances, analogous to a ForeignKey.

    The first positional argument is the related Model class. Internally the
    relationship is stored as the related instance's ``redis_key`` string,
    which is lazily loaded back into a full model instance on access. This
    prevents infinite recursion with circular references.

    The Relationship field enables models to reference other models while maintaining
    queryable indexes. When a model instance with a Relationship is saved, Popoto
    automatically maintains a Redis Set that tracks which instances point to each
    related object.

    This bidirectional indexing is the key innovation: traditional key-value stores
    only allow lookup by key, but Popoto's relationship Sets enable efficient
    reverse lookups ("find all X that reference Y") without scanning all records.

    A field value can be one of three types at runtime:

    * ``Model`` instance - fully loaded relationship.
    * ``str`` - a redis_key (lazy-loaded, not yet resolved).
    * ``None`` - no relationship set.

    Attributes:
        model: The related Model class that this field references. Must be a concrete
            Model subclass, not a string reference.
        many: Reserved for future many-to-many support. Currently not fully implemented.
        null: Whether the relationship can be None. Defaults to True, allowing
            optional relationships.

    Redis Data Structure:
        For each (Model, field_name, related_instance) combination, Popoto maintains
        a Set at key: `$RelationshipF:ModelClass:field_name:related_db_key`
        This Set contains the db_keys of all instances that reference the related object.

    Example:
        class Author(Model):
            name = KeyField()

        class Book(Model):
            title = KeyField()
            author = Relationship(model=Author)

        tolkien = Author.create(name="Tolkien")
        Book.create(title="The Hobbit", author=tolkien)
        Book.create(title="LOTR", author=tolkien)

        # Efficient lookup: "all books by Tolkien"
        books = Book.query.filter(author=tolkien)  # Returns both books

    Note:
        The `type` attribute is set to Model (the base class), not the specific
        related model. This is because type checking happens at the Field base
        class level and needs to accept any Model subclass.
    """

    type: "Model" = None
    model: "Model" = None
    many: bool = False
    null: bool = True

    # Export/import: the reverse-lookup Set is fully rebuilt from the stored
    # redis_key value by on_save(), so a carried redis_key string round-trips
    # with no extra state. That holds under both import modes: preserving
    # keys leaves the stored value alone, and regenerating them
    # (preserve_keys=False) rewrites it through remap_references below before
    # the instance is constructed, so on_save rebuilds the Set from the new
    # key. The policy is about state this field would otherwise have to
    # carry, and there is none either way.
    roundtrip_policy: str = "rebuild"

    @classmethod
    def remap_references(
        cls,
        field_name: str,
        field_value: Any,
        key_map: "dict[str, str]",
        **kwargs: Any,
    ) -> Any:
        """Rewrite a stored ``redis_key`` to the target's regenerated key.

        This is the one field Popoto ships that declares a reference, so it
        is the one field ``preserve_keys=False`` can remap. The value is a
        plain ``redis_key`` string (``encode_popoto_model_obj`` treats such a
        string as already being in storage format), so the remap is a string
        substitution and the related record is never loaded -- deliberately.
        ``Relationship`` lazy-loads to survive circular references, and
        hydrating the target here would recurse right back through the
        pointer being remapped.

        A target absent from ``key_map`` keeps its old key: it either was not
        part of this import or belongs to a model the caller did not chain a
        ``key_map`` from, and inventing a key for it would be worse than a
        pointer the import report counts and names.

        Args:
            field_name: Name of this field on the model.
            field_value: The exported value -- a ``redis_key`` string, or
                ``None`` for an unset relationship (``null`` defaults True).
            key_map: Old ``redis_key`` to new ``redis_key``.
            **kwargs: Reserved for future extension.

        Returns:
            The remapped ``redis_key``, or ``field_value`` unchanged when it
            is not a non-empty string or the target is not in ``key_map``.
        """
        if isinstance(field_value, str) and field_value:
            return key_map.get(field_value, field_value)
        return field_value

    def __init__(self, **kwargs):
        """
        Initialize a Relationship field with the specified related model.

        The initialization defers the Model import to avoid circular dependencies,
        since both Model and Relationship need to reference each other. This is
        a common pattern in ORM design where fields and models are tightly coupled.

        Args:
            **kwargs: Field configuration options including:
                - model: The Model class this relationship points to (required for queries)
                - many: Boolean for many-to-many relationships (future feature)
                - null: Whether None is allowed (default True)
        """
        super().__init__(**kwargs)
        from ..models.base import Model

        relationship_field_defaults = {
            "type": Model,
            "model": None,
            "many": False,
            "null": True,
        }
        self.field_defaults.update(relationship_field_defaults)
        # set field options, let kwargs override
        for k, v in relationship_field_defaults.items():
            setattr(self, k, kwargs.get(k, v))

    @classmethod
    def is_valid(cls, field, value, null_check=True, **kwargs) -> bool:
        """
        Validate a Relationship field value.

        Accepts three value types as valid:
        - A Model instance (the related model)
        - A str containing a redis_key reference (lazy-loaded, format "ClassName:key")
        - None (if field.null is True)

        This override is necessary because the base Field.is_valid() only checks
        isinstance(value, field.type), which rejects the str redis_key values
        that are set during lazy loading from Redis.
        """
        if not null_check and value is None:
            return True
        if field.null and value is None:
            return True
        if value is None:
            logger.error(f"Relationship field {field} is null but null=False")
            return False
        # Accept redis_key strings (lazy-loaded relationship references)
        if isinstance(value, str):
            if ":" not in value:
                logger.error(
                    f"Relationship field {field} has invalid redis_key string: {value}"
                )
                return False
            return True
        # Accept Model instances
        from ..models.base import Model

        if isinstance(value, Model):
            return True
        logger.error(
            f"Relationship field {field} expected Model instance or redis_key string, "
            f"got {type(value)}"
        )
        return False

    def get_filter_query_params(self, field_name) -> set:
        """
        Build the set of valid query parameters for filtering on this relationship.

        This method enables Django-style double-underscore query syntax by traversing
        into the related model and collecting its filterable fields. For example, if
        a Book has an `author` Relationship to Author, and Author has a `name` field,
        this method makes `author__name` a valid filter parameter.

        The implementation deliberately avoids recursive relationship traversal to
        prevent infinite loops (e.g., Person -> Friend -> Person) and to keep query
        compilation tractable.

        Args:
            field_name: The name of this Relationship field on the parent model.

        Returns:
            A set of valid query parameter strings, including:

                - The field name itself (for direct model instance filtering)
                - Chained parameters like `field_name__related_field` for each
                  filterable field on the related model
        """
        related_field_filter_query_params = set()
        for related_field_name, related_field in self.model._meta.fields.items():
            if isinstance(related_field, Relationship):
                continue  # not ready for recursive compilation
            for related_query_param in related_field.get_filter_query_params(
                related_field_name
            ):
                related_field_filter_query_params.add(
                    f"{field_name}__{related_query_param}"
                )

        return (
            super()
            .get_filter_query_params(field_name)
            .union(
                {
                    f"{field_name}",
                }
            )
            .union(related_field_filter_query_params)
        )

    @classmethod
    def on_save(
        cls,
        model_instance: "Model",
        field_name: str,
        field_value: "Model | str | None",
        pipeline=None,
        **kwargs,
    ):
        """
        Maintain the relationship index Set when a model instance is saved.

        This hook is called automatically during model save operations. It updates
        the Redis Set that indexes which instances point to each related object.
        This is the core mechanism that enables efficient reverse lookups.

        When the relationship value changes (e.g., player.team = team_b), the
        instance is removed from the old related object's index set before being
        added to the new one. This prevents ghost entries that would cause stale
        query results. The old value is detected via ``_saved_field_values``,
        which stores field values at the last save/load.

        The index key follows the pattern:
            `$RelationshipF:ModelClass:field_name:related_db_key`

        For example, saving a Membership with person=alice would add the
        Membership's db_key to the Set at `$RelationshipF:Membership:person:Person:Alice`.

        Args:
            model_instance: The model instance being saved.
            field_name: The name of the Relationship field.
            field_value: The related Model instance (or None).
            pipeline: Optional Redis pipeline for batched operations.
            **kwargs: Additional arguments (unused, for extensibility).

        Returns:
            The pipeline (for chaining) or None if no pipeline was provided.

        Note:
            When field_value is None, the instance is removed from the index Set.
            This handles the case where a relationship is being cleared.
        """
        from ..models.base import Model

        # Handle different field_value types:
        # - None: relationship was never set or being cleared
        # - Model instance: fully loaded relationship
        # - str: lazy-loaded relationship (redis_key string due to circular reference protection)
        if field_value is None:
            related_db_key = "None"
        elif isinstance(field_value, Model):
            related_db_key = field_value.db_key
        elif isinstance(field_value, str):
            # field_value is the redis_key string (lazy-loaded but never accessed)
            # Expecting format "ClassName:key_value"
            if ":" not in field_value:
                logger.error(
                    f"Invalid redis_key format for {field_name}: {field_value}. Expected 'ClassName:key_value'"
                )
                return pipeline if pipeline else None
            # Parse the redis_key string into DB_key components
            related_db_key = DB_key.from_redis_key(field_value)
        else:
            # Unknown type, log and return without action
            logger.warning(
                f"Unexpected field_value type in on_save: {type(field_value)} for {field_name}"
            )
            return pipeline if pipeline else None

        # Remove from old index if the Relationship value changed.
        # _saved_field_values tracks values at last save/load.  If the current
        # value differs from the saved value, the old index Set still contains
        # this instance's key -- remove it to prevent ghost entries in queries.
        # This mirrors the proven pattern from KeyFieldMixin.on_save() (PR #150).
        saved_values = getattr(model_instance, "_saved_field_values", {})
        old_value = saved_values.get(field_name)
        if old_value is not None and old_value != field_value:
            # Resolve old value to a related_db_key using the same type-dispatch
            if isinstance(old_value, Model):
                old_related_db_key = old_value.db_key
            elif isinstance(old_value, str):
                if ":" in old_value:
                    old_related_db_key = DB_key.from_redis_key(old_value)
                else:
                    old_related_db_key = None
            else:
                old_related_db_key = None

            if old_related_db_key is not None:
                old_relationship_set_db_key = DB_key(
                    cls.get_special_use_field_db_key(model_instance, field_name),
                    old_related_db_key,
                )
                member_key = model_instance.db_key.redis_key
                if pipeline:
                    pipeline.srem(old_relationship_set_db_key.redis_key, member_key)
                else:
                    get_REDIS_DB().srem(
                        old_relationship_set_db_key.redis_key, member_key
                    )

        # on a one-to-many, save the set of many with the related instance
        # add this instance's id to a relationship set based on the related model
        # example: "$RelationshipF:Membership:person:person_db_key"
        relationship_set_db_key = DB_key(
            cls.get_special_use_field_db_key(model_instance, field_name),
            related_db_key,
        )

        if field_value is None:
            if pipeline:
                return pipeline.srem(
                    relationship_set_db_key.redis_key, model_instance.db_key.redis_key
                )
            else:
                return get_REDIS_DB().srem(
                    relationship_set_db_key.redis_key, model_instance.db_key.redis_key
                )
        else:
            if pipeline:
                return pipeline.sadd(
                    relationship_set_db_key.redis_key, model_instance.db_key.redis_key
                )
            else:
                return get_REDIS_DB().sadd(
                    relationship_set_db_key.redis_key, model_instance.db_key.redis_key
                )

    @classmethod
    def on_delete(
        cls,
        model_instance: "Model",
        field_name: str,
        field_value: "Model | str | None",
        pipeline: redis.client.Pipeline = None,
        **kwargs,
    ):
        """
        Clean up the relationship index Set when a model instance is deleted.

        This hook ensures referential integrity by removing the deleted instance
        from the relationship index. Without this cleanup, queries would return
        stale references to deleted objects.

        Args:
            model_instance: The model instance being deleted.
            field_name: The name of the Relationship field.
            field_value: The related Model instance (or None).
            pipeline: Optional Redis pipeline for batched operations.
            **kwargs: Additional arguments (unused, for extensibility).

        Returns:
            The pipeline (for chaining) or the result of SREM if no pipeline.

        Warning:
            There is a known edge case where lazy-loaded relationships may not
            be fully hydrated at delete time. If the relationship was accessed
            through a chain (e.g., person.friend.friend), the field_value may
            be a key string rather than a Model instance, causing this method
            to fail when accessing field_value.db_key.
        """
        # todo: it's possible this instance is not fully loaded or has been changed.
        #  Need to reload from db before deleting
        #  Example: on person.friend.delete() the person.friend.friend will have field_value as a keystring
        #  It will not be a model instance, so this method will fail on field_value.db_key below
        from ..models.base import Model

        # Handle different field_value types:
        # - None: relationship was never set
        # - Model instance: fully loaded relationship
        # - str: lazy-loaded relationship (redis_key string due to circular reference protection)
        if field_value is None:
            related_db_key = "None"
        elif isinstance(field_value, Model):
            related_db_key = field_value.db_key
        elif isinstance(field_value, str):
            # field_value is the redis_key string (lazy-loaded but never accessed)
            # Expecting format "ClassName:key_value"
            if ":" not in field_value:
                logger.error(
                    f"Invalid redis_key format for {field_name}: {field_value}. Expected 'ClassName:key_value'"
                )
                return pipeline if pipeline else None
            # Parse the redis_key string into DB_key components
            related_db_key = DB_key.from_redis_key(field_value)
        else:
            # Unknown type, log and return without action
            logger.warning(
                f"Unexpected field_value type in on_delete: {type(field_value)} for {field_name}"
            )
            return pipeline if pipeline else None

        relationship_set_db_key = DB_key(
            cls.get_special_use_field_db_key(model_instance, field_name),
            related_db_key,
        )
        # Use saved_redis_key if provided, otherwise fall back to current db_key
        member_key = kwargs.get("saved_redis_key", model_instance.db_key.redis_key)
        if pipeline:
            return pipeline.srem(relationship_set_db_key.redis_key, member_key)
        else:
            return get_REDIS_DB().srem(relationship_set_db_key.redis_key, member_key)

    @classmethod
    def filter_query(cls, model: "Model", field_name: str, **query_params) -> set:
        """
        Execute a filter query on a Relationship field, returning matching db_keys.

        This method handles two types of queries:

        1. **Direct Instance Filtering** (`field_name=instance`):
           Looks up the relationship index Set directly. For example,
           `Membership.query.filter(person=alice)` retrieves the Set at
           `$RelationshipF:Membership:person:Person:Alice`.

        2. **Chained Field Filtering** (`field_name__related_field=value`):
           First queries the related model to find matching instances, then
           looks up each of their relationship index Sets. For example,
           `Membership.query.filter(person__name="Alice")` first finds all
           Persons named Alice, then retrieves all Memberships pointing to them.

        The chained approach supports recursive traversal through multiple
        relationships, though this can become expensive for deep chains.

        Args:
            model: The Model class being queried.
            field_name: The name of the Relationship field.
            **query_params: Filter parameters (e.g., person=alice or person__name="Alice").

        Returns:
            A set of db_key bytes for matching model instances. Returns an empty
            set if no matches are found.

        Raises:
            QueryException: If filtering directly on the field with a non-Model value.

        Performance Note:
            Direct instance filtering is O(1) as it's a single Set lookup.
            Chained filtering requires querying the related model first, then
            performing N Set lookups where N is the number of matching related
            instances. Results are intersected if multiple filters are applied.
        """
        from ..models.base import Model

        keys_lists_to_intersect = list()
        pipeline = get_REDIS_DB().pipeline()

        for query_param, query_value in query_params.items():
            if query_param == f"{field_name}":
                if not isinstance(query_value, Model):
                    raise QueryException(
                        f"Query filter on Relationship expects model instance. Instead, got {query_value}"
                    )

                relationship_set_db_key = DB_key(
                    cls.get_special_use_field_db_key(model, field_name),
                    query_value.db_key,
                )
                keys_lists_to_intersect.append(
                    get_REDIS_DB().smembers(relationship_set_db_key.redis_key)
                )

            elif query_param.startswith(f"{field_name}__"):
                field = model._meta.fields[field_name]
                relationship_field_name = query_param.strip(f"{field_name}__").split(
                    "__"
                )[0]

                relationship_field_values = field.model._meta.fields[
                    relationship_field_name
                ].filter_query(
                    model=field.model,
                    field_name=relationship_field_name,
                    **{query_param.strip(f"{field_name}__"): query_value},
                )
                # note: this will be recursive if references another relationship and so forth

                for relationship_field_value in relationship_field_values:
                    relationship_set_db_key = DB_key(
                        cls.get_special_use_field_db_key(model, field_name),
                        relationship_field_value.db_key,
                    )
                    pipeline.smembers(relationship_set_db_key.redis_key)

        keys_lists_to_intersect += pipeline.execute()
        logger.debug(keys_lists_to_intersect)
        if len(keys_lists_to_intersect):
            return set.intersection(
                *[set(key_list) for key_list in keys_lists_to_intersect]
            )
        return set()

    @classmethod
    def sample_related_keys(
        cls,
        model: "Model",
        field_name: str,
        related_key: Union[str, DB_key],
        count: int,
    ) -> list[str]:
        """
        Return a bounded random sample of the redis_keys pointing at ``related_key``.

        Reads the Relationship reverse index Set — the same
        ``$RelationshipF:<Model>:<field_name>:<related_key>`` Set that
        :meth:`on_save` populates and :meth:`filter_query` reads in full — and
        issues a single ``SRANDMEMBER`` bounded by ``count``.

        This is the bounded counterpart to :meth:`filter_query`'s ``SMEMBERS``.
        On a hub node the full Set can be arbitrarily large, so a caller that
        walks edges (graph traversal, fan-out-limited expansion) needs a sampled
        read rather than an unbounded one. ``filter_query`` cannot serve that
        need: it is unbounded, it requires a loaded ``Model`` instance rather
        than a key, and it wraps its reads in a pipeline.

        Args:
            model: The Model class owning the Relationship field. Typed
                ``"Model"`` rather than ``Type["Model"]`` to match
                :meth:`filter_query` and :meth:`on_save`, which are annotated
                the same way and are likewise called with the class.
            field_name: The name of the Relationship field.
            related_key: The key of the related instance being pointed at.
                A ``DB_key`` is used as-is; a ``str`` is parsed with
                :meth:`DB_key.from_redis_key`, which unescapes the colon
                separators — constructing ``DB_key(some_redis_key)`` instead
                would escape them and silently address a different key. The
                ``DB_key`` arm exists for signature symmetry with
                :meth:`filter_query` and :meth:`on_delete`, which both hold a
                ``DB_key``; the only caller today passes a ``str``.
            count: Maximum number of members to return, passed through to
                ``SRANDMEMBER`` unchanged. A positive count yields distinct
                members with no ordering guarantee; ``0`` yields an empty list.

        Returns:
            A list of member redis_keys decoded to ``str``. Empty if the reverse
            index does not exist.

        Note:
            No exception handling by design — a caller that wants to degrade on
            a failed read owns that policy and should catch around this call.
        """
        from ..backends import get_backend

        backend = get_backend(model)
        if backend.name != "redis":
            return cls._sample_related_keys_planned(
                backend, model, field_name, related_key, count
            )
        related_db_key = (
            related_key
            if isinstance(related_key, DB_key)
            else DB_key.from_redis_key(related_key)
        )
        reverse_index_key = DB_key(
            cls.get_special_use_field_db_key(model, field_name),
            related_db_key,
        ).redis_key
        members: Any = get_REDIS_DB().srandmember(reverse_index_key, count)
        return [
            member.decode("utf-8") if isinstance(member, bytes) else str(member)
            for member in members or []
        ]

    @classmethod
    def _sample_related_keys_planned(
        cls,
        backend: Any,
        model: "Model",
        field_name: str,
        related_key: Union[str, DB_key],
        count: int,
    ) -> list[str]:
        """:meth:`sample_related_keys` on a backend without the reverse-index
        Set (#759 M1.1): an id-only select ``WHERE <field> = <related key>
        ORDER BY random() LIMIT count`` (plan §1, ``OrderTerm`` ``RANDOM``).

        ``SRANDMEMBER``'s count contract is kept: a positive count yields up
        to ``count`` distinct members, ``0`` none, and a negative count
        exactly ``-count`` members with repeats allowed.
        """
        import random

        from ..backends import RANDOM, Cond, Op, QueryPlan

        if count == 0:
            return []
        key = related_key.redis_key if isinstance(related_key, DB_key) else related_key
        if isinstance(key, bytes):
            key = key.decode("utf-8")
        where = Cond(field_name, Op.EXACT, str(key))
        if count > 0:
            plan = QueryPlan(where=where, order_by=(RANDOM,), limit=count, project=())
            rows = backend.select(model._meta.spec, plan)
            return [row["_id"].canonical for row in rows]
        rows = backend.select(model._meta.spec, QueryPlan(where=where, project=()))
        members = [row["_id"].canonical for row in rows]
        return random.choices(members, k=-count) if members else []

`remap_references(field_name, field_value, key_map, **kwargs)` `classmethod` ¶

Rewrite a stored redis_key to the target's regenerated key.

This is the one field Popoto ships that declares a reference, so it is the one field preserve_keys=False can remap. The value is a plain redis_key string (encode_popoto_model_obj treats such a string as already being in storage format), so the remap is a string substitution and the related record is never loaded -- deliberately. Relationship lazy-loads to survive circular references, and hydrating the target here would recurse right back through the pointer being remapped.

A target absent from key_map keeps its old key: it either was not part of this import or belongs to a model the caller did not chain a key_map from, and inventing a key for it would be worse than a pointer the import report counts and names.

Parameters:

Name	Type	Description	Default
`field_name`	`str`	Name of this field on the model.	required
`field_value`	`Any`	The exported value -- a `redis_key` string, or `None` for an unset relationship (`null` defaults True).	required
`key_map`	`dict[str, str]`	Old `redis_key` to new `redis_key`.	required
`**kwargs`	`Any`	Reserved for future extension.	`{}`

Returns:

Type	Description
`Any`	The remapped `redis_key`, or `field_value` unchanged when it
`Any`	is not a non-empty string or the target is not in `key_map`.

Source code in src/popoto/fields/relationship.py

@classmethod
def remap_references(
    cls,
    field_name: str,
    field_value: Any,
    key_map: "dict[str, str]",
    **kwargs: Any,
) -> Any:
    """Rewrite a stored ``redis_key`` to the target's regenerated key.

    This is the one field Popoto ships that declares a reference, so it
    is the one field ``preserve_keys=False`` can remap. The value is a
    plain ``redis_key`` string (``encode_popoto_model_obj`` treats such a
    string as already being in storage format), so the remap is a string
    substitution and the related record is never loaded -- deliberately.
    ``Relationship`` lazy-loads to survive circular references, and
    hydrating the target here would recurse right back through the
    pointer being remapped.

    A target absent from ``key_map`` keeps its old key: it either was not
    part of this import or belongs to a model the caller did not chain a
    ``key_map`` from, and inventing a key for it would be worse than a
    pointer the import report counts and names.

    Args:
        field_name: Name of this field on the model.
        field_value: The exported value -- a ``redis_key`` string, or
            ``None`` for an unset relationship (``null`` defaults True).
        key_map: Old ``redis_key`` to new ``redis_key``.
        **kwargs: Reserved for future extension.

    Returns:
        The remapped ``redis_key``, or ``field_value`` unchanged when it
        is not a non-empty string or the target is not in ``key_map``.
    """
    if isinstance(field_value, str) and field_value:
        return key_map.get(field_value, field_value)
    return field_value

`is_valid(field, value, null_check=True, **kwargs)` `classmethod` ¶

Validate a Relationship field value.

Accepts three value types as valid: - A Model instance (the related model) - A str containing a redis_key reference (lazy-loaded, format "ClassName:key") - None (if field.null is True)

This override is necessary because the base Field.is_valid() only checks isinstance(value, field.type), which rejects the str redis_key values that are set during lazy loading from Redis.

Source code in src/popoto/fields/relationship.py

@classmethod
def is_valid(cls, field, value, null_check=True, **kwargs) -> bool:
    """
    Validate a Relationship field value.

    Accepts three value types as valid:
    - A Model instance (the related model)
    - A str containing a redis_key reference (lazy-loaded, format "ClassName:key")
    - None (if field.null is True)

    This override is necessary because the base Field.is_valid() only checks
    isinstance(value, field.type), which rejects the str redis_key values
    that are set during lazy loading from Redis.
    """
    if not null_check and value is None:
        return True
    if field.null and value is None:
        return True
    if value is None:
        logger.error(f"Relationship field {field} is null but null=False")
        return False
    # Accept redis_key strings (lazy-loaded relationship references)
    if isinstance(value, str):
        if ":" not in value:
            logger.error(
                f"Relationship field {field} has invalid redis_key string: {value}"
            )
            return False
        return True
    # Accept Model instances
    from ..models.base import Model

    if isinstance(value, Model):
        return True
    logger.error(
        f"Relationship field {field} expected Model instance or redis_key string, "
        f"got {type(value)}"
    )
    return False

`get_filter_query_params(field_name)` ¶

Build the set of valid query parameters for filtering on this relationship.

This method enables Django-style double-underscore query syntax by traversing into the related model and collecting its filterable fields. For example, if a Book has an author Relationship to Author, and Author has a name field, this method makes author__name a valid filter parameter.

The implementation deliberately avoids recursive relationship traversal to prevent infinite loops (e.g., Person -> Friend -> Person) and to keep query compilation tractable.

Parameters:

Name	Type	Description	Default
`field_name`		The name of this Relationship field on the parent model.	required

Returns:

Type	Description
`set`	A set of valid query parameter strings, including: The field name itself (for direct model instance filtering) Chained parameters like `field_name__related_field` for each filterable field on the related model

Source code in src/popoto/fields/relationship.py

def get_filter_query_params(self, field_name) -> set:
    """
    Build the set of valid query parameters for filtering on this relationship.

    This method enables Django-style double-underscore query syntax by traversing
    into the related model and collecting its filterable fields. For example, if
    a Book has an `author` Relationship to Author, and Author has a `name` field,
    this method makes `author__name` a valid filter parameter.

    The implementation deliberately avoids recursive relationship traversal to
    prevent infinite loops (e.g., Person -> Friend -> Person) and to keep query
    compilation tractable.

    Args:
        field_name: The name of this Relationship field on the parent model.

    Returns:
        A set of valid query parameter strings, including:

            - The field name itself (for direct model instance filtering)
            - Chained parameters like `field_name__related_field` for each
              filterable field on the related model
    """
    related_field_filter_query_params = set()
    for related_field_name, related_field in self.model._meta.fields.items():
        if isinstance(related_field, Relationship):
            continue  # not ready for recursive compilation
        for related_query_param in related_field.get_filter_query_params(
            related_field_name
        ):
            related_field_filter_query_params.add(
                f"{field_name}__{related_query_param}"
            )

    return (
        super()
        .get_filter_query_params(field_name)
        .union(
            {
                f"{field_name}",
            }
        )
        .union(related_field_filter_query_params)
    )

`on_save(model_instance, field_name, field_value, pipeline=None, **kwargs)` `classmethod` ¶

Maintain the relationship index Set when a model instance is saved.

This hook is called automatically during model save operations. It updates the Redis Set that indexes which instances point to each related object. This is the core mechanism that enables efficient reverse lookups.

When the relationship value changes (e.g., player.team = team_b), the instance is removed from the old related object's index set before being added to the new one. This prevents ghost entries that would cause stale query results. The old value is detected via _saved_field_values, which stores field values at the last save/load.

The index key follows the pattern

$RelationshipF:ModelClass:field_name:related_db_key

For example, saving a Membership with person=alice would add the Membership's db_key to the Set at $RelationshipF:Membership:person:Person:Alice.

Parameters:

Name	Type	Description	Default
`model_instance`	`Model`	The model instance being saved.	required
`field_name`	`str`	The name of the Relationship field.	required
`field_value`	`Model \| str \| None`	The related Model instance (or None).	required
`pipeline`		Optional Redis pipeline for batched operations.	`None`
`**kwargs`		Additional arguments (unused, for extensibility).	`{}`

Returns:

Type	Description
	The pipeline (for chaining) or None if no pipeline was provided.

Note

When field_value is None, the instance is removed from the index Set. This handles the case where a relationship is being cleared.

Source code in src/popoto/fields/relationship.py

@classmethod
def on_save(
    cls,
    model_instance: "Model",
    field_name: str,
    field_value: "Model | str | None",
    pipeline=None,
    **kwargs,
):
    """
    Maintain the relationship index Set when a model instance is saved.

    This hook is called automatically during model save operations. It updates
    the Redis Set that indexes which instances point to each related object.
    This is the core mechanism that enables efficient reverse lookups.

    When the relationship value changes (e.g., player.team = team_b), the
    instance is removed from the old related object's index set before being
    added to the new one. This prevents ghost entries that would cause stale
    query results. The old value is detected via ``_saved_field_values``,
    which stores field values at the last save/load.

    The index key follows the pattern:
        `$RelationshipF:ModelClass:field_name:related_db_key`

    For example, saving a Membership with person=alice would add the
    Membership's db_key to the Set at `$RelationshipF:Membership:person:Person:Alice`.

    Args:
        model_instance: The model instance being saved.
        field_name: The name of the Relationship field.
        field_value: The related Model instance (or None).
        pipeline: Optional Redis pipeline for batched operations.
        **kwargs: Additional arguments (unused, for extensibility).

    Returns:
        The pipeline (for chaining) or None if no pipeline was provided.

    Note:
        When field_value is None, the instance is removed from the index Set.
        This handles the case where a relationship is being cleared.
    """
    from ..models.base import Model

    # Handle different field_value types:
    # - None: relationship was never set or being cleared
    # - Model instance: fully loaded relationship
    # - str: lazy-loaded relationship (redis_key string due to circular reference protection)
    if field_value is None:
        related_db_key = "None"
    elif isinstance(field_value, Model):
        related_db_key = field_value.db_key
    elif isinstance(field_value, str):
        # field_value is the redis_key string (lazy-loaded but never accessed)
        # Expecting format "ClassName:key_value"
        if ":" not in field_value:
            logger.error(
                f"Invalid redis_key format for {field_name}: {field_value}. Expected 'ClassName:key_value'"
            )
            return pipeline if pipeline else None
        # Parse the redis_key string into DB_key components
        related_db_key = DB_key.from_redis_key(field_value)
    else:
        # Unknown type, log and return without action
        logger.warning(
            f"Unexpected field_value type in on_save: {type(field_value)} for {field_name}"
        )
        return pipeline if pipeline else None

    # Remove from old index if the Relationship value changed.
    # _saved_field_values tracks values at last save/load.  If the current
    # value differs from the saved value, the old index Set still contains
    # this instance's key -- remove it to prevent ghost entries in queries.
    # This mirrors the proven pattern from KeyFieldMixin.on_save() (PR #150).
    saved_values = getattr(model_instance, "_saved_field_values", {})
    old_value = saved_values.get(field_name)
    if old_value is not None and old_value != field_value:
        # Resolve old value to a related_db_key using the same type-dispatch
        if isinstance(old_value, Model):
            old_related_db_key = old_value.db_key
        elif isinstance(old_value, str):
            if ":" in old_value:
                old_related_db_key = DB_key.from_redis_key(old_value)
            else:
                old_related_db_key = None
        else:
            old_related_db_key = None

        if old_related_db_key is not None:
            old_relationship_set_db_key = DB_key(
                cls.get_special_use_field_db_key(model_instance, field_name),
                old_related_db_key,
            )
            member_key = model_instance.db_key.redis_key
            if pipeline:
                pipeline.srem(old_relationship_set_db_key.redis_key, member_key)
            else:
                get_REDIS_DB().srem(
                    old_relationship_set_db_key.redis_key, member_key
                )

    # on a one-to-many, save the set of many with the related instance
    # add this instance's id to a relationship set based on the related model
    # example: "$RelationshipF:Membership:person:person_db_key"
    relationship_set_db_key = DB_key(
        cls.get_special_use_field_db_key(model_instance, field_name),
        related_db_key,
    )

    if field_value is None:
        if pipeline:
            return pipeline.srem(
                relationship_set_db_key.redis_key, model_instance.db_key.redis_key
            )
        else:
            return get_REDIS_DB().srem(
                relationship_set_db_key.redis_key, model_instance.db_key.redis_key
            )
    else:
        if pipeline:
            return pipeline.sadd(
                relationship_set_db_key.redis_key, model_instance.db_key.redis_key
            )
        else:
            return get_REDIS_DB().sadd(
                relationship_set_db_key.redis_key, model_instance.db_key.redis_key
            )

`on_delete(model_instance, field_name, field_value, pipeline=None, **kwargs)` `classmethod` ¶

Clean up the relationship index Set when a model instance is deleted.

This hook ensures referential integrity by removing the deleted instance from the relationship index. Without this cleanup, queries would return stale references to deleted objects.

Parameters:

Name	Type	Description	Default
`model_instance`	`Model`	The model instance being deleted.	required
`field_name`	`str`	The name of the Relationship field.	required
`field_value`	`Model \| str \| None`	The related Model instance (or None).	required
`pipeline`	`Pipeline`	Optional Redis pipeline for batched operations.	`None`
`**kwargs`		Additional arguments (unused, for extensibility).	`{}`

Returns:

Type	Description
	The pipeline (for chaining) or the result of SREM if no pipeline.

Warning

There is a known edge case where lazy-loaded relationships may not be fully hydrated at delete time. If the relationship was accessed through a chain (e.g., person.friend.friend), the field_value may be a key string rather than a Model instance, causing this method to fail when accessing field_value.db_key.

Source code in src/popoto/fields/relationship.py

@classmethod
def on_delete(
    cls,
    model_instance: "Model",
    field_name: str,
    field_value: "Model | str | None",
    pipeline: redis.client.Pipeline = None,
    **kwargs,
):
    """
    Clean up the relationship index Set when a model instance is deleted.

    This hook ensures referential integrity by removing the deleted instance
    from the relationship index. Without this cleanup, queries would return
    stale references to deleted objects.

    Args:
        model_instance: The model instance being deleted.
        field_name: The name of the Relationship field.
        field_value: The related Model instance (or None).
        pipeline: Optional Redis pipeline for batched operations.
        **kwargs: Additional arguments (unused, for extensibility).

    Returns:
        The pipeline (for chaining) or the result of SREM if no pipeline.

    Warning:
        There is a known edge case where lazy-loaded relationships may not
        be fully hydrated at delete time. If the relationship was accessed
        through a chain (e.g., person.friend.friend), the field_value may
        be a key string rather than a Model instance, causing this method
        to fail when accessing field_value.db_key.
    """
    # todo: it's possible this instance is not fully loaded or has been changed.
    #  Need to reload from db before deleting
    #  Example: on person.friend.delete() the person.friend.friend will have field_value as a keystring
    #  It will not be a model instance, so this method will fail on field_value.db_key below
    from ..models.base import Model

    # Handle different field_value types:
    # - None: relationship was never set
    # - Model instance: fully loaded relationship
    # - str: lazy-loaded relationship (redis_key string due to circular reference protection)
    if field_value is None:
        related_db_key = "None"
    elif isinstance(field_value, Model):
        related_db_key = field_value.db_key
    elif isinstance(field_value, str):
        # field_value is the redis_key string (lazy-loaded but never accessed)
        # Expecting format "ClassName:key_value"
        if ":" not in field_value:
            logger.error(
                f"Invalid redis_key format for {field_name}: {field_value}. Expected 'ClassName:key_value'"
            )
            return pipeline if pipeline else None
        # Parse the redis_key string into DB_key components
        related_db_key = DB_key.from_redis_key(field_value)
    else:
        # Unknown type, log and return without action
        logger.warning(
            f"Unexpected field_value type in on_delete: {type(field_value)} for {field_name}"
        )
        return pipeline if pipeline else None

    relationship_set_db_key = DB_key(
        cls.get_special_use_field_db_key(model_instance, field_name),
        related_db_key,
    )
    # Use saved_redis_key if provided, otherwise fall back to current db_key
    member_key = kwargs.get("saved_redis_key", model_instance.db_key.redis_key)
    if pipeline:
        return pipeline.srem(relationship_set_db_key.redis_key, member_key)
    else:
        return get_REDIS_DB().srem(relationship_set_db_key.redis_key, member_key)

`filter_query(model, field_name, **query_params)` `classmethod` ¶

Execute a filter query on a Relationship field, returning matching db_keys.

This method handles two types of queries:

Direct Instance Filtering (field_name=instance): Looks up the relationship index Set directly. For example, Membership.query.filter(person=alice) retrieves the Set at $RelationshipF:Membership:person:Person:Alice.
Chained Field Filtering (field_name__related_field=value): First queries the related model to find matching instances, then looks up each of their relationship index Sets. For example, Membership.query.filter(person__name="Alice") first finds all Persons named Alice, then retrieves all Memberships pointing to them.

The chained approach supports recursive traversal through multiple relationships, though this can become expensive for deep chains.

Parameters:

Name	Type	Description	Default
`model`	`Model`	The Model class being queried.	required
`field_name`	`str`	The name of the Relationship field.	required
`**query_params`		Filter parameters (e.g., person=alice or person__name="Alice").	`{}`

Returns:

Type	Description
`set`	A set of db_key bytes for matching model instances. Returns an empty
`set`	set if no matches are found.

Raises:

Type	Description
`QueryException`	If filtering directly on the field with a non-Model value.

Performance Note

Direct instance filtering is O(1) as it's a single Set lookup. Chained filtering requires querying the related model first, then performing N Set lookups where N is the number of matching related instances. Results are intersected if multiple filters are applied.

Source code in src/popoto/fields/relationship.py

@classmethod
def filter_query(cls, model: "Model", field_name: str, **query_params) -> set:
    """
    Execute a filter query on a Relationship field, returning matching db_keys.

    This method handles two types of queries:

    1. **Direct Instance Filtering** (`field_name=instance`):
       Looks up the relationship index Set directly. For example,
       `Membership.query.filter(person=alice)` retrieves the Set at
       `$RelationshipF:Membership:person:Person:Alice`.

    2. **Chained Field Filtering** (`field_name__related_field=value`):
       First queries the related model to find matching instances, then
       looks up each of their relationship index Sets. For example,
       `Membership.query.filter(person__name="Alice")` first finds all
       Persons named Alice, then retrieves all Memberships pointing to them.

    The chained approach supports recursive traversal through multiple
    relationships, though this can become expensive for deep chains.

    Args:
        model: The Model class being queried.
        field_name: The name of the Relationship field.
        **query_params: Filter parameters (e.g., person=alice or person__name="Alice").

    Returns:
        A set of db_key bytes for matching model instances. Returns an empty
        set if no matches are found.

    Raises:
        QueryException: If filtering directly on the field with a non-Model value.

    Performance Note:
        Direct instance filtering is O(1) as it's a single Set lookup.
        Chained filtering requires querying the related model first, then
        performing N Set lookups where N is the number of matching related
        instances. Results are intersected if multiple filters are applied.
    """
    from ..models.base import Model

    keys_lists_to_intersect = list()
    pipeline = get_REDIS_DB().pipeline()

    for query_param, query_value in query_params.items():
        if query_param == f"{field_name}":
            if not isinstance(query_value, Model):
                raise QueryException(
                    f"Query filter on Relationship expects model instance. Instead, got {query_value}"
                )

            relationship_set_db_key = DB_key(
                cls.get_special_use_field_db_key(model, field_name),
                query_value.db_key,
            )
            keys_lists_to_intersect.append(
                get_REDIS_DB().smembers(relationship_set_db_key.redis_key)
            )

        elif query_param.startswith(f"{field_name}__"):
            field = model._meta.fields[field_name]
            relationship_field_name = query_param.strip(f"{field_name}__").split(
                "__"
            )[0]

            relationship_field_values = field.model._meta.fields[
                relationship_field_name
            ].filter_query(
                model=field.model,
                field_name=relationship_field_name,
                **{query_param.strip(f"{field_name}__"): query_value},
            )
            # note: this will be recursive if references another relationship and so forth

            for relationship_field_value in relationship_field_values:
                relationship_set_db_key = DB_key(
                    cls.get_special_use_field_db_key(model, field_name),
                    relationship_field_value.db_key,
                )
                pipeline.smembers(relationship_set_db_key.redis_key)

    keys_lists_to_intersect += pipeline.execute()
    logger.debug(keys_lists_to_intersect)
    if len(keys_lists_to_intersect):
        return set.intersection(
            *[set(key_list) for key_list in keys_lists_to_intersect]
        )
    return set()

`sample_related_keys(model, field_name, related_key, count)` `classmethod` ¶

Return a bounded random sample of the redis_keys pointing at related_key.

Reads the Relationship reverse index Set — the same $RelationshipF:<Model>:<field_name>:<related_key> Set that :meth:on_save populates and :meth:filter_query reads in full — and issues a single SRANDMEMBER bounded by count.

This is the bounded counterpart to :meth:filter_query's SMEMBERS. On a hub node the full Set can be arbitrarily large, so a caller that walks edges (graph traversal, fan-out-limited expansion) needs a sampled read rather than an unbounded one. filter_query cannot serve that need: it is unbounded, it requires a loaded Model instance rather than a key, and it wraps its reads in a pipeline.

Parameters:

Name	Type	Description	Default
`model`	`Model`	The Model class owning the Relationship field. Typed `"Model"` rather than `Type["Model"]` to match :meth:`filter_query` and :meth:`on_save`, which are annotated the same way and are likewise called with the class.	required
`field_name`	`str`	The name of the Relationship field.	required
`related_key`	`Union[str, DB_key]`	The key of the related instance being pointed at. A `DB_key` is used as-is; a `str` is parsed with :meth:`DB_key.from_redis_key`, which unescapes the colon separators — constructing `DB_key(some_redis_key)` instead would escape them and silently address a different key. The `DB_key` arm exists for signature symmetry with :meth:`filter_query` and :meth:`on_delete`, which both hold a `DB_key`; the only caller today passes a `str`.	required
`count`	`int`	Maximum number of members to return, passed through to `SRANDMEMBER` unchanged. A positive count yields distinct members with no ordering guarantee; `0` yields an empty list.	required

Returns:

Type	Description
`list[str]`	A list of member redis_keys decoded to `str`. Empty if the reverse
`list[str]`	index does not exist.

Note

No exception handling by design — a caller that wants to degrade on a failed read owns that policy and should catch around this call.

Source code in src/popoto/fields/relationship.py

@classmethod
def sample_related_keys(
    cls,
    model: "Model",
    field_name: str,
    related_key: Union[str, DB_key],
    count: int,
) -> list[str]:
    """
    Return a bounded random sample of the redis_keys pointing at ``related_key``.

    Reads the Relationship reverse index Set — the same
    ``$RelationshipF:<Model>:<field_name>:<related_key>`` Set that
    :meth:`on_save` populates and :meth:`filter_query` reads in full — and
    issues a single ``SRANDMEMBER`` bounded by ``count``.

    This is the bounded counterpart to :meth:`filter_query`'s ``SMEMBERS``.
    On a hub node the full Set can be arbitrarily large, so a caller that
    walks edges (graph traversal, fan-out-limited expansion) needs a sampled
    read rather than an unbounded one. ``filter_query`` cannot serve that
    need: it is unbounded, it requires a loaded ``Model`` instance rather
    than a key, and it wraps its reads in a pipeline.

    Args:
        model: The Model class owning the Relationship field. Typed
            ``"Model"`` rather than ``Type["Model"]`` to match
            :meth:`filter_query` and :meth:`on_save`, which are annotated
            the same way and are likewise called with the class.
        field_name: The name of the Relationship field.
        related_key: The key of the related instance being pointed at.
            A ``DB_key`` is used as-is; a ``str`` is parsed with
            :meth:`DB_key.from_redis_key`, which unescapes the colon
            separators — constructing ``DB_key(some_redis_key)`` instead
            would escape them and silently address a different key. The
            ``DB_key`` arm exists for signature symmetry with
            :meth:`filter_query` and :meth:`on_delete`, which both hold a
            ``DB_key``; the only caller today passes a ``str``.
        count: Maximum number of members to return, passed through to
            ``SRANDMEMBER`` unchanged. A positive count yields distinct
            members with no ordering guarantee; ``0`` yields an empty list.

    Returns:
        A list of member redis_keys decoded to ``str``. Empty if the reverse
        index does not exist.

    Note:
        No exception handling by design — a caller that wants to degrade on
        a failed read owns that policy and should catch around this call.
    """
    from ..backends import get_backend

    backend = get_backend(model)
    if backend.name != "redis":
        return cls._sample_related_keys_planned(
            backend, model, field_name, related_key, count
        )
    related_db_key = (
        related_key
        if isinstance(related_key, DB_key)
        else DB_key.from_redis_key(related_key)
    )
    reverse_index_key = DB_key(
        cls.get_special_use_field_db_key(model, field_name),
        related_db_key,
    ).redis_key
    members: Any = get_REDIS_DB().srandmember(reverse_index_key, count)
    return [
        member.decode("utf-8") if isinstance(member, bytes) else str(member)
        for member in members or []
    ]

popoto.fields.relationship¶