`popoto.fields.indexed_field_mixin`¶

`popoto.fields.indexed_field_mixin` ¶

Indexed Field Mixin - Secondary Indexing for Non-Key Fields¶

This module provides the IndexedFieldMixin class, which enables exact-match secondary indexing on fields that are NOT part of the model's Redis key.

Design Philosophy¶

KeyField conflates two concerns: identity (forming the Redis storage key) and indexing (enabling queries). IndexedFieldMixin decouples these by providing Set-based indexing without making the field part of the Redis key.

This allows developers to query on fields like email, status, or category without making them part of the model's identity.

The implementation follows the exact same pattern as KeyFieldMixin: - on_save(): Maintains a Redis Set at $IdxF:Model:field_name:value - on_delete(): Removes from the Set - filter_query(): Uses SMEMBERS/SUNION for lookups - get_filter_query_params(): Declares supported query lookups

Index Key Pattern¶

$IdxF:ModelName:field_name:value -> Set of redis_keys

This mirrors the $KeyF pattern used by KeyFieldMixin but uses the $IdxF prefix (auto-generated by FieldBase metaclass via field_class_key).

Usage¶

from popoto import Model, Field
from popoto.fields.shortcuts import IndexedField, UniqueField

class User(Model):
    user_id = AutoKeyField()
    email = UniqueField(type=str)       # indexed + unique
    status = IndexedField(type=str)     # indexed, not unique

# Efficient exact-match queries without making email a KeyField
User.query.filter(email="alice@example.com")
User.query.filter(status="active")
User.query.filter(status__in=["active", "pending"])

`IndexedFieldMixin` ¶

Mixin that provides Set-based secondary indexing for non-key fields.

When mixed with Field, this mixin maintains Redis Sets that track which model instances have a given value for the indexed field. This enables efficient exact-match queries without making the field part of the model's Redis key (identity).

Supports the same query lookups as KeyFieldMixin: - Exact match: filter(field=value) - IN queries: filter(field__in=[v1, v2]) - Null checks: filter(field__isnull=True/False) - Pattern matching: filter(field__startswith="prefix") - Pattern matching: filter(field__endswith="suffix")

Uniqueness enforcement is available via unique=True, which checks the index Set before adding a new entry. This is best-effort under concurrent writes (same trade-off as UniqueKeyField).

Attributes:

Name	Type	Description
`indexed`	`bool`	Always True for indexed fields.

Source code in src/popoto/fields/indexed_field_mixin.py

class IndexedFieldMixin:
    """
    Mixin that provides Set-based secondary indexing for non-key fields.

    When mixed with Field, this mixin maintains Redis Sets that track which
    model instances have a given value for the indexed field. This enables
    efficient exact-match queries without making the field part of the
    model's Redis key (identity).

    Supports the same query lookups as KeyFieldMixin:
    - Exact match: ``filter(field=value)``
    - IN queries: ``filter(field__in=[v1, v2])``
    - Null checks: ``filter(field__isnull=True/False)``
    - Pattern matching: ``filter(field__startswith="prefix")``
    - Pattern matching: ``filter(field__endswith="suffix")``

    Uniqueness enforcement is available via ``unique=True``, which checks
    the index Set before adding a new entry. This is best-effort under
    concurrent writes (same trade-off as UniqueKeyField).

    Attributes:
        indexed (bool): Always True for indexed fields.
    """

    indexed: bool = True

    def __init__(self, **kwargs):
        """
        Initialize IndexedFieldMixin defaults.

        Sets ``indexed=True`` and updates field_defaults for introspection.
        Calls super().__init__() for cooperative multiple inheritance.
        """
        super().__init__(**kwargs)
        indexed_defaults = {
            "indexed": True,
        }
        self.field_defaults.update(indexed_defaults)
        for k, v in indexed_defaults.items():
            setattr(self, k, kwargs.get(k, v))

    @classmethod
    def on_save(
        cls,
        model_instance: "Model",
        field_name: str,
        field_value,
        pipeline: redis.client.Pipeline = None,
        **kwargs,
    ):
        """
        Maintain the secondary index Set when a model instance is saved.

        Adds the model instance's Redis key to the Set at
        ``$IdxF:Model:field_name:value``. If the value has changed since
        last save, removes the old index entry first.

        When ``unique=True``, checks the target Set before adding. Raises
        ModelException if another instance already occupies that value.
        Note: this uniqueness check has a race condition under concurrent
        writes -- same trade-off as UniqueKeyField.

        All operations use the provided pipeline for zero extra round-trips.

        Args:
            model_instance: The Model instance being saved.
            field_name: The name of this field on the model.
            field_value: The value being saved for this field.
            pipeline: Optional Redis pipeline for batched operations.

        Returns:
            The pipeline (if provided) or the SADD result.

        Raises:
            ModelException: If unique=True and the value is already taken
                by a different instance.
        """
        field = model_instance._meta.fields[field_name]
        member_key = model_instance.db_key.redis_key

        # Remove from old index if value changed
        saved_values = getattr(model_instance, "_saved_field_values", {})
        old_value = saved_values.get(field_name)
        if old_value is not None and old_value != field_value:
            old_set_key = DB_key(
                cls.get_special_use_field_db_key(model_instance, field_name),
                old_value,
            )
            if pipeline:
                pipeline.srem(old_set_key.redis_key, member_key)
            else:
                POPOTO_REDIS_DB.srem(old_set_key.redis_key, member_key)

        # Uniqueness check: if unique=True, verify no other instance has this value
        if field.unique and field_value is not None:
            target_set_key = DB_key(
                cls.get_special_use_field_db_key(model_instance, field_name),
                field_value,
            )
            existing_members = POPOTO_REDIS_DB.smembers(target_set_key.redis_key)
            # Filter out self (re-saving same instance is OK)
            other_members = {
                m
                for m in existing_members
                if m != member_key.encode() and m != member_key
            }
            if other_members:
                raise ModelException(
                    f"Uniqueness violation on {model_instance.__class__.__name__}.{field_name}: "
                    f"value '{field_value}' is already taken by another instance"
                )

        # Add to index Set
        index_set_key = DB_key(
            cls.get_special_use_field_db_key(model_instance, field_name),
            field_value,
        )
        if pipeline:
            return pipeline.sadd(index_set_key.redis_key, member_key)
        else:
            return POPOTO_REDIS_DB.sadd(index_set_key.redis_key, member_key)

    @classmethod
    def on_delete(
        cls,
        model_instance: "Model",
        field_name: str,
        field_value,
        pipeline: redis.client.Pipeline = None,
        **kwargs,
    ):
        """
        Remove the model instance from the index Set on delete.

        Args:
            model_instance: The Model instance being deleted.
            field_name: The name of this field on the model.
            field_value: The value stored for this field.
            pipeline: Optional Redis pipeline for batched operations.

        Returns:
            The pipeline (if provided) or the SREM result.
        """
        index_set_key = DB_key(
            cls.get_special_use_field_db_key(model_instance, field_name),
            field_value,
        )
        member_key = kwargs.get("saved_redis_key", model_instance.db_key.redis_key)
        if pipeline:
            return pipeline.srem(index_set_key.redis_key, member_key)
        else:
            return POPOTO_REDIS_DB.srem(index_set_key.redis_key, member_key)

    def get_filter_query_params(self, field_name: str) -> set:
        """
        Return the set of valid query parameter names for filtering on this field.

        Supports the same lookups as KeyFieldMixin:
        - exact match
        - __in
        - __isnull
        - __startswith
        - __endswith

        Args:
            field_name: The name of this field on the model.

        Returns:
            set: Valid query parameter strings.
        """
        return (
            super()
            .get_filter_query_params(field_name)
            .union(
                {
                    f"{field_name}",
                    f"{field_name}__isnull",
                    f"{field_name}__startswith",
                    f"{field_name}__endswith",
                    f"{field_name}__in",
                }
            )
        )

    @classmethod
    def filter_query(cls, model: "Model", field_name: str, **query_params) -> set:
        """
        Execute a filter query on this indexed field and return matching Redis keys.

        Uses Set-based lookups for exact match and __in queries. Falls back
        to SCAN for pattern queries (__startswith, __endswith).

        Args:
            model: The Model class being queried.
            field_name: The name of this field on the model.
            **query_params: Dict mapping query parameter names to values.

        Returns:
            set: Redis keys (as bytes) of matching model instances.

        Raises:
            QueryException: If __isnull receives a non-boolean value.
        """
        keys_lists_to_intersect = list()

        redis_set_key_prefix = model._meta.fields[
            field_name
        ].get_special_use_field_db_key(model, field_name)

        for query_param, query_value in query_params.items():
            if query_param.endswith("__in"):
                # Use SUNION for efficient server-side set union
                set_keys = [
                    DB_key(redis_set_key_prefix, query_value_elem).redis_key
                    for query_value_elem in query_value
                ]
                if set_keys:
                    keys_lists_to_intersect.append(POPOTO_REDIS_DB.sunion(set_keys))
                else:
                    keys_lists_to_intersect.append(set())

            elif query_param == f"{field_name}":
                keys_lists_to_intersect.append(
                    POPOTO_REDIS_DB.smembers(
                        DB_key(redis_set_key_prefix, query_value).redis_key
                    )
                )

            elif query_param.endswith("__isnull"):
                if query_value is True:
                    keys_lists_to_intersect.append(
                        POPOTO_REDIS_DB.smembers(
                            DB_key(redis_set_key_prefix, None).redis_key
                        )
                    )
                elif query_value is False:
                    # Use SCAN to find all index keys for this field, then union
                    pattern = f"{redis_set_key_prefix.redis_key}:*"
                    all_keys = scan_keys(pattern)
                    # Filter out the None set
                    none_key = DB_key(redis_set_key_prefix, None).redis_key
                    non_null_keys = set()
                    for key in all_keys:
                        key_str = key.decode() if isinstance(key, bytes) else key
                        if key_str != none_key:
                            non_null_keys.update(POPOTO_REDIS_DB.smembers(key_str))
                    keys_lists_to_intersect.append(non_null_keys)
                else:
                    raise QueryException(f"{query_param} filter must be True or False")

            elif query_param.endswith("__startswith"):
                # Scan index keys matching the prefix pattern
                pattern = (
                    f"{redis_set_key_prefix.redis_key}:{DB_key.clean(query_value)}*"
                )
                matching_index_keys = scan_keys(pattern)
                members = set()
                for idx_key in matching_index_keys:
                    members.update(POPOTO_REDIS_DB.smembers(idx_key))
                keys_lists_to_intersect.append(members)

            elif query_param.endswith("__endswith"):
                # Scan index keys matching the suffix pattern
                pattern = (
                    f"{redis_set_key_prefix.redis_key}:*{DB_key.clean(query_value)}"
                )
                matching_index_keys = scan_keys(pattern)
                members = set()
                for idx_key in matching_index_keys:
                    members.update(POPOTO_REDIS_DB.smembers(idx_key))
                keys_lists_to_intersect.append(members)

        logger.debug(keys_lists_to_intersect)
        if len(keys_lists_to_intersect):
            return set.intersection(
                *[set(key_list) for key_list in keys_lists_to_intersect]
            )
        return set()

`on_save(model_instance, field_name, field_value, pipeline=None, **kwargs)` `classmethod` ¶

Maintain the secondary index Set when a model instance is saved.

Adds the model instance's Redis key to the Set at $IdxF:Model:field_name:value. If the value has changed since last save, removes the old index entry first.

When unique=True, checks the target Set before adding. Raises ModelException if another instance already occupies that value. Note: this uniqueness check has a race condition under concurrent writes -- same trade-off as UniqueKeyField.

All operations use the provided pipeline for zero extra round-trips.

Parameters:

Name	Type	Description	Default
`model_instance`	`Model`	The Model instance being saved.	required
`field_name`	`str`	The name of this field on the model.	required
`field_value`		The value being saved for this field.	required
`pipeline`	`Pipeline`	Optional Redis pipeline for batched operations.	`None`

Returns:

Type	Description
	The pipeline (if provided) or the SADD result.

Raises:

Type	Description
`ModelException`	If unique=True and the value is already taken by a different instance.

Source code in src/popoto/fields/indexed_field_mixin.py

@classmethod
def on_save(
    cls,
    model_instance: "Model",
    field_name: str,
    field_value,
    pipeline: redis.client.Pipeline = None,
    **kwargs,
):
    """
    Maintain the secondary index Set when a model instance is saved.

    Adds the model instance's Redis key to the Set at
    ``$IdxF:Model:field_name:value``. If the value has changed since
    last save, removes the old index entry first.

    When ``unique=True``, checks the target Set before adding. Raises
    ModelException if another instance already occupies that value.
    Note: this uniqueness check has a race condition under concurrent
    writes -- same trade-off as UniqueKeyField.

    All operations use the provided pipeline for zero extra round-trips.

    Args:
        model_instance: The Model instance being saved.
        field_name: The name of this field on the model.
        field_value: The value being saved for this field.
        pipeline: Optional Redis pipeline for batched operations.

    Returns:
        The pipeline (if provided) or the SADD result.

    Raises:
        ModelException: If unique=True and the value is already taken
            by a different instance.
    """
    field = model_instance._meta.fields[field_name]
    member_key = model_instance.db_key.redis_key

    # Remove from old index if value changed
    saved_values = getattr(model_instance, "_saved_field_values", {})
    old_value = saved_values.get(field_name)
    if old_value is not None and old_value != field_value:
        old_set_key = DB_key(
            cls.get_special_use_field_db_key(model_instance, field_name),
            old_value,
        )
        if pipeline:
            pipeline.srem(old_set_key.redis_key, member_key)
        else:
            POPOTO_REDIS_DB.srem(old_set_key.redis_key, member_key)

    # Uniqueness check: if unique=True, verify no other instance has this value
    if field.unique and field_value is not None:
        target_set_key = DB_key(
            cls.get_special_use_field_db_key(model_instance, field_name),
            field_value,
        )
        existing_members = POPOTO_REDIS_DB.smembers(target_set_key.redis_key)
        # Filter out self (re-saving same instance is OK)
        other_members = {
            m
            for m in existing_members
            if m != member_key.encode() and m != member_key
        }
        if other_members:
            raise ModelException(
                f"Uniqueness violation on {model_instance.__class__.__name__}.{field_name}: "
                f"value '{field_value}' is already taken by another instance"
            )

    # Add to index Set
    index_set_key = DB_key(
        cls.get_special_use_field_db_key(model_instance, field_name),
        field_value,
    )
    if pipeline:
        return pipeline.sadd(index_set_key.redis_key, member_key)
    else:
        return POPOTO_REDIS_DB.sadd(index_set_key.redis_key, member_key)

`on_delete(model_instance, field_name, field_value, pipeline=None, **kwargs)` `classmethod` ¶

Remove the model instance from the index Set on delete.

Parameters:

Name	Type	Description	Default
`model_instance`	`Model`	The Model instance being deleted.	required
`field_name`	`str`	The name of this field on the model.	required
`field_value`		The value stored for this field.	required
`pipeline`	`Pipeline`	Optional Redis pipeline for batched operations.	`None`

Returns:

Type	Description
	The pipeline (if provided) or the SREM result.

Source code in src/popoto/fields/indexed_field_mixin.py

@classmethod
def on_delete(
    cls,
    model_instance: "Model",
    field_name: str,
    field_value,
    pipeline: redis.client.Pipeline = None,
    **kwargs,
):
    """
    Remove the model instance from the index Set on delete.

    Args:
        model_instance: The Model instance being deleted.
        field_name: The name of this field on the model.
        field_value: The value stored for this field.
        pipeline: Optional Redis pipeline for batched operations.

    Returns:
        The pipeline (if provided) or the SREM result.
    """
    index_set_key = DB_key(
        cls.get_special_use_field_db_key(model_instance, field_name),
        field_value,
    )
    member_key = kwargs.get("saved_redis_key", model_instance.db_key.redis_key)
    if pipeline:
        return pipeline.srem(index_set_key.redis_key, member_key)
    else:
        return POPOTO_REDIS_DB.srem(index_set_key.redis_key, member_key)

`get_filter_query_params(field_name)` ¶

Return the set of valid query parameter names for filtering on this field.

Supports the same lookups as KeyFieldMixin: - exact match - __in - __isnull - __startswith - __endswith

Parameters:

Name	Type	Description	Default
`field_name`	`str`	The name of this field on the model.	required

Returns:

Name	Type	Description
`set`	`set`	Valid query parameter strings.

Source code in src/popoto/fields/indexed_field_mixin.py

def get_filter_query_params(self, field_name: str) -> set:
    """
    Return the set of valid query parameter names for filtering on this field.

    Supports the same lookups as KeyFieldMixin:
    - exact match
    - __in
    - __isnull
    - __startswith
    - __endswith

    Args:
        field_name: The name of this field on the model.

    Returns:
        set: Valid query parameter strings.
    """
    return (
        super()
        .get_filter_query_params(field_name)
        .union(
            {
                f"{field_name}",
                f"{field_name}__isnull",
                f"{field_name}__startswith",
                f"{field_name}__endswith",
                f"{field_name}__in",
            }
        )
    )

`filter_query(model, field_name, **query_params)` `classmethod` ¶

Execute a filter query on this indexed field and return matching Redis keys.

Uses Set-based lookups for exact match and __in queries. Falls back to SCAN for pattern queries (__startswith, __endswith).

Parameters:

Name	Type	Description	Default
`model`	`Model`	The Model class being queried.	required
`field_name`	`str`	The name of this field on the model.	required
`**query_params`		Dict mapping query parameter names to values.	`{}`

Returns:

Name	Type	Description
`set`	`set`	Redis keys (as bytes) of matching model instances.

Raises:

Type	Description
`QueryException`	If __isnull receives a non-boolean value.

Source code in src/popoto/fields/indexed_field_mixin.py

@classmethod
def filter_query(cls, model: "Model", field_name: str, **query_params) -> set:
    """
    Execute a filter query on this indexed field and return matching Redis keys.

    Uses Set-based lookups for exact match and __in queries. Falls back
    to SCAN for pattern queries (__startswith, __endswith).

    Args:
        model: The Model class being queried.
        field_name: The name of this field on the model.
        **query_params: Dict mapping query parameter names to values.

    Returns:
        set: Redis keys (as bytes) of matching model instances.

    Raises:
        QueryException: If __isnull receives a non-boolean value.
    """
    keys_lists_to_intersect = list()

    redis_set_key_prefix = model._meta.fields[
        field_name
    ].get_special_use_field_db_key(model, field_name)

    for query_param, query_value in query_params.items():
        if query_param.endswith("__in"):
            # Use SUNION for efficient server-side set union
            set_keys = [
                DB_key(redis_set_key_prefix, query_value_elem).redis_key
                for query_value_elem in query_value
            ]
            if set_keys:
                keys_lists_to_intersect.append(POPOTO_REDIS_DB.sunion(set_keys))
            else:
                keys_lists_to_intersect.append(set())

        elif query_param == f"{field_name}":
            keys_lists_to_intersect.append(
                POPOTO_REDIS_DB.smembers(
                    DB_key(redis_set_key_prefix, query_value).redis_key
                )
            )

        elif query_param.endswith("__isnull"):
            if query_value is True:
                keys_lists_to_intersect.append(
                    POPOTO_REDIS_DB.smembers(
                        DB_key(redis_set_key_prefix, None).redis_key
                    )
                )
            elif query_value is False:
                # Use SCAN to find all index keys for this field, then union
                pattern = f"{redis_set_key_prefix.redis_key}:*"
                all_keys = scan_keys(pattern)
                # Filter out the None set
                none_key = DB_key(redis_set_key_prefix, None).redis_key
                non_null_keys = set()
                for key in all_keys:
                    key_str = key.decode() if isinstance(key, bytes) else key
                    if key_str != none_key:
                        non_null_keys.update(POPOTO_REDIS_DB.smembers(key_str))
                keys_lists_to_intersect.append(non_null_keys)
            else:
                raise QueryException(f"{query_param} filter must be True or False")

        elif query_param.endswith("__startswith"):
            # Scan index keys matching the prefix pattern
            pattern = (
                f"{redis_set_key_prefix.redis_key}:{DB_key.clean(query_value)}*"
            )
            matching_index_keys = scan_keys(pattern)
            members = set()
            for idx_key in matching_index_keys:
                members.update(POPOTO_REDIS_DB.smembers(idx_key))
            keys_lists_to_intersect.append(members)

        elif query_param.endswith("__endswith"):
            # Scan index keys matching the suffix pattern
            pattern = (
                f"{redis_set_key_prefix.redis_key}:*{DB_key.clean(query_value)}"
            )
            matching_index_keys = scan_keys(pattern)
            members = set()
            for idx_key in matching_index_keys:
                members.update(POPOTO_REDIS_DB.smembers(idx_key))
            keys_lists_to_intersect.append(members)

    logger.debug(keys_lists_to_intersect)
    if len(keys_lists_to_intersect):
        return set.intersection(
            *[set(key_list) for key_list in keys_lists_to_intersect]
        )
    return set()

popoto.fields.indexed_field_mixin¶