Indexed Fields¶

Indexed fields provide secondary indexing for non-key fields, enabling efficient exact-match queries without making the field part of the model's Redis key (identity).

The Problem¶

In Popoto, KeyField conflates two concerns:

Identity -- the field's value forms part of the Redis storage key
Indexing -- the field is queryable via filter()

This means that to query on a field like email or status, you must make it a KeyField, which changes the model's Redis key structure. If you later rename an email, the entire Redis key changes, potentially orphaning references.

The Solution¶

IndexedField and UniqueField decouple querying from identity. They maintain Redis Set indexes (identical to KeyField's indexing mechanism) but do not participate in the Redis key.

from popoto import Model, AutoKeyField, IndexedField, UniqueField, Field

class User(Model):
    user_id = AutoKeyField()
    email = UniqueField(type=str)       # indexed + unique, NOT part of key
    status = IndexedField(type=str)     # indexed, NOT part of key
    name = Field(type=str)              # not indexed, not part of key

The Redis key for this model is based solely on user_id (e.g., User:a1b2c3d4...). The email and status fields are indexed separately, enabling queries like:

User.query.filter(email="alice@example.com")
User.query.filter(status="active")
User.query.filter(status__in=["active", "pending"])

Field(indexed=True)¶

You can also enable indexing on a plain Field by passing indexed=True:

class Product(Model):
    sku = AutoKeyField()
    category = Field(type=str, indexed=True)          # indexed, not unique
    barcode = Field(type=str, indexed=True, unique=True)  # indexed + unique

This is functionally identical to using IndexedField and UniqueField shortcuts.

IndexedField¶

IndexedField is a shortcut for Field(indexed=True). It creates a non-key field with Set-based secondary indexing.

from popoto import Model, AutoKeyField, IndexedField

class Order(Model):
    order_id = AutoKeyField()
    status = IndexedField(type=str)
    region = IndexedField(type=str, null=True)

Supported Query Lookups¶

IndexedField supports the same lookups as KeyField:

Lookup	Description	Redis Operation
`status=`	Exact match	SMEMBERS
`status__in=`	Match any value in list	SUNION
`status__isnull=`	Null / non-null check	SMEMBERS / SCAN
`status__startswith=`	Prefix match	SCAN + SMEMBERS
`status__endswith=`	Suffix match	SCAN + SMEMBERS

# Exact match
Order.query.filter(status="shipped")

# IN query -- efficient server-side SUNION
Order.query.filter(status__in=["pending", "processing"])

# Null check
Order.query.filter(region__isnull=True)

# Pattern matching (uses SCAN, slower on large datasets)
Order.query.filter(region__startswith="US-")

Combining with Other Filters¶

Indexed field filters compose with all other filter types using AND logic:

from popoto import SortedField

class Product(Model):
    product_id = AutoKeyField()
    category = IndexedField(type=str)
    price = SortedField(type=float)

# Combine indexed field filter with sorted field range query
affordable_electronics = Product.query.filter(
    category="electronics",
    price__lte=50.0,
)

UniqueField¶

UniqueField is a shortcut for Field(indexed=True, unique=True). It adds a per-value uniqueness constraint on top of the secondary index.

from popoto import Model, AutoKeyField, UniqueField, Field

class User(Model):
    user_id = AutoKeyField()
    email = UniqueField(type=str)
    name = Field(type=str)

user1 = User.create(email="alice@example.com", name="Alice")

# Attempting a duplicate email raises ModelException
try:
    user2 = User.create(email="alice@example.com", name="Not Alice")
except Exception as e:
    print(e)
    # => Uniqueness violation on User.email: value 'alice@example.com' is already taken

Constraints¶

UniqueField cannot be null (null=False is enforced)
Setting unique=False raises ModelException
Setting null=True raises ModelException

Uniqueness Trade-offs¶

The uniqueness check reads the index Set before writing. Under concurrent writes, two processes could both read an empty Set and both proceed to write, resulting in a duplicate. This is the same best-effort trade-off as UniqueKeyField.

For applications requiring strict uniqueness guarantees under high concurrency, implement application-level locking or use Redis transactions.

How It Works¶

Index Key Pattern¶

Indexed fields maintain Redis Sets following this pattern:

$IdxF:ModelName:field_name:value -> Set of redis_keys

For example, a User model with status = IndexedField(type=str):

$IdxF:User:status:active    -> {User:abc123, User:def456}
$IdxF:User:status:inactive  -> {User:ghi789}

This mirrors the $KeyF pattern used by KeyField but uses the $IdxF prefix.

Save Behavior¶

When a model instance is saved:

If the indexed value changed, the old Set entry is removed
If unique=True, the target Set is checked for existing members
The instance's Redis key is added to the Set for the current value

All operations are pipelined with the main model save for zero extra round-trips.

Delete Behavior¶

When a model instance is deleted, the instance's Redis key is removed from the index Set.

Comparison Table¶

Feature	Field	IndexedField	UniqueField	KeyField	UniqueKeyField
Part of Redis key	No	No	No	Yes	Yes
Queryable via filter()	No	Yes	Yes	Yes	Yes
Exact match	No	Yes	Yes	Yes	Yes
`__in` lookup	No	Yes	Yes	Yes	Yes
`__startswith` / `__endswith`	No	Yes	Yes	Yes	Yes
`__isnull` lookup	No	Yes	Yes	Yes	Yes
`__contains` lookup	No	No	No	Yes	No
Uniqueness enforced	No	No	Yes	No	Yes
Can be null	Yes	Yes	No	Yes	No

Immutable Keys and Key Migration¶

KeyField values are immutable by default after the initial save. Changing a KeyField value and calling save() raises KeyMutationError:

from popoto import KeyMutationError

instance = MyModel.query.get(name="old_name")
instance.name = "new_name"

try:
    instance.save()
except KeyMutationError as e:
    print(e)
    # => KeyField 'name' changed from 'old_name' to 'new_name'. Use save(migrate_key=True).

To intentionally migrate a key, pass migrate_key=True:

instance.name = "new_name"
instance.save(migrate_key=True)  # Succeeds, old key is cleaned up

The migrate_key=True flag handles the full migration: deleting the old Redis hash, removing old index entries (sorted sets, geo sets, unique constraints, indexed fields), and creating the new key with all indexes pointing to it.

Warning

Key migration changes the Redis key (identity) of the instance. Any external references to the old key will break. Use this intentionally, not accidentally.