popoto.fields.sorted_field_mixin¶
popoto.fields.sorted_field_mixin
¶
Sorted Field Mixin for Redis Sorted Set-backed range queries.
This module provides the SortedFieldMixin class, which enables efficient range-based filtering on numeric and temporal fields by leveraging Redis Sorted Sets (ZSET).
Design Philosophy
Redis stores all data as strings, making range queries on numeric values impossible without additional data structures. The SortedFieldMixin solves this by maintaining a parallel Sorted Set index alongside each model instance. When a model is saved, its Redis key is added to the Sorted Set with the field value as the score, enabling O(log(N)) range queries via ZRANGEBYSCORE.
Trade-offs
- Storage: Each sorted field creates an additional Redis key per unique partition (or one global key if no partitioning). This is a deliberate trade-off of storage space for query performance.
- Consistency: The Sorted Set index must be updated on every save/delete. This is handled automatically through the on_save() and on_delete() hooks.
- Null values: Currently not supported. A sorted field cannot be null because Redis Sorted Sets require a numeric score. Future versions may support nulls via a separate index set.
Integration with Query System
The Query class (models/query.py) prioritizes sorted field filters because they typically return smaller result sets than key field filters. The filter_query() method returns a set of Redis keys that can be intersected with results from other field filters.
Example
class Product(Model): name = KeyField() price = SortedField(type=float) category = KeyField()
Range query - uses ZRANGEBYSCORE under the hood¶
Product.query.filter(price__gte=10.0, price__lte=50.0)
With partitioning for better performance on large datasets¶
class Product(Model): name = KeyField() price = SortedField(type=float, partition_by='category') category = KeyField()
Query must include partition field¶
Product.query.filter(category='electronics', price__lte=100.0)
SortedFieldMixin
¶
Mixin that adds Redis Sorted Set indexing to enable range-based queries.
This mixin transforms a regular Field into one that supports Django-style range lookups (__gt, __gte, __lt, __lte) by maintaining a Redis Sorted Set index. The design follows the mixin pattern so it can be combined with other field behaviors (e.g., SortedKeyField combines sorting with key field identity).
Why a Mixin
Popoto uses composition over inheritance for field capabilities. This allows creating field types like SortedKeyField that combines range query support with key field identity, without complex inheritance hierarchies.
Partitioning via partition_by
For large datasets, a single global Sorted Set becomes a bottleneck. The partition_by parameter partitions the index by one or more key fields, creating separate Sorted Sets per partition. For example, sorting products by price within each category creates one Sorted Set per category rather than one global set.
.. deprecated::
The sort_by parameter is deprecated in favor of partition_by.
sort_by still works but emits a DeprecationWarning.
Supported Types
- int, float: Used directly as Redis scores
- Decimal: Converted to float (some precision loss possible)
- datetime: Converted to Unix timestamp
- date: Converted to ordinal (days since year 1)
- time: Converted to timestamp
Attributes:
| Name | Type | Description |
|---|---|---|
type |
type
|
The Python type for this field (must be numeric or temporal). |
null |
bool
|
Must be False; sorted fields cannot be null. |
default |
Default value when none provided. |
|
partition_by |
Tuple of field names to partition the sorted index by. |
Example
Basic sorted field¶
price = SortedField(type=float)
Partitioned by category for better scaling¶
price = SortedField(type=float, partition_by='category')
Partitioned by multiple fields¶
timestamp = SortedField(type=datetime.datetime, partition_by=('exchange', 'symbol'))
Deprecated but still supported¶
price = SortedField(type=float, sort_by='category') # emits DeprecationWarning
Source code in src/popoto/fields/sorted_field_mixin.py
63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 | |
sort_by
property
writable
¶
Deprecated: use partition_by instead.
get_filter_query_params(field_name)
¶
Register the Django-style lookup parameters this field supports.
This method is called during model class construction to build a registry of valid query parameters. The Query class uses this registry to validate filter() arguments and route them to the appropriate field's filter_query() method.
The sorted field mixin adds range comparison operators following Django's double-underscore convention. The exact match parameter (field_name without suffix) is also included for equality filtering.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
field_name
|
The attribute name of this field on the model. |
required |
Returns:
| Type | Description |
|---|---|
set
|
A set of valid query parameter strings that can be passed to |
set
|
Model.query.filter(). Includes parameters from parent classes |
set
|
via super() to support mixin composition. |
Source code in src/popoto/fields/sorted_field_mixin.py
is_valid(field, value, null_check=True, **kwargs)
classmethod
¶
Validate that the value is compatible with sorted set storage.
Extends the base Field validation to ensure the value matches the declared type. This is important because the value will be converted to a numeric score for Redis storage, and type mismatches could cause conversion errors or incorrect sorting behavior.
For fields with auto_now or auto_now_add=True, null values are allowed since they will be automatically populated in format_value_pre_save().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
field
|
The Field instance being validated. |
required | |
value
|
The value to validate. |
required | |
null_check
|
Whether to enforce null constraints. |
True
|
|
**kwargs
|
Additional validation context. |
{}
|
Returns:
| Type | Description |
|---|---|
bool
|
True if the value is valid for this sorted field, False otherwise. |
Source code in src/popoto/fields/sorted_field_mixin.py
format_value_pre_save(field_value, skip_auto_now=False)
¶
Normalize the field value before saving to the model's Redis hash.
This method handles the value stored in the model's primary hash (the actual model data), not the sorted set index. Native numeric and temporal types are preserved as-is for msgpack serialization, while other types (like Decimal) are converted to float.
For numeric types (int, float), this method also applies auto_now_add and auto_now logic to automatically set Unix timestamps.
Note that this is separate from convert_to_numeric(), which handles the score conversion for the sorted set index.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
field_value
|
The raw value to be saved. |
required | |
skip_auto_now
|
If True, suppress auto_now timestamp updates. Useful for data migrations where existing timestamps should be preserved. Does not affect auto_now_add behavior. |
False
|
Returns:
| Type | Description |
|---|---|
|
The normalized value suitable for msgpack serialization. |
Source code in src/popoto/fields/sorted_field_mixin.py
convert_to_numeric(field, field_value)
classmethod
¶
Convert a field value to a numeric score for Redis Sorted Set storage.
Redis Sorted Sets require numeric scores for ordering. This method provides the conversion strategy for each supported type, ensuring that the natural ordering of the original type is preserved in the numeric representation.
Conversion strategies
- int/float: Used directly (no conversion needed)
- Decimal: Cast to float (may lose precision for very large values)
- date: Converted to ordinal (days since year 1), preserving day-level ordering
- datetime: Converted to Unix timestamp, preserving second-level ordering
- time: Converted to timestamp (note: may not behave as expected without a date)
This method is used both when saving (to compute the score) and when filtering (to convert query values to comparable scores).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
field
|
The Field instance containing type information. |
required | |
field_value
|
The value to convert. |
required |
Returns:
| Type | Description |
|---|---|
|
A numeric value suitable for use as a Redis Sorted Set score. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the value cannot be converted to a numeric score. |
Source code in src/popoto/fields/sorted_field_mixin.py
get_sortedset_db_key(model, field_name, *partition_field_names)
classmethod
¶
Generate the Redis key for this field's Sorted Set index.
Each sorted field maintains a Redis Sorted Set that maps model instance keys to their field values (as scores). This method constructs the key for that Sorted Set, incorporating any partition field values.
Key structure
$SortedF:{ModelName}:{field_name}:{partition_value1}:{partition_value2}:...
The $SortedF prefix identifies this as a sorted field index (following Popoto's convention of prefixing special-purpose keys). Partition values are appended to create separate sorted sets per partition.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
The Model class or instance. |
required | |
field_name
|
The name of the sorted field. |
required | |
*partition_field_names
|
Values for partition fields (from partition_by). |
()
|
Returns:
| Type | Description |
|---|---|
DB_key
|
A DB_key instance representing the Redis key for the Sorted Set. |
Source code in src/popoto/fields/sorted_field_mixin.py
get_partitioned_sortedset_db_key(model_instance, field_name)
classmethod
¶
Build the fully-qualified Sorted Set key including partition values.
This method reads the actual partition field values from a model instance to construct the complete Sorted Set key. It is used during save/delete operations where we have a concrete instance with all field values.
For queries (where we have filter parameters but not an instance), use get_sortedset_db_key() with explicit partition values instead.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_instance
|
A Model instance with populated field values. |
required | |
field_name
|
The name of the sorted field. |
required |
Returns:
| Type | Description |
|---|---|
DB_key
|
A DB_key for the partition-specific Sorted Set. |
Raises:
| Type | Description |
|---|---|
QueryException
|
If a required partition field value is missing. |
Source code in src/popoto/fields/sorted_field_mixin.py
on_save(model_instance, field_name, field_value, pipeline=None, **kwargs)
classmethod
¶
Update the Sorted Set index when a model instance is saved.
This hook is called by the Model.save() method for each field. For sorted fields, it adds or updates the model's entry in the Sorted Set index using ZADD. The model's Redis key becomes the member, and the field value (converted to numeric) becomes the score.
Redis ZADD is idempotent for updates: if the member already exists, its score is simply updated. This handles both create and update operations without needing to check existence first.
When a partition key changes (e.g., moving an item from category A to category B), the sorted set key changes entirely. This method detects the change by comparing saved partition field values to current ones and removes the member from the old partition's sorted set.
Pipeline Support
When a pipeline is provided, the ZADD command is queued for batch execution. This is crucial for atomic saves where the model hash and all indexes must be updated together.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_instance
|
Model
|
The Model instance being saved. |
required |
field_name
|
str
|
The name of this sorted field. |
required |
field_value
|
Union[int, float]
|
The value being saved (will be converted to score). |
required |
pipeline
|
Pipeline
|
Optional Redis pipeline for batch execution. |
None
|
**kwargs
|
Additional context (unused but accepted for compatibility). |
{}
|
Returns:
| Type | Description |
|---|---|
|
The pipeline (if provided) or the ZADD result. |
Source code in src/popoto/fields/sorted_field_mixin.py
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 | |
on_delete(model_instance, field_name, field_value, pipeline=None, **kwargs)
classmethod
¶
Remove the model instance from the Sorted Set index when deleted.
This hook ensures index consistency when a model is deleted. It removes the model's Redis key from the Sorted Set using ZREM. Without this cleanup, deleted models would remain in the index as orphaned entries, causing ghost results in range queries.
When called during key migration (saved_redis_key is provided), the partition key values may have changed. In that case, the old sorted set key is computed from _saved_field_values rather than the current (mutated) instance attributes.
ZREM is safe to call even if the member doesn't exist (returns 0), so this works correctly even if the index is somehow out of sync.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_instance
|
Model
|
The Model instance being deleted. |
required |
field_name
|
str
|
The name of this sorted field. |
required |
field_value
|
The current field value (used to find the right partition). |
required | |
pipeline
|
Pipeline
|
Optional Redis pipeline for batch execution. |
None
|
**kwargs
|
Additional context including saved_redis_key for key migrations. |
{}
|
Returns:
| Type | Description |
|---|---|
|
The pipeline (if provided) or the ZREM result. |
Source code in src/popoto/fields/sorted_field_mixin.py
536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 | |
filter_query(model_class, field_name, **query_params)
classmethod
¶
Execute a range query against the Sorted Set index.
This is the core query method that translates Django-style filter parameters into a Redis ZRANGEBYSCORE command. It returns a set of Redis keys for model instances whose field values fall within the specified range.
Query Parameter Parsing
The method parses __gt, __gte, __lt, __lte suffixes to determine the range bounds. Redis uses special syntax for exclusive bounds: a leading '(' makes the bound exclusive. For example: - price__gte=10 becomes min="10" (inclusive) - price__gt=10 becomes min="(10" (exclusive)
Partition Handling
For partitioned sorted fields (those with partition_by), the query parameters MUST include values for all partition fields. This is required because each partition has its own Sorted Set, and we need to know which one to query.
Performance
ZRANGEBYSCORE is O(log(N)+M) where N is the set size and M is the number of results. For large datasets, partitioning via partition_by reduces N significantly, improving query performance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_class
|
Model
|
The Model class to query. |
required |
field_name
|
str
|
The name of the sorted field being filtered. |
required |
**query_params
|
Filter parameters (e.g., price__gte=10, price__lt=100). |
{}
|
Returns:
| Type | Description |
|---|---|
set
|
A set of Redis keys (as bytes) for matching model instances. |
set
|
This set can be intersected with results from other field filters. |
Raises:
| Type | Description |
|---|---|
QueryException
|
If a partitioned field is queried without providing values for all partition fields. |
Example
Direct call (usually called by Query.filter_for_keys_set)¶
keys = SortedFieldMixin.filter_query( Product, 'price', price__gte=10.0, price__lt=50.0, category='electronics' )
Source code in src/popoto/fields/sorted_field_mixin.py
613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 | |