Skip to content

popoto.models.canonical_key

popoto.models.canonical_key

Canonical string forms for values that become Redis key bytes.

Popoto derives a row's identity from its KeyField values: DB_key joins them with colons and that string is the Redis key. Historically each value was rendered with str(), which works for str/int/Decimal but is unstable for datetime: str() carries the UTC offset when the value is aware and omits it when the value is naive, so identity depended on how the value happened to decode rather than on the instant it denotes.

That instability is the root of two defects (#537, #538). Before 1.8.2 an aware value reloaded naive, so a re-save derived a different key, wrote a second hash and orphaned the first. And it blocked #521's proposed "assume UTC for legacy offset-free rows" change, because stamping an offset on read shifts str() and therefore shifts the key.

canonical_key_str fixes the projection nobody had touched. Identity becomes a function of the instant; the stored value still keeps the caller's exact offset (#521), because key and value are deliberately different projections.

Dispatch is on datetime.datetime only. Every other type -- including datetime.date and datetime.time -- returns str(value), byte-for-byte what 1.8.2 produced. See :func:canonical_key_str for why the two siblings are excluded rather than deferred.

canonical_key_str(value, *, force=False)

Render value as the bytes it contributes to a Redis key.

For a datetime.datetime: normalize to UTC and render %Y-%m-%dT%H:%M:%S.%f plus a literal Z. An aware value is converted with astimezone; a naive value is assumed UTC and converted with replace(tzinfo=utc), which is the same doctrine convert_to_numeric has applied to sorted-set scores since #519 and that auto_now has applied to stamping since #421. Consequence, and it is the point: an aware 12:00+07:00, its UTC equivalent 05:00+00:00, and the naive 05:00 all render identically and therefore address one row.

For every other value: str(value), byte-identical to 1.8.2. None renders "None", which Model.db_key and _has_unstable_db_key both depend on.

datetime.date has no offset to lose. datetime.time is excluded for PR #532's reason: a naive time.isoformat() with microseconds is byte-identical to the legacy %H:%M:%S.%f form, so a legacy time cannot be told apart from a deliberately naive one, and an aware time's str() already carries its offset stably. Neither type has the bug, so neither is a deferral.

This runs strictly before DB_key.clean(). The canonical form contains colons and hyphens, which clean() already escapes (#525); the two compose and this function must never be taught to escape anything itself.

Setting POPOTO_DATETIME_KEY_LEGACY=1 restores 1.8.2's str(value) for datetimes without a model-code edit, so an adopter can roll readers forward before moving key bytes. See the migration cookbook, recipe 19.

force=True ignores that switch and always renders the canonical form. This exists for exactly one caller: the #537/#538 audit and migration in datetime_key_migration.py, whose entire job is to find and move rows that are not on their canonical key. If the audit computed its target through the gated default, the switch would make every row look already-canonical -- is_clean would be True and the migration would report nothing to move, even over a genuine #538 duplicate pair -- which defeats the tool the switch exists to let operators run safely. Every other caller (DB_key.__str__, the two index-hash helpers in base.py) must stay gated, because those sit on the write path: while the switch is set, live writes still need to derive 1.8.2 key bytes so a fleet mid-rollout keeps reading its own writes. Only the read-only audit (and the migration, which delegates to it) needs the truth regardless of the switch.

Source code in src/popoto/models/canonical_key.py 
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
def canonical_key_str(value, *, force: bool = False) -> str:
    """Render *value* as the bytes it contributes to a Redis key.

    For a ``datetime.datetime``: normalize to UTC and render
    ``%Y-%m-%dT%H:%M:%S.%f`` plus a literal ``Z``. An aware value is converted
    with ``astimezone``; a naive value is *assumed UTC* and converted with
    ``replace(tzinfo=utc)``, which is the same doctrine ``convert_to_numeric``
    has applied to sorted-set scores since #519 and that ``auto_now`` has
    applied to stamping since #421. Consequence, and it is the point: an aware
    ``12:00+07:00``, its UTC equivalent ``05:00+00:00``, and the naive
    ``05:00`` all render identically and therefore address one row.

    For every other value: ``str(value)``, byte-identical to 1.8.2. ``None``
    renders ``"None"``, which ``Model.db_key`` and ``_has_unstable_db_key``
    both depend on.

    ``datetime.date`` has no offset to lose. ``datetime.time`` is excluded for
    PR #532's reason: a naive ``time.isoformat()`` with microseconds is
    byte-identical to the legacy ``%H:%M:%S.%f`` form, so a legacy time cannot
    be told apart from a deliberately naive one, and an *aware* time's ``str()``
    already carries its offset stably. Neither type has the bug, so neither is
    a deferral.

    This runs strictly *before* ``DB_key.clean()``. The canonical form contains
    colons and hyphens, which ``clean()`` already escapes (#525); the two
    compose and this function must never be taught to escape anything itself.

    Setting ``POPOTO_DATETIME_KEY_LEGACY=1`` restores 1.8.2's ``str(value)``
    for datetimes without a model-code edit, so an adopter can roll readers
    forward before moving key bytes. See the migration cookbook, recipe 19.

    ``force=True`` ignores that switch and always renders the canonical form.
    This exists for exactly one caller: the #537/#538 audit and migration in
    ``datetime_key_migration.py``, whose entire job is to find and move rows
    that are *not* on their canonical key. If the audit computed its target
    through the gated default, the switch would make every row look
    already-canonical -- ``is_clean`` would be ``True`` and the migration
    would report nothing to move, even over a genuine #538 duplicate pair --
    which defeats the tool the switch exists to let operators run safely.
    Every other caller (``DB_key.__str__``, the two index-hash helpers in
    ``base.py``) must stay gated, because those sit on the *write* path: while
    the switch is set, live writes still need to derive 1.8.2 key bytes so a
    fleet mid-rollout keeps reading its own writes. Only the read-only audit
    (and the migration, which delegates to it) needs the truth regardless of
    the switch.
    """
    if isinstance(value, datetime.datetime) and (
        force or not Defaults.DATETIME_KEY_LEGACY
    ):
        if value.tzinfo is None:
            as_utc = value.replace(tzinfo=datetime.timezone.utc)
        else:
            as_utc = value.astimezone(datetime.timezone.utc)
        return as_utc.strftime(CANONICAL_DATETIME_FORMAT) + CANONICAL_UTC_SUFFIX
    return str(value)