PolicyCache¶
A capstone recipe composing all shipped Popoto memory primitives into an RL-style action selection cache. Agents accumulate state-action-outcome events; a StreamConsumer crystallization handler detects repeated successful patterns and creates PolicyEntry records.
Overview¶
PolicyCache implements learned action selection:
- Events arrive via EventStreamMixin — state-action-outcome triples.
- Crystallization detects repeated successful patterns (via StreamConsumer handler).
- PolicyEntry records are created composing DecayingSortedField, ConfidenceField, CoOccurrenceField, ExistenceFilter, and PredictionLedgerMixin.
- Retrieval via CompositeScoreQuery ranks policies by multi-factor scores.
- Q-value updates via temporal difference learning refine policy values over time.
Components¶
PolicyEntry Model¶
Composes the shipped memory primitives:
from popoto.recipes.policy_cache import PolicyEntry, compute_fingerprint
policy = PolicyEntry(
agent_id="agent-1",
state_fingerprint=compute_fingerprint({"task": "deploy", "env": "staging"}),
state_features={"task": "deploy", "env": "staging"},
action_type="run_playbook",
action_spec={"playbook": "deploy.yml"},
q_value=Decimal("0.5"), # optional: seed initial Q-value at construction
)
policy.save()
Q-value and recency clock: two separate storage slots¶
PolicyEntry separates learned value from access recency:
| Field | Storage | Role |
|---|---|---|
q_value |
Model hash (TDValueField) |
Learned Q-value; updated by TD(0) |
expected_value |
Sorted set score on Redis; its own column on Postgres | Pure recency clock; decays with time |
expected_value is a DecayingSortedField(partition_by="agent_id", base_score_field="q_value"). The clock (the sorted-set score on Redis) is the decay-weighted access timestamp; q_value supplies the base magnitude. Because the two slots are independent, a save(), touch(), or "acted" outcome that refreshes the decay clock does not overwrite the Q-value, and a TD update to q_value does not alter the recency clock.
Crystallization Handler¶
Automatic pattern detection via StreamConsumer:
from popoto.recipes.policy_cache import crystallization_handler
from popoto.streams.consumer import StreamConsumer
consumer = StreamConsumer(
stream_key="stream:policy_mutations",
group_name="crystallizer",
consumer_name="worker-1",
handler=crystallization_handler,
)
The handler counts events with the same (state_fingerprint, action_type). When the count exceeds MIN_EVENTS_FOR_CRYSTALLIZATION and the Wilson CI lower bound exceeds WILSON_CI_THRESHOLD, a PolicyEntry is crystallized. The Wilson CI lower bound is passed as q_value at construction — a single save() persists both the model fields and the initial Q-value. No separate initialize_q_value call is needed.
Q-Value Updates¶
Temporal difference learning for policy refinement:
from popoto.recipes.policy_cache import update_q_value
# After observing reward from taking an action
update_q_value(policy, reward=0.8, max_future_q=0.6)
The update uses: Q_new = Q_old + alpha * (reward + gamma * max_future_q - Q_old)
update_q_value() is a thin wrapper: the script and the arithmetic belong to
TDValueField, the DecimalField subclass q_value is
declared as. Call TDValueField.td_update() directly to get the same atomic
update on a model that is not a PolicyEntry.
Temporal Discovery¶
Detects cyclical patterns in event timing:
Uses chi-squared tests to detect non-uniform temporal distributions, then creates CyclicDecayField cycles for discovered patterns.
Tuning Constants¶
All constants configurable via Defaults:
| Constant | Default | Optimal Range | Description |
|---|---|---|---|
MIN_EVENTS_FOR_CRYSTALLIZATION |
3 | [1, 10] | Minimum events before crystallization |
WILSON_CI_THRESHOLD |
0.6 | [0.3, 0.8] | Wilson CI lower bound for crystallization |
TD_ALPHA |
0.1 | [0.01, 0.5] | Q-value learning rate |
TD_GAMMA |
0.95 | [0.8, 0.99) | Q-value discount factor |
CHI_SQUARED_P_THRESHOLD |
0.05 | — | p-value threshold for temporal discovery |
INITIAL_CYCLE_AMPLITUDE |
0.5 | — | Initial amplitude for discovered cycles |
Utility Functions¶
compute_fingerprint(features)— stable hash from feature dicts (sorted JSON + SHA-256)wilson_ci_lower(successes, total)— Wilson score confidence interval lower boundchi_squared_uniform(observed)— chi-squared test against uniform distribution
On Postgres¶
PolicyEntry runs unchanged on a Postgres-bound model (#759 M5); its tests
run on both backends. q_value is a numeric column and td_update one
UPDATE, the decay clock is the expected_value column rather than a sorted
set score, the ledger, confidence state, co-occurrence edges and existence
tokens are Postgres columns and tables, and the EventStreamMixin mutation
log is appended to the backend's events tables in the save's own transaction,
where the crystallization and temporal-discovery StreamConsumer handlers
read it. Two behaviours follow the backend: the existence filter is exact
rather than a bloom filter, and a reload after touch() sees the touched
clock (both rows of
Documented divergences).
See Long-tail fields and
Event streams and pub/sub.
See Also¶
- ContextAssembler — retrieval-to-injection bridge
- Policy Cache Recipe Guide — detailed architecture guide
- Agent Memory overview — full primitives reference