Skip to content

Memory Telemetry

popoto.recipes.memory_telemetry turns every live ContextAssembler.assemble() call into a durable, TTL-bounded, Valkey-safe event record — the injected memory set (ids, ranks, scores), retrieval mode, budget consumed, and latency — joinable to the later outcome reported through the ObservationProtocol (acted / used / dismissed / deferred / contradicted). An offline analyzer reads those records into injection-precision, confidence-calibration, and decay-regret reports.

Live agents give what offline benchmarks cannot: the real query/turn distribution and real outcome labels. The outcome signal already exists — it is used to nudge ConfidenceField / CyclicDecayField and then evaporates. This recipe records it so every live agent becomes a continuous, real-workload benchmark.

Privacy default is ids-only

Content capture is opt-in. The default capture="ids" records memory ids, ranks, scores, and metadata only — never memory content or query text. Enabling capture="content" is a per-store decision made explicitly in code (never a global env var or runtime toggle) for fully-open deployments whose operator asserts the store is publishable. Even in all-open deployments, humans talking to an agent can volunteer personal details; enabling content means the operator takes responsibility for that store being public-safe.

Components

Component Role
AssemblyEvent Popoto Model (msgpack, Meta.ttl) — one record per assemble() call
TelemetryRecorder Wraps a ContextAssembler; writes one event per call; fail-open, sampled
report_outcomes() Joins later outcomes onto the matching event record
TelemetryAnalyzer Offline, read-only report generator

Quick start

from popoto.recipes.context_assembler import ContextAssembler
from popoto.recipes.memory_telemetry import (
    TelemetryRecorder, TelemetryAnalyzer, report_outcomes,
)

assembler = ContextAssembler(Memory, score_weights={"relevance": 1.0})
recorder = TelemetryRecorder(assembler)            # capture="ids" (private)

result = recorder.assemble({"topic": "deploy"}, agent_id="agent-1")
event_id = result.metadata["telemetry_event_id"]

# ... later, when the agent's response is observed ...
report_outcomes(event_id, {mem.db_key.redis_key: "acted"})

# ... offline ...
print(TelemetryAnalyzer(agent_id="agent-1").report())

TelemetryRecorder.assemble() delegates to the wrapped assembler unchanged and returns the real AssemblyResult, adding metadata["telemetry_event_id"] when an event was written. It forces the assembler's off-by-default emit_trace hook on so the injection trace is available.

AssemblyEvent

One record per call. Queryable by agent_id (partition), TTL-bounded so telemetry can never grow a store past the scale posture.

Field Notes
event_id Auto key
agent_id Partition — matches assemble(agent_id=...)
model_name Class the assembler queried
ts Unix timestamp (float)
retrieval_mode "hybrid" | "lexical" | "composite"
capture "ids" (default) | "content" (opt-in)
injected [{key, rank, score, source[, content]}], rank order
budget_tokens / budget_items Budget consumed
latency_ms assemble() wall time
overhead_ms Telemetry preparation cost
query_text Populated only under capture="content"
outcomes Joined by report_outcomes(): [{key, outcome, at}]

TelemetryRecorder

TelemetryRecorder(
    inner,                 # a ContextAssembler
    *,
    capture="ids",         # "ids" (private) | "content" (opt-in). Invalid -> ValueError
    sample_rate=1.0,       # fraction of calls recorded, [0.0, 1.0]
    ttl=DEFAULT_EVENT_TTL,  # per-instance TTL (seconds)
    rng=None,              # random.Random for deterministic sampling in tests
)
  • Fail-open. Any error in the telemetry path is caught and logged — it never propagates into assemble() and never blocks a live agent. (Popoto has no Sentry dependency; a deployment routes these warnings to its own reporter.)
  • Sampling. Lower sample_rate on hot paths where write volume would distort latency benchmarks.
  • Overhead reported. recorder.overhead_stats returns {count, mean_ms, p95_ms, max_ms} over the measured per-call telemetry cost — so telemetry can be shown not to distort latency numbers.

report_outcomes()

report_outcomes(event_id, outcome_map, *, apply_effects=False, instances=None)

For each memory_key -> outcome in outcome_map whose key was injected in this event, appends {key, outcome, at} to the event's outcomes. Keys are the same record keys stored in injected (i.e. instance.db_key.redis_key, the same string on both backends); outcomes are validated against ObservationProtocol.VALID_OUTCOMES.

Record-only by default — telemetry is pure observation and does not double-apply confidence/decay effects. Pass apply_effects=True (with instances=) to also forward to ObservationProtocol.on_context_used() so one call owns both sides. A missing event (e.g. TTL-expired) is a logged no-op returning None, not an error. Re-saving refreshes the event's TTL to the model default.

TelemetryAnalyzer

Offline, read-only. All metrics are computed over injected memories that carry a joined outcome; injections without one are counted as pending.

Method Signal
injection_precision() Fraction of injected-and-labeled memories acted on, overall and by rank
confidence_calibration() Acted-rate bucketed by injection-time score
decay_regret() Injected-then-dismissed/contradicted vs injected-then-acted, by score bucket
report() Markdown summary of all three
an = TelemetryAnalyzer(agent_id="agent-1")
an.injection_precision()      # {'injected', 'labeled', 'pending', 'acted_rate', 'by_rank', ...}
an.confidence_calibration()   # {'buckets': [{'label', 'labeled', 'acted', 'acted_rate'}, ...]}
an.decay_regret()             # {'regret', 'acted', 'regret_rate', 'by_bucket': [...]}
print(an.report())            # markdown

Fusion-disagreement and refusal-threshold analyses are fast-follows (the trace already stores the fused score, so they need no schema change).

Scope and constraints

  • Valkey-safe: only core commands. On Redis the injection trace uses read-only pipelined ZSCORE; the event write is an ordinary model save(). No modules, no Lua.
  • TTL mandatory: AssemblyEvent.Meta.ttl bounds the store; the recorder can shorten but the record always expires.
  • Not self-benchmarking: no before/after gates and no automated tuning. This produces evidence for the maintainer-reserved policy-default conversations (refusal threshold, decay rates, outcome semantics) — it never changes a default on its own.
  • Namespace: popoto owns model-keyed, queryable, TTL'd records (AssemblyEvent:*); a deployment's own counter namespace (e.g. analytics:*) is separate by design.

On Postgres

Telemetry runs unchanged on a Postgres-bound process (#759 M5); every test in tests/test_memory_telemetry.py runs on both backends. AssemblyEvent is a table row whose Meta.ttl sets the _expires_at column, so an event expires there as on Redis and is invisible to every read from that instant, and the trace's score proxy reads the backend (the partition's decay ranking or the sorted column) instead of pipelined ZSCORE. See Record expiry and Recipes, mixins and the queue.

See Also