popoto.backends.postgres.schema¶
popoto.backends.postgres.schema
¶
The Postgres schema compiler (#759 M1b, plan §3).
A :class:~popoto.backends.types.ModelSpec compiles to one typed table:
_pk text PRIMARY KEYholds the canonical key string -- the sameClassName:…string Redis uses as the hash key, soModel.pkis the same on both backends.- one typed column per field (the type map below); a
DatetimeFieldadds<f>__utcoff integer(offset seconds,NULL= naive), which keeps the #521 round-trip contract thattimestamptzalone drops. - engine-owned
_created_at/_updated_at([PG-only]). UNIQUEover the key fields, a B-tree per further key field, and a B-tree(partition cols…, f)perSortedField.- M1.1 (plain-field breadth): an
IndexedField/Relationshipcolumn gets a B-tree (UNIQUEfor a unique field), aTagFieldistext[]with a GIN index, the collection fields arejsonb(:mod:.codec),BytesFieldisbytea, andMeta.indexescompiles to composite B-tree /UNIQUEindexes.
The compiled shape is fingerprinted and recorded in popoto_schema the
first time a process binds the model. Create and additive changes (a new
nullable column, a new index) apply automatically under
pg_advisory_xact_lock; anything else -- a dropped or retyped column, a key
change, a newer format version, a schema some other client extended --
raises :class:SchemaDriftError rather than writing (§3 Migrations).
Pure except for :func:ensure_table, which takes an open connection. Never
imports redis.
SCHEMA_FORMAT_VERSION = 1
module-attribute
¶
Bumped when the compiler's output shape changes incompatibly. A client
that finds a newer version in popoto_schema refuses to write.
ENGINE_COLUMNS = (('_created_at', 'timestamptz NOT NULL DEFAULT now()'), ('_updated_at', 'timestamptz NOT NULL DEFAULT now()'), ('_migrated_from', 'jsonb'), ('_estimated_fields', 'text[]'))
module-attribute
¶
Engine-owned columns on every table (plan §1.1, [PG-only]).
COLUMNLESS_KINDS = frozenset({'BM25Field', 'ExistenceFilter', 'FrequencySketch', 'CoOccurrenceField'})
module-attribute
¶
(M4 adds CoOccurrenceField: its edges live in .graph's table.)
Column
dataclass
¶
One stored column. field is the popoto field it belongs to (None
for _pk); role is "value", "utcoff" or "pk".
Source code in src/popoto/backends/postgres/schema.py
TableSpec
dataclass
¶
What one model compiles to. indexes maps index name -> the
CREATE [UNIQUE] INDEX body after ON <table>; both are part of the
fingerprint.
Source code in src/popoto/backends/postgres/schema.py
156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 | |
unique_indexes = dataclass_field(default_factory=dict)
class-attribute
instance-attribute
¶
Unique index name -> the fields it covers (to name a 23505).
field_kinds = dataclass_field(default_factory=dict)
class-attribute
instance-attribute
¶
Field name -> its popoto kind (FieldSpec.kind).
capped_fields = frozenset()
class-attribute
instance-attribute
¶
ListField(max_length=N) columns: elements are type-tagged (.codec).
meta_indexes = dataclass_field(default_factory=dict)
class-attribute
instance-attribute
¶
Meta.indexes unique index name -> its fields (to word a 23505).
companions = ()
class-attribute
instance-attribute
¶
Companion tables (M2b): (name, DDL statements), each statement
idempotent (IF NOT EXISTS), created with the table and on an additive
migration. Part of the fingerprint when present.
search = None
class-attribute
instance-attribute
¶
The compiled search layout (:class:.search.SearchLayout) for a model
with BM25 / embedding / membership fields, else None.
ttl = False
class-attribute
instance-attribute
¶
Meta.ttl (M5): the table has _expires_at and every read of it
filters expired rows out (:mod:.ttl).
datetime_fields
property
¶
Fields stored as timestamptz + <f>__utcoff.
CatalogBefore
dataclass
¶
What :func:grant_created compares against: the relations already
in the schema, and whether the schema itself existed, when the DDL
transaction began.
Source code in src/popoto/backends/postgres/schema.py
quote_ident(name)
¶
Double-quote an identifier. Names reach here from field and model names, which Python already restricts; anything else is refused.
Source code in src/popoto/backends/postgres/schema.py
table_name_for(model_name)
¶
compile_table(spec, schema)
¶
Compile spec to a :class:TableSpec. Pure.
Source code in src/popoto/backends/postgres/schema.py
265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 | |
schema_lock_key(schema)
¶
The advisory-lock key every first use of schema serialises on.
CREATE SCHEMA IF NOT EXISTS is not race-free: two sessions that both
see the schema missing both insert into pg_namespace and one fails
with a unique violation. So every path that may create the schema --
:func:ensure_table and :func:engine_table_ddl -- takes this one key
first, then its per-table key (:func:table_lock_key), always in that
order so the two never deadlock (#759 M4b B1).
Source code in src/popoto/backends/postgres/schema.py
table_lock_key(schema, table)
¶
ddl_timeout_sql(*, waiting=False)
¶
SET LOCAL lock_timeout and statement_timeout for one DDL
transaction, from Defaults (#776, PR #793 review).
Two phases, because they bound different waits:
waiting=True-- before the advisory locks are taken, and while the catalog is read:lock_timeoutisDefaults.PG_DDL_SCHEMA_LOCK_TIMEOUT_MS, a much longer backstop. The schema-wide DDL advisory lock is held for as long as another process's migration runs, and a process whose model is already current just waits its turn (as it always did) instead of failing at the short limit. The wait cannot be a deadlock: every holder of these locks is itself bounded by the DDL phase below, so the chain ends.waiting=False(default) -- immediately before a DDL statement:lock_timeoutisDefaults.PG_DDL_LOCK_TIMEOUT_MS, so a DDL that waits on a table lock another session's open transaction holds gives up instead of hanging with the backend's first-use lock held.
statement_timeout is PG_DDL_STATEMENT_TIMEOUT_MS in both.
SET LOCAL ends with the transaction, so a pooled connection is never
left modified.
Source code in src/popoto/backends/postgres/schema.py
set_ddl_timeouts(cur, *, waiting=False)
¶
Run :func:ddl_timeout_sql on cur, inside its transaction.
Source code in src/popoto/backends/postgres/schema.py
engine_table_ddl(schema, table, body)
¶
One message that creates an engine-owned side table on first use.
It takes the schema lock, then the table lock, then creates the schema and
the table, all in the one implicit transaction the caller runs it in, so a
concurrent :func:ensure_table or another engine table's first use waits
on the schema rather than racing it into pg_namespace. Returns the SQL
and its parameters for PostgresBackend._run(..., write=True).
Source code in src/popoto/backends/postgres/schema.py
ensure_table(conn, ts, *, auto, grant_to=None)
¶
Create or check ts on conn inside one transaction, under an
advisory transaction lock (PgBouncer transaction-mode safe). Returns what
it did: "created", "migrated" or "current".
Raises :class:SchemaDriftError for every case it will not reconcile.
grant_to (#800, opt-in grant_main_role) names a role to grant
access to what this call creates -- the schema, the registry, the table
and its companions and sequences -- in the same transaction
(:func:grant_created). A "current" table grants nothing.
Source code in src/popoto/backends/postgres/schema.py
quote_role(role)
¶
Double-quote a role name as Postgres quotes any identifier. A role
name is whatever the server reports (current_user), not a field
name, so it is escaped rather than refused (:func:quote_ident).
Source code in src/popoto/backends/postgres/schema.py
catalog_before(conn, schema)
¶
Read :class:CatalogBefore for schema on conn (a connection
or cursor), inside the DDL transaction it precedes.
Source code in src/popoto/backends/postgres/schema.py
grant_created(conn, schema, role, before)
¶
Grant role access to exactly what this transaction created in
schema (#800): run on conn after the DDL and before its commit.
"Created here" is two facts together: the object's catalog row was
written by this transaction (its xmin is this transaction's id), and
it was not there when the transaction began (before). The first
excludes everything another session made, however concurrently; the
second excludes a table this transaction only altered (ADD COLUMN
rewrites its pg_class row). So nothing an admin made by hand, no
other role's table and no object a previous process created is ever
granted on, and a warm start -- nothing created -- grants nothing, which
leaves any REVOKE an admin has run in place.
Tables get :data:APP_TABLE_PRIVILEGES (the registry only
:data:APP_REGISTRY_PRIVILEGES), sequences
:data:APP_SEQUENCE_PRIVILEGES, and the schema USAGE only when this
transaction created it. No ALTER DEFAULT PRIVILEGES, no ON ALL
TABLES. Returns the statements it ran.