ensure_schema: reference

The ensure_schema: block on a BackingDataset declares the ontology entity-type schema Frank manages on apply. Frank reads it, calls the ontology service to make the live schema match, and only writes the BD row once the schemas agree.

Shipped in PR #462 (master tracker #457 § Gap #4). The mechanism is additive-only by default; narrowing requires explicit opt-in.

Where it lives

kind: BackingDataset
metadata: { name: air_quality_observed }
spec:
  pipeline: air_quality_observed
  entity_type_id: air_quality_observed       # ← schema is keyed by this
  ontology_tenant_id: ts_demo                 # ← optional; null = single-tenant
  # ...other BD fields...
  ensure_schema:
    display_name: "Air Quality Observed"
    fields:
      - { field_key: external_id, field_type: { type: string }, required: true, indexed: true }
      - { field_key: dt,          field_type: { type: integer }, required: true }
      - { field_key: pm2p5,       field_type: { type: number }, required: false }

When apply hits this BD, Frank calls:

• OntologyClient.create_entity_type(code) if the entity type doesn't exist yet.
• OntologyClient.publish_version(add_fields=...) for every field in ensure_schema: not already in the live schema.
• ?allow_deprecate=true is required to drop fields (see below).
• Type mismatches on existing fields always 409 — Frank refuses to silently change a field's type even with allow_deprecate.

Valid field_type values

type	Description	Maps to Iceberg
string	Variable-length string. The default for free-form values.	string
text	Long-form text (multi-paragraph). Stored same as string.	string
integer	Whole numbers (int64). Use for counts, sequence ids, Unix epochs.	long
number	Floating-point. Use for measurements, percentages, anything fractional.	double
boolean	True/false. Never use int for booleans — bool_coercer rejects.	boolean
datetime	ISO-8601 timestamp. Stored with timezone normalized to UTC.	timestamptz
date	Calendar date only (no time component).	date
geometry	GeoJSON-shaped object (point/line/polygon). Requires srid.	string (silver decodes)

For geometry, the full field_type is:

field_type:
  type: geometry
  geometry_type: point      # or line | polygon | multipoint | multipolygon
  srid: 4326                # WGS84 by default

For everything else, field_type: { type: <one of the above> } is the whole declaration. Other keys (geometry_type, srid, choices) default to null and are stripped from equality comparisons during idempotency checks (otherwise re-apply would silently re-publish).

Field-key rules

• Must match the regex ^[a-z_][a-z0-9_]*$. Lowercase letters, digits, underscores; first char must be a letter or underscore.
• dateObserved (camelCase) 422s. Use date_observed.
• count is a reserved field name in the ontology service — using it will 409. Pick something else (event_count, n_records, etc.).

Required vs indexed

• required: true — value cannot be null. Sync will reject rows missing this field. Use sparingly: in practice every field is optional at bronze because upstreams are dirty.
• indexed: true — the ontology service builds a secondary index on this field for fast query. Use for natural keys, foreign-key-like references, and anything you'll filter on. Default false. Don't over-index — every index costs write throughput.

Additive evolution

The common case. Run apply with a new field declared in ensure_schema:; Frank calls publish_version(add_fields=[<new>]) and the live schema gains the field at a new version. Existing rows have null for the new field until backfill.

ensure_schema:
  fields:
    - { field_key: external_id, field_type: { type: string }, required: true, indexed: true }
    - { field_key: dt,          field_type: { type: integer }, required: true }
    - { field_key: pm2p5,       field_type: { type: number } }
    - { field_key: pm10,        field_type: { type: number } }   # ← new

apply -f is idempotent — re-running this against a schema that already has pm10 is a no-op. The publish_version call is skipped when there's nothing to add.

Narrowing (deprecating a field)

Removing a field from ensure_schema: is a manifest-level change. By default Frank treats it as a mistake and 409s with {error: "narrowing", removed: ["pm10"]}. To actually deprecate the field on the ontology side, apply with ?allow_deprecate=true:

frankctl pipelines apply -f vertical.yaml --allow-deprecate

The flag is per-apply, not per-doc. It applies to every BD doc in the file. Frank calls publish_version(deprecate_fields=["pm10"]); the field stays in the schema but is marked deprecated. Consumers can still read it from existing rows; new rows can't set it.

Type changes

There is no escape hatch. Type mismatches on an existing field always 409, regardless of --allow-deprecate or --allow-recreate.

field_type mismatch on `pm2p5`:
  declared:  {"type": "integer"}
  live:      {"type": "number"}

To change a field's type:

1. Pick a new field_key (pm2p5_int).
2. Add it to ensure_schema: with the new type.
3. Run apply (additive — succeeds).
4. Backfill silver to map the old field to the new.
5. Once consumers migrate, remove the old field with --allow-deprecate.

This is intentional: silent type changes break every consumer that already has the schema cached.

Async publish

OntologyClient.publish_version returns 202 + {job_id, status} and Frank polls /api/v1/schema/types/<code>/versions/jobs/<job_id> until the job is completed or failed. The default poll budget is 10s (configurable via ONTOLOGY_JOB_WAIT_TIMEOUT_SECONDS).

If apply timeouts you'll see:

ensure_schema: publish_version timed out after 10s for entity_type=air_quality_observed

This is recoverable — the publish happens server-side regardless. Re-run apply once the schema has settled (typically within 30s).

Reading back

curl https://ts.ubp.pt/ontology/api/v1/schema/types/air_quality_observed

Shows current version, full field list, and history. Compare against your ensure_schema: block to confirm apply landed what you expected.

Gotchas (from prod live-verify)

These are real PR-A learnings and have bitten people:

• Resolved field_type is fatter than the manifest's.{"type":"string"} in the manifest comes back as {"type":"string","geometry_type":null,"srid":null,"choices":null}. Frank's diff helper strips None-valued keys before equality comparison; if you write a custom integration on top, do the same or every re-apply will look like a narrowing change.
• Relationships need rel_code (not link_code) and reverse_code: null to opt out of auto-creating a reverse relationship. The bootstrap reference impl in dev_docs/ts_demo/scripts/bootstrap.py:65 is stale on this point.
• count is a reserved field_key. Pre-existing test test_ontology_client.py::test_should_create_entity_type breaks because of this. Use event_count or similar.