Ontology Integration

Frank turns curated Iceberg tables into semantic entities. The ontology integration lets teams publish pipeline outputs into ontology-core-v2 so applications can consume typed, versioned, relationship-aware data.

The model

Gold / Silver Iceberg table
        |
        v
Backing dataset
        |
        v
Ontology entity type
        |
        v
Ontology entities

A backing dataset says: this Iceberg table backs this ontology entity type, using these column-to-property mappings and this primary key.

Entity types

Entity types are schemas served by ontology-core-v2. Frank proxies the entity type surface so data builders can work inside the same UI and API:

GET    /api/v1/ontology/status
GET    /api/v1/ontology/entity-types
GET    /api/v1/ontology/entity-types/{code}
POST   /api/v1/ontology/entity-types
POST   /api/v1/ontology/entity-types/{code}/versions
PATCH  /api/v1/ontology/entity-types/{code}
DELETE /api/v1/ontology/entity-types/{code}
GET    /api/v1/ontology/entity-types/{code}/versions

GET /ontology/status is public and reports availability only. Entity-type and version-history reads require an authenticated bearer token. Creating, publishing, updating, or deleting an entity type requires an admin or pipeline_editor user; Frank service and delegation tokens remain supported for internal automation. User requests must also send X-Tenant-ID for an organization in the token's organizations claim. Service and delegation callers may omit the header for cross-tenant automation, or send it to target one tenant.

Entity types can include fields and relationships. Frank synthesizes relationship references into field-like mapping targets so users can map station_name or route_id style columns into relationship refs during backing dataset setup.

Backing datasets

A backing dataset contains:

Field	Meaning
iceberg_namespace / iceberg_table	The materialized table to publish.
entity_type_id / entity_type_name	The ontology type being backed.
schema_library_ref	Optional source schema reference such as fiware:Transportation/Vehicle.
property_mappings	Column-to-property mapping array.
primary_key_column	Stable entity key column.
title_key_column	Human-readable entity label column.
sync_mode	When the dataset should publish.
cursor_column	Optional incremental sync cursor.
transform_id / pipeline_id	Optional lineage back to the producer.

Backing dataset lifecycle:

pending -> syncing -> synced
synced -> syncing
synced -> needs_remapping
needs_remapping -> pending
error -> pending | syncing

Property mappings

Mappings are explicit and reviewable:

[
  {
    "column": "vehicle_id",
    "property": "id",
    "is_primary_key": true,
    "type": "string"
  },
  {
    "column": "observed_at",
    "property": "dateObserved",
    "type": "datetime"
  },
  {
    "column": "station_name",
    "property": "ref_station",
    "is_relationship": true,
    "target_type": "station",
    "target_key": "name"
  }
]

Relationship mappings let the sync activity resolve business keys into ontology entity UUIDs.

File publication

An Iceberg column can populate an ontology file field. Which way depends on the file's storage policy within Frank — what the pipeline did with the file between extraction and publication:

storage policy	column holds	who reads the bytes
not stored — reference passes through	an HTTP(S) URL	the ontology fetches the origin
stored in the table	the bytes (binary)	Frank, from the column
stored in Frank's object store	a tenant-relative object key	Frank, from its own store

All three end in the same place: the bytes are copied into ontology storage and the returned managed key is attached to the entity, under one ledger, one serialized claim and one retry contract. Add a nested file block to the mapping; type stays the physical Iceberg type. The block doesn't create the storage policy — it declares which one the pipeline applied to that column, so the publisher knows how to read it.

Authoring file mappings in the UI

The backing-dataset wizard and the detail page's Mapping edit both author property_mappings[].file. When the entity type has a field of type file, the mapping editor shows a File tab next to Source:

• Publish as: URL (the column holds a link Frank fetches at sync time) or Upload (Frank reads the bytes from the row). For Upload, Bytes come from picks inline bytes (a binary column) or an object reference (a key inside Frank's own bucket).
• File column: only columns whose physical type the chosen mode accepts are offered — string/varchar/char/text for URL and object reference, binary/varbinary for inline bytes — so a wrong pick is impossible rather than rejected on save.
• Identity, fingerprint, filename and content-type columns never offer the file column itself. URL mode needs a fingerprint column or the URL is immutable declaration; the two are exclusive. Upload mode fingerprints the bytes itself, and never accepts immutable.
• On error: Omit is disabled when the target field is required.
• Modes the deployment cannot serve are disabled with the reason reported by GET /api/v1/backing-datasets/capabilities (for example object_store_unavailable); the object-reference size cap is shown when the deployment enforces one.

The row the UI sends is an ordinary column mapping plus the physical type hint and the nested file block — exactly what the API and frankctl apply accept, so a mapping authored in the UI and one declared in YAML are interchangeable.

URL mode

A string column holding an HTTP(S) URL:

property_mappings:
  - column: image_url
    property: image
    type: string
    file:
      mode: url
      identity_column: image_id
      fingerprint_column: image_etag
      filename_column: image_name
      content_type_column: image_mime
      on_error: fail

Frank passes the complete URL — including the query string a presigned URL needs — to the ontology, which fetches and stores the bytes and returns a managed key. Frank writes that key into the entity. Frank never fetches, rewrites, strips, or stages the bytes or the URL itself, and it never writes a raw URL into a file field.

Upload mode (inline binary)

A binary column holding the file's bytes:

property_mappings:
  - column: attachment_bytes
    property: attachment
    type: binary
    file:
      mode: upload
      input: blob
      identity_column: attachment_id
      filename_column: attachment_name
      content_type_column: attachment_mime
      on_error: fail

Frank reads the bytes, computes their SHA-256, and uploads them to the same ontology endpoint as multipart. The bytes stay in the sync activity's own memory for the length of one row: they never appear in a workflow input or result, a Temporal search attribute, memo or heartbeat, a log, a span, an error message, a persisted sync spec, or a ledger row.

fingerprint_column is optional here. With no column declared, the SHA-256 Frank computes from the bytes it read is the fingerprint, so an unchanged blob reuses the existing key and a changed one publishes a new asset. Declare a column only when you already carry a trusted content hash.

filename_column and content_type_column travel on the multipart part itself, which is where the ontology reads them; with neither declared, the upload defaults to upload and application/octet-stream. They are metadata hints and do not bypass the ontology's own controls.

Upload mode (object reference)

A string column holding a tenant-relative object key into the deployment's own transit bucket — for real files (documents, images) whose bytes should sit once in object storage rather than ride the table:

property_mappings:
  - column: attachment_ref
    property: attachment
    type: string
    file:
      mode: upload
      input: object_ref
      identity_column: attachment_id
      fingerprint_column: attachment_version
      filename_column: attachment_name
      content_type_column: attachment_mime
      on_error: fail

The column carries the relative key and nothing else. It cannot carry a scheme, a host, a bucket, a tenant, a port, userinfo or a worker-local path, because the grammar has nowhere to put one — file://, s3://bucket/key, https://…, https://user:pass@host/x, /etc/shadow, ../../ traversal and empty path segments are all rejected before any request is made. Frank composes the real key server-side as <key-prefix>/<frank-tenant-id>/<column-key> from the BackingDataset's own tenant, so another tenant's objects are unaddressable by construction.

fingerprint_column is optional here too, and it is the field to reach for. With a trustworthy version or ETag column, Frank knows before touching the store whether the asset is unchanged, so an unchanged rerun performs zero object reads. Without one, Frank has to hash the bytes, so it reads the object once per run — in bounded chunks, hashing as it goes, never twice.

filename_column and content_type_column fall back to the object's own basename and Content-Type when they are not mapped.

Every failure is typed and appears in run evidence: object_store_not_configured, invalid_object_key, object_key_traversal, object_not_found, object_unauthorized, object_too_large, object_read_failed, object_store_unavailable. All of them happen before anything is uploaded, so none can leave bytes stranded in ontology storage.

Configuring the transit store

The store is Frank's own — the same MinIO/S3 deployment the Iceberg warehouse lives on — so configuration is three settings on the API and the ontology worker:

setting	meaning
FRANK_FILE_OBJECT_BUCKET	the transit bucket (required to use the mode)
FRANK_FILE_OBJECT_KEY_PREFIX	optional root prefix inside the bucket
FRANK_FILE_OBJECT_MAX_BYTES	per-object ceiling; default 64 MiB

Endpoint, region and credentials are the worker's existing AWS_ENDPOINT_URL / AWS_REGION / AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY platform secrets (_FILE forms included). No credential is ever expressible in a mapping, a SyncSpec, or anywhere else along the path. The size ceiling is enforced both from the object's declared length and while reading, so a store that understates a size cannot make Frank buffer an unbounded object.

With no bucket configured, an object_ref mapping is rejected when the backing dataset is created or applied — frankctl apply --dry-run-server reports it — and again by the sync activity before the first row is read.

See ADL-030 for the full resolver, credential and tenant boundary.

Block reference

Key	Required	Meaning
mode	yes	url (the ontology fetches) or upload (Frank sends the bytes).
input	upload only	blob — the mapped column holds the bytes — or object_ref — the column holds an object key. Not accepted for mode: url.
identity_column	yes	Stable asset identity. Neither an expiring URL nor the bytes themselves is an identity.
fingerprint_column	conditional	Content/version fingerprint, e.g. an ETag. Required in URL mode unless immutable: true. Optional in upload mode, where Frank computes SHA-256 from the bytes instead.
immutable	url only	Declares that the bytes never change for a given identity. Mutually exclusive with fingerprint_column, and rejected for upload mode — Frank always has a real content hash there.
filename_column	no	Original filename passed to the ontology.
content_type_column	no	MIME type passed to the ontology.
on_error	no	fail (default) surfaces a publication failure. omit drops the value; allowed only against an optional ontology field, and always visible in run evidence.

The physical type must match the mode: a string type (string, varchar, char, text) for url and for upload with input: object_ref, and a binary one (binary, varbinary) for upload with input: blob. The mapped file column can never double as identity_column, fingerprint_column, filename_column or content_type_column — those are persisted, logged and exported, and neither a credential-bearing URL, raw bytes, nor a raw object key belongs on those surfaces.

A NULL column value means the row has no asset: the property is dropped and the entity keeps whatever it already had. In either upload mode a zero-length blob or object is a data error, not an absent asset, and follows on_error.

Republication and retry

Whether a file is republished depends on identity_column and the fingerprint, never on the URL, the bytes' position, or the object key. A rotated presigned URL for the same identity and fingerprint uploads nothing and keeps the existing key; so does a rerun over an unchanged blob, and so does a rerun over an object whose version column has not moved — which in that case also reads nothing from the store.

The ontology's file upload has no idempotency key and mints a new key per accepted upload, so Frank keeps a durable per-asset ledger:

• The returned key is persisted before the entity is written. If the entity write fails, the retry attaches the same key with zero re-uploads.
• Concurrent runs serialize per asset, so one claim yields at most one acknowledged upload.
• A changed fingerprint uploads once, attaches the new key, and leaves the previous key visible as an orphan candidate. Frank never deletes ontology storage.
• An unknown transport outcome — a lost response, a gateway error — is quarantined rather than retried, because bytes may already be stored under a key Frank never saw. Quarantine is not softened by on_error: omit. It needs an operator to inspect the ontology target.

File-bearing rows always take the per-row REST path. The gRPC bulk path is insert-only and the REST bulk path has no sequencing point between upload and entity write, so neither can carry this contract.

Each sync run reports asset outcomes: requested, uploaded, reused, attached, failed, omitted, outcome-unknown, orphan-candidate, bytes-sent, and object-reads. frankctl backing-datasets sync prints them as files_* rows when a run published files. files_bytes_sent counts only bytes Frank itself sent, so it is always 0 for URL-mode publication; files_object_reads counts objects read from the transit store, so it is 0 for the other two modes and 0 on an unchanged object-ref rerun.

Provenance

Frank-owned persistence, logs, spans, errors, and exports never contain a URL query, fragment, or credential, and never contain file bytes. Every published asset carries a credential-free provenance record instead:

Mode	Scheme	Host	Path
url	https	URL host	URL path, no query
upload + blob	iceberg	Iceberg namespace	table.column
upload + object_ref	s3	bucket	object key

All three are stored with a SHA-256 digest of the locator, so one query answers "where did these bytes come from" across every mode, and the scheme tells you which mode produced the row.

Durable evidence and observability are not the same surface. The object key is recorded in the ledger but is withheld from logs, spans and errors, which carry the scheme, the bucket and the digest instead — enough to identify the object without printing a raw locator.

For URL mode, SSRF, redirect, MIME, size, and egress controls remain the ontology's; private and internal addresses, s3://, worker-local paths, inline blobs, and object references are not URL mode. Object-ref mode reads only the one configured transit bucket, over the deployment's own private endpoint — there is nothing a row or mapping can do to point it anywhere else.

File publication capabilities and ledger

Before publishing files, check what modes the deployment supports:

frankctl backing-datasets capabilities

Output includes file_mode.upload_object_ref with available and, when unavailable, a reason code:

• object_store_not_configured — FRANK_FILE_OBJECT_BUCKET is unset; use URL or inline-blob mode instead.
• object_store_misconfigured — bucket is set but the endpoint is missing or the size limit is invalid; contact the platform operator.

To inspect the per-entity file publication ledger after a sync:

# All assets for a backing dataset
frankctl backing-datasets file-assets <bd-id>

# Filter by state (repeatable)
frankctl backing-datasets file-assets <bd-id> --state upload_failed --state outcome_unknown

# Scope to a single run
frankctl backing-datasets file-assets <bd-id> --run <sync_run_id>

# Machine-readable
frankctl backing-datasets file-assets <bd-id> --json

The ledger shows state, entity key, field key, filename, size, and failure codes. Security-sensitive fields (ontology_key, asset_identity, content_fingerprint, source_url_*) are never returned. failure_detail is redacted of URLs and query tokens before leaving the API.

States requiring operator attention:

State	Meaning	Action
upload_failed	Transport or server rejection	Investigate; will retry on next sync
outcome_unknown	Quarantined: bytes may exist under an unseen key	Inspect the ontology target; do not retry automatically
orphan_candidate	Superseded key Frank never deletes	No action needed; ontology storage cleanup is out of scope

Mapping assistance

Frank can suggest backing dataset mappings:

POST /api/v1/backing-datasets/suggest-mappings

The suggestion request includes the Iceberg table and target entity type. Frank uses table schema, target property names, and AI assistance to propose column-to-property matches.

Sync

Backing datasets sync rows from Iceberg into ontology-core-v2. The sync path tracks:

• Workflow ID and workflow run ID.
• Status: pending, running, synced, error, skipped.
• Started and completed timestamps.
• Rows synced.
• Snapshot ID, including an exact decimal-string representation.
• Full vs incremental sync.
• Error message.
• Trigger source.
• Replay reason and whether force replay was requested.
• Attempted and successfully applied effective-SyncSpec fingerprints.

API v1 SyncRun JSON keeps deprecated numeric snapshot_id for existing clients and adds snapshot_id_exact so JavaScript and other number-limited consumers can read the full 64-bit Iceberg snapshot ID without rounding. Official Frank clients expose that exact value as the canonical string-valued snapshot_id.

Frank treats the Iceberg snapshot and effective SyncSpec as independent checkpoints. A change to mappings, resolved runtime transforms, entity target, REST/gRPC endpoint, or effective ontology tenant replays the unchanged snapshot through REST upsert. The next run skips with zero entity writes only when both checkpoints match. Failed runs retain the previously applied fingerprint. Mapping or ensure_schema changes return 409 while a sync is running, so a later spec cannot overtake and regress the frozen run's ontology/checkpoint. An exact declarative re-apply without schema convergence remains a read-only 200 and does not disturb the running sync.

REST bulk and per-row safety

When a compatible gRPC endpoint is configured, Frank keeps using the gRPC batch path. Otherwise, REST selection is cardinality-aware:

Environment variable	Default	Meaning
ONTOLOGY_REST_BULK_THRESHOLD_ROWS	1000	Estimated rows at or above this value require REST bulk. A missing estimate is treated as high-cardinality.
ONTOLOGY_REST_BULK_MAX_ITEMS	10000	Maximum entities in one bulk request.
ONTOLOGY_REST_BULK_MAX_PAYLOAD_BYTES	52428800	Maximum UTF-8 bytes in the exact serialized {items: [...]} request.
ONTOLOGY_REST_BULK_PAGE_SIZE	200	Target enumeration page size.
ONTOLOGY_REST_BULK_JOB_TIMEOUT_SECONDS	300	Maximum wait for one asynchronous bulk job.
ONTOLOGY_REST_BULK_POLL_INTERVAL_SECONDS	0.5	Bulk-job polling interval.

Before a REST write, Frank pins the selected Iceberg snapshot, validates every source primary key, and builds duplicate-checked target and relationship-key indexes. Missing/duplicate source keys or unresolved/ambiguous relationships therefore fail before source-entity mutation. Bulk inserts use POST /api/v1/{type}/bulk; updates use PATCH with target UUIDs. Every request, enumeration page, and job poll carries the configured ontology tenant.

A bulk acknowledgement must contain one valid job ID. Frank polls that exact job until completed and requires its completed count to equal the submitted count. Completed job ID, operation, submitted count, and completed count are persisted on the exact SyncRun. Payloads and credentials are not evidence and are never stored there.

A lost or malformed submission acknowledgement is ambiguous because the CIRA REST API has no caller-supplied idempotency key or actor-scoped job lookup. Frank therefore does not resubmit that batch automatically. The run fails closed for operator inspection. A retry may poll a pending job only when its exact ID was checkpointed. HTTP 404/405 means bulk is unsupported; a required high-cardinality bulk sync fails rather than falling back to per-row writes.

Smaller REST datasets retain stable-key per-row upsert. Business-key GET and UUID-addressed PATCH retry bounded transport failures, HTTP 408/429, and 5xx responses because those operations are idempotent. POST is never blindly retried. After an ambiguous POST result, Frank re-reads by stable key and confirms or patches the accepted entity when found; otherwise the row remains unconfirmed for Temporal retry. Retry logs exclude URLs, row values, bodies, and credentials. Neither REST strategy deletes ontology entities.

Useful endpoints:

GET  /api/v1/backing-datasets/capabilities
POST /api/v1/backing-datasets/{id}/sync?force=true
GET  /api/v1/backing-datasets/{id}/sync-history
GET  /api/v1/backing-datasets/{id}/sync-history/{run_id}
GET  /api/v1/backing-datasets/{id}/sync-history/{run_id}/logs
GET  /api/v1/backing-datasets/{id}/health

The capabilities response advertises sync contract version 2, exact-run polling, and force-replay support. Clients must confirm those capabilities before requesting force replay; absence is not evidence that an older API will honor force=true.

For rolling upgrades, the historical OntologySyncWorkflow and its three V1 activities remain on ontology-sync-task-queue. New starts use the isolated OntologySyncWorkflowV2 activities on ontology-sync-v2-task-queue, served by ontology-worker-v2. This separation lets V1 histories and retries drain without receiving V2 replay semantics. It is a deployment contract, not a claim that a particular environment has completed the rollout.

The health endpoint checks the mapping, table state, ontology status, and schema drift signals that matter before publication.

Schema libraries

Schema libraries provide target schemas for transforms and backing datasets:

GET  /api/v1/schema-libraries
GET  /api/v1/schema-libraries/{library_id}/domains
GET  /api/v1/schema-libraries/{library_id}/domains/{domain}/schemas
GET  /api/v1/schema-libraries/{library_id}/schemas/{schema_id}
GET  /api/v1/schema-libraries/schema/{full_id}
GET  /api/v1/schema-libraries/search
POST /api/v1/schema-libraries/validate/{full_id}

The registry combines FIWARE Smart Data Models and custom schemas behind one browsing and validation surface.

Identity policies

Identity policies define stable keys for semantic entities. Strategies include:

• passthrough: use the normalized source field.
• composite: concatenate normalized fields.
• hash: hash the composite key.
• uuid: generate a UUID-form key from normalized values.

Policies can normalize values with operations such as trim, upper/lower, space stripping, and NFC normalization. They can be system-level or tenant-level.

Important endpoints:

GET    /api/v1/identity-policies
GET    /api/v1/identity-policies/{id}
POST   /api/v1/identity-policies
PUT    /api/v1/identity-policies/{id}
DELETE /api/v1/identity-policies/{id}
POST   /api/v1/identity-policies/{id}/dry-run

Use dry runs to verify identifier output before a transform or backing dataset depends on it.

Recommended workflow

1. Build and run a transform into a Silver or Gold table.
2. Choose or create an ontology entity type.
3. Register a backing dataset for the table.
4. Use mapping suggestions, then review field and relationship mappings.
5. Pick primary key and title key columns.
6. Run a health check.
7. Trigger sync.
8. Monitor sync history and logs.