Operations

Frank is built to be operated as a set of clear runtime surfaces: API, UI, source worker, transform worker, Temporal workflows, Dagster assets, Iceberg tables, logs, traces, and run records.

Services

Service	Purpose
api	FastAPI application, route registration, pattern sync, schema library access, AI endpoints, and admin APIs.
ui	SvelteKit application for source, transform, pipeline, model, ontology, and settings workflows.
source-worker	Temporal worker for source discovery and extraction.
transform-worker	Temporal worker for transform lifecycle and reconciliation work.
worker	General Temporal worker for AI and platform workflows.
Dagster	Asset materialization, schedules, sensors, and pipeline execution visibility.
Temporal	Durable workflow execution for async source, transform, and orchestration jobs.
Iceberg REST + MinIO/S3	Lakehouse catalog and object storage.
Trino	Query engine for transform execution and previews.
Loki	Persistent log querying for run details.
OpenTelemetry collector	Trace export for API and worker paths.

Local stack

cd ../common-infra
docker-compose up -d

cd ../frank-low-code-pipeline
make up
make status

Common access points:

Surface	URL
API docs	http://localhost:8002/docs
UI	http://localhost:5175
Health	http://localhost:8002/health
Dagster	http://localhost:3000 or configured Dagster URL

Startup work

Production database migration is a separate, fail-closed release step. The API does not start until the one-shot migration service reaches the single Alembic head. Local development retains the image entrypoint migration behavior, but a non-zero Alembic result stops startup rather than becoming a warning.

After that gate, API startup performs platform initialization:

• Runs through FastAPI lifespan setup.
• Initializes Iceberg client.
• Ensures raw namespace.
• Initializes AI transformer services.
• Initializes FIWARE SDM registry and schema libraries.
• Syncs source patterns from JSON files into the database.
• Syncs SQL transforms and transform patterns from filesystem config.
• Registers transform event listeners for Dagster code location reloads.
• Initializes OpenTelemetry and Langfuse instrumentation.
• Registers the complete API router set.

Run lifecycle

Frank stores lightweight run summaries in Postgres and sends detailed work to the relevant runtime.

Work	Runtime	User-facing records
Source discovery	Temporal source worker	Discovery workflow status.
Source sync	Temporal source worker + Iceberg	Sync run history, logs, source status.
Transform materialization	Dagster + API callback	TransformRun records, Dagster run ID, logs, lineage.
Pipeline sandbox	API + worker orchestration	Sandbox workflow status and step results.
Ontology sync	Temporal / Dagster sensor path	OntologySyncRun records and backing dataset history.
AI assistance	Martha workflow execution	AI trace/execution IDs and structured response payloads.

Dagster K8s-run warm-up

The staging Grafana board Dagster / K8s Run Lifecycle is the decision surface for disposable Dagster worker start-up. Its Worker Bootstrap Breakdown row measures only terminal runs and has no per-run metric label; use a selected Dagster run ID in Loki or the durable orchestration observation when a single run needs inspection.

The API's /api/v1/metrics endpoint exposes the bounded event-time families:

• frank_dagster_orchestration_window_phase_seconds_{bucket,sum,count,max} carries fixed window, job_name, and phase labels.
• frank_dagster_orchestration_window_worker_bootstrap_timing_capture_failures counts terminal rows without a valid worker report. It must be zero before interpreting the stage timings as a cohort result.

A valid worker report creates these consecutive phases:

container_to_worker_entrypoint
worker_entrypoint_to_python_startup_hook
python_startup_hook_to_definitions_module
definitions_module_to_definitions_import
worker_entrypoint_to_definitions_import
definitions_import_to_observability_initialized
observability_initialized_to_dagster_imports
dagster_imports_to_resources_import
resources_import_to_registry_module_import
registry_module_import_to_sensors_import
sensors_import_to_registry_build
definitions_import_to_registry_build
registry_build_to_bronze_assets
bronze_to_silver_assets
silver_assets_to_definitions_ready
definitions_ready_to_run_start

For a staging experiment, first confirm every phase's sample count equals the completed cohort and capture failures are zero. Then compare p50, p95, and max for the same phase across baseline and experiment. A missing stage is missing data, not zero seconds; do not use it to select a warm-up optimization.

Logs

Useful CLI commands:

frankctl sources logs <source-id> <run-id> -f
frankctl transforms logs <transform-id> <run-id> -f
frankctl runs get <workflow-id>
frankctl runs wait <workflow-id>

Useful Compose commands:

make logs
make logs-api
make logs-ui
docker-compose logs -f source-worker
docker-compose logs -f transform-worker

The API and workers use structured JSON logging so Loki queries can filter by fields such as workflow ID, Dagster run ID, transform ID, source ID, and trace ID.

Traces

OpenTelemetry is initialized in the API and workers. Dagster-triggered transform paths, source worker paths, and AI paths include trace context where available.

Relevant env:

OTEL_EXPORTER_OTLP_ENDPOINT=alloy:4317

Source operations

Operational checklist:

1. Source is ready or active.
2. Discovery schema is current.
3. Streams are enabled and configured.
4. Incremental streams have cursor fields.
5. Merge streams have primary keys.
6. Target config matches the desired Bronze namespace/table convention.
7. Sync history shows successful runs.

CLI:

frankctl sources get <source-id>
frankctl sources streams list <source-id>
frankctl sources history <source-id>
frankctl sources sync <source-id>

Source credential operations

Connector fields declared with credential: true or type: password are write-only on every read, export, log, trace, and error surface. Runtime storage is an explicit deployment policy:

Mode	Deployment contract
legacy_inline	Current mutable/base Compose compatibility during Generation A. Write-only values are persisted inside redacted Source.source_config; ordinary updates preserve hidden values. No Vault reference or credential-lifecycle operation is available.
vault	Immutable release contract. Values live in the shared platform Vault KV v2 Source namespace; PostgreSQL and Temporal carry only an opaque tenant-owned reference and exact version.

Base docker-compose.yml selects SOURCE_CREDENTIAL_MODE=legacy_inline. Supply values through the Source create/update UI or its write-only credential_values request field. Keep transient requests outside Git and shell arguments. The API merges values only at persistence and never returns them. sources credentials set|rotate are Vault-only and fail in this mode.

The immutable release overlay selects SOURCE_CREDENTIAL_MODE=vault. Create a Source with ordinary config, then stream the exact declared credential object into the CLI:

frankctl sources create -f source.yaml
frankctl sources credentials set <source-id> \
  --name <tenant-local-name> --values-file - < credential-values.yaml
frankctl sources credentials rotate <source-id> \
  --values-file - < rotated-values.yaml

Keep any temporary values file untracked and mode 0600, or supply stdin from the approved secret manager. Vault-mode Source readback and pipeline export contain only safe metadata and the opaque credential_ref. Rotation updates the exact Vault version without recreating Source, stream, schedule, cursor, ledger, or run state.

The one-shot cutover gate writes and exact-version verifies every registered credential before removing only structurally declared fields from PostgreSQL. All affected rows commit together after the locked inventory succeeds. Before commit, a failure rolls back PostgreSQL and quarantines unreferenced Vault versions; after commit, recovery fixes forward or restores a matched, checksummed PostgreSQL/Vault artifact pair. After commit, returning that database to legacy_inline or an older binary is unsupported. Detaching or deleting a Source never deletes Vault data; revocation requires an unreferenced credential and retains Vault history for operator-managed recovery.

Transform operations

Operational checklist:

1. Transform is hydrated.
2. can_run_now is true in the API/UI.
3. Current artifact runtime matches the expected execution engine.
4. Dagster code location has loaded the asset.
5. Last run outcome is not already running.
6. Logs are available for the run.
7. Output table and lineage edges match expectations.

CLI:

frankctl transforms get <transform-id>
frankctl transforms trigger <transform-id>
frankctl transforms runs <transform-id>
frankctl transforms logs <transform-id> <run-id>

Pipeline operations

Pipeline deployment path:

1. Draft or update pipeline.
2. Validate DAG.
3. Run sandbox.
4. Review step results.
5. Activate.
6. Monitor runs.

CLI:

frankctl pipelines get <pipeline-id> --include-version
frankctl pipelines validate <pipeline-id> --timeout 600

Source worker liveness

The unified source worker runs synchronous discovery and extraction activities. Its activity-slot limit must not exceed its executor-thread count. The shipped defaults are eight executor threads, eight activity slots, 100 workflow slots, and five pollers for each task type. Set the TEMPORAL_SOURCE_* capacity and liveness variables in the ecosystem ../.env; Compose deliberately does not repeat them so operator recovery settings are not shadowed.

Check the same probe used by Docker:

docker compose exec source-worker \
  python -m backend.temporal.source_worker_health

A healthy result requires a fresh source-worker event-loop watchdog and a current matching workflow poller on TEMPORAL_SOURCE_TASK_QUEUE. The result also reports activity-poller freshness, but a missing activity poller is diagnostic rather than fatal while every activity slot is occupied. A nonexistent queue, stale watchdog, connection failure, or missing workflow poller returns a non-zero status. This is a process and workflow-queue liveness check, not evidence that activities are progressing or that the sync backlog is draining. Use Temporal backlog, activity latency, heartbeat, and sync-run outcomes for that readiness decision. If the event-loop watchdog itself cannot run, the worker exits instead of continuing without liveness evidence.

Do not blindly restart a worker into an existing workflow backlog. First record the queue backlog and pollers, pause or bound new scheduled starts under the approved recovery plan, and monitor CPU, memory, failures, and schedule-to-start latency while work drains.

Ontology operations

Before syncing a backing dataset:

1. Entity type exists and is the intended version.
2. Iceberg table exists and has expected columns.
3. Property mappings include the primary key column.
4. Relationship mappings include target type and target key.
5. Health check passes.
6. Sync history is reviewed after trigger.

In production and shared staging, every Frank tenant must have an exact entry in ONTOLOGY_TENANT_BINDINGS_JSON. Wildcards and whitespace-normalized aliases are rejected. Service/delegation callers also require the ontology:sync capability. The API checks this policy on declaration, update, preflight, health, entity read, and trigger; the V2 worker re-checks it before Iceberg or ontology I/O so a revoked binding also stops delayed work.

API:

GET  /api/v1/backing-datasets/capabilities
GET  /api/v1/backing-datasets/{id}/health
POST /api/v1/backing-datasets/{id}/sync?force=true
GET  /api/v1/backing-datasets/{id}/sync-history
GET  /api/v1/backing-datasets/{id}/sync-history/{run_id}

Compare attempted_sync_spec_fingerprint with applied_sync_spec_fingerprint on the exact run. effective_spec_changed means mappings or another effective execution input replayed an unchanged snapshot; unchanged must report status: skipped and rows_synced: 0. A failed run must retain the prior applied fingerprint. When reading API v1 directly, record snapshot_id_exact; the numeric snapshot_id remains only for legacy clients and can lose precision when parsed as a JavaScript number. frankctl and the browser client normalize the exact value into a canonical string-valued snapshot_id.

Rolling out sync contract v2

The V1 Temporal contract is immutable during the drain window: OntologySyncWorkflow and activities sync_to_ontology, update_sync_status, and emit_ontology_lineage_edge stay on ontology-sync-task-queue. V2 uses OntologySyncWorkflowV2, the corresponding *_v2 activities, ontology-sync-v2-task-queue, and the separate ontology-worker-v2 service. Only the V1 worker owns reconciliation schedules.

Use this migration-first order:

1. Apply the additive s39_bd_sync_spec_fingerprint migration.
2. Start ontology-worker-v2 while retaining the V1 worker.
3. Use Temporal task-queue describe/visibility evidence to verify live pollers and one executable workflow on both queues. Container health alone is not sufficient.
4. Route new API sync starts to V2 only after the V2 poller is proven live.
5. Update consumers and the CLI, then allow V1 histories and retries to drain.

Rollback routes new API starts back to V1 first. Keep the V2 worker until its workflows are terminal, and do not downgrade the additive schema while any V2 API, worker, history, or retry can still use it. Local verification of these steps is not production deployment evidence.

Rootless Airbyte protocol connector execution

Connector compatibility remains declarative: a pattern supplies the Airbyte source name and image. The source-worker image includes the Docker CLI but not the Airbyte Python package. Frank's thin runtime writes protocol inputs under the vetted /local mount, then executes docker run <image> <operation> against the explicit rootless daemon socket. The runtime constructs the entire fail-closed command vector itself; it has no upstream executable seam to trust.

Connector images are no longer republished, allowlisted, bundled, or preloaded by Frank. Docker pulls the pattern-declared image on demand through the dedicated rootless daemon. The worker has no registry credential, so the daemon — not the worker — is the only component with public registry egress.

Source and discovery workers receive only the read-only rootless Docker socket directory and their writable per-role connector workspace. They never receive /var/run/docker.sock. Each worker remains non-root with a read-only root filesystem, capability drop, no-new-privileges, and bounded noexec /tmp. The protocol runtime's Docker command adds connector limits and isolation: bridge networking, no privileges or capabilities, read-only root, private IPC/cgroup namespaces, bounded memory/CPU/PIDs, two bounded noexec,nosuid,nodev tmpfs mounts, no host mounts other than the vetted /local workspace, and tenant/source/execution/attempt labels.

The release deployer requires the installed rootless daemon socket, a successful Docker version response through that socket, and the rootless daemon UID's nft egress chain before it starts workers. It also resolves Compose and rejects a worker boundary that exposes a host socket, a different socket path, an unreviewed workspace mount, or a missing connector-socket group. Restoring the host Docker socket is never a rollback.

scripts/bootstrap_airbyte_executor.py now bootstraps only the rootless daemon: the engine identity, socket ownership, its systemd unit, the per-role workspaces, nft egress fence, and readiness. It does not copy an application release, install a broker or AuthZ plugin, preload images, or accept image bundles. A daemon that cannot bind the dedicated socket or pass its explicit readiness check remains unavailable to workers.

The daemon owns its data directory and exposes only /run/frank-airbyte-executor/docker/docker.sock. Its systemd post-start hook sets that single socket to the reviewed worker group and mode 0660 on every restart, then verifies that it is a Unix socket. The broker and Docker AuthZ plugin have been removed; neither is part of daemon readiness or recovery.

The rootless daemon uses systemd readiness. Bootstrap waits for the socket, checks it is a Unix socket, and runs Docker version through that exact socket. There is no preload unit, image import, offline archive, or fallback to the host daemon. Failure leaves the connector runtime unavailable rather than silently broadening authority.

The health probe and deployer invoke Docker only with DOCKER_HOST=unix:///run/frank-airbyte-executor/docker/docker.sock; they do not consult the host Docker context or socket. Connector image pull is normal Docker on-demand resolution under the rootless daemon's network policy.

The tool writes a credential-safe deployment journal, but that journal and a green container health state are not live acceptance. Production acceptance also requires exact workflow/activity pollers, representative real discovery and extraction through the rootless daemon, and a daemon restart/recreation proof that preserves the no-host-socket boundary.

Target immutable production releases

This is the target/alternate release topology, not the current mutable box. The current base Compose deployment uses locally built images and does not use Harbor or docker-compose.release.yml. Activating this topology is a deliberate runbook cutover.

For the immutable topology, Frank application releases are built and verified by .github/workflows/harbor.yml. Pull requests run source-only, non-deploying release-contract tests; the only secret they consume is the established GH_PAT repository secret needed to fetch the private shared-utils submodule, and checkout does not persist it. This avoids a second undeployed checkout secret and does not authorize production access. Pull requests receive no Harbor, SSH, runtime, or production-environment credentials and cannot publish candidates, promote tags, or deploy. After merge, the protected main workflow builds the four exact candidates, verifies their digests against real PostgreSQL and Temporal, and may deploy through the protected production GitHub environment only when the repository variable PRODUCTION_DEPLOY_ENABLED is exactly true. Keep the opt-in disabled until that environment and the reviewed host boundary are provisioned. The exact source-worker digest also runs the credential JIT, legacy-payload redaction, and poller-health suites; an import-only smoke test is not sufficient.

The production input is a credential-free manifest plus a deterministic, hash-bound bundle containing only the two Compose files, the release deployer, and reviewed runtime assets. CI never sends a Git checkout, source overlay, environment file, or credential to the host. A first protected deployment may initialize the reviewed host launcher and operator paths. Every later release requires the launcher, config, and bootstrap evidence to be byte-identical and mode-correct; drift stops deployment rather than being overwritten.

The launcher reads operator-owned paths from /opt/frank-low-code/config/deploy.json:

{
  "env_file": "/operator/path/frank.env",
  "data_dir": "/operator/path/frank-data"
}

The environment file remains outside release artifacts and is used only for Compose interpolation. No resolved release service inherits it. API, migration, preflight, source-worker, and discovery-worker Vault identities use the same generic VAULT_ROLE_ID_FILE, VAULT_SECRET_ID_FILE, and VAULT_CACERT container paths but distinct host files; every unrelated service receives no Vault identity. Direct AppRole values and unsafe identity files fail release preflight. The stable data directory remains the same across releases.

Harbor is one optional immutable-image publication path. The canonical server runbook also defines the reviewed local-image #665 cutover, so Harbor is not a prerequisite for connector-execution isolation. Neither path copies source into running containers or bind-mounts the host checkout into workers.

The deployer validates all digests and OCI Git-revision labels, proves that the API AppRole has exactly create, update, and read on the reviewed Source credential path and the worker AppRole has exactly read, applies the additive schema migration, and migrates every structurally declared legacy credential as one database transaction. Any identity, schema, Vault readback, or migration failure stops before a new application process starts. The old API and source consumers are stopped during contraction; this is not described as a Temporal queue drain. New source and discovery workers sanitize queued legacy payloads and must prove exact workflow and activity pollers before admission reopens. The deployer then waits for the exact V2 ontology pollers before starting the API and remaining workers. Only a successful deployment writes sanitized evidence under /opt/frank-low-code/evidence/ and moves the current symlink. It never runs Compose down, prunes images or volumes, or executes an Alembic downgrade.

Verify a release with the health endpoint and persisted evidence:

curl --fail https://api.example.invalid/health

The response includes only the full release SHA and API image digest. Compare those values to the reviewed release manifest. Also verify the single Alembic head and current Temporal workflow/activity pollers; container health or a 2xx response alone is not release evidence.

On failure, inspect the persisted deployment journal to identify the last completed stage. The unchanged current symlink does not prove that partially started services reverted. Fix forward after a committed migration. A prior digest manifest may be redeployed only after its binaries are explicitly proven compatible with the current additive schema and live Temporal histories. Never run an automatic schema downgrade, compose down, volume deletion, database recreation, or image prune. Keep the V2 worker available while any V2 workflow history or retry remains non-terminal.

Changing the launcher or operator paths is not an application release. It requires a separate reviewed and explicitly approved re-bootstrap procedure; normal CI deploys intentionally fail on such drift.

Key environment variables

Area	Variables
API and auth	KEYCLOAK_URL, KEYCLOAK_REALM, KEYCLOAK_CLIENT_ID, KEYCLOAK_ISSUER, CORS_ALLOWED_ORIGINS
Database	POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD
Iceberg/S3	ICEBERG_CATALOG_URI, ICEBERG_CATALOG, AWS_ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION
Temporal	TEMPORAL_HOST, TEMPORAL_PORT, TEMPORAL_NAMESPACE, task queue variables
Source worker	TEMPORAL_SOURCE_ACTIVITY_MAX_WORKERS, TEMPORAL_SOURCE_MAX_CONCURRENT_ACTIVITIES, TEMPORAL_SOURCE_MAX_CONCURRENT_WORKFLOW_TASKS, TEMPORAL_SOURCE_MAX_CONCURRENT_ACTIVITY_TASK_POLLS, TEMPORAL_SOURCE_MAX_CONCURRENT_WORKFLOW_TASK_POLLS, role-specific FRANK_*_BROKER_SOCKET_DIR
Source credential mode	SOURCE_CREDENTIAL_MODE is exactly legacy_inline on base Compose or vault on the immutable release overlay during Generation A
Dagster	DAGSTER_URL, DAGSTER_BRONZE_AUTOMATION_SHARD_COUNT, DAGSTER_TRANSFORM_AUTOMATION_SHARD_COUNT
Logs/traces	LOKI_URL, LOKI_AUTH_TOKEN, OTEL_EXPORTER_OTLP_ENDPOINT
AI	MARTHA_API_URL, MARTHA_KEYCLOAK_URL, MARTHA_CLIENT_ID, MARTHA_CLIENT_SECRET
Ontology	ONTOLOGY_ENABLED, ONTOLOGY_SERVICE_URL, ONTOLOGY_API_KEY, ONTOLOGY_TENANT_ID, ONTOLOGY_TENANT_BINDINGS_JSON
Source credential Vault	Shared VAULT_ADDR/CA plus separate service-specific file-backed AppRole identities on the immutable release overlay
Worker platform secrets	Service-specific PostgreSQL and object-store secret file paths; containers receive only generic POSTGRES_PASSWORD_FILE, AWS_ACCESS_KEY_ID_FILE, and AWS_SECRET_ACCESS_KEY_FILE
Pattern registry	PATTERN_WEBHOOK_SECRET, PATTERN_ADMIN_SECRET

Maintenance commands

make up
make down
make status
make logs
make build
make build-no-cache
make init-iceberg
make init-db
make init-sdm
make test-iceberg

For API route-level checks:

curl http://localhost:8002/health
curl http://localhost:8002/api/v1/status
curl http://localhost:8002/api/v1/services/health