Pattern catalog

Patterns are Frank's connector templates. Each one declares how to extract data from a particular kind of source. The pattern catalog is the lookup table for Source.pattern_id in your apply YAML — pick a pattern, fill in its source_config, and Frank wires up the extraction engine, discovery, and incremental cursor logic for you.

# Live list of patterns on your deployment:
frankctl patterns list

# Per-pattern config schema (what source_config keys it accepts):
frankctl patterns get <pattern_id>

This page is curated reference. The authoritative source is backend/config/patterns/*.json in this repo — if a field surprises you, grep there.

At a glance

Pattern	Category	Engine	Typical use
rest_api	API	dlt	Any HTTP REST endpoint with pagination.
rss	API	dlt	RSS / Atom feeds.
graphql	API	dlt	GraphQL queries.
sftp_bulk	File	dlt	Directory of files on SFTP.
s3	File	dlt	S3 / S3-compatible buckets.
archive	File	dlt	Downloadable archives (ZIP / TAR / .gz).
filesystem	File	dlt	Local or NFS-mounted filesystem.
postgres	Database	dlt	PostgreSQL with WAL CDC.
mysql	Database	dlt	MySQL with binlog CDC.
mongodb	Database	dlt	MongoDB collections.
mssql	Database	dlt	SQL Server.
bigquery	Warehouse	dlt	BigQuery query / table extract.
snowflake	Warehouse	dlt	Snowflake query / table extract.
redshift	Warehouse	dlt	Redshift query / table extract.
databricks	Warehouse	dlt	Databricks Delta table extract.
kafka	Stream	dlt	Kafka topic consumption.
google_sheets	SaaS	dlt	Google Sheets cells.
salesforce	SaaS	airbyte	Salesforce objects.
hubspot	SaaS	airbyte	HubSpot CRM.
stripe	SaaS	airbyte	Stripe payments + customers.
zendesk	SaaS	airbyte	Zendesk tickets.
jira	SaaS	airbyte	Jira issues + projects.
github	SaaS	airbyte	GitHub repos + issues.
slack	SaaS	airbyte	Slack messages + channels.
notion	SaaS	airbyte	Notion pages + databases.
airtable	SaaS	airbyte	Airtable bases.
google_ads	SaaS	airbyte	Google Ads reporting.
google_analytics	SaaS	airbyte	GA4 reporting.

Patterns commonly used in prod

These are the patterns most prod sources route through today. Worth knowing in detail.

rest_api

The workhorse. Use this for any HTTP endpoint that returns JSON, including OWM, ipma, ogc_api_wfs, custom ministry APIs, etc.

Minimum source_config:

source_config:
  base_url: https://api.example.com
  endpoint: /v1/observations

Common extensions:

Key	Purpose
query_params	Map of fixed query params. Supports ${VAR} env expansion at CLI load time.
headers	Map of fixed request headers (e.g. Authorization).
record_path	Dotted path into the JSON response to find the records array. Default data or records. Use list for OWM-shaped responses.
pagination	`{type: page

Auth shapes the engine knows: Basic, Bearer, API key in header or query param, Keycloak passthrough. See the auth: examples in dev_docs/examples/pipelines/.

rss

Dead simple. Just feed_url:

source_config:
  feed_url: https://feeds.feedburner.com/example

Items land in bronze with title, link, published_at, summary, guid, plus whatever extension fields the feed includes. Sync mode is full_refresh; the GUID is the natural dedup key.

sftp_bulk

Directory of files on SFTP. Use sync_mode: iterator on the Source so the workflow checkpoints between chunks (default monolithic will time out on directories of >~5k files).

source_config:
  host: sftp.example.com
  port: 22
  username: "${SFTP_USER}"
  password: "${SFTP_PASSWORD}"
  path_prefix: /inbox/2026
  file_glob: "*.csv"
  file_format: csv
  fetch_concurrency: 5

Frank's file-ledger v2 skips files it's already committed, but re-reads any file whose mtime OR size changed. See the sftp-file-walk/ example for the full shape.

s3

S3 or S3-compatible bucket. Same shape as sftp_bulk but with bucket / prefix instead of host / path_prefix.

source_config:
  bucket: my-bucket
  prefix: incoming/2026/
  file_glob: "*.parquet"
  file_format: parquet
  aws_access_key_id: "${AWS_ACCESS_KEY_ID}"
  aws_secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
  region: eu-west-1

postgres

PostgreSQL with WAL CDC for incremental sync.

source_config:
  host: pg.example.com
  port: 5432
  database: production
  username: "${PG_USER}"
  password: "${PG_PASSWORD}"
  schemas: [public, analytics]
  cdc_enabled: true
  replication_slot: frank_repl

Discovery enumerates tables in the listed schemas. The Stream's primary_key_path becomes the dedup key on merge writes.

archive

Downloadable archive (ZIP / TAR / TAR.gz). Use this for sources like GTFS-static where the publisher dumps everything in one big bundle.

source_config:
  archive_url: https://example.com/gtfs/static.zip
  format: zip
  file_glob: "*.txt"      # which files inside the archive to extract
  file_format: csv

The archive itself is the unit of progress — there's no incremental-within-an-archive concept. To re-fetch, increment the archive_url query string (e.g. ?version=2).

Less-common patterns

Patterns from the at-a-glance table not detailed above (graphql, google_sheets, all of the SaaS connectors, etc.) follow the same declare-source_config pattern. Run frankctl patterns get <pattern_id> on your deployment to see the exact JSON-Schema, or read the source at backend/config/patterns/<pattern_id>.json.

Patterns NOT in the public catalog

Frank ships some prod-only / custom patterns that aren't in the catalog reference (e.g. eredes_sgl, bpstat, ren_datahub, ipma). These are wired in backend/services/dlt/builders/ but not exposed as templates — you can't pick them from a fresh Source create flow. If you need one, the corresponding builder is the reference.

Authoring a new pattern

Three pieces, all under backend/:

1. Pattern JSON at backend/config/patterns/<id>.json — declares the source_config_schema (JSON-Schema) the wizard validates against.
2. Config builder at backend/services/dlt/builders/<id>.py (extends BaseConfigBuilder) — translates the operator's source_config into the engine's runtime config.
3. Discovery engine at backend/services/discovery/<id>.py (extends DiscoveryEngine) — produces the list of streams + their json_schema.

Templates for each are at backend/config/patterns/_template.json, backend/services/dlt/builders/_template.py, backend/services/discovery/_template.py. See dev_docs/solutions/custom-connector-guide.md for the full walkthrough.

After registering in registry.py and the engine, your new pattern shows up in frankctl patterns list and is usable from apply -f immediately.