Piddiplatsch

Documentation ยท Overview slides

Build Status License: Apache-2.0 Python Version

Piddiplatsch processes ESGF STAC publication records from Kafka and registers persistent identifiers (PIDs) with the Handle System. It supports CMIP6, CMIP6Plus, CMIP7, and CORDEX-CMIP6, with project-specific routing and mapping.

Curious by nature. Persistent by design.

Quick start

Install with Conda and the development tools:

git clone https://github.com/ESGF/piddiplatsch.git
cd piddiplatsch
conda env create
conda activate piddi
make develop

Create your local configuration and replace the connection and credential placeholders before running:

cp etc/esgf-example.toml custom.toml
# Edit custom.toml for your site; keep credentials out of Git.
piddi config validate
piddi --help

Start harvesting and mapping Kafka messages into prepared Handle JSONL files:

piddi consume

By default, consume saves raw messages and prepares Handles without publishing to a Handle service. Kafka access is required; see Configuration for site settings.

Run in stages

You can also harvest a small sample, then map it separately:

piddi harvest --limit 100
piddi map --project cmip6 --date last

Once your Handle service profile is configured, publish a completed daily file:

piddi publish --project cmip6 --date yesterday

Use piddi COMMAND --help for options. The Operations guide explains date selection, publication, retries, logging, and monitoring.

Learn more