Generate mapfile for ESGF publication¶

Default mapfile generation¶

The default behavior is to pick up the latest version in the DRS. This required version with a date format (e.g., v20151023). If the version is directly specified in positional argument, the version number from supplied directory is used.

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/

Mapfile without files checksums¶

Because this could be time consuming --no-checksum allows you not include the file checksum into your mapfile(s).

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --no-checksum

Mapfile with pre-calculated checksums¶

If your file checksum have been already calculated apart, you can submit a file to esgmapfile with the checksums list. This checksum file must have the same format as the output of the UNIX command-lines “*sum”.

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --checksums-from /PATH/TO/CHECKSUMS/FILE

Mapfile without DRS versions¶

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --no-version

Mapfile name using tokens¶

The mapfile name is composed by the dataset ID and the dataset version dot-separated. Another template can be specified for the output mapfile(s) name using several tokens. Substrings {dataset_id}, {version}, {job_id} or {date} (in YYYYDDMM) will be substituted where found. If {dataset_id} is not present in mapfile name, then all datasets will be written to a single mapfile, overriding the default behavior of producing ONE mapfile PER dataset.

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --mapfile {dataset_id}.{job_id}
$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --mapfile {date}.{job_id}
$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --mapfile MY_MAPFILE.{version}.{date}

Organize your mapfiles¶

The mapfile(s) are generated into a mapfile folder created in your working directory (if exists). This can be changed by submitting an output directory for your mapfiles.

In addition, a mapfile_drs attribute can be added into the corresponding project section of the configuration INI file(s) (see Configuration). In the same way as the directory_format it defines a tree depending on the facets. Each mapfile is then written into the corresponding output directory. This mapfile_drs directory structure will be added to the output directory if submitted.

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --outdir /PATH/TO/MY_MAPFILES/

The output directory is cleaned up prior to mapfile process to avoid uncompleted mapfiles. In the case of several esgmapfile instances run with the same output directory it is recommended to disable the cleanup:

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --no-cleanup

Walking through latest directories only¶

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --latest-symlink

Walking through a particular version only¶

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --version VERSION

Walking through all versions¶

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --all-versions

Add technical notes¶

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --tech-notes-url URL --tech-notes-title TITLE

Overwrite the dataset identifier¶

$> esgmapfile make --project PROJECT_ID /PATH/TO/SCAN/ --dataset-name DATASET_NAME

Show the expected mapfile name and path¶

The show works as a “dry-run” of the make and supports different types inputs. You can show the mapfiles full path to be generated from:

• a directory to scan:

   $> esgmapfile show --project PROJECT_ID --directory /PATH/TO/SCAN

- a text file with one dataset ID per line:

   $> esgmapfile show --project PROJECT_ID --dataset-list /PATH/TO/TXT_FILE

- a unique dataset ID:

$> esgmapfile show --project PROJECT_ID --dataset-id DATASET_ID

In the case of --dataset-list if no file submitted, the standard input is used.

$> esgmapfile show --project PROJECT_ID --dataset-list < /PATH/TO/TXT_FILE

Exit status¶

• Status = 0

  All the files have been successfully scanned and the mapfile(s) properly generated.

• Status > 0

  Some scan errors occurred or files have been skipped. The error code indicates the number of errors.