FAIRSenDD

Appearance

CWL Workflow ​

Common Workflow Language (CWL) is an open standard for describing command line applications to combine them together into workflows. The command line applications often packaged into Docker container to handle dependencies and versions of the corresponding binaries. CWL workflows are the keystone of Earth Observation Application Packages as standardized by OGC.

Real world data set ​

This example is using Sentinel-1 Sigma Nought backscatter data as processed using Wagner et al. 2021 in Equi7Grid projection. Similar world wide data can be accessed via the EODC STAC catalog. The following test dataset contains only a subset of a tile to make it faster to process.

bash
cd $(mktemp -d)
git clone https://github.com/meggart/RQADeforestationTestData
cwltool https://raw.githubusercontent.com/EarthyScience/FAIRSenDD/refs/heads/main/ogc-app-cwl/fairsendd.cwl#cmd-rqa \
    --continent EU \
    --tiles E051N018T3 \
    --start-date 2021-01-01 \
    --end-date 2022-01-01 \
    --in-dir RQADeforestationTestData
find out.zarr
Cloning into 'RQADeforestationTestData'...
INFO /usr/bin/cwltool 3.1.20240112164112
Error: No such object: danlooo/fairsendd_rqa:latest
INFO ['docker', 'pull', 'danlooo/fairsendd_rqa:latest']
latest: Pulling from danlooo/fairsendd_rqa
7cf63256a31a: Pulling fs layer
f378819f9e78: Pulling fs layer
d144084580a1: Pulling fs layer
67434e452dba: Pulling fs layer
107a5528da9e: Pulling fs layer
67434e452dba: Waiting
107a5528da9e: Waiting
f378819f9e78: Verifying Checksum
f378819f9e78: Download complete
d144084580a1: Verifying Checksum
d144084580a1: Download complete
7cf63256a31a: Verifying Checksum
7cf63256a31a: Download complete
7cf63256a31a: Pull complete
107a5528da9e: Verifying Checksum
107a5528da9e: Download complete
67434e452dba: Verifying Checksum
67434e452dba: Download complete
f378819f9e78: Pull complete
d144084580a1: Pull complete
67434e452dba: Pull complete
107a5528da9e: Pull complete
Digest: sha256:b74f3d41d7a916d95c9287ea304d34c79fd0d65382dcab92494cddfe88428ce6
Status: Downloaded newer image for danlooo/fairsendd_rqa:latest
docker.io/danlooo/fairsendd_rqa:latest
INFO [job cmd-rqa] /tmp/0vyizs91$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/0vyizs91,target=/YMHTvn \
    --mount=type=bind,source=/tmp/ckbb93k3,target=/tmp \
    --mount=type=bind,source=/tmp/tmp.iqhTr1VCPD/RQADeforestationTestData,target=/var/lib/cwl/stg20137883-3bee-408a-b492-ace4c981e179/RQADeforestationTestData,readonly \
    --workdir=/YMHTvn \
    --read-only=true \
    --user=0:0 \
    --rm \
    --cidfile=/tmp/g157hmso/20250328085834-636392.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/YMHTvn \
    danlooo/fairsendd_rqa:latest \
    --continent \
    EU \
    --in-dir \
    /var/lib/cwl/stg20137883-3bee-408a-b492-ace4c981e179/RQADeforestationTestData \
    --tiles \
    E051N018T3 \
    --start-date \
    2021-01-01 \
    --end-date \
    2022-01-01
[ Info: new driver key :zarr, updating backendlist.
[ Info: new driver key :gdal, updating backendlist.
[ Info: new driver key :netcdf, updating backendlist.
[ Info: Write output to out.zarr
relorbits = SubString{String}["022"]
length(dates) = 184
  2.367296 seconds (8.76 M allocations: 444.206 MiB, 5.76% gc time, 99.09% compilation time)
path = "out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0"
size(cube) = (50, 74, 61)
size(tcube) = (50, 74, 48)
outpath = "out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr"
  5.415836 seconds (12.28 M allocations: 629.678 MiB, 15.01% gc time, 92.27% compilation time)
INFO [job cmd-rqa] Max memory used: 695MiB
INFO [job cmd-rqa] completed success
{
    "out_cube": {
        "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr",
        "basename": "out.zarr",
        "class": "Directory",
        "listing": [
            {
                "class": "File",
                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.done",
                "basename": "E051N018T3_rqatrend_VH_D022_thresh_3.0.done",
                "size": 0,
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.done"
            },
            {
                "class": "Directory",
                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr",
                "basename": "E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr",
                "listing": [
                    {
                        "class": "Directory",
                        "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer",
                        "basename": "layer",
                        "listing": [
                            {
                                "class": "File",
                                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer/.zattrs",
                                "basename": ".zattrs",
                                "size": 54,
                                "checksum": "sha1$6308c95300c8fa2029526bd1dbaf14145d9acd8a",
                                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer/.zattrs"
                            },
                            {
                                "class": "File",
                                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer/.zarray",
                                "basename": ".zarray",
                                "size": 190,
                                "checksum": "sha1$f6ff9f2503dd57ddace6b59b168ca29f553292f8",
                                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer/.zarray"
                            },
                            {
                                "class": "File",
                                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer/0.0",
                                "basename": "0.0",
                                "size": 12653,
                                "checksum": "sha1$f8af38fd7909464a25eeb7c4a15857deb6c3532b",
                                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer/0.0"
                            }
                        ],
                        "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer"
                    },
                    {
                        "class": "File",
                        "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/.zattrs",
                        "basename": ".zattrs",
                        "size": 2,
                        "checksum": "sha1$bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f",
                        "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/.zattrs"
                    },
                    {
                        "class": "File",
                        "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/.zmetadata",
                        "basename": ".zmetadata",
                        "size": 2098,
                        "checksum": "sha1$1e5cab4fafc967967cd8b5f2ca1c085a73901bf1",
                        "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/.zmetadata"
                    },
                    {
                        "class": "Directory",
                        "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X",
                        "basename": "X",
                        "listing": [
                            {
                                "class": "File",
                                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X/.zattrs",
                                "basename": ".zattrs",
                                "size": 45,
                                "checksum": "sha1$2e7b4c4f612c94c0d3d006e2748a0f4258ad1806",
                                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X/.zattrs"
                            },
                            {
                                "class": "File",
                                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X/.zarray",
                                "basename": ".zarray",
                                "size": 184,
                                "checksum": "sha1$8112ff8d7d6d676db62674b3223953e586e22810",
                                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X/.zarray"
                            },
                            {
                                "class": "File",
                                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X/0",
                                "basename": "0",
                                "size": 114,
                                "checksum": "sha1$78d517402584b41e77edb234cb3fb629108860fd",
                                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X/0"
                            }
                        ],
                        "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X"
                    },
                    {
                        "class": "File",
                        "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/.zgroup",
                        "basename": ".zgroup",
                        "size": 17,
                        "checksum": "sha1$08e424c08f97fd3c7912e3d66a7e42ad818127b1",
                        "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/.zgroup"
                    },
                    {
                        "class": "Directory",
                        "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y",
                        "basename": "Y",
                        "listing": [
                            {
                                "class": "File",
                                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y/.zattrs",
                                "basename": ".zattrs",
                                "size": 45,
                                "checksum": "sha1$56f5964807358bcabd5a465171947d7623cc6dff",
                                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y/.zattrs"
                            },
                            {
                                "class": "File",
                                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y/.zarray",
                                "basename": ".zarray",
                                "size": 184,
                                "checksum": "sha1$73f1be4b6a5094366690999eeb27c5785d0fed2f",
                                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y/.zarray"
                            },
                            {
                                "class": "File",
                                "location": "file:///tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y/0",
                                "basename": "0",
                                "size": 157,
                                "checksum": "sha1$2cddf4f912c2d70033ec1daa83b63bc6af0d14a1",
                                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y/0"
                            }
                        ],
                        "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y"
                    }
                ],
                "path": "/tmp/tmp.iqhTr1VCPD/out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr"
            }
        ],
        "path": "/tmp/tmp.iqhTr1VCPD/out.zarr"
    }
}INFO Final process status is success
out.zarr
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.done
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer/.zattrs
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer/.zarray
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/layer/0.0
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/.zattrs
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/.zmetadata
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X/.zattrs
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X/.zarray
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/X/0
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/.zgroup
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y/.zattrs
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y/.zarray
out.zarr/E051N018T3_rqatrend_VH_D022_thresh_3.0.zarr/Y/0

The directory out.zarr forms a Zarr Dataset that can be opened as an n dimensional array in various ways, e.g. xarray in Python and YAXArrays in Julia.

© Copyright 2025.