Upload New File

# LAFI RDM Task Register

**Planning horizon:** September–December 2026  

**Last updated:** 15 September 2026  

**Purpose:** Working register for planning and tracking the LAFI research data management implementation and the associated RDM paper.

## Status key

- **Complete** — deliverable finished

- **In progress** — actively being worked on

- **Next** — ready to start or a near-term priority

- **Planned** — defined but scheduled later

- **Waiting** — dependent on data, feedback, or another task

## Milestones

| ID | Milestone | Target | Status |

|---|---|---:|---|

| M1 | Complete and validate Doppler lidar conversion scripts for all 2025 GOP scan types | October 2026 | In progress |

| M2 | Convert the remaining 2025 and selected historical Doppler lidar datasets | November 2026 | Planned |

| M3 | Implement a working automated pipeline prototype using Git, Docker, Python, and Bash | November 2026 | Planned |

| M4 | Convert the 2025 GOP Raman lidar L1-equivalent dataset to CF-conformant NetCDF | November 2026 | Planned |

| M5 | Submit an example lidar dataset and conversion workflow to obs4MIPs for initial feedback | November 2026 | Planned |

| M6 | Consolidate technical documentation and paper evidence | December 2026 | Planned |

| M7 | Produce an RDM paper outline and initial manuscript draft | December 2026 | Planned |

## Workstream A — Doppler lidar conversion

### A1. Vertical-stare conversion

| ID | Task | Deliverable | Status | Depends on |

|---|---|---|---|---|

| A1.1 | Convert the 2025 LAFI GOP vertical-stare data | Daily CF-oriented NetCDF files | Complete | — |

| A1.2 | Record conversion results, salvaged records, failures, and known quality issues | Conversion/QC summary | In progress | A1.1 |

| A1.3 | Complete organized post-conversion validation | Validation report and issue list | Planned | A1.1 |

### A2. Other scan types

| ID | Task | Deliverable | Status | Depends on |

|---|---|---|---|---|

| A2.1 | Inventory scan types, instruments, raw-data structures, and date coverage for the 2025 GOP | Dataset/scan inventory | Next | — |

| A2.2 | Compare structures with the vertical-stare input and identify reusable parser/conversion components | Structural comparison and implementation plan | Next | A2.1 |

| A2.3 | Define output data models and CF metadata for each scan type | Metadata and dimensional-model specification | Planned | A2.2 |

| A2.4 | Implement conversion scripts for the remaining scan types | Version-controlled Python scripts | Planned | A2.3 |

| A2.5 | Add representative test fixtures and automated tests | Test data and pytest suite | Planned | A2.4 |

| A2.6 | Run CF checking and scientific/structural validation | Validation reports and issue log | Planned | A2.4, A2.5 |

| A2.7 | Convert all relevant 2025 GOP data for the additional scan types | Validated 2025 NetCDF collection | Planned | A2.6 |

### A3. Historical and future deployment

| ID | Task | Deliverable | Status | Depends on |

|---|---|---|---|---|

| A3.1 | Inventory historical Doppler lidar datasets and determine compatibility with the new scripts | Historical-data conversion matrix | Planned | A2.2 |

| A3.2 | Adapt scripts for documented historical structural variants | Versioned parsing/conversion rules | Planned | A3.1, A2.4 |

| A3.3 | Convert and validate the prioritized historical datasets | Historical CF-oriented NetCDF collection | Planned | A3.2 |

| A3.4 | Prepare a repeatable deployment configuration for incoming 2026 data | 2026-ready configuration and runbook | Planned | Workstream B |

| A3.5 | Run the scripts when the 2026 dataset becomes available | 2026 converted dataset | Waiting | A3.4 and receipt of 2026 data |

## Workstream B — Automated pipeline prototype

**Objective:** Use the lidar conversion scripts as a realistic prototype for an automated research-data pipeline based on Git, Docker, Python, and Bash.

| ID | Task | Deliverable | Status | Depends on |

|---|---|---|---|---|

| B1 | Define the prototype workflow: ingest → inventory/preflight → convert → test → CF-check → report → publish/stage | Pipeline design | Next | A2.2 |

| B2 | Define raw, staging, output, logs, reports, and quarantine directory conventions | Directory and naming specification | Planned | B1 |

| B3 | Refactor conversion scripts into reusable command-line components with configuration files | Pipeline-ready conversion application | Planned | A2.4 |

| B4 | Create a reproducible Docker image with pinned dependencies | Dockerfile and image build instructions | Planned | B3 |

| B5 | Create Bash orchestration for local/VM execution and safe reruns | Driver scripts | Planned | B2, B3 |

| B6 | Implement GitLab CI jobs for testing, CF checking, artifacts, and controlled conversion runs | `.gitlab-ci.yml` prototype | Planned | B3, B4 |

| B7 | Define triggers for manual, scheduled, and new-data runs | Trigger specification and prototype | Planned | B5, B6 |

| B8 | Implement logging, manifests, checksums, provenance, failure handling, and restart behaviour | Auditable run records | Planned | B5, B6 |

| B9 | Test the full prototype with representative lidar data on the intended infrastructure | End-to-end test report | Planned | B4–B8 |

| B10 | Prepare the pipeline for operational use with 2026 data | Deployment/runbook | Planned | B9 |

## Workstream C — Raman lidar conversion

| ID | Task | Deliverable | Status | Depends on |

|---|---|---|---|---|

| C1 | Inventory the 2025 GOP L1-equivalent Raman lidar dataset, variables, dimensions, metadata, formats, and coverage | Dataset assessment | Next | — |

| C2 | Map source variables and metadata to applicable CF conventions and standard names | CF mapping specification | Planned | C1 |

| C3 | Resolve instrument-specific representation and quality-control requirements with the data provider | Agreed data model and QC rules | Planned | C1, C2 |

| C4 | Implement the Raman lidar conversion script | Version-controlled Python script | Planned | C2, C3 |

| C5 | Add automated tests and representative fixtures | Test suite | Planned | C4 |

| C6 | Convert the 2025 GOP dataset | CF-oriented NetCDF collection | Planned | C4 |

| C7 | Run CF checks plus structural and scientific validation; resolve or document findings | Validation report and issue log | Planned | C5, C6 |

## Workstream D — obs4MIPs engagement

| ID | Task | Deliverable | Status | Depends on |

|---|---|---|---|---|

| D1 | Select a small, representative, validated lidar dataset for initial discussion | Example dataset | Planned | A1.3 or A2.6 |

| D2 | Review obs4MIPs Data Specifications 2.6 against the example dataset | Gap analysis | Planned | D1 |

| D3 | Package the example data, relevant conversion code, metadata description, and documented open questions | Consultation package | Planned | D2 |

| D4 | Identify the appropriate obs4MIPs contact and draft the initial request | Contact record and email draft | Planned | D3 |

| D5 | Contact obs4MIPs and request guidance on adapting ground-based lidar observations to their requirements | Submitted request | Planned | D4 |

| D6 | Record feedback, decisions, required changes, and implications for Doppler/Raman workflows | Decision log and implementation backlog | Waiting | D5 and external response |

| D7 | Implement an initial obs4MIPs-compatible output profile if the guidance supports it | Prototype dataset and code changes | Waiting | D6 |

## Workstream E — GitLab documentation

| ID | Task | Deliverable | Status | Depends on |

|---|---|---|---|---|

| E1 | Define a documentation structure separating user guidance, dataset documentation, pipeline operations, and developer reference | Documentation map | Next | — |

| E2 | Document Doppler lidar datasets, structural variants, metadata strategy, conversion use, and validation | Doppler lidar documentation | In progress | Workstream A |

| E3 | Document Raman lidar source data, mappings, conversion, and validation | Raman lidar documentation | Planned | Workstream C |

| E4 | Document Docker image, dependencies, configuration, pipeline stages, deployment, and troubleshooting | Pipeline documentation | Planned | Workstream B |

| E5 | Document provenance, checksums, logs, reruns, failures, and validation-report interpretation | Operations/runbook documentation | Planned | B8, B9 |

| E6 | Document obs4MIPs gap analysis, communication, decisions, and resulting implementation | obs4MIPs documentation | Planned | Workstream D |

| E7 | Review links, commands, examples, terminology, and reproducibility from a new-user perspective | Documentation review | Planned | E2–E6 |

## Workstream F — RDM paper

**Provisional focus:** Implementing FAIR-oriented, standards-based data pipelines for heterogeneous atmospheric observations within general university infrastructure and a time-limited collaborative research project.

| ID | Task | Deliverable | Status | Depends on |

|---|---|---|---|---|

| F1 | Define the research question, contribution, audience, and candidate journal | Paper concept note | Next | — |

| F2 | Create an evidence register linking claims to implemented components and documented results | Evidence matrix | Planned | E1 and ongoing workstreams |

| F3 | Document the project/infrastructure context and constraints | Case-study notes | Planned | — |

| F4 | Describe the architecture and methods: standards, conversion, testing, containerization, automation, provenance, access, and dissemination | Methods outline | Planned | A–E |

| F5 | Define evaluation criteria and collect evidence such as coverage, success/failure counts, reproducibility, validation findings, and operational effort | Evaluation dataset | Planned | A–D |

| F6 | Develop figures and tables for the data lifecycle, pipeline architecture, heterogeneous data challenges, and outputs | Figure/table plan | Planned | F2, F4 |

| F7 | Draft the manuscript outline | Detailed outline | Planned | F1–F3 |

| F8 | Draft an initial manuscript | Manuscript draft | Planned | F4–F7 |

| F9 | Identify remaining evidence gaps and convert them into follow-up implementation tasks | Revised backlog | Planned | F8 |

## Proposed sequence: September–December 2026

### September

- Complete the scan-type and Raman lidar inventories (A2.1, C1).

- Define the documentation map and pipeline design (E1, B1).

- Define the paper concept and begin the evidence register (F1, F2).

- Continue documenting and validating the completed vertical-stare conversion (A1.2–A1.3).

### October

- Design and implement the additional Doppler scan-type conversions (A2.2–A2.5).

- Establish naming, directory, configuration, and command-line conventions (B2–B3).

- Complete the Raman lidar CF mapping and begin implementation (C2–C4).

- Start the obs4MIPs gap analysis using a validated example (D1–D2).

### November

- Convert and validate remaining 2025 Doppler and Raman lidar data (A2.6–A2.7, C5–C7).

- Build and test the Docker/GitLab/Bash pipeline prototype (B4–B9).

- Package and send the initial obs4MIPs consultation request (D3–D5).

- Begin prioritized historical Doppler conversion (A3.1–A3.3).

### December

- Consolidate repository documentation and operational runbooks (E2–E7).

- Prepare the 2026 deployment configuration (A3.4, B10).

- Incorporate any obs4MIPs feedback received (D6–D7).

- Complete the paper evidence matrix, outline, figures/tables plan, and initial draft (F2–F8).

- Review unfinished items and carry justified work into the 2027 backlog.

## Immediate priority queue

1. **A2.1:** Inventory all non-vertical Doppler lidar scan types and 2025 GOP coverage.

2. **C1:** Inventory the Raman lidar L1-equivalent dataset.

3. **A1.3:** Define and complete organized validation of the vertical-stare outputs.

4. **B1:** Draw the end-to-end pipeline workflow and define stage interfaces.

5. **E1:** Agree the GitLab documentation structure.

6. **F1:** Write a one-page RDM paper concept note.

## Decision and issue log

| Date | Type | Item | Decision / issue | Follow-up |

|---|---|---|---|---|

| 2026-09-15 | Decision | Planning scope | Use this register to track LAFI RDM conversion, pipeline, documentation, obs4MIPs, and paper work through December 2026. | Update whenever work starts, finishes, is blocked, or new work appears. |

| 2026-09-15 | Reference | Standards | CF Conventions and obs4MIPs Data Specifications 2.6 are available as primary references for detailed mapping and compliance work. | Cite exact sections when implementing or reviewing data models. |

## Update protocol for this chat

When reporting progress, use any natural phrasing such as “A2.1 is complete,” “the Raman conversion is blocked,” or “add a task for …”. The register should then be updated as follows:

1. Change the relevant task status.

2. Add the completion date or blocker where useful.

3. Record decisions that affect later implementation.

4. Add newly discovered tasks under the appropriate workstream.

5. Reorder the immediate priority queue when dependencies or deadlines change.

6. Preserve completed tasks so the register also becomes a project record and a source of evidence for the paper.