README.md for production doppler lidar conversion documents

# Production Doppler Lidar Conversion

This directory documents the production workflow used to convert Doppler lidar

data within LAFI from the original Halo Photonics HPL format into structured

NetCDF products suitable for long-term storage, scientific use, and

CF-oriented data publication.

The documentation is organised around two complementary aspects of the

conversion process:

1. **Data-oriented documentation**, describing the characteristics of the

   source datasets, scan geometries, structural variants, exceptional files,

   and the scientific reasoning behind individual processing decisions.

2. **Technical documentation**, describing the implementation of the

   conversion workflow, the processing stages, scripts, output structure,

   logging, and procedures required to reproduce the conversion.

The corresponding production scripts are maintained under:

`../../scripts/production_doppler_lidar_conversion/`

---

## Scope

The workflow currently focuses primarily on Doppler lidar observations acquired

in vertical or near-vertical stare mode.

The first production datasets include observations from:

- **DL011**

- **DL141**

- **DL262**

These datasets contain similar measurements but differ in instrument

configuration, range-gate structure, acquisition periods, and in some cases

scan geometry.

The workflow is therefore designed to preserve the characteristics of the

original observations rather than forcing all instruments or files into a

single uniform structure.

Non-vertical stare observations and scanning measurements are retained as a

separate processing category and will be documented under

`other_scan_types/`.

---

## Processing philosophy

The production conversion follows a preservation-oriented approach.

The primary principle is that conversion and scientific quality control are

treated as separate tasks.

Where possible:

- all parseable source files are retained;

- all parseable rays and range gates are retained;

- observations are not removed because they appear scientifically unusual;

- structural inconsistencies are documented rather than silently corrected;

- original instrument geometry, including azimuth and elevation, is retained;

- incompatible native range structures are kept separate rather than

  interpolated onto a common grid;

- exceptional observations are preserved even when their final CF

  representation has not yet been defined.

Scientific filtering and higher-level quality control can therefore be applied

later without losing information from the original observations.

---

## Production processing levels

The current conversion consists of three main processing stages:

```text

Original HPL files

        |

        v

Source-aligned Level-1 NetCDF

        |

        v

Daily Level-1 NetCDF

        |

        v

CF-oriented daily NetCDF

```

### 1. Source-aligned Level-1

Each source HPL file is converted independently to NetCDF.

The aim of this stage is to preserve the original observations in a structured

binary format while retaining information about the source file and any

structural issues encountered during parsing.

Typical variables include:

- radial velocity;

- intensity;

- attenuated backscatter;

- time;

- range;

- azimuth;

- elevation;

- pitch;

- roll;

- data-presence and parser-quality indicators.

No scientific screening is applied at this stage.

---

### 2. Daily Level-1 aggregation

Source-aligned Level-1 files are subsequently combined into daily products.

Aggregation is performed only between files with compatible native structures.

Files with different range-gate lengths, numbers of range gates, or otherwise

incompatible range coordinates are kept as separate daily products.

Consequently, a single UTC date can legitimately produce more than one daily

NetCDF file.

No temporal averaging, interpolation, scientific filtering, or duplicate

removal is performed during aggregation.

---

### 3. CF-oriented daily products

Daily Level-1 files representing confirmed vertical-profile observations are

converted into concise CF-oriented NetCDF products.

This stage adds or standardises metadata including:

- CF conventions;

- station identification;

- instrument location;

- instrument altitude;

- spatial coordinates;

- vertical coordinate information;

- variable standard names where scientifically appropriate;

- radial-velocity sign convention;

- feature-type metadata.

Instrument-specific latitude, longitude, altitude, and station identifiers are

provided by the production configuration rather than inferred from another

instrument.

Observations whose geometry is not compatible with the current vertical-profile

representation are retained at Level-1 and their CF conversion is deferred.

---

## Vertical and non-vertical observations

The documentation is divided into two main categories:

```text

production_doppler_lidar_conversion/

|

|-- vertical_scan/

|

`-- other_scan_types/

```

### `vertical_scan/`

Contains documentation for observations verified as vertical or near-vertical

stares.

This includes:

- dataset-specific inventories;

- scan-angle checks;

- range-gate configurations;

- structural variants;

- exceptional records;

- instrument-specific processing decisions;

- CF metadata strategy.

The current production work on DL011, DL141, and the vertical component of

DL262 belongs to this category.

### `other_scan_types/`

Reserved for measurements requiring a different spatial representation,

including:

- non-vertical fixed stares;

- scanning measurements;

- overlapping or specialised scan modes where the geometry differs from a

  simple vertical profile.

These observations may share the same source HPL reader and Level-1 conversion

logic but can require different aggregation and CF spatial metadata.

---

## Dataset-specific documentation

Detailed information for individual instruments is maintained separately so

that differences between instruments remain explicit.

For example:

```text

vertical_scan/

|-- dl011.md

|-- dl141.md

`-- dl262.md

```

These documents record information such as:

- number of source files;

- acquisition period;

- scan types;

- range-gate configurations;

- pointing-angle distributions;

- known structural anomalies;

- exceptional files;

- conversion decisions;

- instrument coordinates and altitude.

This separation is important because the production converter is shared across

multiple instruments, while the source datasets themselves are not identical.

---

## Structural variants

Doppler lidar files within the same instrument record can contain different

numbers of range gates or different range-gate lengths.

These differences are treated as characteristics of the source observations,

not automatically as errors.

The daily aggregation therefore preserves structurally incompatible

observations separately.

Further discussion is provided in:

`vertical_scan/structural_variants.md`

---

## CF metadata strategy

The CF-oriented conversion is currently designed primarily for vertical

time-height Doppler lidar observations.

The implementation retains the original pointing information and uses the

instrument location together with range and elevation to construct appropriate

vertical spatial metadata.

A separate treatment is required for strongly non-vertical beams or scanning

geometries.

The rationale and individual metadata decisions are documented in:

`vertical_scan/cf_metadata_strategy.md`

---

## Reproducibility

The production conversion scripts and reusable Python processing code are

maintained separately from this documentation under:

`../../scripts/production_doppler_lidar_conversion/`

The technical documentation there describes:

- software environment;

- source and output directory layout;

- reusable Python modules;

- command-line conversion scripts;

- production wrapper scripts;

- logging;

- rerunning interrupted conversions;

- basic post-conversion checks.

Dataset-specific production wrappers provide the paths and metadata required

for each instrument while sharing the same lower-level conversion code.

---

## Validation

Basic checks are performed after conversion to confirm that the expected files

were processed and that no command-level conversion or aggregation failures

occurred.

Additional post-conversion validation, including systematic metadata and CF

checks across instruments, will be developed separately.