# CF Metadata Strategy for Vertical-Scan Doppler Lidar Data
## Purpose
This document records the metadata strategy used to produce CF-oriented daily NetCDF files from the LAFI vertical-scan Doppler lidar Level-1 products.
The current implementation targets the CF 1.13 conventions and is designed primarily for repeated vertical Doppler lidar profiles at a fixed instrument location.
The strategy is intentionally conservative: CF metadata are added where the scientific meaning of the source quantity is sufficiently clear, while uncertain interpretations are retained through descriptive metadata rather than being forced into a standard name.
---
## CF conventions used
The current production files are designed according to CF 1.13.
CF defines a `timeSeriesProfile` as a series of vertical profile features at the same horizontal position. This provides the conceptual basis for representing repeated vertical lidar profiles at a fixed station. fileciteturn30file1
CF also requires that observations in a discrete sampling geometry can be unambiguously associated with their spatial and temporal coordinates, and it uses the global `featureType` attribute to identify the discrete sampling geometry represented by the file. fileciteturn30file7
The production files therefore use:
```text
featureType = "timeSeriesProfile"
```
for the confirmed vertical-profile observations.
---
## CF version and checker compatibility
The production conversion currently targets **CF 1.13**, while the CF checker available in the current workflow validates against **CF 1.8**.
This difference is important when interpreting checker output.
A checker run can therefore report a conventions-version error or warning even when the file has intentionally been written according to the newer CF 1.13 conventions.
The checker is currently used as a useful compatibility and structural diagnostic rather than as the sole definition of correctness for the CF-1.13 products.
Some implementation details were adjusted after testing with the CF-1.8-capable checker.
In particular, station identifiers and station names were changed from variable-length NetCDF string variables to **fixed-length character arrays** because this provided better compatibility with the available checker.
This should not be interpreted as a CF-1.13 requirement. CF permits character strings to be represented using either `char` or string-valued variables; for character arrays, the final dimension represents the maximum string length. fileciteturn30file3
The fixed-length representation is therefore an implementation and interoperability choice made to improve behaviour with the available validation tooling.
A more systematic post-conversion validation against a checker supporting the current CF version is planned separately.
---
## Global conventions attribute
The final products identify the intended CF convention through the global `Conventions` attribute.
The current production implementation uses:
```text
Conventions = "CF-1.13"
```
The uploaded CF 1.13 document used during development is labelled as a draft and describes its draft convention identifier accordingly. fileciteturn29file1
The exact convention identifier used for final publication should therefore be reviewed during the organised post-conversion validation stage against the finalized CF 1.13 release and the validator used for publication.
---
## Spatial representation
### Horizontal location
Vertical-scan products represent observations above a fixed instrument location.
The following scalar coordinates are therefore included:
-`latitude`;
-`longitude`;
- instrument altitude.
Latitude and longitude use the standard CF coordinate metadata:
```text
latitude:standard_name = "latitude"
latitude:units = "degrees_north"
longitude:standard_name = "longitude"
longitude:units = "degrees_east"
```
Instrument coordinates are supplied explicitly by the dataset-specific production configuration and must not be inherited from another lidar.
---
## Beam range and altitude
The source HPL `range` represents distance along the lidar beam to the centre of each range gate.
It is retained because it describes the native instrument sampling geometry.
For the final vertical-profile products, `range` is not treated as the CF vertical geolocation itself.
Instead, an auxiliary `altitude` coordinate is calculated for every ray and range gate using:
```text
altitude = instrument_altitude + range × sin(elevation)
```
where:
-`instrument_altitude` is the lidar altitude above mean sea level;
-`range` is the source beam range to the gate centre;
-`elevation` is the recorded beam elevation for the individual ray.
The resulting altitude variable uses:
```text
standard_name = "altitude"
units = "m"
positive = "up"
```
CF discrete sampling geometries allow a vertical auxiliary coordinate to vary across profiles and vertical levels, which is consistent with representing altitude as a multidimensional coordinate when needed. fileciteturn30file4
The use of the measured elevation also avoids silently assuming an exact 90-degree beam when small pointing variations are present.
Pitch and roll are retained as source engineering variables but are not currently applied separately to the altitude calculation.
---
## Time coordinate
Time is stored as elapsed seconds from the Unix epoch:
```text
units = "seconds since 1970-01-01T00:00:00+00:00"
calendar = "proleptic_gregorian"
standard_name = "time"
axis = "T"
```
The source HPL timestamps are treated as UTC.
Daily aggregation concatenates and sorts the observations chronologically but does not average or deduplicate them.
---
## Station identification
The production files contain station-identification metadata so that the observations can be associated with the fixed observing platform.
The station identifier is assigned:
```text
cf_role = "timeseries_id"
```
CF strongly recommends a station variable with `cf_role = "timeseries_id"` for time series of profiles. fileciteturn30file1
Current station identifiers are instrument-specific, for example:
```text
GLAFO-UHOH-DL011
GLAFO-UHOH-DL141
GLAFO-UHOH-DL262
```
The station name identifies the common observing site, while the station ID distinguishes the individual lidar.
As noted above, station identifier and name are currently written as fixed-length character arrays for compatibility with the available CF-1.8 checker.
---
## Radial velocity
The measured Doppler velocity remains represented as **radial velocity** at Level-1.
It is not automatically renamed to vertical air velocity, even for vertical stares.
The current standard name is:
```text
radial_velocity_of_scatterers_toward_instrument
```
where the instrument sign convention has been confirmed as positive toward the instrument.
The sign convention is also described explicitly in metadata so that interpretation does not depend only on the standard-name string.
No scientific filtering is applied before assigning the Level-1 radial-velocity metadata.
for attenuated backscatter where the physical interpretation of the source quantity is considered sufficiently consistent with that CF quantity.
The source units are:
```text
m-1 sr-1
```
The production conversion also supports omitting this standard name when the calibration or physical interpretation is not sufficiently confirmed.
This conservative behaviour is intentional.
---
## Intensity
The HPL source describes the intensity quantity as:
```text
SNR + 1
```
No CF standard name is assigned to this variable.
Instead, the quantity is described using:
- a descriptive variable name;
-`long_name`;
- units of `1`;
- a comment recording the source interpretation.
This avoids assigning a standard name whose physical definition is not known to match the instrument quantity exactly.
---
## Use of the CF Assistant
The **CF Assistant** developed within the LAFI data-management workflow was used during metadata design to search the CF conventions and CF Standard Name Table for potentially appropriate standard names.
It was particularly useful for determining whether established CF standard names existed for the lidar quantities before introducing local descriptive names.
The CF Assistant was used as a discovery and interpretation aid rather than as an independent authority.
Candidate names identified through the tool were checked against their CF definitions and the physical interpretation of the source HPL variables before being adopted.
Where no sufficiently appropriate standard name was established, the production workflow retains a descriptive `long_name` instead.
---
## Ancillary structural information
The Level-1 products contain ancillary variables describing structural recovery from the HPL source, including:
-`ray_quality_flag`;
-`gate_data_present`;
-`valid_gate_count`.
These variables describe parsing and source-record completeness rather than scientific measurement quality.
Science variables reference these through `ancillary_variables` where appropriate.
This allows later quality-control processing to distinguish structural provenance from scientific screening.
---
## Coordinate associations
Science variables are associated with the coordinates required to interpret their observations, including as appropriate:
- time;
- latitude;
- longitude;
- altitude;
- instrument altitude;
- azimuth;
- elevation;
- station identifier.
CF requires data in discrete sampling geometries to be unambiguously associated with the coordinates needed to geolocate each observation. fileciteturn30file7
Azimuth, elevation, pitch, and roll are retained even for vertical scans because they are part of the original instrument geometry and can reveal small deviations or exceptional pointing configurations.
---
## Non-vertical observations
The current CF strategy applies only to observations verified as vertical or near-vertical profiles.
A source label such as `Stare` is not sufficient by itself to establish that geometry.
For example, one DL262 file labelled `Stare` was found to have:
```text
azimuth = 256.01 degrees
elevation = -2.00 degrees
```
That observation is preserved through Level-1 processing but is deliberately excluded from the current vertical-profile CF conversion.
Non-vertical fixed stares and scanning observations require a separate spatial representation and will be treated under the `other_scan_types` workflow.
---
## Human-readable provenance
CF recommends global descriptive attributes such as `title`, `institution`, `source`, `history`, `references`, and `comment` for documenting the origin and processing of a dataset. fileciteturn29file4
The production workflow retains concise provenance describing:
- the observational source;
- processing level;
- conversion method;
- time coverage;
- instrument identity;
- aggregation method;
- CF conversion step.
Large internal parser or aggregation structures are not intended to dominate the final CF-oriented metadata. Detailed source-aligned provenance remains available at earlier processing stages and in production logs.
---
## Validation status
The current CF-oriented products have been iteratively checked with the available CF checker operating against CF 1.8.
These checks have already influenced practical encoding choices, particularly the use of fixed-length character arrays for station labels.
Because the files target CF 1.13, an error or warning concerning the declared CF convention version is expected when they are checked with a validator limited to CF 1.8.
The current checker results should therefore be interpreted as compatibility diagnostics, not as a complete validation of CF-1.13 compliance.
A dedicated post-conversion validation phase will later include:
- systematic CF checks across the converted instruments;
- checks for accidental instrument-specific metadata leakage;
- verification of coordinates, station information, and standard names;
- review against validation tooling that supports the target CF version.
# Structural Variants in Vertical-Scan Doppler Lidar Data
## Purpose
This document describes how structural differences within the LAFI vertical-scan Doppler lidar datasets are identified and handled during production conversion.
A structural variant is a source file or group of files whose native measurement layout differs from the dominant configuration of the corresponding instrument. Examples include changes in:
- number of range gates;
- range-gate length;
- scan label;
- observed maximum gate index;
- pointing geometry;
- completeness of ray or gate records.
These differences are treated primarily as characteristics of the source data rather than as reasons to discard observations.
---
## Preservation principle
The Level-1 conversion is preservation-oriented.
The production workflow therefore aims to:
- retain all parseable source files;
- retain all parseable rays;
- retain all parseable range-gate records with non-negative gate indices;
- preserve native azimuth and elevation information;
- represent unavailable observations as missing values;
- record structural anomalies rather than silently correcting or filtering them.
Scientific quality control is intentionally separated from structural conversion.
A file that is unusual is therefore not automatically considered invalid.
---
## Why structural variants matter
HPL files nominally belonging to the same instrument or scan family do not necessarily use the same native range grid.
For example, files may differ in:
- the number of declared range gates;
- the physical length of each gate;
- the maximum gate index actually present in the file;
- specialised or overlapping acquisition configurations.
Combining observations with incompatible native range grids into one rectangular daily array would require one or more of the following:
- interpolation;
- truncation;
- padding onto an artificial common grid;
- modification of the original sampling geometry.
These operations are deliberately avoided at the current processing level.
The daily aggregation therefore combines only observations whose native range structures are compatible.
As a result, a single UTC date can legitimately produce more than one daily Level-1 file and, subsequently, more than one CF-oriented daily product.
This behaviour is intentional.
---
## Range definition
The source HPL format describes the range-gate centre using the relationship:
```text
range = (zero-based gate index + 0.5) × range-gate length
```
The `range` coordinate in the Level-1 NetCDF therefore remains an instrument beam coordinate.
Files with different range-gate lengths or different output gate counts consequently have different native range coordinates and are not combined during daily aggregation.
---
## Header-declared and observed gate counts
A source file can contain range-gate records beyond the number of gates declared in the HPL header.
The production converter does not automatically discard these observations.
Instead, the conversion determines the maximum parseable non-negative gate index present in the source data and allocates the output range dimension so that these records can be retained.
For example, a file may declare 100 gates in its header but contain later rays extending to gate index 134. In this case, the resulting Level-1 file can contain 135 output gates while preserving the original header value separately as provenance metadata.
The distinction between:
- source-declared gate count; and
- output gate count
is therefore intentional.
---
## Missing and incomplete records
Some HPL files contain incomplete gate blocks, malformed lines, null bytes, or other structural irregularities.
Where the data remain parseable, the converter retains the recoverable observations and records the structural issue through ancillary information such as:
-`ray_quality_flag`;
-`gate_data_present`;
-`valid_gate_count`;
- parser-status and provenance metadata.
Unavailable cells in the rectangular NetCDF representation are stored as missing values.
The current Level-1 conversion does not interpret these flags as scientific quality indicators. They describe the success and completeness of structural recovery from the source file.
---
## DL011 structural variants
The DL011 source dataset contains 1884 HPL files.
The dominant configuration is:
- 400 gates;
- 30 m range-gate length;
-`Stare`.
Additional configurations observed during the 2025 record include:
- 160 gates × 12 m;
- 160 gates × 30 m;
- 100 gates × 30 m;
- 135 gates × 30 m;
- 7920 gates × 120 m associated with a `Stare - overlapping` configuration.
Five source files were identified during strict parsing as structurally unusual. These included incomplete gate blocks, null bytes, and one case where later rays extended beyond the header-declared gate count.
All were retained in the production conversion.
The completed DL011 production run aggregated 80 UTC dates into 85 daily CF products. The larger number of output files reflects the preservation of incompatible native range structures on some dates.
---
## DL141 structural variants
The vertical-scan subset of DL141 consists of 2343 `Stare_141_*.hpl` files.
The observed configurations are:
| File count | Range gates | Gate length | Scan type |
|---:|---:|---:|---|
| 2303 | 400 | 30.0 m | `Stare` |
| 21 | 3990 | 30.0 m | `Stare - overlapping` |
| 16 | 100 | 30.0 m | `Stare` |
| 3 | 1668 | 18.0 m | `Stare - overlapping` |
All 2343 Stare-family files were checked for pointing direction and were found to be effectively vertical, with elevation values close to 90 degrees.
The different range structures are therefore treated as vertical-profile observations but remain structurally distinct during daily aggregation.
The `User5` files present elsewhere in the DL141 source directory are not part of this vertical-scan production subset and will be treated separately with other scan geometries.
---
## DL262 structural variants
The DL262 source dataset contains 2810 `Stare_262_*.hpl` files.
The structural breakdown is:
- 2809 files with 400 gates × 30 m;
- 1 file with 100 gates × 30 m.
A pointing-direction check showed that the same exceptional 100-gate file is also strongly non-vertical:
```text
Stare_262_20250630_12.hpl
azimuth = 256.01 degrees
elevation = -2.00 degrees
```
The remaining 2809 files are effectively vertical, with elevations approximately 90 degrees.
The exceptional file is still retained through source-aligned Level-1 and daily Level-1 processing. Its final CF conversion is deferred because the current vertical-profile CF treatment is not appropriate for its geometry.
It will be revisited together with other non-vertical stare and scanning observations.
---
## Aggregation policy
Daily aggregation is based on structural compatibility rather than date alone.
Compatible source-aligned Level-1 files are concatenated along time and sorted chronologically.
The aggregation does not:
- interpolate observations;
- average observations;
- remove duplicate timestamps;
- scientifically screen observations;
- force incompatible range grids into a common coordinate.
Where a date contains more than one compatible structural group, more than one daily file is produced.
---
## Relationship to future quality control
Structural preservation should not be confused with endorsement of every observation as scientifically valid.
The current workflow is intended to preserve the source information sufficiently well that later scientific quality-control procedures can make informed decisions.
Possible later quality-control steps may include:
- signal-to-noise screening;
- physical plausibility checks;
- cloud or aerosol filtering;
- treatment of instrumental artefacts;
- removal or classification of scientifically unusable observations.
These higher-level decisions are outside the scope of the current production format conversion.
