Loading docs/production_doppler_lidar_conversion/vertical_scan/dataset_overview.md 0 → 100644 +282 −0 Original line number Diff line number Diff line # Vertical Scan Dataset Overview This document summarises the Doppler lidar datasets currently included in the vertical-scan production conversion workflow. The purpose is to record the observed characteristics of each source dataset, including file counts, scan geometry, native range structures, and exceptional records that affect the conversion strategy. The information below reflects checks performed on the production source data before and during conversion. --- ## DL011 ### Source dataset Source directory: `/lafi/doppler_vertical_GOP/DL_011_xr_plus_vertical/Proc/2025` Source-file pattern: `Stare_011_*.hpl` Total source files: **1884** ### Instrument location - Latitude: **48.71076389° N** - Longitude: **9.19021944° E** - Instrument altitude: **394 m ASL** - Station: **GLAFO-Hohenheim** - Station ID: **GLAFO-UHOH-DL011** ### Dominant configuration The dominant acquisition configuration is: - 400 range gates - 30 m range-gate length - scan type `Stare` Several shorter periods use different gate counts or gate lengths. Observed structural variants include: - 160 gates × 12 m - 160 gates × 30 m - 100 gates × 30 m - 135 gates × 30 m - 7920 gates × 120 m for a `Stare - overlapping` configuration ### Known exceptional source files Five files were identified during strict parsing as structurally unusual: 1. `Stare_011_20250617_10.hpl` — NUL bytes 2. `Stare_011_20250623_08.hpl` — incomplete gate block 3. `Stare_011_20250625_05.hpl` — incomplete gate block 4. `Stare_011_20250626_08.hpl` — header declares 100 gates while later rays contain up to 135 gates 5. `Stare_011_20251028_10.hpl` — incomplete gate block These files were not removed from production conversion. The Level-1 conversion follows a preservation-oriented policy and retains all parseable non-negative gate records while recording structural issues. ### Production outcome The completed production run reported: - HPL files discovered: **1884** - HPL conversion failures: **0** - UTC dates aggregated: **80** - aggregation failures: **0** - daily CF files produced: **85** - CF conversion failures: **0** The number of daily CF files exceeds the number of UTC dates because incompatible native range structures are preserved as separate daily products. --- ## DL141 ### Source dataset Source directory: `/lafi/doppler_vertical_GOP/DL_141_vertical_until_17_june_2025/Proc/2025` The full directory contains multiple filename and scan families. Total HPL files observed: **4019** Filename families: - `Stare_141_*.hpl`: **2343** - `User5_141_*.hpl`: **1676** The current vertical-scan production workflow includes only: `Stare_141_*.hpl` The `User5` files are associated with other scan geometries and are therefore reserved for later treatment under the non-vertical/scanning workflow. ### Scan types within the Stare family The 2343 `Stare_141` files contain: - **2319** `Stare` - **24** `Stare - overlapping` ### Structural configurations The verified structural breakdown is: - **2303** files: 400 gates × 30.0 m, `Stare` - **21** files: 3990 gates × 30.0 m, `Stare - overlapping` - **16** files: 100 gates × 30.0 m, `Stare` - **3** files: 1668 gates × 18.0 m, `Stare - overlapping` These four groups sum to the full set of **2343** Stare-family files. ### Pointing geometry All 2343 Stare-family files were checked using the first ray direction in each file. Observed azimuth/elevation combinations were: - 1526 files: 90.00° / 90.01° - 338 files: 360.00° / 90.00° - 310 files: 90.00° / 90.02° - 81 files: 360.00° / 90.01° - 49 files: 359.99° / 90.00° - 17 files: 90.00° / 90.00° - 10 files: 90.01° / 90.00° - 7 files: 359.99° / 90.01° - 5 files: 0.00° / 90.00° All measured elevations are effectively vertical. The variation in azimuth is not significant for a beam at approximately 90° elevation. ### Instrument location - Latitude: **48.716226° N** - Longitude: **9.189143° E** - Instrument altitude: **406.0 m ASL** - Station: **GLAFO-Hohenheim** - Station ID: **GLAFO-UHOH-DL141** ### Conversion decision All **2343 `Stare_141_*.hpl` files** are treated as vertical-scan observations. All structural variants are retained. Files with incompatible native range structures are kept separate during daily aggregation rather than being interpolated or forced onto a common range grid. The `User5` files are not part of the current vertical conversion and will be handled separately when the other scan geometries are addressed. --- ## DL262 ### Source dataset Source directory: `/lafi/doppler_vertical_GOP/DL_metek_vertical_DIAL_from_30_Jun_1244_to_28_Oct_2025_0700_UTC_2025/Proc/2025` Source-file pattern: `Stare_262_*.hpl` Total source files: **2810** All source files declare: `Scan type: Stare` ### Structural configurations The dataset contains: - **2809** files: 400 gates × 30.0 m - **1** file: 100 gates × 30.0 m ### Pointing geometry A full pointing-direction check identified: - **2472** files: azimuth 90.00°, elevation 90.02° - **326** files: azimuth 90.00°, elevation 90.03° - **11** files: azimuth 90.00°, elevation 90.00° - **1** file: azimuth 256.01°, elevation -2.00° Therefore: - **2809 files are effectively vertical** - **1 file is strongly non-vertical** The exceptional non-vertical file is: `Stare_262_20250630_12.hpl` This is also the only source file using the 100-gate configuration. ### Instrument location - Latitude: **48.716226° N** - Longitude: **9.189143° E** - Instrument altitude: **406.0 m ASL** - Station: **GLAFO-Hohenheim** - Station ID: **GLAFO-UHOH-DL262** ### Conversion decision All **2810 source files** are retained through the source-aligned Level-1 and daily Level-1 stages. The exceptional non-vertical observation is not discarded. However, the current vertical-profile CF representation is not applied to the daily product containing the strongly non-vertical observation. Its CF conversion is deferred until non-vertical stare and scanning geometries are handled explicitly. The remaining vertical observations are eligible for the vertical-profile CF conversion. ### Level-1 production outcome The completed Level-1 production run reported: - HPL files discovered: **2810** - HPL conversion failures: **0** - UTC dates aggregated: **121** - aggregation failures: **0** - daily Level-1 files: **121** The CF stage was initially deferred and subsequently run separately for the vertical products. --- ## Cross-dataset observations The three datasets demonstrate why the production workflow does not assume that all files from a given instrument share a single configuration. Important differences include: - changing range-gate counts; - changing range-gate lengths; - overlapping scan configurations; - different acquisition modes within the same source directory; - scan labels that do not by themselves guarantee vertical pointing; - isolated observations whose geometry differs substantially from the dominant dataset geometry. For this reason, the production workflow relies on source metadata and observed geometry rather than filename or scan label alone. The common preservation principle is that structural differences are retained and documented rather than removed or homogenised during Level-1 conversion. docs/production_doppler_lidar_conversion/vertical_scan/vertical_scan_README.md 0 → 100644 +154 −0 Original line number Diff line number Diff line # Vertical Scan Doppler Lidar Conversion This directory documents the production conversion of Doppler lidar observations that have been verified as vertical or near-vertical stare measurements. The current production datasets include: - **DL011** - **DL141** - **DL262** The purpose of this documentation is to capture the characteristics of the source data and the reasoning behind processing decisions that are specific to vertical-profile observations. The corresponding production scripts are maintained under: `../../../scripts/production_doppler_lidar_conversion/vertical_scan/` --- ## Scope The vertical-scan workflow is intended for Doppler lidar observations whose measurement beam is directed vertically or sufficiently close to vertical that the observations can be represented as time-height profiles above a fixed instrument location. The scan label contained in the source HPL files is not used by itself to determine whether a dataset belongs to this category. Actual pointing information, particularly elevation, is inspected before the vertical-profile CF treatment is applied. This distinction is important because a source file labelled `Stare` can still represent a non-vertical fixed beam. --- ## Processing approach Vertical-scan observations follow the common production sequence: ```text Original HPL files | v Source-aligned Level-1 NetCDF | v Daily Level-1 NetCDF | v CF-oriented daily NetCDF ``` The conversion is preservation-oriented. In particular: - all parseable observations are retained at Level-1; - scientific quality screening is not applied during format conversion; - azimuth and elevation are retained even for vertical observations; - different native range-gate structures are not interpolated onto a common grid; - structurally incompatible observations can therefore produce separate daily files for the same UTC date; - observations that are not compatible with the vertical-profile CF representation remain preserved at Level-1 and can be deferred for later treatment. --- ## Data documentation Detailed information about the source datasets is maintained in: `dataset_overview.md` This includes: - source-file inventories; - scan-type distributions; - pointing-angle checks; - range-gate configurations; - structural variants; - exceptional observations; - instrument location and altitude; - dataset-specific conversion decisions. The dataset overview is organised by instrument so that differences between DL011, DL141, and DL262 remain explicit without requiring separate instrument-specific documentation files. --- ## Structural variants The number of range gates and the range-gate length are not assumed to remain constant throughout an instrument record. Where incompatible native range structures occur, they are retained separately during daily aggregation. This means that: - a UTC date may result in more than one daily Level-1 file; - the number of daily CF products may exceed the number of UTC dates; - this behaviour is intentional and does not by itself indicate an aggregation failure. Further details will be documented in: `structural_variants.md` --- ## CF metadata strategy The final CF-oriented products are intended to describe vertical time-height profiles at a fixed instrument location. Instrument-specific metadata such as latitude, longitude, altitude, station ID, and instrument ID must be supplied explicitly for each dataset. The original pointing geometry remains part of the converted observations, and the vertical spatial coordinate is derived using the instrument altitude together with the measured range and elevation. Strongly non-vertical fixed beams and scanning geometries require a different spatial representation and are therefore handled separately under: `../other_scan_types/` Further metadata decisions will be documented in: `cf_metadata_strategy.md` --- ## Current implementation The same reusable lower-level conversion code is used across the vertical-scan datasets. Dataset-specific production wrappers provide: - source and output paths; - instrument-specific metadata; - production logging; - dataset-specific inclusion or deferral rules. The lower-level scripts will be renamed and reorganised after the currently running production conversions are complete so that their filenames reflect their shared, instrument-independent role. Loading
docs/production_doppler_lidar_conversion/vertical_scan/dataset_overview.md 0 → 100644 +282 −0 Original line number Diff line number Diff line # Vertical Scan Dataset Overview This document summarises the Doppler lidar datasets currently included in the vertical-scan production conversion workflow. The purpose is to record the observed characteristics of each source dataset, including file counts, scan geometry, native range structures, and exceptional records that affect the conversion strategy. The information below reflects checks performed on the production source data before and during conversion. --- ## DL011 ### Source dataset Source directory: `/lafi/doppler_vertical_GOP/DL_011_xr_plus_vertical/Proc/2025` Source-file pattern: `Stare_011_*.hpl` Total source files: **1884** ### Instrument location - Latitude: **48.71076389° N** - Longitude: **9.19021944° E** - Instrument altitude: **394 m ASL** - Station: **GLAFO-Hohenheim** - Station ID: **GLAFO-UHOH-DL011** ### Dominant configuration The dominant acquisition configuration is: - 400 range gates - 30 m range-gate length - scan type `Stare` Several shorter periods use different gate counts or gate lengths. Observed structural variants include: - 160 gates × 12 m - 160 gates × 30 m - 100 gates × 30 m - 135 gates × 30 m - 7920 gates × 120 m for a `Stare - overlapping` configuration ### Known exceptional source files Five files were identified during strict parsing as structurally unusual: 1. `Stare_011_20250617_10.hpl` — NUL bytes 2. `Stare_011_20250623_08.hpl` — incomplete gate block 3. `Stare_011_20250625_05.hpl` — incomplete gate block 4. `Stare_011_20250626_08.hpl` — header declares 100 gates while later rays contain up to 135 gates 5. `Stare_011_20251028_10.hpl` — incomplete gate block These files were not removed from production conversion. The Level-1 conversion follows a preservation-oriented policy and retains all parseable non-negative gate records while recording structural issues. ### Production outcome The completed production run reported: - HPL files discovered: **1884** - HPL conversion failures: **0** - UTC dates aggregated: **80** - aggregation failures: **0** - daily CF files produced: **85** - CF conversion failures: **0** The number of daily CF files exceeds the number of UTC dates because incompatible native range structures are preserved as separate daily products. --- ## DL141 ### Source dataset Source directory: `/lafi/doppler_vertical_GOP/DL_141_vertical_until_17_june_2025/Proc/2025` The full directory contains multiple filename and scan families. Total HPL files observed: **4019** Filename families: - `Stare_141_*.hpl`: **2343** - `User5_141_*.hpl`: **1676** The current vertical-scan production workflow includes only: `Stare_141_*.hpl` The `User5` files are associated with other scan geometries and are therefore reserved for later treatment under the non-vertical/scanning workflow. ### Scan types within the Stare family The 2343 `Stare_141` files contain: - **2319** `Stare` - **24** `Stare - overlapping` ### Structural configurations The verified structural breakdown is: - **2303** files: 400 gates × 30.0 m, `Stare` - **21** files: 3990 gates × 30.0 m, `Stare - overlapping` - **16** files: 100 gates × 30.0 m, `Stare` - **3** files: 1668 gates × 18.0 m, `Stare - overlapping` These four groups sum to the full set of **2343** Stare-family files. ### Pointing geometry All 2343 Stare-family files were checked using the first ray direction in each file. Observed azimuth/elevation combinations were: - 1526 files: 90.00° / 90.01° - 338 files: 360.00° / 90.00° - 310 files: 90.00° / 90.02° - 81 files: 360.00° / 90.01° - 49 files: 359.99° / 90.00° - 17 files: 90.00° / 90.00° - 10 files: 90.01° / 90.00° - 7 files: 359.99° / 90.01° - 5 files: 0.00° / 90.00° All measured elevations are effectively vertical. The variation in azimuth is not significant for a beam at approximately 90° elevation. ### Instrument location - Latitude: **48.716226° N** - Longitude: **9.189143° E** - Instrument altitude: **406.0 m ASL** - Station: **GLAFO-Hohenheim** - Station ID: **GLAFO-UHOH-DL141** ### Conversion decision All **2343 `Stare_141_*.hpl` files** are treated as vertical-scan observations. All structural variants are retained. Files with incompatible native range structures are kept separate during daily aggregation rather than being interpolated or forced onto a common range grid. The `User5` files are not part of the current vertical conversion and will be handled separately when the other scan geometries are addressed. --- ## DL262 ### Source dataset Source directory: `/lafi/doppler_vertical_GOP/DL_metek_vertical_DIAL_from_30_Jun_1244_to_28_Oct_2025_0700_UTC_2025/Proc/2025` Source-file pattern: `Stare_262_*.hpl` Total source files: **2810** All source files declare: `Scan type: Stare` ### Structural configurations The dataset contains: - **2809** files: 400 gates × 30.0 m - **1** file: 100 gates × 30.0 m ### Pointing geometry A full pointing-direction check identified: - **2472** files: azimuth 90.00°, elevation 90.02° - **326** files: azimuth 90.00°, elevation 90.03° - **11** files: azimuth 90.00°, elevation 90.00° - **1** file: azimuth 256.01°, elevation -2.00° Therefore: - **2809 files are effectively vertical** - **1 file is strongly non-vertical** The exceptional non-vertical file is: `Stare_262_20250630_12.hpl` This is also the only source file using the 100-gate configuration. ### Instrument location - Latitude: **48.716226° N** - Longitude: **9.189143° E** - Instrument altitude: **406.0 m ASL** - Station: **GLAFO-Hohenheim** - Station ID: **GLAFO-UHOH-DL262** ### Conversion decision All **2810 source files** are retained through the source-aligned Level-1 and daily Level-1 stages. The exceptional non-vertical observation is not discarded. However, the current vertical-profile CF representation is not applied to the daily product containing the strongly non-vertical observation. Its CF conversion is deferred until non-vertical stare and scanning geometries are handled explicitly. The remaining vertical observations are eligible for the vertical-profile CF conversion. ### Level-1 production outcome The completed Level-1 production run reported: - HPL files discovered: **2810** - HPL conversion failures: **0** - UTC dates aggregated: **121** - aggregation failures: **0** - daily Level-1 files: **121** The CF stage was initially deferred and subsequently run separately for the vertical products. --- ## Cross-dataset observations The three datasets demonstrate why the production workflow does not assume that all files from a given instrument share a single configuration. Important differences include: - changing range-gate counts; - changing range-gate lengths; - overlapping scan configurations; - different acquisition modes within the same source directory; - scan labels that do not by themselves guarantee vertical pointing; - isolated observations whose geometry differs substantially from the dominant dataset geometry. For this reason, the production workflow relies on source metadata and observed geometry rather than filename or scan label alone. The common preservation principle is that structural differences are retained and documented rather than removed or homogenised during Level-1 conversion.
docs/production_doppler_lidar_conversion/vertical_scan/vertical_scan_README.md 0 → 100644 +154 −0 Original line number Diff line number Diff line # Vertical Scan Doppler Lidar Conversion This directory documents the production conversion of Doppler lidar observations that have been verified as vertical or near-vertical stare measurements. The current production datasets include: - **DL011** - **DL141** - **DL262** The purpose of this documentation is to capture the characteristics of the source data and the reasoning behind processing decisions that are specific to vertical-profile observations. The corresponding production scripts are maintained under: `../../../scripts/production_doppler_lidar_conversion/vertical_scan/` --- ## Scope The vertical-scan workflow is intended for Doppler lidar observations whose measurement beam is directed vertically or sufficiently close to vertical that the observations can be represented as time-height profiles above a fixed instrument location. The scan label contained in the source HPL files is not used by itself to determine whether a dataset belongs to this category. Actual pointing information, particularly elevation, is inspected before the vertical-profile CF treatment is applied. This distinction is important because a source file labelled `Stare` can still represent a non-vertical fixed beam. --- ## Processing approach Vertical-scan observations follow the common production sequence: ```text Original HPL files | v Source-aligned Level-1 NetCDF | v Daily Level-1 NetCDF | v CF-oriented daily NetCDF ``` The conversion is preservation-oriented. In particular: - all parseable observations are retained at Level-1; - scientific quality screening is not applied during format conversion; - azimuth and elevation are retained even for vertical observations; - different native range-gate structures are not interpolated onto a common grid; - structurally incompatible observations can therefore produce separate daily files for the same UTC date; - observations that are not compatible with the vertical-profile CF representation remain preserved at Level-1 and can be deferred for later treatment. --- ## Data documentation Detailed information about the source datasets is maintained in: `dataset_overview.md` This includes: - source-file inventories; - scan-type distributions; - pointing-angle checks; - range-gate configurations; - structural variants; - exceptional observations; - instrument location and altitude; - dataset-specific conversion decisions. The dataset overview is organised by instrument so that differences between DL011, DL141, and DL262 remain explicit without requiring separate instrument-specific documentation files. --- ## Structural variants The number of range gates and the range-gate length are not assumed to remain constant throughout an instrument record. Where incompatible native range structures occur, they are retained separately during daily aggregation. This means that: - a UTC date may result in more than one daily Level-1 file; - the number of daily CF products may exceed the number of UTC dates; - this behaviour is intentional and does not by itself indicate an aggregation failure. Further details will be documented in: `structural_variants.md` --- ## CF metadata strategy The final CF-oriented products are intended to describe vertical time-height profiles at a fixed instrument location. Instrument-specific metadata such as latitude, longitude, altitude, station ID, and instrument ID must be supplied explicitly for each dataset. The original pointing geometry remains part of the converted observations, and the vertical spatial coordinate is derived using the instrument altitude together with the measured range and elevation. Strongly non-vertical fixed beams and scanning geometries require a different spatial representation and are therefore handled separately under: `../other_scan_types/` Further metadata decisions will be documented in: `cf_metadata_strategy.md` --- ## Current implementation The same reusable lower-level conversion code is used across the vertical-scan datasets. Dataset-specific production wrappers provide: - source and output paths; - instrument-specific metadata; - production logging; - dataset-specific inclusion or deferral rules. The lower-level scripts will be renamed and reorganised after the currently running production conversions are complete so that their filenames reflect their shared, instrument-independent role.
