chore: 🔧 Minor tweaks for neatness
This commit is contained in:
@@ -5,6 +5,7 @@ This project provides tools for processing UK Met Office Rain Radar NIMROD image
|
|||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
The project consists of a main pipeline workflow that processes multiple modules in sequence:
|
The project consists of a main pipeline workflow that processes multiple modules in sequence:
|
||||||
|
|
||||||
- `main.py`: Main pipeline orchestrator that calls on the modules as needed
|
- `main.py`: Main pipeline orchestrator that calls on the modules as needed
|
||||||
- `batch_nimrod.py`: Module for batch processing multiple NIMROD files with configurable bounding boxes
|
- `batch_nimrod.py`: Module for batch processing multiple NIMROD files with configurable bounding boxes
|
||||||
- `generate_timeseries.py`: Module for extracting cropped rain data and creating rainfall timeseries
|
- `generate_timeseries.py`: Module for extracting cropped rain data and creating rainfall timeseries
|
||||||
@@ -13,22 +14,26 @@ The project consists of a main pipeline workflow that processes multiple modules
|
|||||||
## Features
|
## Features
|
||||||
|
|
||||||
### main.py
|
### main.py
|
||||||
|
|
||||||
- Orchestrates the entire workflow pipeline
|
- Orchestrates the entire workflow pipeline
|
||||||
- Processes DAT files to ASC format
|
- Processes DAT files to ASC format
|
||||||
- Generates timeseries data for specified locations
|
- Generates timeseries data for specified locations
|
||||||
- Combines grouped CSV files into consolidated datasets
|
- Combines grouped CSV files into consolidated datasets
|
||||||
|
|
||||||
### batch_nimrod.py
|
### batch_nimrod.py
|
||||||
|
|
||||||
- Process multiple NIMROD dat files
|
- Process multiple NIMROD dat files
|
||||||
- Automatically extract datetime from file data
|
- Automatically extract datetime from file data
|
||||||
- Export clipped raster data to ASC format
|
- Export clipped raster data to ASC format
|
||||||
|
|
||||||
### generate_timeseries.py
|
### generate_timeseries.py
|
||||||
|
|
||||||
- Extract cropped rain data based on specified locations
|
- Extract cropped rain data based on specified locations
|
||||||
- Create rainfall timeseries CSVs for each location
|
- Create rainfall timeseries CSVs for each location
|
||||||
- Parse datetime from filename and create proper datetime index
|
- Parse datetime from filename and create proper datetime index
|
||||||
|
|
||||||
### combine_timeseries.py
|
### combine_timeseries.py
|
||||||
|
|
||||||
- Combine multiple timeseries CSV files into grouped datasets
|
- Combine multiple timeseries CSV files into grouped datasets
|
||||||
- Group locations by specified output groups
|
- Group locations by specified output groups
|
||||||
- Create consolidated CSV files for each group
|
- Create consolidated CSV files for each group
|
||||||
@@ -49,29 +54,32 @@ It is recommended to use UV for environment and package handling.
|
|||||||
1. RunMain Pipeline `uv run main.py
|
1. RunMain Pipeline `uv run main.py
|
||||||
1. find the output in the COMBINED_FOLDER (as per config location)
|
1. find the output in the COMBINED_FOLDER (as per config location)
|
||||||
|
|
||||||
|
|
||||||
The main pipeline will:
|
The main pipeline will:
|
||||||
|
|
||||||
1. Process DAT files to ASC format if needed
|
1. Process DAT files to ASC format if needed
|
||||||
2. Generate timeseries data for specified locations
|
1. Generate timeseries data for specified locations
|
||||||
3. Combine grouped CSV files into consolidated datasets
|
1. Combine grouped CSV files into consolidated datasets
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
The `config.py` file defines folder paths:
|
The `config.py` file defines folder paths:
|
||||||
|
|
||||||
- DAT_TOP_FOLDER: "./dat_files"
|
- DAT_TOP_FOLDER: "./dat_files"
|
||||||
- ASC_TOP_FOLDER: "./asc_files"
|
- ASC_TOP_FOLDER: "./asc_files"
|
||||||
- CSV_TOP_FOLDER: "./csv_files"
|
- CSV_TOP_FOLDER: "./csv_files"
|
||||||
- COMBINED_FOLDER: "./combined_files"
|
- COMBINED_FOLDER: "./combined_files"
|
||||||
|
|
||||||
Example of how the zone csv files should look:
|
Example of how the zone csv files should look:
|
||||||
```
|
|
||||||
filler, zone_name, easting, northing, other_filler, last_filler, zone_number
|
```csv
|
||||||
aa, TM0816, 608500, 216500, a, a, 1
|
1K Grid, easting, northing, zone_number
|
||||||
aa, TF6842, 568500, 342500, a, a, 1
|
TM0816, 608500, 216500, 1
|
||||||
|
TF6842, 568500, 342500, 1
|
||||||
```
|
```
|
||||||
|
|
||||||
## Acknowledgments
|
## Acknowledgments
|
||||||
|
|
||||||
Thank you to the following projects for their inspiration and code:
|
Thank you to the following projects for their inspiration and code:
|
||||||
* [Richard Thomas - Original Nimrod dat to asc file conversion](https://github.com/richard-thomas/MetOffice_NIMROD)
|
|
||||||
* [Declan Valters - building the timeseries from the asc files](https://github.com/dvalters/NIMROD-toolbox)
|
- [Richard Thomas - Original Nimrod dat to asc file conversion](https://github.com/richard-thomas/MetOffice_NIMROD)
|
||||||
|
- [Declan Valters - building the timeseries from the asc files](https://github.com/dvalters/NIMROD-toolbox)
|
||||||
|
|||||||
@@ -24,11 +24,11 @@ if __name__ == "__main__":
|
|||||||
reader = csv.reader(csvfile)
|
reader = csv.reader(csvfile)
|
||||||
header = next(reader) # Skip header row
|
header = next(reader) # Skip header row
|
||||||
for row in reader:
|
for row in reader:
|
||||||
# Extract the relevant fields: Ossheet (location ID), Easting, Northing, Zone
|
# Extract the relevant fields: 1K Grid, Easting, Northing, Zone
|
||||||
zone_id = row[1] # Ossheet column
|
zone_id = row[0] # Ossheet column
|
||||||
easting = int(row[2]) # Easting column
|
easting = int(row[1]) # Easting column
|
||||||
northing = int(row[3]) # Northing column
|
northing = int(row[2]) # Northing column
|
||||||
zone = int(row[6]) # ZoneID column
|
zone = int(row[3]) # ZoneID column
|
||||||
locations.append([zone_id, easting, northing, zone])
|
locations.append([zone_id, easting, northing, zone])
|
||||||
|
|
||||||
batch = BatchNimrod(Config)
|
batch = BatchNimrod(Config)
|
||||||
|
|||||||
Reference in New Issue
Block a user