UK Met Office Rain Radar NIMROD Data Processor

This project provides tools for processing UK Met Office Rain Radar NIMROD image files. It allows extraction of raster data from NIMROD .dat format files and conversion to ESRI ASCII (.asc) format. It also allows the creation of timeseries data from the ASC files, formatted for Infoworks ICM.

Overview

The project consists of a main pipeline workflow that processes multiple modules in sequence:

main.py: Main pipeline orchestrator that calls on the modules as needed
batch_nimrod.py: Module for batch processing multiple NIMROD files with configurable bounding boxes
generate_timeseries.py: Module for extracting cropped rain data and creating rainfall timeseries
extract.py: Module for extracting the dat files from the .gz.tar files that are downloaded from source

Features

main.py

Orchestrates the entire workflow pipeline
Uncompress the packed .gz.tar files to DAT files
Processes DAT files to ASC format
Generates timeseries data for specified locations
Combines grouped CSV files into consolidated datasets formatted for Infoworks ICM

extract.py

Converts all .gz.tar files first to 288 (1 day) of .gz files
Converts all .gz files to .dat files ready for processing.

batch_nimrod.py

Process multiple NIMROD dat files
Automatically extract datetime from file data
Export clipped raster data to ASC format

generate_timeseries.py

Extract cropped rain data based on specified locations
Create rainfall timeseries CSVs for each location
Parse datetime from filename and create proper datetime index
Group locations by specified output groups
Create consolidated CSV files for each group

Requirements

This is a multi-threaded application and requires Python 3.14t (free-threaded) to run correctly and efficiently. Please ensure you are using the free-threaded build of Python 3.14.

It is recommended to use UV for environment and package handling. Link to uv install

Usage

Ensure all required packages are installed uv sync
Adjust the config.py file to match your needs.
Ensure your .gz.tar files are in the TAR_TOP_FOLDER (as per config location)
Ensure your zone csv files are in the ZONE_FOLDER (as per config location)
RunMain Pipeline uv run main.py Note that you will have to set your environment variable PYTHON_GIL=0 first
find the output in the COMBINED_FOLDER (as per config location)

The main pipeline will:

Uncompress the .gz.tar files ready for processing
Process DAT files to ASC format
Generate timeseries data for specified locations
Combine grouped locations into consolidated datasets

Configuration

The config.py file defines folder paths and file deletion options:

TAR_TOP_FOLDER = "./tar_files"
GZ_TOP_FOLDER = "./gz_files"
DAT_TOP_FOLDER = "./dat_files"
ASC_TOP_FOLDER = "./asc_files"
COMBINED_FOLDER = "./combined_files"
ZONE_FOLDER = "./zone_inputs"

Example of how the zone csv files should look:

1K Grid, easting, northing, zone_number
TM0816, 608500, 216500, 1
TF6842, 568500, 342500, 1

Acknowledgments

Thank you to the following projects for their inspiration and code:

3.4 KiB Raw Blame History