Jake/met_office_radar_data
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Now deleting existing combined csv files after confirmation at start.

Browse Source
This commit is contained in:
Jake Pullen
2025-12-15 10:13:11 +00:00
parent a43edb1148
commit 9aaf8a5e88
4 changed files with 8 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.MD
+1 -1
View File
@@ -15,7 +15,7 @@ The project consists of a main pipeline workflow that processes multiple modules
### main.py
- **Startup Safety Check**: Scans the `COMBINED_FOLDER` at startup and warns the user if existing files are found, offering a chance to abort to prevent accidental data mixing.
- **Startup Safety Check**: Scans the `COMBINED_FOLDER` at startup and warns the user if existing files are found, Deleting existing files if continue is accepted.
- **Batch Processing**: Processes input tar files in configurable batches to manage resource usage.
- **End-to-End Processing**: Extracts GZ files, processes DAT/ASC, and appends to CSV in a single thread per file.
- **Concurrency**: Uses multi-threading to process individual GZ files within a batch concurrently.