-
Notifications
You must be signed in to change notification settings - Fork 2
realtime
These instructions are for setting up realtime LPT from scratch on a Linux system with an Anaconda Python environment.
First, download Anaconda Python for Linux using the download link at https://www.anaconda.com/. The current version is for Python 3.7. Subsequent Python versions should work as long as they maintain backwards compatibility. The current direct download link is here.
This is usually done by running the installer script (e.g., Anaconda3-2020.02-Linux-x86_64.sh). The script will ask for a directory where to install Python environments. Generally your home directory on your system will do. The Python environment on my system takes up 1.7 GB of space.
To set up a Python environment with the modules needed to run LPT, you can issue the following command:
conda create --name lpt numpy pandas matplotlib basemap netCDF4 h5py scipy wrf-python gdal
The command may first ask you to update Anaconda (optional). Answer yes when prompted with the list of modules to be installed, and it will install the environment. To activate the environment before running LPT, use source activate lpt.
The code is on GitHub here.
Navigate to a directory where you want to run LPT, and get the code using the following:
git clone https://github.com/brandonwkerns/lpt-python-public.git
This should download the code and put it under the directory lpt-python-public/.
NOTE: If you want to use a different directory name, you can call the git clone command like this:
git clone https://github.com/brandonwkerns/lpt-python-public.git custom_directory_name
When/if code updates are needed, doing a git pull anywhere under the directory should update all of the relevant files.
The code repository has "MASTER" files which are to be copied and edited. The idea is that the MASTER files may be updated on the GitHub repository. If you directly edit the "MASTER" files, it may cause issues when git pull is used.
Get into the realtime directory using cd lpt-public/realtime (or whatever custom directory you specified). Then, copy the following "MASTER" files. Also, make sure they are executable:
cp lpt.cron.MASTER lpt.cron
cp download_cfs_rt.MASTER.sh download_cfs_rt.sh
cp download_cmorph_rt.MASTER.sh download_cmorph_rt.sh
cp lpt_crontab_wrapper.MASTER.sh lpt_crontab_wrapper.sh
cp lpt_run_cmorph.MASTER.py lpt_run_cmorph.py
cp lpt_run_cfs_fcst.MASTER.py lpt_run_cfs_fcst.py
chmod +x download_cfs_rt.sh download_cmorph_rt.sh lpt_crontab_wrapper.sh
Edit the directories in the following lines:
In lpt.cron, for this script directory.
LPT_REALTIME_DIR=/path/to/this/realtime/script/directory
In download_cfs_rt.sh, for the directory you want to download the raw realtime data to.
download_parent_dir=/path/to/keep/your/data
In download_cmorph_rt.sh, for the directory you want to download the raw realtime data to.
workdir=/path/to/keep/your/data
In lpt_crontab_wrapper.sh, for this script directory and the path to your anaconda3 directory. You can get the anaconda3 directory using which anaconda and it's the path with taking out the bin/anaconda at the end.
WORKDIR=/path/to/this/realtime/script/directory/
ANACONDA_DIR=/path/to/anaconda3
In lpt_run_cmorph.py, for the path to CMORPH data (same as specified in download_cmorph_rt.sh).
dataset['raw_data_parent_dir'] = '/path/to/cmorph/rt/'
In lpt_run_cfs_fcst.py, for the path to CFS forecast data (same as specified in download_cfs_rt.sh).
dataset['raw_data_parent_dir'] = '/path/to/model_fcst_grib/cfs'
If you won't want the LPT scripts to make plots (runs faster), they can be turned off in the .py scripts using the line
# Plotting
plotting['do_plotting'] = True # True or False -- Should I make plots?
This step tests whether the directories are set up properly, and also downloads some raw data that can be used to run LPT.
To get today's latest CMORPH data: ./download_cmorph_rt.sh.
To get CMORPH data for a specific date: ./download_cmorph_rt.sh YYYYMMDD where YYYYMMDD are year, month, date like 20200401. To build up a long enough period to run LPT, get at least 45 days of data.
To get today's latest CFS data, and the past 7 days: ./download_cfs_rt.sh.
LPT can be run manually using the script lpt_crontab_wrapper.sh. For today's date, run it like ./lpt_crontab_wrapper.sh, for a specific date, run it like ./lpt_crontab_wrapper.sh YYYYMMDD.
To run the scripts automatically, either set the crontab using the lpt.cron file crontab lpt.cron, or add the line by line entries to your existing crontab commands. The crontab is set to run download_cfs_rt.sh at 09:20, download_cmorph_rt.sh every 3 hours at :10 past the hour, and LPT calculations lpt_crontab_wrapper.sh at 09:30. These times can be modified using the lines below. For example, to test run it, set the time to a few minutes later than the current time, and wait to see whether it runs properly.
#Min H D M DoW /path/to/scripts/script.sh
20 9 * * * $LPT_REALTIME_DIR/download_cfs_rt.sh >& $LPT_REALTIME_DIR/logs/download.cfs.rt.crontab.log
10 */3 * * * $LPT_REALTIME_DIR/download_cmorph_rt.sh >& $LPT_REALTIME_DIR/logs/download.cmorph.rt.crontab.log
30 9 * * * export $DISPLAY=:0 ; $LPT_REALTIME_DIR/lpt_crontab_wrapper.sh >& $LPT_REALTIME_DIR/logs/run.rt.crontab.log
Digital data are set to be saved under ./data/ and images under ./images/. Log files from scripts running on crontab will go under ./logs/.
The data and images output from LPT can be placed in custom directories by changing the lines with output['img_dir'] and output['data_dir'] in the .py scripts.