@@ -77,57 +77,67 @@ Example: java -Xmx20G -jar pathTo/GSync_X.X.jar -r -u -k -b hcibioinfo_gsync_rep
77
77
78
78
79
79
80
- u0028003$ java -jar -Xmx1G ~/Code/AwsApps/target/JobRunner_0.2 .jar
80
+ u0028003$ java -jar -Xmx1G ~/Code/AwsApps/target/JobRunner_0.3 .jar
81
81
82
- ****************************************************************************************************************************
83
- ** AWS Job Runner : December 2021 **
84
- ****************************************************************************************************************************
85
- JR is an app for running bash scripts on AWS EC2 nodes. It downloads and uncompressed your resource bundle and looks for
86
- xxx.sh_JR_START files in your S3 Jobs directories. For each, it copies over the directory contents, executes the
87
- associated xxx.sh script, and transfers back the results. This is repeated until no unrun jobs are found. Launch many
88
- EC2 JR nodes, each running an instance of the JR, to process hundreds of jobs in parallel. Use spot requests and
89
- hibernation to reduce costs.
82
+ **************************************************************************************
83
+ ** AWS Job Runner : January 2021 **
84
+ **************************************************************************************
85
+ JR is an app for running bash scripts on AWS EC2 nodes. It downloads and uncompressed
86
+ your resource bundle and looks for xxx.sh_JR_START files in your S3 Jobs directories.
87
+ For each, it copies over the directory contents, executes the associated xxx.sh
88
+ script, and transfers back the results. This is repeated until no unrun jobs are
89
+ found. Launch many EC2 JR nodes, each running an instance of the JR, to process
90
+ hundreds of jobs in parallel. Use spot requests and hibernation to reduce costs.
91
+ Upon termination, JR will cancel the spot request and kill the instance.
90
92
91
93
To use:
92
- 1) Install and configure the aws cli on your local workstation, see https://aws.amazon.com/cli/
93
- 2) Upload your aws credentials file into a private bucket on aws, e.g.
94
- aws s3 cp ~/.aws/credentials s3://my-jr/aws.cred.txt
94
+ 1) Install and configure the aws cli on your local workstation, see
95
+ https://aws.amazon.com/cli/
96
+ 2) Upload a [default] aws credential file containing a single set of region,
97
+ aws_access_key_id, and aws_secret_access_key info into a private bucket, e.g.
98
+ aws s3 cp ~/.aws/credentials s3://my-jr/aws.cred.txt
95
99
3) Generate a secure 24hr timed URL for the credentials file, e.g.
96
100
aws --region us-west-2 s3 presign s3://my-jr/aws.cred.txt --expires-in 259200
97
101
4) Upload a zip archive containing resources needed to run your jobs into S3, e.g.
98
102
aws s3 cp ~/TNRunnerResourceBundle.zip s3://my-jr/TNRunnerResourceBundle.zip
99
103
This will be copied into the /JRDir/ directory and then unzipped.
100
104
5) Upload script and job files into a 'Jobs' directory on S3, e.g.
101
105
aws s3 cp ~/JRJobs/A/ s3://my-jr/Jobs/A/ --recursive
102
- 6) Optional, upload bash script files ending with JR_INIT.sh and or JR_TERM.sh. These are executed by JR before and after
103
- running the main bash script. Use these to copy in sample specific resources, e.g. fastq/ cram/ bam files, and to run
106
+ 6) Optional, upload bash script files ending with JR_INIT.sh and or JR_TERM.sh. These
107
+ are executed by JR before and after running the main bash script. Use these to
108
+ copy in sample specific resources, e.g. fastq/ cram/ bam files, and to run
104
109
post job clean up.
105
- 7) Upload a file named XXX_JR_START to let the JobRunner know the bash script named XXX is ready to run, e.g.
110
+ 7) Upload a file named XXX_JR_START to let the JobRunner know the bash script named
111
+ XXX is ready to run, e.g.
106
112
aws s3 cp s3://my-jr/emptyFile s3://my-jr/Jobs/A/dnaAlignQC.sh_JR_START
107
- 8) Launch the JobRunner.jar on one or more JR configured EC2 nodes. See https://ri-confluence.hci.utah.edu/x/gYCgBw
113
+ 8) Launch the JobRunner.jar on one or more JR configured EC2 nodes. See
114
+ https://ri-confluence.hci.utah.edu/x/gYCgBw
108
115
109
- Job Runner Options:
116
+ Job Runner Required Options:
110
117
-c URL to your secure timed config credentials file.
111
118
-r S3URI to your zipped resource bundle.
112
119
-j S3URI to your root Jobs directory containing folders with job scripts to execute.
113
120
-l S3URI to your Log folder for node logs.
114
121
115
122
Default Options:
116
- -d Directory on the local worker node, full path, in which resources and job files will be processed, defaults to /JRDir/
123
+ -d Directory on the local worker node, full path, in which resources and job files
124
+ will be processed, defaults to /JRDir/
117
125
-a Aws credentials directory, defaults to ~/.aws/
118
- -t Terminate the EC2 node upon job completion. Defaults to looking for jobs for the min2Wait.
119
- -w Minutes to wait when jobs are not found before termination, defaults to 10.
120
- -x Replace S3 job directories with processed analysis, defaults to syncing local with S3. WARNING, if selected, don't place
121
- any files in these S3 jobs directories that cannot be replaced. JR will delete them.
126
+ -t Terminate the EC2 node upon program exit, defaults to leaving it running.
127
+ -w Minutes to wait looking for jobs before exiting, defaults to 10.
128
+ -x Replace S3 job directories with processed analysis, defaults to syncing local with
129
+ S3. WARNING, if selected, don't place any files in these S3 jobs directories that
130
+ cannot be replaced. JR will delete them.
122
131
-v Verbose debugging output.
123
132
124
- Example: java -jar -Xmx1G JobRunner.jar
133
+ Example: java -jar -Xmx1G JobRunner.jar -x -t
125
134
-r s3://my-jr/TNRunnerResourceBundle.zip
126
135
-j s3://my-jr/Jobs/
127
136
-l s3://my-jr/NodeLogs/
128
- -c 'https://my-jr.s3.us-west-2.amazonaws.com/aws.cred.txt?X-Amz-Algorithm=AWS4-HMXXX...'
137
+ -c 'https://my-jr.s3.us-west-2.amazonaws.com/aws.cred.txt?X-AmRun...'
138
+
139
+ **************************************************************************************
129
140
130
- ****************************************************************************************************************************
131
141
132
142
133
143
0 commit comments