Skip to content

Commit 6650a48

Browse files
Update README.md
1 parent 97f57c2 commit 6650a48

File tree

1 file changed

+35
-25
lines changed

1 file changed

+35
-25
lines changed

README.md

Lines changed: 35 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -77,57 +77,67 @@ Example: java -Xmx20G -jar pathTo/GSync_X.X.jar -r -u -k -b hcibioinfo_gsync_rep
7777

7878

7979

80-
u0028003$ java -jar -Xmx1G ~/Code/AwsApps/target/JobRunner_0.2.jar
80+
u0028003$ java -jar -Xmx1G ~/Code/AwsApps/target/JobRunner_0.3.jar
8181

82-
****************************************************************************************************************************
83-
** AWS Job Runner : December 2021 **
84-
****************************************************************************************************************************
85-
JR is an app for running bash scripts on AWS EC2 nodes. It downloads and uncompressed your resource bundle and looks for
86-
xxx.sh_JR_START files in your S3 Jobs directories. For each, it copies over the directory contents, executes the
87-
associated xxx.sh script, and transfers back the results. This is repeated until no unrun jobs are found. Launch many
88-
EC2 JR nodes, each running an instance of the JR, to process hundreds of jobs in parallel. Use spot requests and
89-
hibernation to reduce costs.
82+
**************************************************************************************
83+
** AWS Job Runner : January 2021 **
84+
**************************************************************************************
85+
JR is an app for running bash scripts on AWS EC2 nodes. It downloads and uncompressed
86+
your resource bundle and looks for xxx.sh_JR_START files in your S3 Jobs directories.
87+
For each, it copies over the directory contents, executes the associated xxx.sh
88+
script, and transfers back the results. This is repeated until no unrun jobs are
89+
found. Launch many EC2 JR nodes, each running an instance of the JR, to process
90+
hundreds of jobs in parallel. Use spot requests and hibernation to reduce costs.
91+
Upon termination, JR will cancel the spot request and kill the instance.
9092

9193
To use:
92-
1) Install and configure the aws cli on your local workstation, see https://aws.amazon.com/cli/
93-
2) Upload your aws credentials file into a private bucket on aws, e.g.
94-
aws s3 cp ~/.aws/credentials s3://my-jr/aws.cred.txt
94+
1) Install and configure the aws cli on your local workstation, see
95+
https://aws.amazon.com/cli/
96+
2) Upload a [default] aws credential file containing a single set of region,
97+
aws_access_key_id, and aws_secret_access_key info into a private bucket, e.g.
98+
aws s3 cp ~/.aws/credentials s3://my-jr/aws.cred.txt
9599
3) Generate a secure 24hr timed URL for the credentials file, e.g.
96100
aws --region us-west-2 s3 presign s3://my-jr/aws.cred.txt --expires-in 259200
97101
4) Upload a zip archive containing resources needed to run your jobs into S3, e.g.
98102
aws s3 cp ~/TNRunnerResourceBundle.zip s3://my-jr/TNRunnerResourceBundle.zip
99103
This will be copied into the /JRDir/ directory and then unzipped.
100104
5) Upload script and job files into a 'Jobs' directory on S3, e.g.
101105
aws s3 cp ~/JRJobs/A/ s3://my-jr/Jobs/A/ --recursive
102-
6) Optional, upload bash script files ending with JR_INIT.sh and or JR_TERM.sh. These are executed by JR before and after
103-
running the main bash script. Use these to copy in sample specific resources, e.g. fastq/ cram/ bam files, and to run
106+
6) Optional, upload bash script files ending with JR_INIT.sh and or JR_TERM.sh. These
107+
are executed by JR before and after running the main bash script. Use these to
108+
copy in sample specific resources, e.g. fastq/ cram/ bam files, and to run
104109
post job clean up.
105-
7) Upload a file named XXX_JR_START to let the JobRunner know the bash script named XXX is ready to run, e.g.
110+
7) Upload a file named XXX_JR_START to let the JobRunner know the bash script named
111+
XXX is ready to run, e.g.
106112
aws s3 cp s3://my-jr/emptyFile s3://my-jr/Jobs/A/dnaAlignQC.sh_JR_START
107-
8) Launch the JobRunner.jar on one or more JR configured EC2 nodes. See https://ri-confluence.hci.utah.edu/x/gYCgBw
113+
8) Launch the JobRunner.jar on one or more JR configured EC2 nodes. See
114+
https://ri-confluence.hci.utah.edu/x/gYCgBw
108115

109-
Job Runner Options:
116+
Job Runner Required Options:
110117
-c URL to your secure timed config credentials file.
111118
-r S3URI to your zipped resource bundle.
112119
-j S3URI to your root Jobs directory containing folders with job scripts to execute.
113120
-l S3URI to your Log folder for node logs.
114121

115122
Default Options:
116-
-d Directory on the local worker node, full path, in which resources and job files will be processed, defaults to /JRDir/
123+
-d Directory on the local worker node, full path, in which resources and job files
124+
will be processed, defaults to /JRDir/
117125
-a Aws credentials directory, defaults to ~/.aws/
118-
-t Terminate the EC2 node upon job completion. Defaults to looking for jobs for the min2Wait.
119-
-w Minutes to wait when jobs are not found before termination, defaults to 10.
120-
-x Replace S3 job directories with processed analysis, defaults to syncing local with S3. WARNING, if selected, don't place
121-
any files in these S3 jobs directories that cannot be replaced. JR will delete them.
126+
-t Terminate the EC2 node upon program exit, defaults to leaving it running.
127+
-w Minutes to wait looking for jobs before exiting, defaults to 10.
128+
-x Replace S3 job directories with processed analysis, defaults to syncing local with
129+
S3. WARNING, if selected, don't place any files in these S3 jobs directories that
130+
cannot be replaced. JR will delete them.
122131
-v Verbose debugging output.
123132

124-
Example: java -jar -Xmx1G JobRunner.jar
133+
Example: java -jar -Xmx1G JobRunner.jar -x -t
125134
-r s3://my-jr/TNRunnerResourceBundle.zip
126135
-j s3://my-jr/Jobs/
127136
-l s3://my-jr/NodeLogs/
128-
-c 'https://my-jr.s3.us-west-2.amazonaws.com/aws.cred.txt?X-Amz-Algorithm=AWS4-HMXXX...'
137+
-c 'https://my-jr.s3.us-west-2.amazonaws.com/aws.cred.txt?X-AmRun...'
138+
139+
**************************************************************************************
129140

130-
****************************************************************************************************************************
131141

132142

133143

0 commit comments

Comments
 (0)