rush v0.1.6

Changelog

• ignoring input only containing newline character ("\n")

• value of flag -D/--record-delimiter can be empty

  $ echo abc | rush -D "" -k 'echo {}'
  a
  b
  c
  

rush v0.1.5

Changelog

• fix continue (-c): immediately saving ignored commands when successful commands file is not empty (usually when running command with flag -c for 3+th times)

rush v0.1.4

Changelog

• fix: remove duplicated commands in file of successful commands (-C/--succ-cmd-file)

rush v0.1.3

Changelog

• fix bug of closing closed channel when using flag -e

rush v0.1.2

Changelog

• fix job id for multiple input file
• make sure message "stop on first error(s)" be printed
• better windows support
• better solution to avoid closing closed channel (cancel)
• doc: add section: special cases

rush v0.1.1

Changelog

• safer replacement string: return original replacement string if it is

  • unsupported

  • undefined variable by flag -v. It's a common mistake, leaving it be blank was wrong and some times dangerous. e.g.

      # can be dangerous
      rush -v d=$dir 'rm -rf {w}/bin'            
    
      # old versions produced wrong result: ".", it should be: "a"
      echo a.txt | rush 'v={}; echo ${v/.txt/}'  
    

Benchmark

Softwares:

• parallel: GNU parallel (Perl)
• rust-parallel: https://github.com/mmstick/parallel (Rust)
• gargs: https://github.com/brentp/gargs (Go)
• rush: https://github.com/shenwei356/rush (Go)

Result:

Note that rust-parallel always print "parallel: reading inputs from standard input" to STDOUT,
so the MD5 sum of results are different from others.

===============================[ a few stdout ]===============================
--------------------------[ keep order: False ]--------------------------

commands:
seq 1 10 | gargs -p 4  "echo job:{}; seq 1 10" > t.gargs
seq 1 10 | rush -j 4  "echo job:{}; seq 1 10" > t.rush
seq 1 10 | parallel -j 4  "echo job:{}; seq 1 10" > t.parallel
seq 1 10 | rust-parallel -j 4  "echo job:{}; seq 1 10" > t.rust-parallel

time:
gargs: 0.0312929153442
parallel: 0.186486005783
rush: 0.0261890888214
rust-parallel: 0.0669119358063

md5sum:
8a18e003c70f3a32c86078f01f7bee8b  t.gargs
6d82dfa19a9f0546bb376b2042139ded  t.parallel
9f747a9de51af62fdc85d363b403de11  t.rush
8099e6ad94165e8334ef54e3f1571204  t.rust-parallel

---------------------------[ keep order: True ]---------------------------

commands:
seq 1 10 | gargs -p 4 -o "echo job:{}; seq 1 10" > t.gargs
seq 1 10 | rush -j 4 -k "echo job:{}; seq 1 10" > t.rush
seq 1 10 | parallel -j 4 -k "echo job:{}; seq 1 10" > t.parallel
seq 1 10 | rust-parallel -j 4  "echo job:{}; seq 1 10" > t.rust-parallel

time:
gargs: 0.021341085434
parallel: 0.178062200546
rush: 0.0201990604401
rust-parallel: 0.0584881305695

md5sum:
6d82dfa19a9f0546bb376b2042139ded  t.gargs
6d82dfa19a9f0546bb376b2042139ded  t.parallel
6d82dfa19a9f0546bb376b2042139ded  t.rush
8099e6ad94165e8334ef54e3f1571204  t.rust-parallel


==============================[ lots of stdout ]==============================
--------------------------[ keep order: False ]--------------------------

commands:
seq 1 10 | gargs -p 4  "echo job:{}; seq 1 1000000" > t.gargs
seq 1 10 | rush -j 4  "echo job:{}; seq 1 1000000" > t.rush
seq 1 10 | parallel -j 4  "echo job:{}; seq 1 1000000" > t.parallel
seq 1 10 | rust-parallel -j 4  "echo job:{}; seq 1 1000000" > t.rust-parallel

time:
gargs: 0.209739923477
parallel: 0.302277088165
rush: 0.228335857391
rust-parallel: 0.242180109024

md5sum:
d682bac7cb5b5139706246df6ed9ab40  t.gargs
509086b10292759096d6db278bb2add8  t.parallel
add4a9f2252a368447a50275c55bc352  t.rush
627790a71dfb0553b82af71d2e5e4f0a  t.rust-parallel                                                        
                                                                                                         
---------------------------[ keep order: True ]---------------------------                               
                                                                                                         
commands:
seq 1 10 | gargs -p 4 -o "echo job:{}; seq 1 1000000" > t.gargs
seq 1 10 | rush -j 4 -k "echo job:{}; seq 1 1000000" > t.rush
seq 1 10 | parallel -j 4 -k "echo job:{}; seq 1 1000000" > t.parallel
seq 1 10 | rust-parallel -j 4  "echo job:{}; seq 1 1000000" > t.rust-parallel

time:
gargs: 0.230170965195
parallel: 0.315106868744
rush: 0.254786968231
rust-parallel: 0.246857881546

md5sum:
509086b10292759096d6db278bb2add8  t.gargs
509086b10292759096d6db278bb2add8  t.parallel
509086b10292759096d6db278bb2add8  t.rush
627790a71dfb0553b82af71d2e5e4f0a  t.rust-parallel


==========================[ a few stdout & run long ]==========================
--------------------------[ keep order: False ]--------------------------

commands:
seq 1 10 | gargs -p 4  "echo job:{}; sleep 2; seq 1 10" > t.gargs
seq 1 10 | rush -j 4  "echo job:{}; sleep 2; seq 1 10" > t.rush
seq 1 10 | parallel -j 4  "echo job:{}; sleep 2; seq 1 10" > t.parallel
seq 1 10 | rust-parallel -j 4  "echo job:{}; sleep 2; seq 1 10" > t.rust-parallel

time:
gargs: 6.03412008286
parallel: 6.22608017921
rush: 6.04201102257
rust-parallel: 6.10499191284

md5sum:
41d3bd9bd2e2f77e1249cf5dcee246f2  t.gargs
6d82dfa19a9f0546bb376b2042139ded  t.parallel
ab3dcbd7451a886c70068ddd1d16bbf8  t.rush
8099e6ad94165e8334ef54e3f1571204  t.rust-parallel

---------------------------[ keep order: True ]---------------------------

commands:
seq 1 10 | gargs -p 4 -o "echo job:{}; sleep 2; seq 1 10" > t.gargs
seq 1 10 | rush -j 4 -k "echo job:{}; sleep 2; seq 1 10" > t.rush
seq 1 10 | parallel -j 4 -k "echo job:{}; sleep 2; seq 1 10" > t.parallel
seq 1 10 | rust-parallel -j 4  "echo job:{}; sleep 2; seq 1 10" > t.rust-parallel

time:
gargs: 6.03302717209
parallel: 6.21512699127
rush: 6.03814911842
rust-parallel: 6.1030228138

md5sum:
6d82dfa19a9f0546bb376b2042139ded  t.gargs
6d82dfa19a9f0546bb376b2042139ded  t.parallel
6d82dfa19a9f0546bb376b2042139ded  t.rush
8099e6ad94165e8334ef54e3f1571204  t.rust-parallel


=========================[ lots of stdout & run long ]=========================
--------------------------[ keep order: False ]--------------------------

commands:
seq 1 10 | gargs -p 4  "echo job:{}; sleep 2; seq 1 1000000" > t.gargs
seq 1 10 | rush -j 4  "echo job:{}; sleep 2; seq 1 1000000" > t.rush
seq 1 10 | parallel -j 4  "echo job:{}; sleep 2; seq 1 1000000" > t.parallel
seq 1 10 | rust-parallel -j 4  "echo job:{}; sleep 2; seq 1 1000000" > t.rust-parallel

time:
gargs: 6.18096399307
parallel: 6.30418491364
rush: 6.1610519886
rust-parallel: 6.2177939415

md5sum:
878ec8fc1531cf4a912b489a9a679e3f  t.gargs
509086b10292759096d6db278bb2add8  t.parallel
ded23702158f839ed35637b5fb1436e3  t.rush
627790a71dfb0553b82af71d2e5e4f0a  t.rust-parallel

---------------------------[ keep order: True ]---------------------------

commands:
seq 1 10 | gargs -p 4 -o "echo job:{}; sleep 2; seq 1 1000000" > t.gargs
seq 1 10 | rush -j 4 -k "echo job:{}; sleep 2; seq 1 1000000" > t.rush
seq 1 10 | parallel -j 4 -k "echo job:{}; sleep 2; seq 1 1000000" > t.parallel
seq 1 10 | rust-parallel -j 4  "echo job:{}; sleep 2; seq 1 1000000" > t.rust-parallel
                                                                                                         
time:                                                                                                    
gargs: 6.17494392395                                                                                     
parallel: 6.30704498291
rush: 6.1639289856
rust-parallel: 6.18260288239

md5sum:
509086b10292759096d6db278bb2add8  t.gargs
509086b10292759096d6db278bb2add8  t.parallel
509086b10292759096d6db278bb2add8  t.rush
627790a71dfb0553b82af71d2e5e4f0a  t.rust-parallel

rush v0.1.0

Changelog

• fix indexing function for record delimiter, support multi-characters record delimiter (-D) now.
• support continue for multi-line commands
• immediately flush after a job successfully done
• add flag alias -C for --succ-cmd-file

rush v0.0.9

Changelog:

• fix replacement strings: {.} and {:} only take effect on basename

rush v0.0.8

Changelog

• add replacement string {^suffix} for removing suffix. For example:

  $ echo dir/sample_1.fq.gz | rush 'echo {^.fq.gz}, {%:^_1}'
  dir/sample_1, sample
  

rush v0.0.7

Changes:

• fix help message of flag -D
• add flag: -J/--records-join-seq, record separator for joining multi-records (default is "\n")