- Batch transfers, 200 files at the time.

 - New options related to batch transfers:
 - - x, number of files per batch tranfer (default 200).
 - - Max number of handles for linux is calculated by the number of sources times number of handles per file.
 - - h, handles per file, default 2.
 - Preprocess of sources occur concurrently.
 - Default blocksize for blocks is 8MB instead of 4 MB
 - Reduced default per core number of readers and workers (5, 8).
 - p option keeps the directory structure from storage account while downloading from the storage account.

Author: giventocode

.gitignore

@@ -12,6 +12,7 @@ _build/windows_amd64
 # Architecture specific extensions/prefixes
 *.[568vq]
 [568vq].out
+debug
 
 *.cgo1.go
 *.cgo2.c

.vscode/launch.json

@@ -11,8 +11,11 @@
             "port": 2345,
             "host": "127.0.0.1",
             "program": "${fileDirname}",
-            "env": {},
-            "args": [" -f \"http://video.ch9.ms/ch9/d36c/6cfce1e3-63e5-47a7-b3b6-8c813a23d36c/SnackPackPlanetXamarin.mp3\" -f \"http://video.ch9.ms/ch9/d36c/6cfce1e3-63e5-47a7-b3b6-8c813a23d36c/SnackPackPlanetXamarin.mp3\" -n v1.mp3 -n v2.mp3 -t http-file "] ,
+            "env": {
+                "ACCOUNT_NAME":"storagejaa",
+                "ACCOUNT_KEY":"/mXWG4aXMWvftzR+Ed5URccwrDSvvv5cUsKCqq5gbyPXvtlyfQ9gjq462kuH/2dErjwR8QcAS74vLaNXynd74g=="
+            },
+            "args": ["-c", "many", "-t","blob-file","-p"] ,
             "showLog": true
         }
     ]

README.md

@@ -29,7 +29,7 @@ Sources and targets are decoupled, this design enables the composition of variou
 Download, extract and set permissions:
 
 ```bash
-wget -O bp_linux.tar.gz https://github.com/Azure/blobporter/releases/download/v0.5.04/bp_linux.tar.gz
+wget -O bp_linux.tar.gz https://github.com/Azure/blobporter/releases/download/v0.5.10/bp_linux.tar.gz
 tar -xvf bp_linux.tar.gz linux_amd64/blobporter
 chmod +x ~/linux_amd64/blobporter
 cd ~/linux_amd64
@@ -46,7 +46,7 @@ export ACCOUNT_KEY=<STORAGE_ACCOUNT_KEY>
 
 ### Windows
 
-Download [BlobPorter.exe](https://github.com/Azure/blobporter/releases/download/v0.5.04/bp_windows.zip)
+Download [BlobPorter.exe](https://github.com/Azure/blobporter/releases/download/v0.5.10/bp_windows.zip)
 
 Set environment variables (if using the command prompt):
 
@@ -84,6 +84,7 @@ If you want to rename multiple files, you can use the -n option:
 
 `./blobporter -f /datadrive/f1.tar -f /datadrive/f2.md -n b1 -n b2 -c mycontainer`
 
+
 ### Upload to Azure Page Blob Storage
 
 Same as uploading to block blob storage, but with the transfer definiton (-t option) set to file-pageblob.
@@ -127,6 +128,11 @@ Without the -n option all files in the container will be downloaded.
 
 `./blobporter  -c mycontainer -t blob-file`
 
+By default files are downloaded to the same directory where you are running blobporter. If you want to keep the same directory structure of the storage account use the -p option.
+
+`./blobporter -p -c mycontainer -t blob-file`
+
+
 ### Download a file via HTTP to a local file
 
 `./blobporter -f "http://mysource/file.bam"  -n /datadrive/file.bam -t http-file`
@@ -171,15 +177,23 @@ Without the -n option all files in the container will be downloaded.
 
 ## Performance Considerations
 
-By default, BlobPorter creates 6 readers and 9 workers for each core on the computer. You can overwrite these values by using the options -r (number of readers) and -g (number of workers). When overriding these options there are few considerations:
+By default, BlobPorter creates 5 readers and 8 workers for each core on the computer. You can overwrite these values by using the options -r (number of readers) and -g (number of workers). When overriding these options there are few considerations:
 
 - If during the transfer the buffer level is constant at 000%, workers could be waiting for data. Consider increasing the number of readers. If the level is 100% the opposite applies; increasing the number of workers could help.
 
 - In BlobPorter, each reader or worker correlates to one goroutine. Goroutines are lightweight and a Go program can create a high number of goroutines, however, there's a point where the overhead of context switching impacts overall performance. Increase these values in small increments, e.g. 5.
 
-- For transfers from fast disks (SSD) or HTTP sources a lesser number readers or workers could provide the same performance than the default values. You could reduce these values if you want to minimize resource utilization. Lowering these numbers reduces contention and the likelihood of experiencing throttling conditions.
+- For transfers from fast disks (SSD) or HTTP sources reducing the number readers or workers could provide better performance than the default values. Reduce these values if you want to minimize resource utilization. Lowering these numbers reduces contention and the likelihood of experiencing throttling conditions.
+
+- Starting with version 0.5.10:
+
+- - Transfers are batched. Each batch transfer will concurrently read and transfer up to 200 files (default value) from the source. The batch size can be modified using the -x option, the maximum value is 500.
+
+- - Blobs smaller than the block size are transferred in a single operation. With relatively small files (<32MB) performance may be higher if you set a block size equal to the size of the files. Setting the number of workers and readers to the number of files could also, yeild performance gains.
+
+## Issues and Feedback
 
-- In Linux, BlobPorter reduces the number of readers if the number of open files during the transfer is greater than 1024. Linux restricts the number of files open by a process and since each reader holds a handle to the file to transfer, you can reach this limit if you want transfer multiple files even with a relatively low number of readers. For example, if you have 10 readers and want to transfer more than 102 files you will reach this limit. In this case BlobPorter will issue a warning displaying the new number of readers. If the resulting number of readers impacts performance, consider running multiple instances of BlobPorter with a smaller source list.
+If you have a question or find a bug, open a new issue in this repository. BlobPorter is an OSS project maintained by the contributors.
 
 ## Contribute