- fix use exact name not working

- documentation updates

Author: giventocode

README.md

@@ -29,7 +29,7 @@ Sources and targets are decoupled, this design enables the composition of variou
 Download, extract and set permissions:
 
 ```bash
-wget -O bp_linux.tar.gz https://github.com/Azure/blobporter/releases/download/v0.6.10/bp_linux.tar.gz
+wget -O bp_linux.tar.gz https://github.com/Azure/blobporter/releases/download/v0.6.12/bp_linux.tar.gz
 tar -xvf bp_linux.tar.gz linux_amd64/blobporter
 chmod +x ~/linux_amd64/blobporter
 cd ~/linux_amd64
@@ -46,7 +46,7 @@ export ACCOUNT_KEY=<STORAGE_ACCOUNT_KEY>
 
 ### Windows
 
-Download [BlobPorter.exe](https://github.com/Azure/blobporter/releases/download/v0.6.10/bp_windows.zip)
+Download [BlobPorter.exe](https://github.com/Azure/blobporter/releases/download/v0.6.12/bp_windows.zip)
 
 Set environment variables (if using the command prompt):
 

args.go

@@ -174,7 +174,8 @@ func (p *paramParserValidator) parseAndValidate() error {
 		p.pvgDupCheck,
 		p.pvgParseBlockSize,
 		p.pvgQuietMode,
-		p.pvgKeepDirectoryStructure)
+		p.pvgKeepDirectoryStructure,
+		p.pvgUseExactMatch)
 
 	if err != nil {
 		return err
@@ -257,6 +258,11 @@ func (p *paramParserValidator) getSourceRules() ([]parseAndValidationRule, error
 //**************************
 
 //Global rules....
+func (p *paramParserValidator) pvgUseExactMatch() error {
+	p.params.useExactMatch = p.args.exactNameMatch
+	return nil
+}
+
 func (p *paramParserValidator) pvgTransferStatusPathIsPresent() error {
 
 	if p.args.transferStatusPath != "" {

docs/conf.py

@@ -15,7 +15,7 @@
 # import os
 # import sys
 # sys.path.insert(0, os.path.abspath('.'))
-
+import sphinx_bootstrap_theme
 
 # -- Project information -----------------------------------------------------
 
@@ -75,6 +75,8 @@
 # a list of builtin themes.
 #
 html_theme = 'sphinx_rtd_theme'
+#html_theme = 'bootstrap'
+#html_theme_path = sphinx_bootstrap_theme.get_html_theme_path()
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the

docs/examples.rst

@@ -1,43 +1,52 @@
-========
 Examples
 ========
 
+
 Upload to Azure Block Blob Storage
--------------------------------------------
+-----------------------------------
 
 Single file upload:
 
-``./blobporter -f /datadrive/myfile.tar -c mycontainer -n myfile.tar``
+::
+
+    ./blobporter -f /datadrive/myfile.tar -c mycontainer -n myfile.tar
 
-    **Note:** If the container does not exist, it will be created.
+.. note:: BlobPorter will create the container if it doesn't exist.
 
 Upload all files that match the pattern:
 
-``./blobporter -f "/datadrive/*.tar" -c mycontainer``
+::
+
+    ./blobporter -f "/datadrive/*.tar" -c mycontainer
 
 You can also specify a list of files or patterns explicitly:
 
-``./blobporter -f "/datadrive/*.tar" -f "/datadrive/readme.md" -f "/datadrive/log" -c mycontainer``
+::
+
+    ./blobporter -f "/datadrive/*.tar" -f "/datadrive/readme.md" -f "/datadrive/log" -c mycontainer
 
 If you want to rename the target file name, you can use the -n option:
 
-``./blobporter -f /datadrive/f1.tar  -n newname.tar -c mycontainer``
+::
+
+    ./blobporter -f /datadrive/f1.tar  -n newname.tar -c mycontainer
 
 Upload to Azure Page Blob Storage
---------------------------------------
+----------------------------------
 
-Same as uploading to block blob storage, but with the transfer definiton (-t option) set to ``file-pageblob``.
+Same as uploading to block blob storage, but with the transfer definiton (-t option) set to ``file-pageblob``.  For example, a single file upload to page blob:
 
-For example, a single file upload to page blob:
+::
 
-``./blobporter -f /datadrive/mydisk.vhd -c mycontainer -n mydisk.vhd -t file-pageblob``
+    ./blobporter -f /datadrive/mydisk.vhd -c mycontainer -n mydisk.vhd -t file-pageblob
 
-    **Note:** The file size and block size must be a multiple of 512 (bytes). The maximum block size is 4MB.
 
-Transfer Data from an S3 endpoint to Azure Storage
---------------------------------------------------
+.. note:: The file size and block size must be a multiple of 512 (bytes). The maximum block size is 4MB.
 
-You can upload data from an S3 compatible endpoint.
+Transfer data from S3 to Azure Storage
+---------------------------------------
+
+You can transfer data from an S3 compatible endpoint.
 
 First you must specify the access and secret keys via environment variables.
 
@@ -48,71 +57,117 @@ First you must specify the access and secret keys via environment variables.
 
 Then you can specify an S3 URI, with the following format:
 
-``[HOST]/[BUCKET]/[PREFIX]``
+::
+
+    [HOST]/[BUCKET]/[PREFIX]
 
 For example:
 
-``./blobporter -f s3://mys3api.com/mybucket/mydata -c froms3 -t s3-blockblob``
+::
 
-    **Note:** For better performance, consider running this tranfer from a high-bandwidth VM running in the same region as source or the target. Data is uploaded as it is downloaded from the source, therefore the transfer is bound to the bandwidth of the VM for performance.
-    Synchronously Copy data between Azure Blob Storage targets and sources
+    ./blobporter -f s3://mys3api.com/mybucket/mydata -c froms3 -t s3-blockblob
 
+.. note::
 
-Transfer Data Between Azure Storage Accounts, Containers and Blob Types
+    BlobPorter will upload the data as it downloads it from the source.
+    The performance of the transfer will be constraint by the bandwidth of the host running BlobPorter. Consider running this type of transfer from a Virtual Machine running in the same Azure region as the target or the source.
+
+Transfer data between Azure Storage accounts, containers and blob types
 -----------------------------------------------------------------------
 
 First, you must set the account key of the source storage account.
 
-``export SOURCE_ACCOUNT_KEY=<YOUR  KEY>``
+::
+
+    export SOURCE_ACCOUNT_KEY=<YOUR  KEY>
+
 
 Then you can specify the URI of the source. The source could be a page, block or append blob. Prefixes are supported.
 
-``./blobporter -f "https://mysourceaccount.blob.core.windows.net/container/myblob" -c mycontainer -t blob-blockblob``
+::
+
+    ./blobporter -f "https://mysourceaccount.blob.core.windows.net/container/myblob" -c mycontainer -t blob-blockblob
 
-    **Note:** For better performance, consider running this tranfer from a high-bandwidth VM running in the same region as source or the target. Data is uploaded as it is downloaded from the source, therefore the transfer is bound to the bandwidth of the VM for performance.
-    Synchronously Copy data between Azure Blob Storage targets and sources
+.. note::
 
+    BlobPorter will upload the data as it downloads it from the source.
+    The performance of the transfer will be constraint by the bandwidth of the host running BlobPorter. Consider running this type of transfer from a Virtual Machine running in the same Azure region as the target or the source.
 
 Transfer from an HTTP/HTTPS source to Azure Blob Storage
 --------------------------------------------------------
 
 To block blob storage:
 
-``./blobporter -f "http://mysource/file.bam" -c mycontainer -n file.bam -t http-blockblob``
+::
+
+    ./blobporter -f "http://mysource/file.bam" -c mycontainer -n file.bam -t http-blockblob
 
 To page blob storage:
 
-``./blobporter -f "http://mysource/my.vhd" -c mycontainer -n my.vhd -t http-pageblob``
+::
+
+    ./blobporter -f "http://mysource/my.vhd" -c mycontainer -n my.vhd -t http-pageblob
 
-    **Note:** For better performance, consider running this tranfer from a high-bandwidth VM running in the same region as source or the target. Data is uploaded as it is downloaded from the source, therefore the transfer is bound to the bandwidth of the VM for performance.
-    Synchronously Copy data between Azure Blob Storage targets and sources
+.. note::
+
+    BlobPorter will upload the data as it downloads it from the source.
+    The performance of the transfer will be constraint by the bandwidth of the host running BlobPorter. Consider running this type of transfer from a Virtual Machine running in the same Azure region as the target or the source.
 
 Download from Azure Blob Storage
---------------------------
+--------------------------------
 
 For download scenarios, the source can be a page, append or block blob:
 
-``./blobporter -c mycontainer -n file.bam -t blob-file``
+::
+
+    ./blobporter -c mycontainer -n file.bam -t blob-file
 
 You can use the -n option to specify a prefix. All blobs that match the prefix will be downloaded. 
 
 The following will download all blobs in the container that start with f:
 
-``./blobporter -c mycontainer -n f -t blob-file``
+::
+
+    ./blobporter -c mycontainer -n f -t blob-file
+
 
 Without the -n option all files in the container will be downloaded.
 
-``./blobporter -c mycontainer -t blob-file``
+::
+
+    ./blobporter -c mycontainer -t blob-file
+
 
 By default files are downloaded keeping the same directory structure as the remote source. 
 
-If you want download to the same directory where you are running blobporter, set -i option.
+If you want download to the same directory where you are running blobporter set -i option.
+
+::
+
+    ./blobporter -c mycontainer -t blob-file -i
+
+
+For scenarios where blob endpoint is from a soverign cloud (e.g. China and Germany), Azure Gov or Azure Stack, you can specify the fully qualified domain name:
+
+::
+
+    ./blobporter -f "https://[ACCOUNT_NAME].[BASE_URL]/[CONTAINER_NAME]/[PREFIX]" -t blob-file
+
+And the source account key, must be set via an environment variable. 
+
+::
+
+    export SOURCE_ACCOUNT_KEY=<YOUR  KEY>
+
 
-``./blobporter -p -c mycontainer -t blob-file -i``
 
 Download a file from a HTTP source
 ----------------------------------
 
-``./blobporter -f "http://mysource/file.bam" -n /datadrive/file.bam -t http-file``
+::
+
+    ./blobporter -f "http://mysource/file.bam" -n /datadrive/file.bam -t http-file
+
+.. note::
 
-    **Note:** The ACCOUNT_NAME and ACCOUNT_KEY environment variables are not required in this scenario.
+    The ACCOUNT_NAME and ACCOUNT_KEY environment variables are not required.

docs/gettingstarted.rst

@@ -9,7 +9,7 @@ Download, extract and set permissions
 
 ::
 
-    wget -O bp_linux.tar.gz https://github.com/Azure/blobporter/releases/download/v0.6.09/bp_linux.tar.gz
+    wget -O bp_linux.tar.gz https://github.com/Azure/blobporter/releases/download/v0.6.12/bp_linux.tar.gz
     tar -xvf bp_linux.tar.gz linux_amd64/blobporter
     chmod +x ~/linux_amd64/blobporter
     cd ~/linux_amd64
@@ -24,7 +24,7 @@ Set environment variables: ::
 Windows
 -------
 
-Download `BlobPorter.exe <https://github.com/Azure/blobporter/releases/download/v0.6.10/bp_windows.zip>`_
+Download `BlobPorter.exe <https://github.com/Azure/blobporter/releases/download/v0.6.12/bp_windows.zip>`_
 
 Set environment variables (if using the command prompt): ::
 
@@ -36,3 +36,50 @@ Set environment variables (if using PowerShell): ::
     $env:ACCOUNT_NAME="<STORAGE_ACCOUNT_NAME>"
     $env:ACCOUNT_KEY="<STORAGE_ACCOUNT_KEY>"
 
+
+Command Options
+---------------
+
+ -f, --source_file          (string) URL, Azure Blob or S3 Endpoint, file or files (e.g. /data/\*.gz) to upload.
+ -c, --container_name       (string) Container name (e.g. mycontainer).
+ -n, --blob_name            (string) Blob name (e.g. myblob.txt) or prefix for download scenarios.
+ -g, --concurrent_workers   (int) Number of go-routines for parallel upload.
+ -r, --concurrent_readers   (int) Number of go-routines for parallel reading of the input.
+ -b, --block_size           (string) Desired size of each blob block.
+
+                            Can be specified as an integer byte count or integer suffixed with B, KB or MB. 
+
+ -a, --account_name         (string) Storage account name (e.g. mystorage).
+
+                            Can also be specified via the ACCOUNT_NAME environment variable.
+
+ -k, --account_key          (string) Storage account key string.
+                            
+                            Can also be specified via the ACCOUNT_KEY environment variable.
+ -s, --http_timeout         (int) HTTP client timeout in seconds. Default value is 600s.
+ -d, --dup_check_level      (string) Desired level of effort to detect duplicate data blocks to minimize upload size.
+ 
+                            Must be one of None, ZeroOnly, Full (default "None")
+ -t, --transfer_type        (string) Defines the source and target of the transfer.
+ 
+                            Must be one of file-blockblob, file-pageblob, http-blockblob, http-pageblob, blob-file, pageblock-file (alias of blob-file), blockblob-file (alias of blob-file), http-file, blob-pageblob, blob-blockblob, s3-pageblob and s3-blockblob.
+ -m, --compute_blockmd5     (bool) If set, block level MD5 has will be computed and included as a header when the block is sent to blob storage.
+ 
+                            Default is false.
+ -q, --quiet_mode           (bool) If set, the progress indicator is not displayed. 
+
+                            The files to transfer, errors, warnings and transfer completion summary is still displayed.
+ -x, --files_per_transfer   (int) Number of files in a batch transfer. Default is 500.
+ -h, --handles_per_file     (int) Number of open handles for concurrent reads and writes per file. Default is 2.
+ -i, --remove_directories   (bool) If set blobs are downloaded or uploaded without keeping the directory structure of the source. 
+                            
+                            Not applicable when the source is a HTTP endpoint.
+ -o, --read_token_exp       (int) Expiration in minutes of the read-only access token that will be generated to read from S3 or Azure Blob sources.
+                            
+                            Default value: 360.
+ -l, --transfer_status      (string) Transfer status file location.
+                            If set, blobporter will use this file to track the status of the transfer. 
+                            
+                            In case of failure and the same file is referrenced, the source files that were transferred will be skipped.
+                            
+                            If the transfer is successful a summary will be appended.

docs/index.rst

@@ -8,6 +8,9 @@ BlobPorter
 
 BlobPorter is a data transfer tool for Azure Blob Storage that maximizes throughput through concurrent reads and writes that can scale up and down independently.
 
+::
+    
+
 .. image :: bptransfer.png
 
 .. toctree::
@@ -16,6 +19,7 @@ BlobPorter is a data transfer tool for Azure Blob Storage that maximizes through
 
    gettingstarted
    examples
-   performance/perfmode
-   resumable_transfers
-   options
+   resumabletrans
+   perfmode
+    
+