Merge pull request #572 from microsoft/staging

Staging to Master

Author: saidbleik

.flake8

@@ -11,6 +11,8 @@
 # E402  module level import not at top of file
 # E731  do not assign a lambda expression, use a def
 # F821  undefined name 'get_ipython' --> from generated python files using nbconvert
+# E722: do not use bare except
+# E231: missing white space after "," --> black generates autoformat [,] which fails flake8
+ignore = E203, E266, W503, F403, F405, E402, E731, F821, E722, E231
 
-ignore = E203, E266, W503, F403, F405, E402, E731, F821
 max-line-length = 88

README.md

@@ -50,7 +50,7 @@ The following is a summary of the commonly used NLP scenarios covered in the rep
 |-------------------------|  ------------------- |-------|---|
 |Text Classification                     |BERT, XLNet, RoBERTa| Text classification is a supervised learning method of learning and predicting the category or the class of a document given its text content. |English, Hindi, Arabic|
 |Named Entity Recognition                |BERT| Named entity recognition (NER) is the task of classifying words or key phrases of a text into predefined entities of interest. |English|
-|Text Summarization|BERTSum|Text summarization is a language generation task of summarizing the input text into a shorter paragraph of text.|English
+|Text Summarization|BERTSumExt <br> BERTSumAbs <br> UniLM (s2s-ft)|Text summarization is a language generation task of summarizing the input text into a shorter paragraph of text.|English
 |Entailment                              |BERT, XLNet, RoBERTa| Textual entailment is the task of classifying the binary relation between two natural-language texts,  *text* and *hypothesis*, to determine if the *text* agrees with the *hypothesis* or not. |English|
 |Question Answering                      |BiDAF, BERT, XLNet| Question answering (QA) is the task of retrieving or generating a valid answer for a given query in natural language, provided with a passage related to the query. |English|
 |Sentence Similarity                     |BERT, GenSen| Sentence similarity is the process of computing a similarity score given a pair of text documents. |English|

SETUP.md

@@ -58,31 +58,84 @@ You can specify the environment name as well with the flag `-n`.
 Click on the following menus to see how to install the Python GPU environment:
 
 <details>
-<summary><strong><em>Python GPU environment on Linux, MacOS</em></strong></summary>
+<summary><strong><em>Python GPU environment</em></strong></summary>
 
-Assuming that you have a GPU machine, to install the Python GPU environment, which by default installs the CPU environment:
+Assuming that you have a GPU machine, to install the Python GPU environment, 
+1. Check the CUDA **driver** version on your machine by running
 
-    cd nlp-recipes
-    python tools/generate_conda_file.py --gpu
-    conda env create -n nlp_gpu -f nlp_gpu.yaml
-
-</details>
+        nvidia-smi
+    The top of the output shows the CUDA **driver** version, which is 10.0 in the example below.   
+    +-----------------------------------------------------------------------------+  
+    | NVIDIA-SMI 410.79 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Driver Version: 410. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;CUDA Version: 10.0     |  
+    |-------------------------------+----------------------+----------------------+
+2. Decide which cuda **runtime** version you should install.   
+The cuda **runtime** version is the version of the cudatoolkit that will be installed in the conda environment in the next step, which should be <= the CUDA **driver** version found in step 1.  
+Currently, this repo uses PyTorch 1.4.0 which is compatible with cuda 9.2 and cuda 10.1. The conda environment file generated in step 3 installs cudatoolkit 10.1 by default. If your CUDA **driver** version is < 10.1, you should add additional argument "--cuda_version 9.2" when calling generate_conda_files.py.   
 
-<details>
-<summary><strong><em>Python GPU environment on Windows</em></strong></summary>
+3. Install the GPU environment:  
+If CUDA **driver** version >= 10.1
 
-Assuming that you have an Azure GPU DSVM machine, here are the steps to setup the Python GPU environment:
-1. Make sure you have CUDA Toolkit version 9.0 above installed on your Windows machine. You can run the command below in your terminal to check.
-
-         nvcc --version
-    If you don't have CUDA Toolkit or don't have the right version, please download it from here: [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit)
+        cd nlp-recipes
+        python tools/generate_conda_file.py --gpu
+        conda env create -n nlp_gpu -f nlp_gpu.yaml
 
-2. Install the GPU environment.
+    If CUDA **driver** version < 10.1
 
         cd nlp-recipes
-        python tools/generate_conda_file.py --gpu
+        python tools/generate_conda_file.py --gpu --cuda_version 9.2
         conda env create -n nlp_gpu -f nlp_gpu.yaml
 
+4. Enable mixed precision training (optional)  
+Mixed precision training is particularly useful if your model takes a long time to train. It usually reduces the training time by 50% and produces the same model quality. To enable mixed precision training, run the following command 
+
+        conda activate nlp_gpu
+        git clone https://github.com/NVIDIA/apex.git
+        cd apex
+        pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
+
+    **Troubleshooting**:  
+    If you run into an error message "RuntimeError: Cuda extensions are being compiled with a version of Cuda that does not match the version used to compile Pytorch binaries.", you need to make sure your NVIDIA Cuda compiler driver (nvcc) version and your cuda **runtime** version are exactly the same. To check the nvcc version, run   
+
+        nvcc -V
+
+    If the nvcc version is 10.0, it's recommended to upgrade to 10.1 and re-create your conda environment with cudatoolkit=10.1.
+    
+    **Steps to upgrade CUDA **driver** version and nvcc version**  
+    We have tested the following steps. Alternatively, you can follow the official instructions [here](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)  
+    a. Update apt-get and reboot your machine
+
+        sudo apt-get update
+        sudo apt-get upgrade --fix-missing
+        sudo reboot
+    b. Download the CUDA toolkit .run file from https://developer.nvidia.com/cuda-10.1-download-archive-base based on your target platform. For example, on a Linux machine with Ubuntu 16.04, run   
+
+        wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.105_418.39_linux.run  
+
+    c. Upgrade CUDA driver by running  
+
+        sudo sh cuda_10.1.105_418.39_linux.run
+    First, accept the user agreement.  
+    ![](https://nlpbp.blob.core.windows.net/images/upgrade_cuda_driver/1agree_to_user_agreement.PNG)  
+    Next, choose the components to install.  
+    It's possible that you already have NVIDIA driver 418.39 and CUDA 10.1, but nvcc 10.0. In this case, you can uncheck the "DRIVER" box and upgrade nvcc by re-installing CUDA toolkit only.   
+    ![](https://nlpbp.blob.core.windows.net/images/upgrade_cuda_driver/2install_cuda_only.PNG)  
+
+    If you choose to install all components, follow the instructions on the screen to uninstall existing NVIDIA driver and CUDA toolkit first.  
+    ![](https://nlpbp.blob.core.windows.net/images/upgrade_cuda_driver/3install_all.PNG)   
+    Then re-run   
+
+        sudo sh cuda_10.1.105_418.39_linux.run
+    Select "Yes" to update the cuda symlink.   
+    ![](https://nlpbp.blob.core.windows.net/images/upgrade_cuda_driver/4Upgrade_symlink.PNG)  
+
+    d. Run the following commands again to make sure you have NVIDIA driver 418.39, CUDA driver 10.1 and nvcc 10.1
+
+        nvidia-smi
+        nvcc -V
+
+    e. Repeat steps 3 & 4 to recreate your conda environment with cudatoolkit **runtime** 10.1 and apex installed for mixed precision training. 
+
+
 </details>
 
 ### Register Conda Environment in DSVM JupyterHub

cgmanifest.json

@@ -18,7 +18,18 @@
         }
       },
       "license": "Apache-2.0"
+   },
+   {
+      "component": { 
+      "type": "git", 
+      "git": { 
+        "repositoryUrl": "https://github.com/nlpyang/PreSumm", 
+        "commitHash": "2df3312582a3a014aacbc1be810841705c67d06e" 
+        }
+      },
+      "license": "MIT License"
    }
+
  ],
  "Version": 1
-}
+}

examples/README.md

@@ -6,7 +6,7 @@ This folder contains examples and best practices, written in Jupyter notebooks,
 |---| ------------------------ | ------------------- |---|
 |[Text Classification](text_classification)|Topic Classification|BERT, XLNet, RoBERTa, DistilBERT|en, hi, ar|
 |[Named Entity Recognition](named_entity_recognition) |Wikipedia NER|BERT|en|
-|[Text Summarization](text_summarization)|News Summarization, Headline Generation|Extractive: BERTSumExt <br> Abstractive: WIP, ETA: Mar. 2020|en
+|[Text Summarization](text_summarization)|News Summarization, Headline Generation|Extractive: BERTSumExt <br> Abstractive: UniLM (s2s-ft)|en
 |[Entailment](entailment)|MultiNLI Natural Language Inference|BERT|en|
 |[Question Answering](question_answering) |SQuAD|BiDAF, BERT, XLNet, DistilBERT|en|
 |[Sentence Similarity](sentence_similarity)|STS Benchmark|BERT, GenSen|en|

examples/model_explainability/interpret_dnn_layers.ipynb

@@ -46,6 +46,7 @@
    "outputs": [],
    "source": [
     "import sys\n",
+    "from tempfile import TemporaryDirectory\n",
     "\n",
     "sys.path.append(\"../../\")\n",
     "import json\n",
@@ -322,11 +323,12 @@
     "text = \"rare bird has more than enough charm to make it memorable.\"\n",
     "\n",
     "# get the tokenized words.\n",
-    "tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\")\n",
+    "cache_dir = TemporaryDirectory().name\n",
+    "tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\", cache_dir=cache_dir)\n",
     "words = [\"[CLS]\"] + tokenizer.tokenize(text) + [\"[SEP]\"]\n",
     "\n",
     "# load BERT base model\n",
-    "model = BertModel.from_pretrained(\"bert-base-uncased\").to(device)\n",
+    "model = BertModel.from_pretrained(\"bert-base-uncased\", cache_dir=cache_dir).to(device)\n",
     "for param in model.parameters():\n",
     "    param.requires_grad = False\n",
     "model.eval()\n",