updated all notebooks - Adithya S K

Author: adithya-s-k

docs/demo.ipynb

@@ -13,6 +13,70 @@
     "This makes it easy to test and experiment with different approaches in real-time."
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "!git clone https://github.com/adithya-s-k/VARAG\n",
+    "%cd VARAG\n",
+    "%pwd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "!apt-get update && apt-get install -y && apt-get install -y poppler-utils"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "%pip install -e .\n",
+    "\n",
+    "## We will be using Docling for OCR\n",
+    "%pip install docling"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run Gradio"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "!python demo.py --share"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

docs/simpleRAG.ipynb

@@ -396,93 +396,15 @@
     },
     {
       "cell_type": "code",
-      "execution_count": 11,
+      "execution_count": null,
       "metadata": {
         "colab": {
           "base_uri": "https://localhost:8080/"
         },
         "id": "m4izqMiY-knW",
         "outputId": "d6427938-f1f2-4f08-b051-73e841a48840"
       },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "This was the retrieved Context\n",
-            "======================================================================================================================================================\n",
-            "\n",
-            "\n",
-            "Chunk 1:\n",
-            "Text: The table uses \"W\" and \"T\" markers to denote which system or department serves as the primary source (writer) or storage location (trailer) for each type of document.\n",
-            "\n",
-            "## C More similarity maps\n",
-            "\n",
-            "In Figure 7, ColPali assigns a high similarity to all patches with the word \"Kazakhstan\" when given the token <_Kazakhstan> . Moreover, our model seems to exhibit world knowledge capabilities as the patch around the word \"Kashagan\" - an offshore oil field in Kazakhstan - also shows a high similarity score. On the other hand, in Figure 8, we observe that ColPali is also capable of complex image understanding. Not only are the patches containing the word \"formulations\" highly similar to the query token _formula , but so is the upper-left molecule structure.\n",
-            "\n",
-            "It is also interesting to highlight that both similarity maps showcase a few white patches with high similarity scores. This behavior might first seem surprising as the white patches should not carry a meaningful signal from the original images\n",
-            "Chunk Index: 68\n",
-            "Document Name: colpali.pdf\n",
-            "\n",
-            "\n",
-            "======================================================================================================================================================\n",
-            "======================================================================================================================================================\n",
-            "\n",
-            "\n",
-            "Chunk 2:\n",
-            "Text: 4% ) on the aggregated benchmark.\n",
-            "\n",
-            "## Can the model adapt to new tasks?\n",
-            "\n",
-            "Contrary to more complex multi-step retrieval pipelines, ColPali can be trained end-to-end, directly optimizing the downstream retrieval task which greatly facilitates fine-tuning to boost performance on specialized domains, multilingual retrieval, or specific visual elements the model struggles with. To demonstrate, we add 1552 samples representing French tables and associated queries to the training set. This represents the only French data in the training set, with all other examples being kept unchanged. We see significant NDCG@5 improvements (Figure 4) and even starker Recall@1 gains ( +6 . 63% ) on the TabFQuAD benchmark, with no performance degradation on the rest of the benchmark tasks ( +0\n",
-            "Chunk Index: 36\n",
-            "Document Name: colpali.pdf\n",
-            "\n",
-            "\n",
-            "======================================================================================================================================================\n",
-            "======================================================================================================================================================\n",
-            "\n",
-            "\n",
-            "Chunk 3:\n",
-            "Text: Optimized late interaction engines (Santhanam et al., 2022; Lee et al., 2023) enable to easily scale corpus sizes to millions of documents with reduced latency degradations.\n",
-            "\n",
-            "Offline Indexing. (R3) Standard retrieval methods using bi-encoders represent each chunk as a single vector embedding, which is easy to store and fast to compute. However, processing a PDF to get the different chunks is the most time-consuming part (layout detection, OCR, chunking), and using captioning to handle multimodal data will only exacerbate this already lengthy process. On the other hand, ColPali directly encodes pages from their image representation. Although the encoder model is larger than standard retrieval encoders, skipping the preprocessing allows large speedups at indexing 10 (Figure 3).\n",
-            "\n",
-            "Memory Footprint. Our method requires storing a vector per image patch\n",
-            "Chunk Index: 31\n",
-            "Document Name: colpali.pdf\n",
-            "\n",
-            "\n",
-            "======================================================================================================================================================\n",
-            "======================================================================================================================================================\n",
-            "\n",
-            "\n",
-            "Chunk 4:\n",
-            "Text: This is particularly notable as our training dataset does not contain non-English samples.\n",
-            "\n",
-            "ColPali : Adding Late Interaction. One benefit of inputting image patch embeddings through a language model is that they are natively mapped to a latent space similar to textual input (query). This enables leveraging the ColBERT strategy to compute interactions between text tokens and image patches, which enables a step-change improvement in performance compared to BiPali. Results in Table 2 show that our ColPali model also largely outperforms the strong baselines based on Unstructured and captioning, as well as all evaluated text-image embedding models. The difference is particularly stark on the more visually complex benchmark tasks, such as InfographicVQA, ArxivQA, and TabFQuAD representing respectively infographics, figures, and tables\n",
-            "Chunk Index: 25\n",
-            "Document Name: colpali.pdf\n",
-            "\n",
-            "\n",
-            "======================================================================================================================================================\n",
-            "======================================================================================================================================================\n",
-            "\n",
-            "\n",
-            "Chunk 5:\n",
-            "Text: Subsequently, our vision is to combine visual retrieval and visually grounded query answering to create RAG systems that purely function from visual features. An interesting line of research could be attempting to generate answers leveraging information stored in the indexed multivector patch embeddings.\n",
-            "\n",
-            "## Limitations\n",
-            "\n",
-            "Focus. In this work, we evaluate models on document retrieval tasks, covering several modalities (figures, text, tables, infographics). We however primarily focus on PDF-type documents, and evaluating systems on image retrieval with documents stemming from web page screenshots or handwritten documents might be an interesting generalization. We also focus on high-resource languages (English and French) and although we have shown the capacity of the ColPali model to generalize to languages outside of its fine-tuning set, it is unclear how the model would perform on languages that are not as represented in the model's language backbone\n",
-            "Chunk Index: 38\n",
-            "Document Name: colpali.pdf\n",
-            "\n",
-            "\n",
-            "======================================================================================================================================================\n"
-          ]
-        }
-      ],
+      "outputs": [],
       "source": [
         "query = \"what is colpali ?\"\n",
         "num_results = 5\n",
@@ -501,7 +423,7 @@
     },
     {
       "cell_type": "code",
-      "execution_count": 12,
+      "execution_count": null,
       "metadata": {
         "colab": {
           "base_uri": "https://localhost:8080/",
@@ -510,20 +432,7 @@
         "id": "2flyc-fc-knX",
         "outputId": "00f23adb-0387-4de3-f7f4-bd4ac61d458d"
       },
-      "outputs": [
-        {
-          "data": {
-            "text/markdown": [
-              "ColPali is a model designed for document retrieval tasks that combines visual retrieval with language processing to enhance performance, particularly in the context of multimodal documents like PDFs, figures, tables, and infographics. It utilizes a late interaction mechanism to compute interactions between text tokens and image patches, improving retrieval capabilities significantly compared to previous models. ColPali can be trained end-to-end, allowing it to adapt to new tasks and specialized domains efficiently. Additionally, it can handle various languages and leverages visual features for query answering, aiming to create systems that function purely from visual information."
-            ],
-            "text/plain": [
-              "<IPython.core.display.Markdown object>"
-            ]
-          },
-          "metadata": {},
-          "output_type": "display_data"
-        }
-      ],
+      "outputs": [],
       "source": [
         "from IPython.display import display, Markdown, Latex\n",
         "\n",
@@ -557,45 +466,11 @@
         "id": "cTvt2kHk-knY",
         "outputId": "3d4d0aba-6b57-4401-ae0e-4692d65da39a"
       },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "/content/VARAG/examples\n",
-            "2024-09-28 09:45:43.314833: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
-            "2024-09-28 09:45:43.339060: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
-            "2024-09-28 09:45:43.347860: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
-            "2024-09-28 09:45:44.422675: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
-            "INFO:sentence_transformers.SentenceTransformer:Use pytorch device_name: cuda\n",
-            "INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: all-MiniLM-L6-v2\n",
-            "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1617: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be deprecated in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
-            "  warnings.warn(\n",
-            "Using device: cuda\n",
-            "INFO:httpx:HTTP Request: GET https://api.gradio.app/pkg-version \"HTTP/1.1 200 OK\"\n",
-            "Running on local URL:  http://127.0.0.1:7860\n",
-            "INFO:httpx:HTTP Request: GET http://127.0.0.1:7860/startup-events \"HTTP/1.1 200 OK\"\n",
-            "INFO:httpx:HTTP Request: HEAD http://127.0.0.1:7860/ \"HTTP/1.1 200 OK\"\n",
-            "\n",
-            "To create a public link, set `share=True` in `launch()`.\n",
-            "INFO:httpx:HTTP Request: GET https://checkip.amazonaws.com/ \"HTTP/1.1 200 \"\n",
-            "INFO:httpx:HTTP Request: GET https://checkip.amazonaws.com/ \"HTTP/1.1 200 \"\n"
-          ]
-        }
-      ],
+      "outputs": [],
       "source": [
         "%cd examples\n",
         "!python textDemo.py --share"
       ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "fRFHnpC7BW_K"
-      },
-      "outputs": [],
-      "source": []
     }
   ],
   "metadata": {
@@ -609,7 +484,16 @@
       "name": "python3"
     },
     "language_info": {
-      "name": "python"
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.11.9"
     },
     "widgets": {
       "application/vnd.jupyter.widget-state+json": {