update setup and requirements (#241)

* update requirements

* update setup

* add sentencepiece back in

* black formatter

* change 'dev' to 'eval', add scipy back in

* move scipy

* separate out requirements, update environment.yml

* fix pycocotools

* update README

* moved tqdm to eval, wandb only in trainnig

* update version

* black format

---------

Co-authored-by: Alexander Loftus <alex@creyonbio.com>

Author: loftusa

README.md

@@ -36,6 +36,20 @@ or to create a conda environment for running OpenFlamingo, run
 conda env create -f environment.yml
 ```
 
+To install training or eval dependencies, run one of the first two commands. To install everything, run the third command.
+```
+pip install open-flamingo[training]
+pip install open-flamingo[eval]
+pip install open-flamingo[all]
+```
+
+There are three `requirements.txt` files: 
+- `requirements.txt` 
+- `requirements-training.txt`
+- `requirements-eval.txt`
+
+Depending on your use case, you can install any of these with `pip install -r <requirements-file.txt>`. The base file contains only the dependencies needed for running the model.
+
 # Approach
 OpenFlamingo is a multimodal language model that can be used for a variety of tasks. It is trained on a large multimodal dataset (e.g. Multimodal C4) and can be used to generate text conditioned on interleaved images/text. For example, OpenFlamingo can be used to generate a caption for an image, or to generate a question given an image and a text passage. The benefit of this approach is that we are able to rapidly adapt to new tasks using in-context learning.
 

environment.yml

@@ -7,4 +7,6 @@ dependencies:
   - pip
   - pip:
     - -r requirements.txt
+    - -r requirements-training.txt
+    - -r requirements-eval.txt
     - -e .

requirements-dev.txt

@@ -0,0 +1,13 @@
+scipy
+torchvision
+nltk
+inflection
+pycocoevalcap
+pycocotools
+tqdm
+
+black
+mypy
+pylint
+pytest
+requests

requirements-eval.txt

@@ -0,0 +1,5 @@
+torchvision
+braceexpand
+webdataset
+tqdm
+wandb

requirements-training.txt

@@ -2,17 +2,6 @@ einops
 einops-exts
 transformers>=4.28.1
 torch==2.0.1
-torchvision
 pillow
-more-itertools
-datasets
-braceexpand
-webdataset
-wandb
-nltk
-scipy
-inflection
-sentencepiece==0.1.98
-pycocoevalcap
-pycocotools
-open_clip_torch>=2.16.0
+open_clip_torch>=2.16.0 
+sentencepiece==0.1.98 

requirements.txt

@@ -6,47 +6,51 @@
     with Path(Path(__file__).parent, "README.md").open(encoding="utf-8") as file:
         long_description = file.read()
 
-    # TODO: This is a hack to get around the fact that we can't read the requirements.txt file, we should fix this.
-    # def _read_reqs(relpath):
-    #     fullpath = os.path.join(Path(__file__).parent, relpath)
-    #     with open(fullpath) as f:
-    #         return [
-    #             s.strip()
-    #             for s in f.readlines()
-    #             if (s.strip() and not s.startswith("#"))
-    #         ]
-
     REQUIREMENTS = [
         "einops",
         "einops-exts",
         "transformers>=4.28.1",
         "torch==2.0.1",
-        "torchvision",
         "pillow",
-        "more-itertools",
-        "datasets",
-        "braceexpand",
-        "webdataset",
-        "wandb",
-        "nltk",
+        "open_clip_torch>=2.16.0",
+        "sentencepiece==0.1.98",
+    ]
+
+    EVAL = [
         "scipy",
+        "torchvision",
+        "nltk",
         "inflection",
-        "sentencepiece==0.1.98",
-        "open_clip_torch>=2.16.0",
+        "pycocoevalcap",
+        "pycocotools",
+        "tqdm",
+    ]
+
+    TRAINING = [
+        "wandb",
+        "torchvision",
+        "braceexpand",
+        "webdataset",
+        "tqdm",
     ]
 
     setup(
         name="open_flamingo",
         packages=find_packages(),
         include_package_data=True,
-        version="2.0.0",
+        version="2.0.1",
         license="MIT",
         description="An open-source framework for training large multimodal models",
         long_description=long_description,
         long_description_content_type="text/markdown",
         data_files=[(".", ["README.md"])],
         keywords=["machine learning"],
         install_requires=REQUIREMENTS,
+        extras_require={
+            "eval": EVAL,
+            "training": TRAINING,
+            "all": list(set(REQUIREMENTS + EVAL + TRAINING)),
+        },
         classifiers=[
             "Development Status :: 4 - Beta",
             "Intended Audience :: Developers",