Use Langchain DirectoryLoader， but facing error:" raise BadZipFile("File is not a zip file")"

[BlackSiao]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.document_loaders import DirectoryLoader


def load_documents(directory="book"):
    loader = DirectoryLoader(directory, show_progress=True)
    documents = loader.load()
    for document in documents:
        print(document)


load_documents('book')

Description

I tried to use langchain to load all the txt text files in a certain folder, but an error was reported: raise BadZipFile("File is not a zip file"); I saw the same problem on GitHub, but the method it gives is to reinstall NLTK, but I tried it without success, and the error message is as follows: File "C:\Users\BlackSiao\Desktop\毕业设计\ChatGlm_testcode\pythonProject1\file_load.py", line 12, in
load_documents('book')
File "C:\Users\BlackSiao\Desktop\毕业设计\ChatGlm_testcode\pythonProject1\file_load.py", line 7, in load_documents
documents = loader.load()
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\langchain\document_loaders\directory.py", line 156, in load
self.load_file(i, p, docs, pbar)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\langchain\document_loaders\directory.py", line 105, in load_file
raise e
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\langchain\document_loaders\directory.py", line 99, in load_file
sub_docs = self.loader_cls(str(item), **self.loader_kwargs).load()
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\langchain\document_loaders\unstructured.py", line 86, in load
elements = self._get_elements()
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\langchain\document_loaders\unstructured.py", line 172, in get_elements
return partition(filename=self.file_path, **self.unstructured_kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\partition\auto.py", line 406, in partition
elements = partition_md(
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\documents\elements.py", line 518, in wrapper
elements = func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\file_utils\filetype.py", line 604, in wrapper
elements = func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\file_utils\filetype.py", line 559, in wrapper
elements = func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\chunking_init.py", line 69, in wrapper
elements = func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\partition\md.py", line 104, in partition_md
return partition_html(
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\documents\elements.py", line 518, in wrapper
elements = func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\file_utils\filetype.py", line 604, in wrapper
elements = func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\file_utils\filetype.py", line 559, in wrapper
elements = func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\chunking_init.py", line 69, in wrapper
elements = func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\partition\html.py", line 141, in partition_html
document_to_element_list(
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\partition\common.py", line 559, in document_to_element_list
num_pages = len(document.pages)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\documents\xml.py", line 54, in pages
self._pages = self._parse_pages_from_element_tree()
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\documents\html.py", line 176, in _parse_pages_from_element_tree
element = _parse_tag(tag_elem)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\documents\html.py", line 421, in _parse_tag
return _text_to_element(
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\documents\html.py", line 469, in _text_to_element
elif is_narrative_tag(text, tag):
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\documents\html.py", line 517, in is_narrative_tag
return tag not in HEADING_TAGS and is_possible_narrative_text(text)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\partition\text_type.py", line 87, in is_possible_narrative_text
if "eng" in languages and (sentence_count(text, 3) < 2) and (not contains_verb(text)):
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\partition\text_type.py", line 189, in contains_verb
pos_tags = pos_tag(text)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\nlp\tokenize.py", line 44, in pos_tag
_download_nltk_package_if_not_present(
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\unstructured\nlp\tokenize.py", line 21, in _download_nltk_package_if_not_present
nltk.find(f"{package_category}/{package_name}")
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\nltk\data.py", line 555, in find
return find(modified_name, paths)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\nltk\data.py", line 542, in find
return ZipFilePathPointer(p, zipentry)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\nltk\compat.py", line 41, in _decorator
return init_func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\nltk\data.py", line 394, in init
zipfile = OpenOnDemandZipFile(os.path.abspath(zipfile))
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\nltk\compat.py", line 41, in _decorator
return init_func(*args, **kwargs)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\site-packages\nltk\data.py", line 935, in init
zipfile.ZipFile.init(self, filename)
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\zipfile.py", line 1269, in init
self._RealGetContents()
File "C:\Users\BlackSiao\miniconda3\envs\chatglm\lib\zipfile.py", line 1336, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
33%|███▎ | 1/3 [00:03<00:07, 3.60s/it]

System Info

My python version :3.10
Langchain：0.0.314
NLTK: 3.8.1

[dosubot[bot]]

🤖

Hey @BlackSiao! Great to see you diving into another adventure with LangChain. How's everything going?

Based on the error message you provided, it seems that the issue is related to the NLTK library when it tries to find a package and fails to open it as a zip file. This could be due to a corrupted NLTK data package.

To address this issue, I suggest manually downloading the NLTK packages. You can do this by running the following code:

import nltk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')

These packages ('punkt' and 'averaged_perceptron_tagger') are commonly used for tokenization and part-of-speech tagging, which might be used in the document loading process.

After downloading these packages, please try running your code again. If the error persists, it might be due to a different NLTK package. In that case, you might need to download the entire NLTK package collection using nltk.download('all'). Please be aware that downloading all NLTK packages will require a significant amount of disk space.

If you're still encountering issues, it might be helpful to check the file types in your directory. The DirectoryLoader in LangChain uses the UnstructuredFileLoader by default, which supports a variety of file types including .txt, .md, .html, .pdf, .docx, .pptx, .xlsx, .csv, .json, .xml, and .zip. If your directory contains a file type not supported by the UnstructuredFileLoader, you might encounter errors. In that case, you can specify a different loader class when initializing the DirectoryLoader.

For example, if your directory contains only .txt files, you can use the TextLoader:

from langchain.document_loaders import DirectoryLoader, TextLoader

loader = DirectoryLoader(directory, loader_cls=TextLoader, show_progress=True)

This will ensure that the DirectoryLoader uses the TextLoader to load your .txt files.

I hope this helps! If you have any other questions or if something is unclear, please let me know.

Sources:

• LangChain DirectoryLoader
• LangChain TextLoader
• NLTK Documentation

Sources

• UnstructuredMarkdownLoader resulting in zipfile.BadZipFile: File is not a zip file
• Error Loading Word Documents - File is not a zip file
• libs/community/langchain_community/document_loaders/directory.py
• libs/langchain/langchain/document_loaders/s3_directory.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[BlackSiao]

Thank you very much, this answear is very useful to me

[zkyuan]

我竟然看懂了，中国学生在做毕业设计碰到问题了。写了一堆英语，nb