DirectoryLoader Slow when there are too many exclusions

[siyanda4navilabs]

I've been trying to upload a lot of files (~15/20k) while also trying to exclude files (100k) that have been already uploaded, let's say something like:

 loader_pdf = DirectoryLoader(
        dirs,
        glob="*.pdf",
        loader_cls=PyPDFLoader,
        show_progress=True,
        use_multithreading=False,
        recursive=True,
        silent_errors=True,
        exclude=excluded

it was taking days which was quite unusual and was hanging, then I saw this:

        items = [
            path
            for path in paths
            if not (self.exclude and any(path.match(glob) for glob in self.exclude))
            and path.is_file()
        ]

if both self.exclude and paths are large (like in my case )this becomes O( N*M ~N^2).

I propose something that would be a bit faster than this:
import fnmatch

    self.exclude_patterns = [re.compile(fnmatch.translate(p)) for p in self.exclude]
    
    
    items = [path for path in paths
    if not (self.exclude and any(pattern.match(str(path)) for pattern in self.exclude_patterns))
    and path.is_file()]

This solution will still not completely resolve the problem but using-fnmatch will definitely reduce it