[Proposal] Out-of-tree Document Loader

[yaohui-wyh]

Background

Document Loaders are currently all "in-tree," meaning that their implementation is included in the LangChain repository. This increases the size of the LangChain framework and potentially introduces security risks, particularly for poorly implemented Document Loaders that fetch and parse remote resources.

This also places a heavy burden on LangChain contributors and Data vendors to upgrade or fix the current Loader implementations. A significant portion of issues & pull requests are related to Document Loaders, as can be seen here: https://github.com/hwchase17/langchain/issues?q=document+loader

Proposal (Draft)

Similar to the Kubernetes CSI Volume Plugin, we could draft a specification for the "LangChain Document Loader" plugin and allow the development & release of Document Loaders outside of the core LangChain codebase/release.

This would be beneficial for the LangChain ecosystem and also increase the production stability.

The Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. Using CSI third-party storage providers can write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code.

• CSI Volume Plugins in Kubernetes Design Doc kubernetes/community#1258
• https://kubernetes.io/blog/2021/12/10/storage-in-tree-to-csi-migration-status-update/