Indexing API not reducing the ingestion time with OpenSearch with exactly the same input file

[arjunsohanlal]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

INDEX_NAME="test-vector-index-5"
VECTOR_FIELD='vector_field'

CONNECTION_STRING = "postgresql+psycopg://admin:admin@10.177.245.11:5433/vectordb"
COLLECTION_NAME = "vectordb"
namespace = f"aoss/{COLLECTION_NAME}"

# Setting up record manager on PostgreSQL
record_manager = SQLRecordManager(namespace, db_url=CONNECTION_STRING)
record_manager.create_schema()

# Initializing OpenSearch LangChain client
osvectorsearch = OpenSearchVectorSearch(
  embedding_function = embeddings,
  index_name = INDEX_NAME,
  http_auth = awsauth,
  use_ssl = True,
  verify_certs = True,
  http_compress = True, # enables gzip compression for request bodies
  connection_class = RequestsHttpConnection,
  opensearch_url="https://" + OPENSEARCH_ENDPOINT
)

cleanup = "incremental"

result = index(
    documents,
    record_manager,
    osvectorsearch,
    cleanup=cleanup.value,
    source_id_key="source",
    batch_size=10
)

Description

I've used the same code for LangChain's Indexing API with Weaviate and PGVector where the feature works as expected. For example, for 2185 documents to be ingested on the first go with cleanup='full', it would take about 10 minutes to do so. I'd modify a single character, and reingestion using cleanup=incremental would take a few seconds.

I'd tried the same with OpenSearch, where the first ingestion took around 700 seconds, and re-ingesting exactly the same data took about 590 seconds. Is there any reason why this might be happening?

System Info

LangChain package versions:

langchain                0.3.3
langchain-community      0.3.2
langchain-core           0.3.10
langchain-openai         0.2.2
langchain-text-splitters 0.3.0

Platform: Windows
Python Version: 3.12.4