parse: support partitioning files before parsing

[pmrowla]

• Adds LlamaParse.partition_pages to allow partitioning multiple page docs into separate parsing jobs
  • Defaults to None (docs are not partitioned)
  • When set, docs will be split into segments of up to partition_pages and each segment will be parsed in a separate job
  • Works in conjunction with target_pages and max_pages
• When parsing multiple documents, partition segments will be parsed sequentially per document (num_workers batching is handled at the file level, and segments within a single file are not split into additional batches)
• When parsing a single document, partitioned segment jobs will be batched by num_workers if target_pages is set. When target_pages is None, segments are parsed sequentially. This is done since we have to naively try to parse each segment until we either reach max_pages or error out from trying to parse past the end of the file.

Known limitations:

• Docs are re-uploaded for each partitioned segment. This can be resolved once parse can reference previously uploaded docs in the backend.
• Finding the end of a document is naive in order to avoid needing to open docs locally (and adding dependencies to handle determining page count for supported doc types). This could be resolved if we had a way to retrieve total page count for previously uploaded docs via the API

Requires:

• parse: fix num_workers/parse job batching #708

[pmrowla]

May be worth considering returning pagination information as a part of job_metadata in the API? Currently we return job_pages which corresponds to the number of pages which were parsed (but may be less than the actual number of pages returned in the response if this was a cache hit)

[logan-markewich]

The methods in this lib are starting to get very confusing to follow 😅 Going to checkout the code locally to better review this...

[logan-markewich]

If we already have the JSON result, no need to make another API call right? We can get any info we need out of the JSON?

[pmrowla]

fixed in 0d6c3bf

[logan-markewich]

If specific errors are expected, we should catch them explicitly

[pmrowla]

fixed in 72b8b63

[logan-markewich]

oh boy this loop took me a while to understand. To confirm

1. The user has enabled parition_pages
2. The user has set max_pages
3. The user has not set target pages

So, we parse in batches of pages, until we hit max pages, and return the result

Is that correct?

[pmrowla]

Yes that's correct.

The issue is that we don't have a way to get the total number of pages in the doc locally. So in the case where the user wants to parse the entire doc (and is not using target_pages), we have to take the naive approach and loop and parse each batch of pages until either:

• we reach max_pages if it is set by the user
• we reach EOF, which can be indicated by either:
  • parse returns fewer pages than the batch range we requested, meaning the requested range starts on a valid page and then runs past the end of the doc
  • API returns a page does not exist error, meaning the requested range starts past the end of the doc (this should only happen when the previous range ends exactly on the last page of the doc)

In the future, we can simplify this behavior if we actually return the total number of available pages within a job result (and then we would know exactly how many remaining batches we need to request, and can run them concurrently instead of doing this sequential loop)