"EOFError: end of file reached" during manual chunking

[khaledh]

We're hitting an issue when an object is backfilled in Salesforce, causing 13M+ records to be retrieved on the next extraction (i.e. similar to a full import). The job uses manual chunking, and fails at the first step when it's querying all IDs in preparation for chunking (as opposed to querying the actual data). See the stack trace below.

From the logs it seems that the batch for fetching the IDs succeeds, but when actually fetching the CSV list of IDs, it takes about 160 seconds before failing with EOFError: end of file reached, which is caused by IOError: Connection reset by peer. It sounds like Salesforce is terminating the connection? I tried to find docs on the default request timeout when retrieving results, but couldn't find it.

The CSV response is about 340MB uncompressed, and when manually downloaded (through the Bulk Data Load Jobs UI) takes only several seconds to download. Not sure why it takes over 2.5 minutes to download via the gem.

[khaledh]

@csholmes Is this related to #50? Any idea how to proceed?

[csholmes]

I suspect this is either a JRuby or a Salesforce issue. I'm having trouble reproducing the issue though, except for once actually I caught the same issue with PK Chunking trying to download a very large batch.

It doesn't seem to take that long "normally", when there isn't an issue. It takes less than a minute to download and decompress.

To attempt to solve this issue, I added IOError to the retry handler: #69

[khaledh]

@csholmes OK, thanks. We switched to PK chunking a while ago to avoid hitting this issue. We currently have one job that uses manual chunking, and the total data size is in the range of 150K, so most probably it's not going to be impacted. But thanks for adding the retry for IOError, just in case.