Behavior of push_to_hub with create_pr=True with large number of files

[klamike]

When create_pr=True, push_to_hub ends up submitting several pull requests with one commit each rather than one pull request with several commits. Is that intentional? The latter seems more natural to me but not sure about downstream impacts.

Relevant snippet:

datasets/src/datasets/dataset_dict.py

Lines 1870 to 1905 in e8ee24a

	commit_message = commit_message if commit_message is not None else "Upload dataset"
	if len(additions) <= config.UPLOADS_MAX_NUMBER_PER_COMMIT:
	commit_info = api.create_commit(
	repo_id,
	operations=additions + deletions,
	commit_message=commit_message,
	commit_description=commit_description,
	token=token,
	repo_type="dataset",
	revision=revision,
	create_pr=create_pr,
	)
	else:
	logger.info(
	f"Number of files to upload is larger than {config.UPLOADS_MAX_NUMBER_PER_COMMIT}. Splitting the push into multiple commits."
	)
	num_commits = math.ceil(len(additions) / config.UPLOADS_MAX_NUMBER_PER_COMMIT)
	for i in range(0, num_commits):
	operations = additions[
	i * config.UPLOADS_MAX_NUMBER_PER_COMMIT : (i + 1) * config.UPLOADS_MAX_NUMBER_PER_COMMIT
	] + (deletions if i == 0 else [])
	commit_info = api.create_commit(
	repo_id,
	operations=operations,
	commit_message=commit_message + f" (part {i:05d}-of-{num_commits:05d})",
	commit_description=commit_description,
	token=token,
	repo_type="dataset",
	revision=revision,
	create_pr=create_pr,
	)
	logger.info(
	f"Commit #{i + 1} completed"
	+ (f" (still {num_commits - i - 1} to go)" if num_commits - i - 1 else "")
	+ "."
	)

Note how in the else branch, the create_commit call in the for loop has create_pr=create_pr -- I think it should be create_pr=False with a call to create_branch beforehand to pass the PR's revision?

Came across this working on a PR for #7527 -- it wouldn't be as much of a nuisance if push_to_hub worked like that.

[klamike]

Closing since convert to parquet has been removed... #7592 (comment)