Adds FileReader
#1499
Adds FileReader
#1499
Conversation
meta-cla
bot
added
the
CLA Signed
torchdata/nodes/io/file_read.py
Outdated
| import logging | ||
| from typing import Any, Dict, Optional | ||
|
|
||
| from smart_open import open |
There was a problem hiding this comment.
Can this open clash with Python's in-built open for some other operations?
Is it safer to use it as smart_open.open?
| except Exception as e: | ||
| logger.error(f"Error reading {file_path}: {e}") | ||
| raise | ||
|
|
There was a problem hiding this comment.
We should also add a shutdown method.
There was a problem hiding this comment.
similar question with #1498 - what is needed to be implemented there?
torchdata/nodes/io/file_read.py
Outdated
| encoding=None if "b" in self.mode else self.encoding, | ||
| transport_params=self.transport_params, | ||
| ) as f: | ||
| content = f.read() |
There was a problem hiding this comment.
- Is this use case for large files
- Does this line load all the contents at once
if yes for 1. and No for 2., then we might want to add a chunked reader or streamer.
There was a problem hiding this comment.
But I think that might be out of scope for this PR and we can work on that later.
There was a problem hiding this comment.
it includes large files. but i have a separate piece of code that does read text file line-by-line, without using this FileReader. i.e. text streamer.
agree that we those would be great to have as a separate PR.
torchdata/nodes/io/file_read.py
Outdated
|
|
||
| return {self.DATA_KEY: content, self.METADATA_KEY: metadata} | ||
|
|
||
| except Exception as e: |
There was a problem hiding this comment.
We can add a retry logic
what if there were some network hiccups, or some other retryable error
Is there a way to find out if the error was because of network (we can retry in that case) or some other factor (file not present, etc., so do not retry)
There was a problem hiding this comment.
sounds good, added in the new commit.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/data/1499
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit ec10460 with merge base b8eb16f ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
2 participants
similar to #1498, but for file reading.