feat: allow mime_type to be guessed for ByteStream #9573

kanenorman · 2025-06-30T22:47:35Z

Related Issues

fixes feat: allow mime_type to be guessed for ByteStream #9563

Proposed Changes:

Add guess_mime_type parameter to ByteStream
Refactor FileTypeRouter
Add guess_mime_type parameter to haystack.components.converters.utils.get_bytestream_from_source

How did you test it?

Added unit test for ByteStream
Added unit test for get_bytestream_from_source
Did not change test for FileTypeRouter

Notes for the reviewer

This is a breaking change so would like feedback 😄

Checklist

I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
I documented my code
I ran pre-commit hooks and fixed any issue

julian-risch

Hello @kanenorman Thank you for opening this pull request!
I left a few comments and also saw that three unit tests are failing here:

FAILED test/components/converters/test_multi_file_converter.py::TestMultiFileConverter::test_run_error_handling - haystack.core.errors.PipelineRuntimeError: The following component failed to run:
Component name: 'router'
Component type: 'FileTypeRouter'
Error: [Errno 2] No such file or directory: '/home/runner/work/haystack/haystack/test/test_files/non_existent.txt'
FAILED test/components/routers/test_file_router.py::TestFileTypeRouter::test_unlisted_extensions - FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/haystack/haystack/test/test_files/audio/ignored.mp3'
FAILED test/components/routers/test_file_router.py::TestFileTypeRouter::test_no_extension - FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/haystack/haystack/test/test_files/txt/doc_2'

Please let me know if you need help looking into those. Other than need the pull request only needs a release note and we describe how you can generate one here. Please let me know in case you need help with that.
I was wondering why you wrote in the pull request description that it is a breaking change? With guess_mime_type being False as default and with FileTypeRouter using guess_mime_type True, the behavior remains the same, no?

Other than that the pull request looks quite good to me already! 🙂

julian-risch · 2025-07-01T14:49:37Z

haystack/components/converters/utils.py

@@ -8,20 +8,24 @@
 from haystack.dataclasses import ByteStream


-def get_bytestream_from_source(source: Union[str, Path, ByteStream]) -> ByteStream:
+def get_bytestream_from_source(
+    source: Union[str, Path, ByteStream], guess_mime_type: Optional[bool] = False


Given that we have False as a default value, there is no reason to make this Optional. Let's change the type to bool

Suggested change

source: Union[str, Path, ByteStream], guess_mime_type: Optional[bool] = False

source: Union[str, Path, ByteStream], guess_mime_type: bool = False

and in from_file_path you also use bool instead of Optional[bool]

julian-risch · 2025-07-01T15:04:29Z

test/components/converters/test_utils.py

@@ -31,3 +33,38 @@ def test_normalize_metadata_list_of_wrong_size():
 def test_normalize_metadata_other_type():
    with pytest.raises(ValueError, match="meta must be either None, a dictionary or a list of dictionaries."):
        normalize_metadata(({"a": 1},), sources_count=1)
+
+
+def test_get_bytestream_from_path_object(tmp_path):


In addition to the test cases here, I suggest we add a test case for FileTypeRouter in test/components/routers/test_file_router.py checking that the parameter additional_mimetypes still works as expected. We're doing mimetypes.add_type(mime, ext) there and need to check that when get_bytestream_from_source it's taking into account these additional mimetypes, for example: {"application/vnd.openxmlformats-officedocument.wordprocessingml.document": ".docx"}.

kanenorman · 2025-07-02T04:35:24Z

Thanks for the feedback @julian-risch. I’ll get the changes implemented shortly.

I was wondering why you wrote in the pull request description that it is a breaking change? With guess_mime_type being False by default and FileTypeRouter using guess_mime_type=True, the behavior remains the same, no?

You’re absolutely right. I misspoke. Thanks for pointing that out! 😄

Regarding the failing unit test, I’ve identified the issue and will push a fix.

Before proceeding, I reviewed the latest haystack-ai source code (v2.15.1) and noticed that FileTypeRouter raises a FileNotFoundError only when a non-existent file is passed with the meta parameter. Without meta, the component does not raise an error and still classifies the file based on its extension.

Here’s a minimal example:

python-version: 3.11.3
mac-os: Sequoia 15
haystack-ai: 2.15.1

from haystack.components.routers import FileTypeRouter

router = FileTypeRouter(mime_types=[r'text/plain'])

# No meta - does not raise error
router.run(sources=["non_existent.txt"])
# → {'text/plain': [PosixPath('non_existent.txt')]}

# With meta - raises FileNotFoundError
router.run(sources=["non_existent.txt"], meta={"spam": "eggs"})
# → FileNotFoundError: [Errno 2] No such file or directory: 'non_existent.txt'

This feels inconsistent. Should metadata change how we treat missing files? Should the component:

Maintain existing behavior?
Skip missing files?
Raise an error in all cases?
Still classify the path even if the file doesn't exist?

Happy to align with whatever behavior is intended, just wanted to clarify before committing a fix.

julian-risch · 2025-07-07T13:44:56Z

Hi @kanenorman thanks for your patience! We discussed in the team and concluded that for now we would like to keep the inconsistency and not fix it in this PR. That makes this PR a bit more complex but doesn't break any current behavior.
We'll then open a separate issue on the inconsistency and plan to have a deprecation warning in the 2.16.0 release, and consistent behavior in 2.17.0.

coveralls · 2025-07-11T08:41:32Z

Pull Request Test Coverage Report for Build 16437756318

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.01%) to 90.464%

Totals
Change from base Build 16421645889:	0.01%
Covered Lines:	12077
Relevant Lines:	13350

💛 - Coveralls

julian-risch

Looks good to me! 👍 Congratulations on your first pull request merged into Haystack!
The pull request now doesn't introduce any breaking changes, which is great. How you handled windows/unix mimetype differences in the test makes sense to me.

kanenorman added 3 commits June 30, 2025 14:05

feat(bytestream): add guess_mime_type parameter

239766f

refactor(FileTypeRouter): refactor guess mimetype

6284787

feat(bytestream): add guess_mime_type to util

ef15e95

kanenorman requested a review from a team as a code owner June 30, 2025 22:47

kanenorman requested review from julian-risch and removed request for a team June 30, 2025 22:47

github-actions bot added topic:tests type:documentation Improvements on the docs labels Jun 30, 2025

style(ruff): add trailing whitespace

ac399fb

julian-risch requested changes Jul 1, 2025

View reviewed changes

kanenorman added 2 commits July 1, 2025 11:47

fix: fix type annotation

8096fe6

test(file_type_router): add test for additional_mimetypes param

18946c9

kanenorman added 2 commits July 7, 2025 10:14

fix(file_type_router): non-existent file behavior

bff0c07

feat(file_type_router): add release notes

a5a2e2d

kanenorman requested a review from a team as a code owner July 7, 2025 15:43

kanenorman requested review from dfokina and removed request for a team July 7, 2025 15:43

kanenorman added 4 commits July 7, 2025 10:52

Merge branch 'main' into feat_bytestream_guess_mime_type

a375a0b

fix(file_type_router): remove unused logger

8b0908c

Merge branch 'main' into feat_bytestream_guess_mime_type

250bca7

style: fix ruff formatting magic values

8da2267

kanenorman and others added 2 commits July 11, 2025 10:02

test(bytestream): handle windows/unix mimetype differences

a32aac8

Merge branch 'main' into feat_bytestream_guess_mime_type

92e794a

julian-risch approved these changes Jul 22, 2025

View reviewed changes

julian-risch changed the title ~~feat: bytestream guess mime type~~ feat: allow mime_type to be guessed for ByteStream Jul 22, 2025

julian-risch enabled auto-merge (squash) July 22, 2025 07:38

julian-risch merged commit 9420492 into deepset-ai:main Jul 22, 2025
19 checks passed

anakin87 mentioned this pull request Jul 22, 2025

feat: Add DocumentTypeRouter #9634

Merged

julian-risch mentioned this pull request Jul 22, 2025

FileTypeRouter handles non-existent files inconsistently #9638

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: allow mime_type to be guessed for ByteStream #9573

feat: allow mime_type to be guessed for ByteStream #9573

Uh oh!

kanenorman commented Jun 30, 2025

Uh oh!

julian-risch left a comment

Uh oh!

julian-risch Jul 1, 2025

Uh oh!

julian-risch Jul 1, 2025

Uh oh!

julian-risch Jul 1, 2025

Uh oh!

kanenorman commented Jul 2, 2025

Uh oh!

julian-risch commented Jul 7, 2025

Uh oh!

coveralls commented Jul 11, 2025 •

edited

Loading

Uh oh!

julian-risch left a comment

Uh oh!

Uh oh!

Uh oh!

	source: Union[str, Path, ByteStream], guess_mime_type: Optional[bool] = False
	source: Union[str, Path, ByteStream], guess_mime_type: bool = False

feat: allow mime_type to be guessed for ByteStream #9573

feat: allow mime_type to be guessed for ByteStream #9573

Uh oh!

Conversation

kanenorman commented Jun 30, 2025

Related Issues

Proposed Changes:

How did you test it?

Notes for the reviewer

Checklist

Uh oh!

julian-risch left a comment

Choose a reason for hiding this comment

Uh oh!

julian-risch Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

julian-risch Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

julian-risch Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

kanenorman commented Jul 2, 2025

Uh oh!

julian-risch commented Jul 7, 2025

Uh oh!

coveralls commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 16437756318

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Uh oh!

julian-risch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coveralls commented Jul 11, 2025 •

edited

Loading