Add fast sentence detection #6
Merged
+1,194
−27
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…int to ensure compatibility and enhance text processing capabilities in the project.
…pendencies: Enhance the clarity of the requirements section, specify the use of `uv` for installation, and detail the installation process for Hugging Face support and PyTorch, ensuring users have a better understanding of the setup process.
…ntroduce optional parameters for sentence detection, including language and cleaning heuristics, and refactor the function to handle segmented input more effectively. Update CLI to lazily load analyze_text and related functions for improved performance.
…ion handling to catch both ImportError and OSError, improving robustness in model loading. Additionally, include sentences module in processing exports for better accessibility.
… entity predictions by introducing a metadata field, allowing for additional contextual information. Update grouping logic to consider sentence indices and improve JSON output formatting to include metadata attributes, ensuring richer data representation.
…nce segmentation using pySBD, including the SentenceSpan class for representing sentences and their character boundaries. Implement caching for segmenter instances and fallback logic for span generation when character offsets are unavailable.
…e exception handling to catch both ImportError and OSError, improving robustness in tokenizer initialization.
…te text files, 'clinical_note.txt' and 'long_clinical_note.txt', to enhance test coverage for clinical documentation scenarios, ensuring comprehensive validation of processing functionalities.
…ion tests in `test_sentence_detection_real.py` to validate sentence detection functionality with real models, ensuring consistent behavior and proper handling of placeholder-only segments.
…ure `analyze_text`, `get_model_max_length`, and `list_models` are lazily imported, improving performance and allowing for easier testing by exposing these functions for patching without eager imports.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
Brief description of what this PR does.
Type of Change
Changes Made
Testing
Documentation
Code Quality
Dependencies
Checklist
Related Issues
Closes #(issue number)
Related to #(issue number)
Screenshots/Examples
If applicable, add screenshots or example outputs to help explain your changes.