Skip to content

Speed up tokenization #1601

@rien

Description

@rien

The tokenization performed by tree-sitter can be slow for large datasets: up to 50% or even more is spent on this part.

Multiple improvements are possible:

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions