What is now the intended method of compiling/using a language that does not have a pypi package?

[MartV0]

Now that Language.build_library is deprecated since version 0.22.0 what is the intended way to parse a language that does not have a dedicated pypi package. In my case i would like to parse R. Any help would be greatly appreciated.

[ObserverOfTime]

In the general case: submit an issue or PR to the language repo.

In your case:

pip install git+https://github.com/r-lib/tree-sitter-r@next

[doyled-it]

I do not like this solution. Now we are relying on all other language libraries to create Python packages and upload them to PyPi?

I have access to so many other tree-sitter grammars if I can build my own .so. For example, this part of my project dynamically creates it based on the OS:

    def _load_parser(self) -> None:
        """Load the parser for the given language.

        Sets `self.parser`'s language to the one specified in `self.language`.
        """
        # Get the directory to store the file in from environment (or default)
        build_dir: Path = Path.home() / ".janus/tree-sitter/build-files"
        if (custom_dir := os.environ.get("TREE_SITTER_BUILD_DIR")) is not None:
            build_dir = Path(custom_dir)

        # Locate the .so file, generate the file if necessary
        platform_str = f"{platform.system()}_{platform.processor()}"
        so_file = build_dir / f"{self.language}_parser_{platform_str}.so"
        if not so_file.exists():
            log.warning(
                f"Could not load {so_file}, building one for {platform.system()} "
                f"system, with {platform.processor()} processor"
            )
            self._create_parser(so_file)

        # string required for Windows, as 'WindowsPath' is not iterable
        so_file = str(so_file)

        # Load the parser using the generated .so file
        self.parser: tree_sitter.Parser = tree_sitter.Parser()
        pointer = self._so_to_pointer(so_file)
        self.parser.set_language(tree_sitter.Language(pointer, self.language))

    def _so_to_pointer(self, so_file: str) -> int:
        """Convert the .so file to a pointer.

        Taken from `treesitter.Language.__init__` to get past deprecated warning.

        Arguments:
            so_file: The path to the so file for the language.

        Returns:
            The pointer to the language.
        """
        lib = cdll.LoadLibrary(os.fspath(so_file))
        # Added this try-except block to handle the case where the language is not
        # supported in lowercase by the creator of the grammar. Ex: COBOL
        # https://github.com/yutaro-sakamoto/tree-sitter-cobol/blob/main/grammar.js#L13
        try:
            language_function = getattr(lib, f"tree_sitter_{self.language}")
        except AttributeError:
            language = self.language.upper()
            language_function = getattr(lib, f"tree_sitter_{language}")

        language_function.restype = c_void_p
        pointer = language_function()
        return pointer

    def _create_parser(self, so_file: Path | str) -> None:
        """Create the parser for the given language.

        Arguments:
            so_file: The path to the so file for the language.
        """
        # Store the library in the `build` directory
        tree_sitter_dir = Path.home() / ".tree-sitter"
        tree_sitter_dir.mkdir(exist_ok=True)
        lang_dir = tree_sitter_dir / f"tree-sitter-{self.language}"

        if not lang_dir.exists():
            github_url = LANGUAGES[self.language]["url"]
            if github_url is None:
                message = f"Tree-sitter does not support {self.language} yet."
                log.error(message)
                raise ValueError(message)
            if LANGUAGES[self.language].get("branch"):
                self._git_clone(github_url, lang_dir, LANGUAGES[self.language]["branch"])
            else:
                self._git_clone(github_url, lang_dir)

        tree_sitter.Language.build_library(str(so_file), [str(lang_dir)])

But with the new versions of py-tree-sitter (>0.22.0) we're relegated to a much smaller subset of languages. And we're having issues like this when running on some systems (that I think would be fixed if we upgraded to the new py-tree-sitter versions. But, obviously, we can't because we would severely limit the number of languages that our app can support):

>       self.parser.set_language(tree_sitter.Language(pointer, self.language))
E       ValueError: Incompatible Language version 15. Must be between 13 and 14

I think it's an unrealistic expectation to require these languages repositories to create Python packages and upload them to PyPi, because tree-sitter grammars aren't only used by py-tree-sitter. It makes sense to add back the backwards compatibility to point to our own .so files.

[Akuli]

You might like https://pypi.org/project/tree-sitter-language-pack/.