DirectoryTree Supporting Different File Systems (AWS S3, Google Cloud Storage, Azure Storage)

[juftin]

I've been spinning my wheels trying to get the DirectoryTree widget to play nicely with AWS S3 using boto3 and I realized there's an easier way to do that, with fsspec and s3fs.

fsspec is a "specification for pythonic filesystems" and s3fs is an implementation of fsspec for AWS S3.

fsspec also has a great extension, universal_pathlib, which extends the fsspec to the pathlib API. It also supports other filesystems like Google Cloud Storage, HDFS, Azure Stoage, and more.

Put them all together, and you have a really simple way to extend the DirectoryTree API where the only difference is swapping pathlib.Path for upath.UPath.

"""
Universal File Browser
"""

import pathlib
import sys

from textual.app import App, ComposeResult
from textual.containers import Container
from textual.widgets import DirectoryTree, Footer, Header, TreeNode
from textual.widgets._directory_tree import DirEntry
from upath import UPath


class UniversalDirectoryTree(DirectoryTree):
    """
    A Universal DirectoryTree supporting different filesystems
    """

    @classmethod
    def load_directory(cls, node: TreeNode[DirEntry]) -> None:
        """
        Load Directory Using Universal Pathlib
        """
        assert node.data is not None
        dir_path = UPath(node.data.path)
        node.data.loaded = True
        directory = sorted(
            list(dir_path.iterdir()),
            key=lambda path: (not path.is_dir(), path.name.lower()),
        )
        for path in directory:
            node.add(
                path.name,
                data=DirEntry(str(path), path.is_dir()),
                allow_expand=path.is_dir(),
            )
        node.expand()


class UniversalFileBrowser(App):
    """
    Textual File Browser App.
    """

    BINDINGS = [
        ("q", "quit", "Quit"),
        ("d", "toggle_dark", "Toggle dark mode"),
    ]

    def compose(self) -> ComposeResult:
        """
        Compose our UI.
        """
        yield Header()
        target_file = str(pathlib.Path.cwd() if len(sys.argv) == 1 else sys.argv[1])
        yield Container(UniversalDirectoryTree(target_file, id="tree-view"))
        yield Footer()


if __name__ == "__main__":
    UniversalFileBrowser().run()

Finally, run it on an S3 bucket. For this example I found a public S3 bucket using a quick Google search:

pip install "textual[dev]" s3fs universal_pathlib

or if you're more inclined to pipx:

pipx install "textual[dev]"
pipx inject textual s3fs universal_pathlib

textual run "universal_directory_tree.py s3://ryft-public-sample-data/"

Hopefully this is useful for some TUIs out there 🎉

[juftin]

Here it is with Google Cloud Storage and one of Google's public buckets:

pipx install "textual[dev]"
pipx inject textual gcsfs universal_pathlib

textual run "universal_directory_tree.py gs://gcp-public-data-arco-era5"