Feature Proposal: Pager for large (lazy loading) tables

[oefe]

I have a few use cases where I need to browse large amounts of tabular data. They are provided by a REST API with pagination.

Rendering the entire data set into a Table via a Pager is not practical, as this would require loading all data upfront, would take a very long time to render, and the user won't see anything until everything is loaded and rendered.

Currently I load and display the data one (API) page at a time using a Live display. Then I wait for the user to press a key before proceeding to the next page. This works reasonably as long as one row of data renders as one line on screen (or at least a fixed number of lines). I can just request as many rows from the API as there are lines on the screen (minus room headers etc.) But if the row height varies, this doesn't work well – if I request too many rows, some data are cut off. If I don't request enough rows, there is empty unused space below the table.

I think this can be solved, but it requires a lot of bookkeeping and caching of already retrieved rows, and an efficient means to determine how many rows fit into the available screen space (a naive algorithm that just fills the table row by row and measures its height each time would have O(n²) performance. In a quick test, it started to feel slightly sluggish at about 20 rows). Therefore I believe this would be best done as a reusable component.

I'm thinking of creating such a component, but I have been wondering where it should go:

• Add it as a new component in Rich itself? This might make it easer to access interna of the way Rich measures and renders tables, which we might need to get good performance.
• As the pager would be an interactive feature, essentially a TUI, maybe it should be in Textual instead? I see that "Lazy loading API" is on the roadmap for the Textual DataTable. However, DataTable is not the same as a Rich Table. Also, I'm not familiar with Textual.
• Or should it be separate component built on top of Rich?

Thoughts?

[willmcgugan]

This is really an application for Textual. Rich renderables aren't really aware of height, there is no easy way to fit a table to the height of the terminal.

The DataTable widget doesn't have explicit support for lazy loading, but it wouldn't be that hard to implement. You could add rows in a background task, while the user is still able to view the loaded data.

The API is not too different from Rich Table, so I'm sure you could get up and running quickly.

[oefe]

Rich renderables aren't really aware of height, there is no easy way to fit a table to the height of the terminal.

Yeah, I noticed that. It seems that rich.Live has a solution, which I extracted into a function:

def measure_height(console: Console, table: Table) -> int:
    lines = console.render_lines(table)
    shape = Segment.get_shape(lines)
    _, height = shape
    return height

In DataTable, add_row takes a row height, with no "auto" option. So I have to tell it the row height upfront? In that case, I could simply add up all the row heights.

However, in my use case I don't know how tall the rows are going to be. There are some text fields in the data that could take multiple lines, depending on content and available horizontal space. So I rely on Table to adjust the row height dynamically.