feat(data_frame): Add .update_data(data, *, reset) and .update_cell_value(value, *, row, col)

[maxmoro]

In Shiny-R I often use the dataTableProxy() to manipulate the data shown in the DT, so the view is changed without reloading/regenerating the entire table.
Is it possible to do it with the data_frame in Python?

[maxmoro]

I'm trying to edit the data in the data_frame, without re-render the data_frame output, so I can keep the filter/sort the user performed.

Here is a test code, where when pressing the 'click' button, the second column of the second line should become 'xxx'. But it is not working

I'm using the .set_patches_fn inside the click event. I know it is not "by-the-book", I'm just wondering if it is possible in some way.

shiny Live link

and the code

from palmerpenguins import load_penguins
from shiny import App, render, ui, reactive, Outputs

penguins = load_penguins()

app_ui = ui.page_fluid(
    ui.h2("Palmer Penguins"),
    ui.input_action_button('click','click'),
    ui.output_ui("rows"),
    ui.output_data_frame("penguins_df"),
)

def server(input, output, session):
    

    @render.data_frame
    def penguins_df():
        return render.DataTable(penguins, editable=True,)  

    @reactive.Effect
    @reactive.event(input.click)
    def click():
        print('click')
        def edtx() -> list[render.CellPatch]():
            out = ({'row_index': 1, 'column_index': 1, 'value': 'xxx'},)
            print(out)
            return(out)
            
        penguins_df.set_patches_fn(edtx())
        
    #just testing if it works when editing another cell
    @penguins_df.set_patches_fn
    def edt(*, patches: list[render.CellPatch]) -> list[render.CellPatch]:
        out = ({'row_index': 1, 'column_index': 1, 'value': 'e'},)
        return(out)


app = App(app_ui, server)

[schloerke]

I've updated your app to use the new patches handler after being clicked.

There were two subtle changes:

1. Do not call your function when setting it. Ex: penguins_df.set_patches_fn(edtx)
2. Add *, patches parameters to edtx

Final app (shinylive):

from palmerpenguins import load_penguins

from shiny import App, reactive, render, ui

penguins = load_penguins()

app_ui = ui.page_fluid(
    ui.h2("Palmer Penguins"),
    ui.input_action_button("click", "click"),
    ui.output_ui("rows"),
    ui.output_data_frame("penguins_df"),
)


def server(input, output, session):

    @render.data_frame
    def penguins_df():
        return render.DataTable(
            penguins,
            editable=True,
        )

    @reactive.Effect
    @reactive.event(input.click)
    def click():
        print("click")

        def edtx(*, patches) -> list[render.CellPatch]:
            print("new!")
            out = [
                render.CellPatch(({"row_index": 1, "column_index": 1, "value": "xxx"}))
            ]
            print(out)
            return out

        penguins_df.set_patches_fn(edtx)

    # just testing if it works when editing another cell
    @penguins_df.set_patches_fn
    def edt(*, patches: list[render.CellPatch]) -> list[render.CellPatch]:
        print("original")
        out = [render.CellPatch(({"row_index": 1, "column_index": 1, "value": "e"}))]
        return out


app = App(app_ui, server)

---

I saw that the original cell location never escaped a saving state. This is being addressed in #1529 .

[schloerke]

In Shiny-R I often use the dataTableProxy() to manipulate the data shown in the DT, so the view is changed without reloading/regenerating the entire table.
Is it possible to do it with the data_frame in Python?

It is definitely a possible feature! I have a sketch of what it could look like here:

py-shiny/shiny/render/_data_frame.py

Lines 719 to 739 in 6611277

	# TODO-barret-render.data_frame; Add `update_cell_value()` method
	# def _update_cell_value(
	# self, value: CellValue, *, row_index: int, column_index: int
	# ) -> CellPatch:
	# """
	# Update the value of a cell in the data frame.
	#
	# Parameters
	# ----------
	# value
	# The new value to set the cell to.
	# row_index
	# The row index of the cell to update.
	# column_index
	# The column index of the cell to update.
	# """
	# cell_patch_processed = self._set_cell_patch_map_value(
	# value, row_index=row_index, column_index=column_index
	# )
	# # TODO-barret-render.data_frame; Send message to client to update cell value
	# return cell_patch_processed

It is currently not implemented as line 738 hints that we need a "send message to the browser" action that is not implemented in the typescript code. It would be similar to how we can update the sort from the server:

• TS hook:

  py-shiny/js/data-frame/index.tsx

  Lines 430 to 462 in 6611277

  	useEffect(() => {
  	const handleColumnSort = (
  	event: CustomEvent<{ sort: { col: number; desc: boolean }[] }>
  	) => {
  	const shinySorting = event.detail.sort;
  	const columnSorting: SortingState = [];
  	shinySorting.map((sort) => {
  	columnSorting.push({
  	id: columns[sort.col]!,
  	desc: sort.desc,
  	});
  	});
  	setSorting(columnSorting);
  	};
  	if (!id) return;
  	const element = document.getElementById(id);
  	if (!element) return;
  	element.addEventListener(
  	"updateColumnSort",
  	handleColumnSort as EventListener
  	);
  	return () => {
  	element.removeEventListener(
  	"updateColumnSort",
  	handleColumnSort as EventListener
  	);
  	};
  	}, [columns, id, setSorting]);

• Python code:

  py-shiny/shiny/render/_data_frame.py

  Lines 939 to 942 in 6611277

  	await self._send_message_to_browser(
  	"updateColumnSort",
  	{"sort": vals},
  	)

Note: This could also be something similar to update_data(self, data: DataFrameLikeT), but the required infrastructure code changes would be similar.

---

In Shiny-R I often use the dataTableProxy() ....

I do not believe a proxy object will be created within py-shiny. However, Python is pass by reference and we can empower our renderers to have extra methods. These extra methods, (e.g. .data_view() or .update_sort() or even .update_data()) should cover the benefits of proxy object.

---

One open question that I had was "how should the updates be supplied?". Should it be at the cell level or at the "whole data frame" level?

• Cell
  • Efficient and precise
  • Harder to work with as a user. Must retrieve to row, col, value info for every cell.
• Whole data frame
  • Inefficient. Will need to send the whole data frame to the browser
  • Comfortable to work with as a user. Keeps the interface transaction as data frame in and data frame out

Thoughts?

[maxmoro]

Thank you for your prompt reply. Here are a couple of examples of common use cases I can think of:

1. Single Row Edit: A Shiny app displays a list (data frame). The user selects a row and clicks an "Edit" button. A form appears, allowing the user to modify the selected row's information. The form handles the editing logic. When the user clicks "OK," the changes are applied to the row in the list.
2. Full Table Refresh: The user triggers a refresh or recalculation of the entire table. The table needs to be reloaded from scratch with updated data.

In the first case, editing at the cell level is the most efficient and streamlined approach. The second case requires a full table refresh, so resetting the entire data frame is quicker. (reactive on the @render.data_frame) But the user will lose the filters and sort. (even if the new options in 1.0 will help to reset them)

Based on my experience, I would recommend prioritizing cell-level updates . Whole-table refreshes could be a second priority.

One open question that I had was "how should the updates be supplied?". Should it be at the cell level or at the "whole data frame" level?

• Cell

  • Efficient and precise
  • Harder to work with as a user. Must retrieve to row, col, value info for every cell.

• Whole data frame

  • Inefficient. Will need to send the whole data frame to the browser
  • Comfortable to work with as a user. Keeps the interface transaction as data frame in and data frame out

Thoughts?

[maxmoro]

I do not believe a proxy object will be created within py-shiny. However, Python is pass by reference and we can empower our renderers to have extra methods. These extra methods, (e.g. .data_view() or .update_sort() or even .update_data()) should cover the benefits of proxy object.

I fully agree, I think Python's by-reference approach is very useful and easy to code with. I intuitively built an App where the edit of a cell triggers other cells to change, just using the referenced data set (.data() and .data_view()).

[schloerke]

Currently, when a @render.data_frame function executes these qualities are reset:

• column filtering
• column sorting
• selected rows
• user edits

I believe not losing these qualities are the root of the issue.

• Having update_cell_value() would allow us to shim in new values and not reset any of the qualities. (We're in agreement)
• Having a update_data() method could update the data and accept the qualities as parameters to opt-in/opt-out of being updated. If the qualities are maintained, then we could require certain data characteristics to be consistent between the old and the new data. Seems reasonable to reset all at once and not individual qualities. If anything, their corresponding update method can be run.
• I currently taking the stance that every time a renderer function runs, all qualities are reset. Having a renderer run as an update would be a new pattern and I'd like to avoid it.

Pseudo code

def update_data(self, data: DataFrameLikeT, *, reset: bool | None = None) -> None:
    if reset is True:
		# everything naturally resets
		...
	else:
		action_type = warning if reset is None else error # error when reset = False
		for each quality, display any error messages with the action_type
			verify new data and old data have same column types
			verify all existing edits and selections are within the bounds of the new data

	Send message to browser with `data` and `reset`

and for completeness

def update_cell_value(self, value: TagNode, *, row: int, col: int) -> None:
    Add cell patch info to internal cell patches dictionary
    It feels off to call the currently set patches method on this submitted value
    client_value = maybe_as_cell_html(value)
	Send message to browser with `value: client_value`, `row`, and `col`

[maxmoro]

I agree with your points. Your pseudo-code would be awesome. It would streamline the process (creation vs. editing vs. refresh data), keep it simple to code, and avoid getting lost in the @render reactivity (in R we need to use Isolate to avoid re-triggering the rendered)
Thanks!

[kwa]

I think my comment #1560 (comment) probably more applies to this discussion.

This part:

Currently, when a @render.data_frame function executes these qualities are reset:

column filtering
column sorting
selected rows
user edits

I believe not losing these qualities are the root of the issue.

, is also the problem I want to solve.

My use case does not involve changing the data in any cells only controlling what the underlying dataframe in the datagrid component is compared to the original and how it is displayed.
I want access to

1. The original dataframe df (possibly trivial but still, is it available as .data() perhaps)
2. Be able to construct the modified dataframe df_mod which includes
   2.2 The column sorting, selected rows, and user filters (edits are not important for my use case, but probably is in general)
   2.3 External sorting and filtering I want to apply together with 2.2
3. Render df_mod and
4. keep the state of df_mod somewhere

Currently, my external state is (this is a sample):

• a reactive data-range select so that I can filter a column based on multiple date values.
• a reactive group by where a selectize input can choose multiple columns and the order of selecting columns matter
• a reactive sort where a selectize input can choose multiple columns to sort by and the order of selecting columns matter

So the use case is when choosing external state parameters and changing df_mod, so that the dataframe re-renders, I lose the user choices inside the dataframe widget. I want to use my external state + the widget's current inputs simultaneously to decide how the new df_mod should be constructed and rendered. Currently, the user needs to manually re-apply all widget selections as soon as an external input is changed due to re-rendering.

[schloerke]

@kwa

Using the example above of

    @render.data_frame
    def penguins_df():
        return render.DataTable(
            penguins,
            editable=True,
        )

I want access to

1. The original dataframe df (possibly trivial but still, is it available as .data() perhaps)

Correct. This would be available as penguins_df.data()

2. Be able to construct the modified dataframe df_mod which includes
   2.2 The column sorting, selected rows, and user filters (edits are not important for my use case, but probably is in general)
   2.3 External sorting and filtering I want to apply together with 2.2

"Be able to construct the modified data frame". Hopefully this will be resolved with .update_cell() and .update_data() as described above.

3. Render df_mod and

This the .data_view() of the previous data frame can be used to render another data frame. Ex:

    @render.data_frame
    def penguins_df_other():
        return render.DataTable(penguins.data_view())

4 keep the state of df_mod somewhere

Example of using the modified data from the first data frame to display a second data frame:

py-shiny/shiny/api-examples/data_frame_data_view/app-express.py

Line 47 in fa9f8d4

df_original.data_view(),

[schloerke]

Added .update_cell_value(value, *, row: int, col: int | str) and .update_data(data) in #1719. This will be included in the next release: v1.2

Note: Support for .update_data(*, reset=) was dropped as it has the same effect as re-running the render method (which can be triggered through any reactive update).

[kwa]

@schloerke
I had a look at https://github.com/posit-dev/py-shiny/pull/1719/files#diff-45a5511a939e4432331a2418695d983ace4c5f12b12c94a0a3e5ecd2f386ba6f and it seems very similar to what I want to do.

When is the 1.2 release planned?

Or What is a good option to install the unreleased code? -->

# First install htmltools, then shiny
pip install git+https://github.com/posit-dev/py-htmltools.git#egg=htmltools
pip install git+https://github.com/posit-dev/py-shiny.git#egg=shiny

perhaps?