[SPARK-51560] Support cache/persist/unpersist for DataFrame

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to support cache, persist, and unpersist for DataFrame.

Why are the changes needed?

For feature parity.

Does this PR introduce any user-facing change?

No. This is a new addition.

How was this patch tested?

Pass the CIs.

$ swift test --filter DataFrameTests
...
􀟈  Test run started.
􀄵  Testing Library Version: 102 (arm64e-apple-macos13.0)
􀟈  Suite DataFrameTests started.
􀟈  Test orderBy() started.
􀟈  Test isEmpty() started.
􀟈  Test show() started.
􀟈  Test persist() started.
􀟈  Test showCommand() started.
􀟈  Test table() started.
􀟈  Test selectMultipleColumns() started.
􀟈  Test showNull() started.
􀟈  Test schema() started.
􀟈  Test selectNone() started.
􀟈  Test rdd() started.
􀟈  Test sort() started.
􀟈  Test unpersist() started.
􀟈  Test limit() started.
􀟈  Test count() started.
􀟈  Test cache() started.
􀟈  Test selectInvalidColumn() started.
􀟈  Test collect() started.
􀟈  Test countNull() started.
􀟈  Test select() started.
􀟈  Test persistInvalidStorageLevel() started.
􁁛  Test rdd() passed after 0.571 seconds.
􁁛  Test selectNone() passed after 1.347 seconds.
􁁛  Test select() passed after 1.354 seconds.
􁁛  Test selectMultipleColumns() passed after 1.354 seconds.
􁁛  Test selectInvalidColumn() passed after 1.395 seconds.
􁁛  Test schema() passed after 1.747 seconds.
++
||
++

++
􁁛  Test showCommand() passed after 1.885 seconds.
+-----------+-----------+-------------+
| namespace | tableName | isTemporary |
+-----------+-----------+-------------+

+-----------+-----------+-------------+
+------+-------+------+
| col1 | col2  | col3 |
+------+-------+------+
| 1    | true  | abc  |
| NULL | NULL  | NULL |
| 3    | false | def  |
+------+-------+------+
􁁛  Test showNull() passed after 1.890 seconds.
+------+-------+
| col1 | col2  |
+------+-------+
| true | false |
+------+-------+
+------+------+
| col1 | col2 |
+------+------+
| 1    | 2    |
+------+------+
+------+------+
| col1 | col2 |
+------+------+
| abc  | def  |
| ghi  | jkl  |
+------+------+
􁁛  Test show() passed after 1.975 seconds.
􁁛  Test collect() passed after 2.045 seconds.
􁁛  Test countNull() passed after 2.566 seconds.
􁁛  Test persistInvalidStorageLevel() passed after 2.578 seconds.
􁁛  Test cache() passed after 2.683 seconds.
􁁛  Test isEmpty() passed after 2.778 seconds.
􁁛  Test count() passed after 2.892 seconds.
􁁛  Test persist() passed after 2.903 seconds.
􁁛  Test unpersist() passed after 2.917 seconds.
􁁛  Test limit() passed after 3.068 seconds.
􁁛  Test orderBy() passed after 3.101 seconds.
􁁛  Test sort() passed after 3.102 seconds.
􁁛  Test table() passed after 3.720 seconds.
􁁛  Suite DataFrameTests passed after 3.720 seconds.
􁁛  Test run with 21 tests passed after 3.720 seconds.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Could you review this PR, @viirya ?

[viirya]

This checks the result correctness. If it is not actually cached, we still can get the same result. I'm not sure if we have a way to check if it is actually cached or not?

[dongjoon-hyun]

Yes, correct! This is only API test, @viirya .

I checked the cached and uncached result manually via Connect Server UI because we didn't catalog part yet~

[dongjoon-hyun]

BTW, this is a valid error result check~

[dongjoon-hyun]

Thank you! Merged to main.