Skip to content

Commit 9f138f1

Browse files
LuciferYangyhuang-db
authored andcommitted
[SPARK-52386][CORE][TESTS][FOLLOWUP] Add field sorting to JSON strings to ensure the consistency of the content output to golden files
### What changes were proposed in this pull request? This pr adds sorting to the JSON content when regenerating golden files in `HistoryServerSuite` to ensure that the JSON strings output to the golden files are stable. Additionally, this pr refreshes the `HistoryServerExpectations` directory with the new approach for all golden files. ### Why are the changes needed? apache#51072 has fixed and enhanced the way `HistoryServerSuite` generates golden files. However, the current approach is susceptible to the execution environment and cannot guarantee strong consistency in the content output to golden files every time. For instance, when I switched to a different computer to run the command for refreshing golden files, I noticed that the results had changed: ``` diff --git a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json index e485c0a..500a748 100644 --- a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json +++ b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json -9,9 +9,9 "sparkUser" : "lijunqing", "completed" : true, "appSparkVersion" : "3.3.0-SNAPSHOT", + "startTimeEpoch" : 1642039450519, "endTimeEpoch" : 1642039536564, - "lastUpdatedEpoch" : 0, - "startTimeEpoch" : 1642039450519 + "lastUpdatedEpoch" : 0 } ] }, { "id" : "application_1628109047826_1317105", -24,9 +24,9 "sparkUser" : "john", "completed" : true, "appSparkVersion" : "3.1.1.119", + "startTimeEpoch" : 1628637895333, "endTimeEpoch" : 1628638170208, - "lastUpdatedEpoch" : 0, - "startTimeEpoch" : 1628637895333 + "lastUpdatedEpoch" : 0 } ] }, { ... ``` As can be seen, due to differences in the execution environment, the ordering of the output fields has changed. Therefore, field sorting has been added to the output JSON to ensure that the output content is not affected by the environment. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass Github Actions - Executing ``` SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "core/testOnly org.apache.spark.deploy.history.RocksDBBackendHistoryServerSuite" ``` in multiple environments has verified the consistency of the output content. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#51093 from LuciferYang/SPARK-52386-FOLLOWUP. Lead-authored-by: yangjie01 <yangjie01@baidu.com> Co-authored-by: YangJie <yangjie01@baidu.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>
1 parent d0131bb commit 9f138f1

File tree

59 files changed

+13116
-13114
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

59 files changed

+13116
-13114
lines changed

core/src/test/resources/HistoryServerExpectations/app_environment_expectation.json

Lines changed: 6 additions & 6 deletions
Large diffs are not rendered by default.

core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json

Lines changed: 156 additions & 156 deletions
Large diffs are not rendered by default.

core/src/test/resources/HistoryServerExpectations/complete_stage_list_json_expectation.json

Lines changed: 186 additions & 186 deletions
Large diffs are not rendered by default.

core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json

Lines changed: 156 additions & 156 deletions
Large diffs are not rendered by default.

core/src/test/resources/HistoryServerExpectations/excludeOnFailure_for_stage_expectation.json

Lines changed: 562 additions & 562 deletions
Large diffs are not rendered by default.

core/src/test/resources/HistoryServerExpectations/excludeOnFailure_node_for_stage_expectation.json

Lines changed: 775 additions & 775 deletions
Large diffs are not rendered by default.
Lines changed: 38 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,54 @@
11
[ {
2-
"id" : "driver",
3-
"hostPort" : "localhost:57971",
4-
"isActive" : true,
5-
"rddBlocks" : 0,
6-
"memoryUsed" : 0,
7-
"diskUsed" : 0,
8-
"totalCores" : 0,
9-
"maxTasks" : 0,
102
"activeTasks" : 0,
11-
"failedTasks" : 1,
3+
"addTime" : "2015-02-03T16:43:00.906GMT",
4+
"attributes" : { },
5+
"blacklistedInStages" : [ ],
126
"completedTasks" : 31,
13-
"totalTasks" : 32,
14-
"totalDuration" : 9011,
15-
"totalGCTime" : 0,
16-
"totalInputBytes" : 28000288,
17-
"totalShuffleRead" : 0,
18-
"totalShuffleWrite" : 13180,
7+
"diskUsed" : 0,
8+
"excludedInStages" : [ ],
9+
"executorLogs" : { },
10+
"failedTasks" : 1,
11+
"hostPort" : "localhost:57971",
12+
"id" : "driver",
13+
"isActive" : true,
1914
"isBlacklisted" : false,
15+
"isExcluded" : false,
2016
"maxMemory" : 278302556,
21-
"addTime" : "2015-02-03T16:43:00.906GMT",
22-
"executorLogs" : { },
23-
"blacklistedInStages" : [ ],
17+
"maxTasks" : 0,
18+
"memoryUsed" : 0,
2419
"peakMemoryMetrics" : {
20+
"ConcurrentGCCount" : 0,
21+
"ConcurrentGCTime" : 0,
22+
"DirectPoolMemory" : 0,
2523
"JVMHeapMemory" : 0,
2624
"JVMOffHeapMemory" : 0,
27-
"OnHeapExecutionMemory" : 0,
25+
"MajorGCCount" : 0,
26+
"MajorGCTime" : 0,
27+
"MappedPoolMemory" : 0,
28+
"MinorGCCount" : 0,
29+
"MinorGCTime" : 0,
2830
"OffHeapExecutionMemory" : 0,
29-
"OnHeapStorageMemory" : 0,
3031
"OffHeapStorageMemory" : 0,
31-
"OnHeapUnifiedMemory" : 0,
3232
"OffHeapUnifiedMemory" : 0,
33-
"DirectPoolMemory" : 0,
34-
"MappedPoolMemory" : 0,
35-
"ProcessTreeJVMVMemory" : 0,
33+
"OnHeapExecutionMemory" : 0,
34+
"OnHeapStorageMemory" : 0,
35+
"OnHeapUnifiedMemory" : 0,
3636
"ProcessTreeJVMRSSMemory" : 0,
37-
"ProcessTreePythonVMemory" : 0,
38-
"ProcessTreePythonRSSMemory" : 0,
39-
"ProcessTreeOtherVMemory" : 0,
37+
"ProcessTreeJVMVMemory" : 0,
4038
"ProcessTreeOtherRSSMemory" : 0,
41-
"MinorGCCount" : 0,
42-
"MinorGCTime" : 0,
43-
"MajorGCCount" : 0,
44-
"MajorGCTime" : 0,
45-
"TotalGCTime" : 0,
46-
"ConcurrentGCCount" : 0,
47-
"ConcurrentGCTime" : 0
39+
"ProcessTreeOtherVMemory" : 0,
40+
"ProcessTreePythonRSSMemory" : 0,
41+
"ProcessTreePythonVMemory" : 0,
42+
"TotalGCTime" : 0
4843
},
49-
"attributes" : { },
50-
"resources" : { },
44+
"rddBlocks" : 0,
5145
"resourceProfileId" : 0,
52-
"isExcluded" : false,
53-
"excludedInStages" : [ ]
46+
"resources" : { },
47+
"totalCores" : 0,
48+
"totalDuration" : 9011,
49+
"totalGCTime" : 0,
50+
"totalInputBytes" : 28000288,
51+
"totalShuffleRead" : 0,
52+
"totalShuffleWrite" : 13180,
53+
"totalTasks" : 32
5454
} ]

0 commit comments

Comments
 (0)