Skip to content

Commit 6a89f65

Browse files
committed
[SPARK-52130][FOLLOW-UP][ML][CONNECT] Refine error message when model.summary is evicted
### What changes were proposed in this pull request? Refine error message when model.summary is evicted. In Spark Connect ML handler, `model.summary.XXX` invocation has different code path than `model.evaluate`, and it should raise readable error message when model.summary is evicted, otherwise the MLHandler will raise NullPointerException which is not friendly to end-users. ### Why are the changes needed? Refine error message. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #51048 from WeichenXu123/improve-scml-summary-error. Authored-by: Weichen Xu <weichen.xu@databricks.com> Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
1 parent a4bdb37 commit 6a89f65

File tree

1 file changed

+12
-5
lines changed
  • sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml

1 file changed

+12
-5
lines changed

sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ import org.apache.spark.ml.{Estimator, EstimatorUtils, Model, Transformer}
2828
import org.apache.spark.ml.evaluation.Evaluator
2929
import org.apache.spark.ml.param.{ParamMap, Params}
3030
import org.apache.spark.ml.tree.TreeConfig
31-
import org.apache.spark.ml.util.{MLWritable, Summary}
31+
import org.apache.spark.ml.util.{HasTrainingSummary, MLWritable, Summary}
3232
import org.apache.spark.sql.DataFrame
3333
import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
3434
import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
@@ -223,10 +223,17 @@ private[connect] object MLHandler extends Logging {
223223
.build()
224224

225225
case proto.MlCommand.CommandCase.FETCH =>
226-
val helper = AttributeHelper(
227-
sessionHolder,
228-
mlCommand.getFetch.getObjRef.getId,
229-
mlCommand.getFetch.getMethodsList.asScala.toArray)
226+
val objRefId = mlCommand.getFetch.getObjRef.getId
227+
val methods = mlCommand.getFetch.getMethodsList.asScala.toArray
228+
val obj = sessionHolder.mlCache.get(objRefId)
229+
if (obj != null && obj.isInstanceOf[HasTrainingSummary[_]]
230+
&& methods(0).getMethod == "summary"
231+
&& !obj.asInstanceOf[HasTrainingSummary[_]].hasSummary) {
232+
throw MLCacheInvalidException(
233+
objRefId,
234+
sessionHolder.mlCache.getOffloadingTimeoutMinute)
235+
}
236+
val helper = AttributeHelper(sessionHolder, objRefId, methods)
230237
val attrResult = helper.getAttribute
231238
attrResult match {
232239
case s: Summary =>

0 commit comments

Comments
 (0)