Velox, GCS and cURL | CURL error [77]=Problem with the SSL CA cert (path? access rights?)

[A-Mongy]

I am trying to test Gluten with Velox backend with spark 3.4.1

I get the following error in the executor logs

Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)'
Retriable: False
Context: Split [Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245] Task Gluten_Stage_0_TID_3_VTID_3
Additional Context: Operator: TableScan[0] 0
Function: checkGcsStatus
File: /root/src/weiting/gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp
Line: 62
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox12_GLOBAL__N_114checkGcsStatusEN6google5cloud5v2_226StatusERKSt17basic_string_viewIcSt11char_traitsIcEERKSsSD_
# 4  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile13preadInternalEmmPc
# 5  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile6preadvEmRKSt6vectorIN5folly5RangeIPcEESaIS7_EE
# 6  _ZN8facebook5velox4dwio6common19ReadFileInputStream4readERKSt6vectorIN5folly5RangeIPcEESaIS8_EEmNS2_10MetricsLog11MetricsTypeE
# 7  _ZN8facebook5velox4dwio6common19DirectCoalescedLoad8loadDataEb
# 8  _ZN8facebook5velox5cache13CoalescedLoad12loadOrFutureEPN5folly10SemiFutureIbEEb
# 9  _ZN8facebook5velox4dwio6common17DirectInputStream12loadPositionEv
# 10 _ZN8facebook5velox4dwio6common17DirectInputStream4NextEPPKvPi
# 11 _ZN8facebook5velox7parquet10ReaderBase16loadFileMetaDataEv
# 12 _ZN8facebook5velox7parquet10ReaderBaseC1ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 13 _ZN8facebook5velox7parquet13ParquetReaderC2ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 14 _ZN8facebook5velox7parquet20ParquetReaderFactory12createReaderESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 15 _ZN8facebook5velox9connector4hive11SplitReader12createReaderEv
# 16 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsE
# 17 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 18 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 19 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv
# 20 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 21 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 22 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 23 _ZN6gluten24WholeStageResultIterator4nextEv
# 24 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 25 0x00007a07606b9930

        at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
        at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
        at org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext(IteratorsV1.scala:159)
        at org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext(IteratorsV1.scala:71)
        at org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext(IteratorsV1.scala:37)
        at org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext(IteratorsV1.scala:100)
        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
        at scala.collection.Iterator.isEmpty(Iterator.scala:387)
        at scala.collection.Iterator.isEmpty$(Iterator.scala:387)
        at org.apache.spark.InterruptibleIterator.isEmpty(InterruptibleIterator.scala:28)
        at org.apache.gluten.execution.VeloxColumnarToRowExec$.toRowIterator(VeloxColumnarToRowExec.scala:122)
        at org.apache.gluten.execution.VeloxColumnarToRowExec.$anonfun$doExecuteInternal$1(VeloxColumnarToRowExec.scala:78)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:853)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:853)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
        at org.apache.spark.scheduler.Task.run(Task.scala:139)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError

curl seems to be working right, I tried to update the ca certificates also, so I am not sure what is the issue here. Any ideas would be apprecaited

[A-Mongy]

Any help is appreciated here
@amyreese, @dlespiau, @passy, @ot, @JkSelf

[JkSelf]

@A-Mongy Can you help to check the permissions of the certificate file or directory to ensure that the process accessing the GCS object can read the SSL certificate file? Thanks.

[A-Mongy]

Thanks @JkSelf for getting back to me on this!

I have tried to change the permission for the etc/ssl/ folder

total 24
drwxr-xr-x 1 root root  4096 Apr 15 04:41 certs
-rwxr-xr-x 1 root root 12332 Oct 27  2024 openssl.cnf
drwxr-xr-x 1 root root  4096 Oct 27  2024 private

I tried with the nightly jar, still having the same issue

Here is an updated log of the error, as I turned on debug mode:

20250618 00:30:14.379505    91 HiveConnectorUtil.cpp:286] No need to check type lowercase modeVARCHAR
I20250618 00:30:14.379514    91 HiveConnectorUtil.cpp:286] No need to check type lowercase modeVARCHAR
I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1
I20250618 00:30:14.384630    96 EventBase.h:883] expCoeff_ -5e-07 folly::EventBase::SmoothLoopTime::SmoothLoopTime(std::chrono::microseconds)
I20250618 00:30:14.384725    96 EventBase.cpp:543] EventBase(): Starting loop.
I20250618 00:30:14.384757    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551576 next 18446744073709551577
I20250618 00:30:14.384773    96 EventBase.cpp:1058] latest 18446744073709551576 next 18446744073709551577
I20250618 00:30:14.384784    96 EventBase.cpp:801] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) startWork_ 29679038962635
I20250618 00:30:14.384800    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551577 next 18446744073709551577
I20250618 00:30:14.384812    96 EventBase.cpp:1058] latest 18446744073709551577 next 18446744073709551577
I20250618 00:30:14.384876    96 EventBase.cpp:1058] latest 18446744073709551577 next 18446744073709551577
I20250618 00:30:14.384821    96 EventBase.cpp:655] EventBase 0x7ee85c00a760 did not timeout  loop time guess: 76 idle time: 40 busy time: 36 avgLoopTime: 0.001368 maxLatencyLoopTime: 0.001368 maxLatency_: 0us notificationQueueSize: 1 nothingHandledYet(): 0
I20250618 00:30:14.384897    96 EventBase.cpp:686] EventBase 0x7ee85c00a760 loop time: 0
I20250618 00:30:14.384910    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551577 next 18446744073709551578
I20250618 00:30:14.384919    96 EventBase.cpp:1058] latest 18446744073709551577 next 18446744073709551578
I20250618 00:30:14.384930    96 EventBase.cpp:801] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) startWork_ 29679039108575
I20250618 00:30:14.384951    96 EventBase.cpp:794] EventBase 0x7ee85c00a760 virtual void folly::EventBase::bumpHandlingTime() (loop) latest 18446744073709551578 next 18446744073709551578
I20250618 00:30:14.384963    96 EventBase.cpp:1058] latest 18446744073709551578 next 18446744073709551578
I20250618 00:30:14.384982    96 EventBase.cpp:1058] latest 18446744073709551578 next 18446744073709551578
I20250618 00:30:14.384972    96 EventBase.cpp:655] EventBase 0x7ee85c00a760 did not timeout  loop time guess: 151 idle time: 109 busy time: 42 avgLoopTime: 0.008853 maxLatencyLoopTime: 0.008853 maxLatency_: 0us notificationQueueSize: 0 nothingHandledYet(): 0
I20250618 00:30:14.384994    96 EventBase.cpp:686] EventBase 0x7ee85c00a760 loop time: 0
E20250618 00:30:14.421082    91 Exceptions.h:66] Line: /work/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp:62, Function:checkGcsStatus, Expression:  Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)', Source: RUNTIME, ErrorCode: INVALID_STATE
I20250618 00:30:14.421433    91 Task.cpp:2402] Terminating task Gluten_Stage_0_TID_0_VTID_0 with state Failed after running for 43ms
I20250618 00:30:14.421552    91 Task.cpp:1343] All drivers (1) finished for task Gluten_Stage_0_TID_0_VTID_0 after running for 43ms
25/06/18 00:30:14 ERROR TaskResources: Task 0 failed by error: 
org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Failed to get GCS object due to: Path:'gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet', SDK Error Type:, GCS Status Code:Unknown error,  Message:'Permanent error, with a last message of PerformWork() - CURL error [77]=Problem with the SSL CA cert (path? access rights?)'
Retriable: False
Context: Split [Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245] Task Gluten_Stage_0_TID_0_VTID_0
Additional Context: Operator: TableScan[0] 0
Function: checkGcsStatus
File: /work/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/gcs/GcsFileSystem.cpp
Line: 62
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZN8facebook5velox12_GLOBAL__N_114checkGcsStatusEN6google5cloud5v2_226StatusERKSt17basic_string_viewIcSt11char_traitsIcEERKSsSD_
# 4  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile13preadInternalEmmPc
# 5  _ZNK8facebook5velox12_GLOBAL__N_111GcsReadFile6preadvEmRKSt6vectorIN5folly5RangeIPcEESaIS7_EEPNS0_11filesystems4File7IoStatsE
# 6  _ZN8facebook5velox4dwio6common19ReadFileInputStream4readERKSt6vectorIN5folly5RangeIPcEESaIS8_EEmNS2_10MetricsLog11MetricsTypeE
# 7  _ZN8facebook5velox4dwio6common19DirectCoalescedLoad8loadDataEb
# 8  _ZN8facebook5velox5cache13CoalescedLoad12loadOrFutureEPN5folly10SemiFutureIbEEb
# 9  _ZN8facebook5velox4dwio6common17DirectInputStream12loadPositionEv
# 10 _ZN8facebook5velox4dwio6common17DirectInputStream4NextEPPKvPi
# 11 _ZN8facebook5velox7parquet10ReaderBase16loadFileMetaDataEv
# 12 _ZN8facebook5velox7parquet10ReaderBaseC1ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 13 _ZN8facebook5velox7parquet13ParquetReaderC2ESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 14 _ZN8facebook5velox7parquet20ParquetReaderFactory12createReaderESt10unique_ptrINS0_4dwio6common13BufferedInputESt14default_deleteIS6_EERKNS5_13ReaderOptionsE
# 15 _ZN8facebook5velox9connector4hive11SplitReader12createReaderEv
# 16 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsE
# 17 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 18 _ZN8facebook5velox4exec9TableScan8getSplitEv
# 19 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 20 _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv
# 21 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 22 _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 23 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 24 _ZN6gluten24WholeStageResultIterator4nextEv
# 25 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 26 0x00007ee91c6b9930

You can notice the two lines where the gcs bucket is also mentioned are:

I20250618 00:30:14.379588    91 HiveDataSource.cpp:306] Adding split Hive: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet 0 - 14245
I20250618 00:30:14.384384    91 FileHandle.cpp:65] Generating file handle for: gs://bucket/Tenants/default/source/OnlineStore/salesorderdetail/_delta_log/00000001748882003640.checkpoint.parquet uuid: 1

So I would say gluten is able to access the gcs bucket? what do you think?

[A-Mongy]

I have to add to this, when triggering a spark submit to only read data from GCS, reading a CSV or parquet file. There is no issues and it reads and prints the data from the file with no issue. So definitely not a GCS access issue and not a curl issue.

However, if I try to write data I can see the issue.
Also, with a recent jar I can see the issue is now hidden by the following error instead

Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: Empty Optional cannot be unwrapped
Retriable: False
Function: operator()
File: /work/ep/build-velox/build/velox_ep/velox/exec/Driver.cpp
Line: 571
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  _ZZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEEENKUlvE3_clEv.cold
# 4  _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 5  _ZN8facebook5velox4exec6Driver4nextEPN5folly10SemiFutureINS3_4UnitEEERPNS1_8OperatorERNS1_14BlockingReasonE
# 6  _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 7  _ZN6gluten24WholeStageResultIterator4nextEv
# 8  Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 9  0x00007879946b9930

        at org.apache.gluten.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
        at org.apache.gluten.vectorized.ColumnarBatchOutIterator.hasNext0(ColumnarBatchOutIterator.java:57)
        at org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:39)
        ... 28 more
25/06/23 15:04:58 INFO CoarseGrainedExecutorBackend: Got assigned task 1
25/06/23 15:04:58 INFO Executor: Running task 0.1 in stage 0.0 (TID 1)