Skip to content

Hive partition projection date partition values are not parsed with partition_projection_format #25642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
metadaddy opened this issue Apr 22, 2025 · 3 comments · May be fixed by #25657
Open

Hive partition projection date partition values are not parsed with partition_projection_format #25642

metadaddy opened this issue Apr 22, 2025 · 3 comments · May be fixed by #25657
Assignees
@chenjian2664

Comments

@metadaddy
Copy link
Member

When using partition projection in the Hive connector, DATE partition values are parsed with yyyy-M-d rather than the configured partition_projection_format.

To reproduce:

Create a text file with the following content:

a1 b2 c3
d4 e5 f6
g7 h8 i9

Upload the file to an S3 bucket my-bucket (you'll need to change this) with the path /logs/123456789012/us-west-001/metadaddy-test/2025/04/01/2025-04-01-21-32-16-E568B2907131C0C0.

Create a table with partition projection:

CREATE TABLE regex_test(
  foo VARCHAR,
  bar VARCHAR,
  baz VARCHAR,
  timestamp DATE WITH (
    partition_projection_format='yyyy/MM/dd',
    partition_projection_interval=1,
    partition_projection_interval_unit='DAYS', 
    partition_projection_range=ARRAY['NOW-365DAYS', 'NOW'], 
    partition_projection_type='date'
  )
)
WITH (
  format='REGEX',
  partitioned_by=ARRAY['timestamp'],
  regex='([^ ]*) ([^ ]*) \\[(.*?)\\]$',
  external_location='s3://my-bucket/logs/123456789012/us-west-001/metadaddy-test',
  partition_projection_enabled=true, 
  partition_projection_location_template='s3://my-bucket/logs/123456789012/us-west-001/metadaddy-test/${timestamp}'
);

Try to SELECT *:

SELECT * FROM regex_test;
Query 20250422_010824_00035_tttig failed: Invalid partition value '2024/04/22' for DATE partition key: timestamp=2024%2F04%2F22

Stack trace:

2025-04-22T01:08:24.138Z	DEBUG	Query-20250422_010824_00035_tttig-494	io.trino.execution.QueryStateMachine	Query 20250422_010824_00035_tttig failed
io.trino.spi.TrinoException: Invalid partition value '2024/04/22' for DATE partition key: timestamp=2024%2F04%2F22
	at io.trino.plugin.hive.util.HiveUtil.datePartitionKey(HiveUtil.java:449)
	at io.trino.plugin.hive.util.HiveUtil.parsePartitionValue(HiveUtil.java:313)
	at io.trino.plugin.hive.HivePartitionManager.parsePartition(HivePartitionManager.java:265)
	at io.trino.plugin.hive.HivePartitionManager.parseValuesAndFilterPartition(HivePartitionManager.java:216)
	at io.trino.plugin.hive.HivePartitionManager.lambda$getPartitions$3(HivePartitionManager.java:113)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:1034)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:297)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:303)
	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:669)
	at io.trino.plugin.hive.HivePartitionManager.tryLoadPartitions(HivePartitionManager.java:199)
	at io.trino.plugin.hive.HiveMetadata.lambda$getTableProperties$90(HiveMetadata.java:3044)
	at java.base/java.util.Optional.or(Optional.java:313)
	at io.trino.plugin.hive.HiveMetadata.getTableProperties(HiveMetadata.java:3039)
	at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.getTableProperties(ClassLoaderSafeConnectorMetadata.java:992)
	at io.trino.tracing.TracingConnectorMetadata.getTableProperties(TracingConnectorMetadata.java:1162)
	at io.trino.metadata.MetadataManager.getTableProperties(MetadataManager.java:400)
	at io.trino.tracing.TracingMetadata.getTableProperties(TracingMetadata.java:247)
	at io.trino.sql.planner.planprinter.TableInfoSupplier.apply(TableInfoSupplier.java:47)
	at io.trino.sql.planner.planprinter.TableInfoSupplier.apply(TableInfoSupplier.java:31)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitScanFilterAndProjectInfo(PlanPrinter.java:1235)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitProject(PlanPrinter.java:1203)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitProject(PlanPrinter.java:641)
	at io.trino.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitScanFilterAndProjectInfo(PlanPrinter.java:1315)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitProject(PlanPrinter.java:1203)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitProject(PlanPrinter.java:641)
	at io.trino.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitScanFilterAndProjectInfo(PlanPrinter.java:1315)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitProject(PlanPrinter.java:1203)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitProject(PlanPrinter.java:641)
	at io.trino.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitScanFilterAndProjectInfo(PlanPrinter.java:1315)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitProject(PlanPrinter.java:1203)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitProject(PlanPrinter.java:641)
	at io.trino.sql.planner.plan.ProjectNode.accept(ProjectNode.java:81)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.processChildren(PlanPrinter.java:1963)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitOutput(PlanPrinter.java:1444)
	at io.trino.sql.planner.planprinter.PlanPrinter$Visitor.visitOutput(PlanPrinter.java:641)
	at io.trino.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
	at io.trino.sql.planner.planprinter.PlanPrinter.<init>(PlanPrinter.java:240)
	at io.trino.sql.planner.planprinter.PlanPrinter.textLogicalPlan(PlanPrinter.java:385)
	at io.trino.sql.planner.planprinter.PlanPrinter.textLogicalPlan(PlanPrinter.java:362)
	at io.trino.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:255)
	at io.trino.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:244)
	at io.trino.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:239)
	at io.trino.execution.SqlQueryExecution.doPlanQuery(SqlQueryExecution.java:503)
	at io.trino.execution.SqlQueryExecution.planQuery(SqlQueryExecution.java:482)
	at io.trino.execution.SqlQueryExecution.start(SqlQueryExecution.java:420)
	at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:272)
	at io.trino.dispatcher.LocalDispatchQuery.startExecution(LocalDispatchQuery.java:150)
	at io.trino.dispatcher.LocalDispatchQuery.lambda$waitForMinimumWorkers$2(LocalDispatchQuery.java:134)
	at io.airlift.concurrent.MoreFutures.lambda$addSuccessCallback$12(MoreFutures.java:570)
	at io.airlift.concurrent.MoreFutures$3.onSuccess(MoreFutures.java:545)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1139)
	at io.trino.$gen.Trino_474____20250422_003045_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)

The problem seems to be that the Hive partition manager is (correctly) generating partition names of the form 2024/04/22, but they are being parsed by HiveUtil.parseHiveDate using HIVE_DATE_PARSER, which is configured with date formats of the form yyyy-M-d.
@chenjian2664
Copy link
Contributor

Let me investigate it

@chenjian2664 chenjian2664 self-assigned this Apr 23, 2025
@metadaddy
Copy link
Member Author

Hi @chenjian2664 - thanks for picking this up!

I created this test, based on BaseTestHiveOnDataLake.testDatePartitionProjectionOnDateColumnWithDefaults. It currently fails (stack trace below) with the same error that I see when I try to use partition projection. I believe it should pass.

The only difference from BaseTestHiveOnDataLake.testDatePartitionProjectionOnDateColumnWithDefaults is using yyyy/MM/dd for the partition format (with a similarly formatted range) rather than yyyy-MM-dd.

It is important that Trino correctly handles yyyy/MM/dd, since this is a widely used partitioning format; for example, in S3 server access logs.

abstract class BaseTestHiveOnDataLake
        extends AbstractTestQueryFramework
{

    // ...

    @Test
    public void testDatePartitionProjectionOnDateColumnWithSlash()
    {
        String tableName = "nation_" + randomNameSuffix();
        String fullyQualifiedTestTableName = getFullyQualifiedTestTableName(tableName);

        computeActual(
                "CREATE TABLE " + fullyQualifiedTestTableName + " ( " +
                        "  name varchar(25), " +
                        "  comment varchar(152), " +
                        "  nationkey bigint, " +
                        "  regionkey bigint, " +
                        "  short_name1 varchar(152) WITH (" +
                        "    partition_projection_type='enum', " +
                        "    partition_projection_values=ARRAY['PL1', 'CZ1'] " +
                        "  ), " +
                        "  short_name2 date WITH (" +
                        "    partition_projection_type='date', " +
                        "    partition_projection_format='yyyy/MM/dd', " +
                        "    partition_projection_range=ARRAY['2001/1/22', '2001/1/25']" +
                        "  )" +
                        ") WITH ( " +
                        "  partitioned_by=ARRAY['short_name1', 'short_name2'], " +
                        "  partition_projection_enabled=true " +
                        ")");

        assertThat(
                hiveMinioDataLake
                        .runOnHive("SHOW TBLPROPERTIES " + getHiveTestTableName(tableName)))
                .containsPattern("[ |]+projection\\.enabled[ |]+true[ |]+")
                .containsPattern("[ |]+projection\\.short_name1\\.type[ |]+enum[ |]+")
                .containsPattern("[ |]+projection\\.short_name1\\.values[ |]+PL1,CZ1[ |]+")
                .containsPattern("[ |]+projection\\.short_name2\\.type[ |]+date[ |]+")
                .containsPattern("[ |]+projection\\.short_name2\\.format[ |]+yyyy/MM/dd[ |]+")
                .containsPattern("[ |]+projection\\.short_name2\\.range[ |]+2001/1/22,2001/1/25[ |]+");

        computeActual(createInsertStatement(
                fullyQualifiedTestTableName,
                ImmutableList.of(
                        ImmutableList.of("'POLAND_1'", "'Comment'", "0", "5", "'PL1'", "DATE '2001-1-22'"),
                        ImmutableList.of("'POLAND_2'", "'Comment'", "1", "5", "'PL1'", "DATE '2001-1-23'"),
                        ImmutableList.of("'CZECH_1'", "'Comment'", "2", "5", "'CZ1'", "DATE '2001-1-24'"),
                        ImmutableList.of("'CZECH_2'", "'Comment'", "3", "5", "'CZ1'", "DATE '2001-1-25'"),
                        ImmutableList.of("'CZECH_3'", "'Comment'", "4", "5", "'CZ1'", "DATE '2001-1-26'"))));

        assertQuery(
                format("SELECT * FROM %s", getFullyQualifiedTestTableName("\"" + tableName + "$partitions\"")),
                "VALUES ('PL1','2001-1-22'), ('PL1','2001-1-23'), ('PL1','2001-1-24'), ('PL1','2001-1-25')," +
                        "('CZ1','2001-1-22'), ('CZ1','2001-1-23'), ('CZ1','2001-1-24'), ('CZ1','2001-1-25')");

        assertQuery(
                format("SELECT name FROM %s WHERE short_name1='PL1' AND short_name2=(DATE '2001-1-23')", fullyQualifiedTestTableName),
                "VALUES 'POLAND_2'");

        assertQuery(
                format("SELECT name FROM %s WHERE short_name1='PL1' AND ( short_name2=(DATE '2001-1-23') OR short_name2=(DATE '2001-1-22') )", fullyQualifiedTestTableName),
                "VALUES ('POLAND_1'), ('POLAND_2')");

        assertQuery(
                format("SELECT name FROM %s WHERE short_name2 > DATE '2001-1-23'", fullyQualifiedTestTableName),
                "VALUES ('CZECH_1'), ('CZECH_2')");

        assertQuery(
                format("SELECT name FROM %s WHERE short_name2 >= DATE '2001-1-23' AND short_name2 <= DATE '2001-1-25'", fullyQualifiedTestTableName),
                "VALUES ('POLAND_2'), ('CZECH_1'), ('CZECH_2')");

        assertQuery(
                format("SELECT name FROM %s WHERE short_name1='PL1'", fullyQualifiedTestTableName),
                "VALUES ('POLAND_1'), ('POLAND_2')");

        assertQuery(
                format("SELECT name FROM %s", fullyQualifiedTestTableName),
                "VALUES ('POLAND_1'), ('POLAND_2'), ('CZECH_1'), ('CZECH_2')");
    }

// ...

}

I ran ./mvnw test -pl plugin/trino-hive -Dtest=TestHive4OnDataLake#testDatePartitionProjectionOnDateColumnWithDefaults first to verify that I have everything correctly configured, and got Tests run: 1, Failures: 0, Errors: 0, Skipped: 0.

Running the above test:

% ./mvnw test -pl plugin/trino-hive -Dtest=TestHive4OnDataLake#testDatePartitionProjectionOnDateColumnWithSlash   
[INFO] Scanning for projects...
[INFO] Loaded 0 groupId for remote repository google-maven-central-copy
[INFO] Loaded 0 groupId for remote repository maven-central
[INFO] 
[INFO] ------------------------< io.trino:trino-hive >-------------------------
[INFO] Building trino-hive 475-SNAPSHOT
[INFO]   from pom.xml
[INFO] ----------------------------[ trino-plugin ]----------------------------
[INFO] Loaded 0 groupId for remote repository ossrh
[INFO] Loaded 0 groupId for remote repository jitpack.io
[INFO] Loaded 0 groupId for remote repository central.maven.org
[INFO] 
[INFO] --- trino:15:check-spi-dependencies (default-check-spi-dependencies) @ trino-hive ---
[INFO] 
[INFO] --- enforcer:3.5.0:enforce (default) @ trino-hive ---
[INFO] Rule 0: org.apache.maven.enforcer.rules.dependency.BannedDependencies passed
[INFO] Rule 1: org.apache.maven.plugins.enforcer.RestrictImports passed
[INFO] Rule 2: org.apache.maven.plugins.enforcer.RestrictImports passed
[INFO] Rule 3: org.apache.maven.plugins.enforcer.RestrictImports passed
[INFO] Rule 4: org.apache.maven.enforcer.rules.RequireProfileIdsExist passed
[INFO] Rule 5: org.codehaus.mojo.extraenforcer.model.RequirePropertyDiverges passed
[INFO] Rule 6: org.apache.maven.enforcer.rules.RequireJavaVendor passed
[INFO] Rule 7: org.apache.maven.enforcer.rules.BanDuplicatePomDependencyVersions passed
[INFO] Rule 8: org.apache.maven.enforcer.rules.dependency.RequireUpperBoundDeps passed
[INFO] Rule 9: org.apache.maven.enforcer.rules.version.RequireMavenVersion passed
[INFO] Rule 10: org.apache.maven.enforcer.rules.version.RequireJavaVersion passed
[INFO] Rule 11: org.codehaus.mojo.extraenforcer.dependencies.EnforceBytecodeVersion passed
[INFO] Rule 12: org.apache.maven.enforcer.rules.RequirePluginVersions passed
[INFO] 
[INFO] --- sortpom:4.0.0:verify (default) @ trino-hive ---
[INFO] Verifying file /Users/ppatterson/src/trino/plugin/trino-hive/pom.xml
[INFO] 
[INFO] --- dependency-scope:1.0.3:check (default) @ trino-hive ---
[WARNING] Could not find project version for dependency com.google.oauth-client:google-oauth-client:jar:1.37.0 (compile). This is probably a bug in the plugin
[INFO] No test dependency scope issues found
[INFO] 
[INFO] --- license:5.0.0:check (default) @ trino-hive ---
[INFO] Checking licenses...
[INFO] 
[INFO] --- checkstyle:3.6.0:check (checkstyle) @ trino-hive ---
[INFO] You have 0 Checkstyle violations.
[INFO] 
[INFO] --- git-commit-id:9.0.1:revision (default) @ trino-hive ---
[INFO] 
[INFO] --- build-helper:3.6.0:bsh-property (surefire-enable-preview) @ trino-hive ---
[INFO] 
[INFO] --- jacoco:0.8.13:prepare-agent (default) @ trino-hive ---
[INFO] Skipping JaCoCo execution because property jacoco.skip is set.
[INFO] 
[INFO] --- resources:3.3.1:resources (default-resources) @ trino-hive ---
[INFO] skip non existing resourceDirectory /Users/ppatterson/src/trino/plugin/trino-hive/src/main/resources
[INFO] 
[INFO] --- compiler:3.14.0:compile (default-compile) @ trino-hive ---
[INFO] Nothing to compile - all classes are up to date.
[INFO] 
[INFO] --- trino:15:generate-service-descriptor (default-generate-service-descriptor) @ trino-hive ---
[INFO] Wrote io.trino.plugin.hive.HivePlugin to /Users/ppatterson/src/trino/plugin/trino-hive/target/trino-hive-475-SNAPSHOT-services.jar
[INFO] 
[INFO] --- resources:3.3.1:testResources (default-testResources) @ trino-hive ---
[INFO] Copying 30 resources from src/test/resources to target/test-classes
[INFO] 
[INFO] --- compiler:3.14.0:testCompile (default-testCompile) @ trino-hive ---
[INFO] Recompiling the module because of changed source code.
[INFO] Compiling 209 source files with javac [debug deprecation parameters release 23] to target/test-classes
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/HiveQueryRunner.java:[230,43] TestingHivePlugin(java.nio.file.Path,java.util.Optional<io.trino.metastore.HiveMetastore>) in io.trino.plugin.hive.TestingHivePlugin has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java:[536,25] withRole(java.lang.String,io.trino.spi.security.SelectedRole) in io.trino.spi.security.Identity.Builder has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java:[8984,21] nonTrinoExceptionFailure() in io.trino.sql.query.QueryAssertions.QueryAssert has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java:[9247,17] nonTrinoExceptionFailure() in io.trino.sql.query.QueryAssertions.QueryAssert has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java:[9294,17] nonTrinoExceptionFailure() in io.trino.sql.query.QueryAssertions.QueryAssert has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java:[9431,45] getTypeSignature(io.trino.metastore.HiveType) in io.trino.plugin.hive.util.HiveTypeUtil has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveGlueMetadataListing.java:[75,35] TestingHivePlugin(java.nio.file.Path,io.trino.metastore.HiveMetastore) in io.trino.plugin.hive.TestingHivePlugin has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestHiveReaderProjectionsUtil.java:[79,137] getTypeSignature(io.trino.metastore.HiveType) in io.trino.plugin.hive.util.HiveTypeUtil has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestOrcPageSourceMemoryTracking.java:[538,58] getTypeSignature(io.trino.metastore.HiveType) in io.trino.plugin.hive.util.HiveTypeUtil has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/TestingHivePlugin.java:[53,33] TestingHiveConnectorFactory(java.nio.file.Path,java.util.Optional<io.trino.metastore.HiveMetastore>) in io.trino.plugin.hive.TestingHiveConnectorFactory has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/orc/TestOrcReaderConfig.java:[40,17] setLazyReadSmallRanges(boolean) in io.trino.plugin.hive.orc.OrcReaderConfig has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/orc/TestOrcReaderConfig.java:[41,17] setNestedLazy(boolean) in io.trino.plugin.hive.orc.OrcReaderConfig has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/orc/TestOrcReaderConfig.java:[69,17] setLazyReadSmallRanges(boolean) in io.trino.plugin.hive.orc.OrcReaderConfig has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/orc/TestOrcReaderConfig.java:[70,17] setNestedLazy(boolean) in io.trino.plugin.hive.orc.OrcReaderConfig has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/parquet/AbstractTestParquetReader.java:[2253,16] sqlTimestampOf(long) in io.trino.testing.DateTimeTestingUtils has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/parquet/write/TestDataWritableWriteSupport.java:[47,25] init(org.apache.hadoop.conf.Configuration) in org.apache.parquet.hadoop.api.WriteSupport has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/parquet/write/TestDataWritableWriter.java:[370,48] fromByteArray(byte[]) in org.apache.parquet.io.api.Binary has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/parquet/write/TestDataWritableWriter.java:[395,42] setScale(int) in org.apache.hadoop.hive.common.type.HiveDecimal has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/parquet/write/TestDataWritableWriter.java:[401,26] fromByteArray(byte[]) in org.apache.parquet.io.api.Binary has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/parquet/write/TestDataWritableWriter.java:[413,22] fromByteArray(byte[]) in org.apache.parquet.io.api.Binary has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/util/TestHiveBucketing.java:[236,59] getTypeSignature(io.trino.metastore.HiveType) in io.trino.plugin.hive.util.HiveTypeUtil has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/util/TestHiveBucketing.java:[312,54] getTypeSignature(io.trino.metastore.HiveType) in io.trino.plugin.hive.util.HiveTypeUtil has been deprecated
[WARNING] /Users/ppatterson/src/trino/plugin/trino-hive/src/test/java/io/trino/plugin/hive/util/TestLazyMap.java:[54,54] getLazySimpleMapObjectInspector(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector,org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector,byte,byte,org.apache.hadoop.io.Text,boolean,byte) in org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory has been deprecated
[INFO] 
[INFO] --- dependency:3.8.1:analyze-only (default) @ trino-hive ---
[INFO] No dependency problems found
[INFO] 
[INFO] --- dependency:3.8.1:analyze-duplicate (default) @ trino-hive ---
[INFO] No duplicate dependencies found in <dependencies/> or in <dependencyManagement/>
[INFO] 
[INFO] --- duplicate-finder:2.0.1:check (default) @ trino-hive ---
[INFO] Checking compile classpath
[INFO] Checking runtime classpath
[INFO] Checking test classpath
[INFO] 
[INFO] --- modernizer:3.1.0:modernizer (modernizer) @ trino-hive ---
[INFO] 
[INFO] --- modernizer:3.1.0:modernizer (modernizer-production-code) @ trino-hive ---
[INFO] 
[INFO] --- surefire:3.5.3:test (default-test) @ trino-hive ---
[INFO] Tests will run in random order. To reproduce ordering use flag -Dsurefire.runOrder.random.seed=1274359431646208
[INFO] Using configured provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
WARNING: Using incubator modules: jdk.incubator.vector
2025-04-23T11:42:14.377-0600	INFO	ForkJoinPool-1-worker-1	io.airlift.log.Logging	Logging to stderr
[INFO] Running io.trino.plugin.hive.TestHive4OnDataLake
2025-04-23T11:42:14.457-0600	INFO	ForkJoinPool-1-worker-1	org.testcontainers.images.PullPolicy	Image pull policy will be performed by: DefaultPullPolicy()
2025-04-23T11:42:14.459-0600	INFO	ForkJoinPool-1-worker-1	org.testcontainers.utility.ImageNameSubstitutor	Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
2025-04-23T11:42:15.027-0600	INFO	ForkJoinPool-1-worker-1	tc.testcontainers/ryuk:0.11.0	Creating container for image: testcontainers/ryuk:0.11.0
2025-04-23T11:42:15.283-0600	INFO	ForkJoinPool-1-worker-1	tc.testcontainers/ryuk:0.11.0	Container testcontainers/ryuk:0.11.0 is starting: b528c7bee5bdca77d92c1f17dbd64e74ecd54aef44b8830e5266c858c21bd5e4
2025-04-23T11:42:15.561-0600	INFO	ForkJoinPool-1-worker-1	tc.testcontainers/ryuk:0.11.0	Container testcontainers/ryuk:0.11.0 started in PT0.533188S
2025-04-23T11:42:15.567-0600	INFO	ForkJoinPool-1-worker-1	tc.minio/minio:RELEASE.2024-12-18T13-15-44Z	Creating container for image: minio/minio:RELEASE.2024-12-18T13-15-44Z
2025-04-23T11:42:15.709-0600	INFO	ForkJoinPool-1-worker-1	tc.minio/minio:RELEASE.2024-12-18T13-15-44Z	Container minio/minio:RELEASE.2024-12-18T13-15-44Z is starting: 941aaae03fc3b0a443fa6445b154de0acdbf4c256285decd136d79d7d3016ed9
2025-04-23T11:42:16.536-0600	INFO	ForkJoinPool-1-worker-1	tc.minio/minio:RELEASE.2024-12-18T13-15-44Z	Container minio/minio:RELEASE.2024-12-18T13-15-44Z started in PT0.968151S
2025-04-23T11:42:16.537-0600	INFO	ForkJoinPool-1-worker-1	io.trino.testing.containers.Minio	MinIO container started with address for api: http://localhost:52432 and console: http://localhost:52431
2025-04-23T11:42:16.769-0600	INFO	ForkJoinPool-1-worker-1	tc.ghcr.io/trinodb/testing/hive4.0-hive:110	Creating container for image: ghcr.io/trinodb/testing/hive4.0-hive:110
2025-04-23T11:42:16.919-0600	INFO	ForkJoinPool-1-worker-1	tc.ghcr.io/trinodb/testing/hive4.0-hive:110	Container ghcr.io/trinodb/testing/hive4.0-hive:110 is starting: 712bdb19be2e1a343f7f66adf1a2c86354ed6f9096d4cebc244b80c234e977bd
2025-04-23T11:42:24.350-0600	INFO	ForkJoinPool-1-worker-1	tc.ghcr.io/trinodb/testing/hive4.0-hive:110	Container ghcr.io/trinodb/testing/hive4.0-hive:110 started in PT7.5808S
2025-04-23T11:42:24.354-0600	INFO	ForkJoinPool-1-worker-1	io.trino.plugin.hive.containers.HiveHadoop	Hive container started with addresses for metastore: thrift://localhost:52443
2025-04-23T11:42:24.354-0600	INFO	ForkJoinPool-1-worker-1	tc.ghcr.io/trinodb/testing/hive4.0-hive:110	Creating container for image: ghcr.io/trinodb/testing/hive4.0-hive:110
2025-04-23T11:42:24.414-0600	INFO	ForkJoinPool-1-worker-1	tc.ghcr.io/trinodb/testing/hive4.0-hive:110	Container ghcr.io/trinodb/testing/hive4.0-hive:110 is starting: 8c95f563be66f6fb65a9c2b274b24e11e81423b3c4be67b56cd6a72def4c09d2
2025-04-23T11:42:30.684-0600	INFO	ForkJoinPool-1-worker-1	tc.ghcr.io/trinodb/testing/hive4.0-hive:110	Container ghcr.io/trinodb/testing/hive4.0-hive:110 started in PT6.329886S
2025-04-23T11:42:30.691-0600	INFO	ForkJoinPool-1-worker-1	io.trino.plugin.hive.containers.Hive4HiveServer	Hive container started with addresses for hive server: localhost:52474
2025-04-23T11:42:31.077-0600	INFO	ForkJoinPool-1-worker-1	stdout	# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf
2025-04-23T11:42:31.928-0600	INFO	ForkJoinPool-1-worker-1	stdout	# WARNING: Unable to attach Serviceability Agent. You can try again with escalated privileges. Two options: a) use -Djol.tryWithSudo=true to try with sudo; b) echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
2025-04-23T11:42:33.258-0600	INFO	ForkJoinPool-1-worker-1	org.eclipse.jetty.ee10.servlet.ServletContextHandler	Started oeje10s.ServletContextHandler@bf70937{ROOT,/,b=null,a=AVAILABLE,vh=[@http,@https],h=GzipHandler@eec0e94{STARTED,min=32,inflate=-1}}
2025-04-23T11:42:35.526-0600	INFO	ForkJoinPool-1-worker-1	io.trino.connector.CatalogStoreManager	-- Loading catalog store memory --
2025-04-23T11:42:35.527-0600	INFO	ForkJoinPool-1-worker-1	io.trino.connector.CatalogStoreManager	-- Loaded catalog store memory --
2025-04-23T11:42:35.545-0600	INFO	ForkJoinPool-1-worker-1	io.trino.testing.DistributedQueryRunner	Created DistributedQueryRunner in 3.21s (unclosed instances = 1)
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	WARNING: A restricted method in
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	java.lang.foreign.SymbolLookup
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	 has been called
WARNING:
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	java.lang.foreign.SymbolLookup::libraryLookup
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	 has been called by
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	io.airlift.compress.v3.internal.NativeLoader
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	 in
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	an unnamed module
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	
WARNING: Use --enable-native-access=
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	ALL-UNNAMED
2025-04-23T11:42:48.559-0600	INFO	SplitRunner-20250423_174246_00003_z5qzb.1.1.0-6-349	stderr	 to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 37.45 s <<< FAILURE! -- in io.trino.plugin.hive.TestHive4OnDataLake
[ERROR] io.trino.plugin.hive.TestHive4OnDataLake.testDatePartitionProjectionOnDateColumnWithSlash -- Time elapsed: 9.210 s <<< FAILURE!
java.lang.AssertionError: Execution of 'actual' query 20250423_174250_00004_z5qzb failed: SELECT * FROM hive.hive_datalake."nation_97bcid3j35$partitions"
	at io.trino.testing.QueryAssertions.assertDistributedQuery(QueryAssertions.java:297)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:185)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:158)
	at io.trino.testing.AbstractTestQueryFramework.assertQuery(AbstractTestQueryFramework.java:356)
	at io.trino.plugin.hive.BaseTestHiveOnDataLake.testDatePartitionProjectionOnDateColumnWithSlash(BaseTestHiveOnDataLake.java:1040)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1460)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2036)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)
Caused by: io.trino.testing.QueryFailedException: Invalid partition value '2001/01/22' for DATE partition key: short_name1=PL1/short_name2=2001%2F01%2F22
	at io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:138)
	at io.trino.testing.DistributedQueryRunner.executeInternal(DistributedQueryRunner.java:565)
	at io.trino.testing.DistributedQueryRunner.executeWithPlan(DistributedQueryRunner.java:554)
	at io.trino.testing.QueryAssertions.assertDistributedQuery(QueryAssertions.java:288)
	... 9 more
	Suppressed: java.lang.Exception: SQL: SELECT * FROM hive.hive_datalake."nation_97bcid3j35$partitions"
		at io.trino.testing.DistributedQueryRunner.executeInternal(DistributedQueryRunner.java:572)
		... 11 more
Caused by: io.trino.spi.TrinoException: Invalid partition value '2001/01/22' for DATE partition key: short_name1=PL1/short_name2=2001%2F01%2F22
	at io.trino.plugin.hive.util.HiveUtil.datePartitionKey(HiveUtil.java:449)
	at io.trino.plugin.hive.util.HiveUtil.parsePartitionValue(HiveUtil.java:313)
	at io.trino.plugin.hive.HivePartitionManager.parsePartition(HivePartitionManager.java:265)
	at io.trino.plugin.hive.HivePartitionManager.parseValuesAndFilterPartition(HivePartitionManager.java:216)
	at io.trino.plugin.hive.HivePartitionManager.lambda$getPartitions$3(HivePartitionManager.java:113)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:1034)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:297)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:303)
	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1949)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:297)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:303)
	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
	at io.trino.spi.connector.InMemoryRecordSet$InMemoryRecordCursor.advanceNextPosition(InMemoryRecordSet.java:110)
	at io.trino.plugin.base.MappedRecordSet$MappedRecordCursor.advanceNextPosition(MappedRecordSet.java:89)
	at io.trino.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:89)
	at io.trino.spi.connector.ConnectorPageSource.getNextSourcePage(ConnectorPageSource.java:72)
	at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:269)
	at io.trino.operator.Driver.processInternal(Driver.java:403)
	at io.trino.operator.Driver.lambda$process$8(Driver.java:306)
	at io.trino.operator.Driver.tryWithLock(Driver.java:709)
	at io.trino.operator.Driver.process(Driver.java:298)
	at io.trino.operator.Driver.processForDuration(Driver.java:269)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:890)
	at io.trino.execution.executor.dedicated.SplitProcessor.run(SplitProcessor.java:77)
	at io.trino.execution.executor.dedicated.TaskEntry$VersionEmbedderBridge.lambda$run$0(TaskEntry.java:201)
	at io.trino.$gen.Trino_testversion____20250423_174235_71.run(Unknown Source)
	at io.trino.execution.executor.dedicated.TaskEntry$VersionEmbedderBridge.run(TaskEntry.java:202)
	at io.trino.execution.executor.scheduler.FairScheduler.runTask(FairScheduler.java:177)
	at io.trino.execution.executor.scheduler.FairScheduler.lambda$submit$0(FairScheduler.java:164)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:128)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1575)

2025-04-23T11:42:51.855-0600	INFO	main	io.trino.testing.containers.junit.ReportLeakedContainers	Checking for leaked containers
2025-04-23T11:42:51.890-0600	INFO	LifeCycleManager Shutdown Hook	io.airlift.bootstrap.LifeCycleManager	JVM is shutting down, cleaning up
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestHive4OnDataLake>BaseTestHiveOnDataLake.testDatePartitionProjectionOnDateColumnWithSlash:1040->AbstractTestQueryFramework.assertQuery:356 Execution of 'actual' query 20250423_174250_00004_z5qzb failed: SELECT * FROM hive.hive_datalake."nation_97bcid3j35$partitions"
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
WARNING: Using incubator modules: jdk.incubator.vector
[INFO] Running io.trino.plugin.hive.TestHive4OnDataLake
2025-04-23T11:42:52.374-0600 INFO LogTestDurationListener enabled: false
2025-04-23T11:42:52.398-0600 INFO ProgressLoggingListener is disabled!
2025-04-23T11:42:52.456-0600 INFO ManageTestResources.onStart: running checks
2025-04-23T11:42:52.498-0600 INFO ManageTestResources.onFinish: running checks
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.275 s -- in io.trino.plugin.hive.TestHive4OnDataLake
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:01 min
[INFO] Finished at: 2025-04-23T10:42:52-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.5.3:test (default-test) on project trino-hive: There are test failures.
[ERROR] 
[ERROR] See /Users/ppatterson/src/trino/plugin/trino-hive/target/surefire-reports for the individual test results.
[ERROR] See dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

@chenjian2664
Copy link
Contributor

@metadaddy Thanks for the info, very useful

@chenjian2664 chenjian2664 linked a pull request Apr 24, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chenjian2664 chenjian2664

Labels
None yet
Milestone
No milestone
2 participants
@metadaddy @chenjian2664