Rename ending with everyItem/matchingSome/etc is not recognized, despite error message listing it as a valid choice

[MateuszKubuszok]

Discussed in #717

^{Originally posted by elpitero March 26, 2025}
Hello
I recently started using chimney and during experiments I found a case where nested renames do not seem to work:

  case class A_inner_inner(fieldA_A: String)
  case class A_inner(fieldA: Option[A_inner_inner])
  case class A(field: List[A_inner])

  case class B_inner_inner(fieldB_B: String)
  case class B_inner(fieldB: Option[B_inner_inner])
  case class B(field: List[B_inner])

  private val transformer = Transformer.define[A, B]
    .withFieldRenamed(_.field.everyItem.fieldA, _.field.everyItem.fieldB)
    .withFieldRenamed(_.field.everyItem.fieldA.matchingSome.fieldA_A, _.field.everyItem.fieldB.matchingSome.fieldB_B)
    .enableMacrosLogging
    .buildTransformer

Compilation of code above fails with error:

 + Attempting expansion of rule ProductToProduct
                          + Resolved A_inner_inner getters: (`fieldA_A`: java.lang.String (ConstructorVal, declared)) and B_inner_inner constructor (`fieldB_B`: java.lang.String (ConstructorParameter, default = None)), using name-based matching
                          + B_inner_inner
                          |   field fieldB_B: renaming from _.matching[scala.Some[A_inner_inner]].value.fieldA_A is not allowed, try selecting only field names from _.field.everyItem.field.everyItem.fieldA.matching[scala.Some[A_inner_inner]].value (_.field.everyItem.field.everyItem.fieldA.matching[scala.Some[A_inner_inner]].value.field1.field2, etc)
                          | 

Notice double field in logs ( names from _.field.everyItem.field.everyItem.fieldA

To make this code compile it is enough to either remove Option from classes definition (and keep double nested withFieldRenamed but without matchingSome) or make fieldA and fieldB have the same name (have single withFieldRenamed).
I have read documentation, but I coundn't found any info about such limitation, but I could have missed something.
Is there any solution for this case?
I am using Scala 2.13.16 and Chimney 1.7.3

^{Originally posted by MateuszKubuszok March 28, 2025}

It's a bug, I believe that here and additional check would have to be made, to check if non-.field-ending path (Option/Either/Collection/Map) is a valid use case (in past it wasn't and they had to be enabled on an allow-list-like fashion).

[MateuszKubuszok]

@elpitero, if you are interested in fixing this issue, I can guide you. In the foreseeable future, I do not have time for Chimney maintenance, so I can reply and guide others but I am not doing any work here.

[elpitero]

Hi @MateuszKubuszok , I am interested in this issue. I prepared new test case and run it with very basic logging to get familiar with the source code. I think the issue is with prefixPath retrieved from srcJournal. it contains entries with doubled prefix like: _.field.everyItem.field.everyItem.fieldA. I wrote simple function that strips this unnecessary prefix (inside extractSrcByPath for this test case only) and derivation started to work correctly. I would appreciate your guidance where to look for misbehaving journaling function.

[MateuszKubuszok]

The issue lies in handling everything in

chimney/chimney/src/main/scala/io/scalaland/chimney/internal/compiletime/derivation/transformer/rules/TransformProductToProductRuleModule.scala

Lines 488 to 504 in 2d5e2b2

	def extractNestedSource(path: Path, extractedSrcValue: ExistentialExpr): DerivationResult[ExistentialExpr] =
	path match {
	case Path.Root =>
	DerivationResult.pure(extractedSrcValue)
	case Path.AtField(sourceName, path2) =>
	import extractedSrcValue.Underlying as ExtractedSourceValue, extractedSrcValue.value as extractedSrcExpr
	extractSource[ExtractedSourceValue](sourceName, extractedSrcExpr).flatMap { extractedSrcValue2 =>
	extractNestedSource(path2, extractedSrcValue2)
	}
	case path =>
	DerivationResult.notSupportedOperationFromPath[From, To, ExistentialExpr](
	operation,
	toName,
	path,
	ctx.srcJournal.last._1
	)
	}

It only accepts an unbroken chain of _.fieldName1.fieldName2.fieldName3.

The way it is used is that:

• it takes Path segments from the longest up to _

  chimney/chimney/src/main/scala/io/scalaland/chimney/internal/compiletime/derivation/transformer/rules/TransformProductToProductRuleModule.scala

  Line 507 in 2d5e2b2

  (prefixPath, prefixExpr) <- ctx.srcJournal.reverseIterator

• each of them matches against extractNestedSource (so even when some expr in srcJournal would match but contains everyItem, everyMapKey, etc it is dropped) -

  chimney/chimney/src/main/scala/io/scalaland/chimney/internal/compiletime/derivation/transformer/rules/TransformProductToProductRuleModule.scala

  Line 508 in 2d5e2b2

  newSourcePath <- sourcePath.drop(prefixPath).iterator

• then it accepts the first success

  chimney/chimney/src/main/scala/io/scalaland/chimney/internal/compiletime/derivation/transformer/rules/TransformProductToProductRuleModule.scala

  Lines 511 to 516 in 2d5e2b2

  	extractedNestedSourceCandidates.fold(extractedNestedSourceCandidates.next()) { (a, b) =>
  	// We're not using orElse because we want to:
  	// - find the first successful result
  	// - but NOT aggregate the errors, if everything fails, keep only the first error
  	a.recoverWith(errors => b.recoverWith(_ => DerivationResult.fail(errors)))
  	}

The journaling is working as intended, it does not need changing. The fault lies in the simplistic lookup that is a leftover from when the only allowed paths, were nested fields.

The function extractNestedSource needs to be updated in a smart way to not fail at certain path values that currently fall under case path => but which can be safely handled.

What we want is to find the closest expression in journal - which should be the longest prefix Path to sourcePath but apparently it requires some adjustment - and then it's either out value (prefix = path) or we need to additionally extract some nested field.

[elpitero]

Hi. I have prepared initial implementation here. It does fail at tests TotalTransformerSealedHierarchySpec and analogical at PartialTransformerSealedHierarchySpec:

(Some(colors2.Red): Option[colors2.Color])
        .into[Option[colors1.Color]]
        .withFieldComputedFrom(_.matchingSome.matching[colors2.Black.type])(_.matchingSome, blackIsRed)
        .transform ==> Some(colors1.Red)

with java.lang.ClassCastException: class io.scalaland.chimney.fixtures.colors2$Red$ cannot be cast to class io.scalaland.chimney.fixtures.colors2$Black$.
I think additional runtime check would be required, but I dont see any ready to use solution (other than ExprPromises PatternMatchCase).
I am also unsure of use iterator.next, but it does not seem to cause any problems

[MateuszKubuszok]

DerivationResult.pure(extractedSrcExpr.asInstanceOfExpr[Expected].as_??)`

This is unsafe and I don't think it can be fixed with pure + runtime check.

For special cases like this one, it should rather rely on deriveRecursiveTransformationExpr, updating correctly paths and current value, so that nested sealed rule would take care of it, but it might be tricky to get it right.