Handling Java enums defined with some utility methods

[kamilkloch]

It does work for Scala 2.
EDIT: It does work for Scala 3, when all the files are located in a single module.

Repo with the issue: https://github.com/kamilkloch/chimney_issues

For Scala 3 compilation fails with an error:

Chimney can't derive transformation from p1.Side to p2.Side

  macro expansion thrown exception!: java.lang.AssertionError: assertion failed: expected a term symbol, but received class Side:
  	scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
  	scala.quoted.runtime.impl.QuotesImpl$reflect$Ref$.apply(QuotesImpl.scala:475)
  	scala.quoted.runtime.impl.QuotesImpl$reflect$Ref$.apply(QuotesImpl.scala:474)
  	io.scalaland.chimney.internal.compiletime.datatypes.ProductTypesPlatform$ProductType$.parseConstructor$$anonfun$1$$anonfun$1(ProductTypesPlatform.scala:149)
  	io.scalaland.chimney.internal.compiletime.datatypes.SingletonTypes$SingletonType$.parse(SingletonTypes.scala:39)
  	io.scalaland.chimney.internal.compiletime.datatypes.SingletonTypes$SingletonType$.unapply(SingletonTypes.scala:50)
  	io.scalaland.chimney.internal.compiletime.datatypes.SingletonTypes$Singleton$.unapply(SingletonTypes.scala:15)
  	io.scalaland.chimney.internal.compiletime.derivation.transformer.rules.TransformToSingletonRuleModule$TransformToSingletonRule$.expand(TransformToSingletonRuleModule.scala:18)
  	io.scalaland.chimney.internal.compiletime.derivation.transformer.rules.TransformationRules.io$scalaland$chimney$internal$compiletime$derivation$transformer$rules$TransformationRules$Rule$$$_$expandRules$$anonfun$2(TransformationRules.scala:43)
  	io.scalaland.chimney.internal.compiletime.DerivationResult$.namedScope$$anonfun$2(DerivationResult.scala:218)

[kamilkloch]

I simplified the the repo https://github.com/kamilkloch/chimney_issues,
Java enums have no utility methods, the issue persists.

[MateuszKubuszok]

You can also check the constructor - normal enums are part of out test suite.

[kamilkloch]

You can also check the constructor - normal enums are part of out test suite.

I do not quite understand, is there something I could do?

[MateuszKubuszok]

Sorry, I think the repro is ok, I would have to look at the available symbols in the macro and see what is wrong there. Maybe an issue with how we're reading them or maybe another issue with reading Java sources by Scala.

[kamilkloch]

Is such bugfix something which a Scala 3 macro potato like myself could try to attack?

[MateuszKubuszok]

There is a stack trace, so it shuttle be possible:

• recently I moved chimney-macro-commons to a separate repository (in preparation to 2.0.0 pre releases), it might be easier to use some commit from before that
  • an alternative is to add debug prints to macro commons and locally publish a snapshot
• then you could print the symbols and compare what is being printed for cases when it works and cases when it does not

[kamilkloch]

When 2 Java enums are in the same module as the tranformation, it works, enableMacrosLogging gives

[info] -- Info: /Users/kamilk/github/kamilkloch/chimney_issues/module3/src/main/scala/TestTransformer.scala:10:12 
[info] 10 |    println(ss1.into[pp2.Side].enableMacrosLogging.transform)
[info]    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[info]    |
[info]    |+ Start derivation with context: ForTotal[From = pp1.Side, To = pp2.Side](src = side)(TransformerConfig(
[info]    ||   flags = TransformerFlags(displayMacrosLogging),
[info]    ||   localFlagsOverridden = false,
[info]    ||   runtimeOverrides = Vector(),
[info]    ||   preventImplicitSummoningForTypes = None
[info]    || ))
[info]    |+ Deriving Total Transformer expression from pp1.Side to pp2.Side with context:
[info]    || ForTotal[From = pp1.Side, To = pp2.Side](src = side)(TransformerConfig(
[info]    ||   flags = TransformerFlags(displayMacrosLogging),
[info]    ||   localFlagsOverridden = false,
[info]    ||   runtimeOverrides = Vector(),
[info]    ||   preventImplicitSummoningForTypes = None
[info]    || ))
[info]    |  + Attempting expansion of rule Implicit
[info]    |  + Rule Implicit decided to pass on to the next rule
[info]    |  + Attempting expansion of rule ImplicitOuterTransformer
[info]    |  + Rule ImplicitOuterTransformer decided to pass on to the next rule
[info]    |  + Attempting expansion of rule ImplicitConversion
[info]    |  + Rule ImplicitConversion decided to pass on to the next rule - some conditions were fulfilled but at least one failed: Implicit conversions are disabled
[info]    |  + Attempting expansion of rule Subtypes
[info]    |  + Rule Subtypes decided to pass on to the next rule
[info]    |  + Attempting expansion of rule TypeConstraint
[info]    |  + Rule TypeConstraint decided to pass on to the next rule - some conditions were fulfilled but at least one failed: <:< evidence is disabled
[info]    |  + Attempting expansion of rule ToSingleton
[info]    |  + Rule ToSingleton decided to pass on to the next rule
[info]    |  + Attempting expansion of rule OptionToOption
[info]    |  + Rule OptionToOption decided to pass on to the next rule
[info]    |  + Attempting expansion of rule PartialOptionToNonOption
[info]    |  + Rule PartialOptionToNonOption decided to pass on to the next rule
[info]    |  + Attempting expansion of rule ToOption
[info]    |  + Rule ToOption decided to pass on to the next rule
[info]    |  + Attempting expansion of rule ValueClassToValueClass
[info]    |  + Rule ValueClassToValueClass decided to pass on to the next rule
[info]    |  + Attempting expansion of rule ValueClassToType
[info]    |  + Rule ValueClassToType decided to pass on to the next rule
[info]    |  + Attempting expansion of rule TypeToValueClass
[info]    |  + Rule TypeToValueClass decided to pass on to the next rule
[info]    |  + Attempting expansion of rule EitherToEither
[info]    |  + Rule EitherToEither decided to pass on to the next rule
[info]    |  + Attempting expansion of rule MapToMap
[info]    |  + Rule MapToMap decided to pass on to the next rule
[info]    |  + Attempting expansion of rule IterableToIterable
[info]    |  + Rule IterableToIterable decided to pass on to the next rule
[info]    |  + Attempting expansion of rule ProductToProduct
[info]    |  + Rule ProductToProduct decided to pass on to the next rule - some conditions were fulfilled but at least one failed: Type pp2.Side does not have a public primary constructor NOR exactly one (non-primary) public constructor
[info]    |  + Attempting expansion of rule SealedHierarchyToSealedHierarchy
[info]    |    + Resolved pp1.Side subtypes: (pp1.Side.BUY, pp1.Side.SELL) and pp2.Side subtypes (pp2.Side.BUY, pp2.Side.SELL)
[info]    |    + Deriving Total Transformer expression from pp1.Side.BUY to pp2.Side.BUY with context:
[info]    |    | ForTotal[From = pp1.Side.BUY, To = pp2.Side.BUY](src = buy)(TransformerConfig(
[info]    |    |   flags = TransformerFlags(displayMacrosLogging),
[info]    |    |   localFlagsOverridden = false,
[info]    |    |   runtimeOverrides = Vector(),
[info]    |    |   preventImplicitSummoningForTypes = None
[info]    |    | ))
[info]    |      + Attempting expansion of rule Implicit
[info]    |      + Rule Implicit decided to pass on to the next rule
[info]    |      + Attempting expansion of rule ImplicitOuterTransformer
[info]    |      + Rule ImplicitOuterTransformer decided to pass on to the next rule
[info]    |      + Attempting expansion of rule ImplicitConversion
[info]    |      + Rule ImplicitConversion decided to pass on to the next rule - some conditions were fulfilled but at least one failed: Implicit conversions are disabled
[info]    |      + Attempting expansion of rule Subtypes
[info]    |      + Rule Subtypes decided to pass on to the next rule
[info]    |      + Attempting expansion of rule TypeConstraint
[info]    |      + Rule TypeConstraint decided to pass on to the next rule - some conditions were fulfilled but at least one failed: <:< evidence is disabled
[info]    |      + Attempting expansion of rule ToSingleton
[info]    |      + Rule ToSingleton expanded successfully: pp2.Side.BUY
[info]    |    + Derived recursively total expression pp2.Side.BUY
[info]    |    + Deriving Total Transformer expression from pp1.Side.SELL to pp2.Side.SELL with context:
[info]    |    | ForTotal[From = pp1.Side.SELL, To = pp2.Side.SELL](src = sell)(TransformerConfig(
[info]    |    |   flags = TransformerFlags(displayMacrosLogging),
[info]    |    |   localFlagsOverridden = false,
[info]    |    |   runtimeOverrides = Vector(),
[info]    |    |   preventImplicitSummoningForTypes = None
[info]    |    | ))
[info]    |      + Attempting expansion of rule Implicit
[info]    |      + Rule Implicit decided to pass on to the next rule
[info]    |      + Attempting expansion of rule ImplicitOuterTransformer
[info]    |      + Rule ImplicitOuterTransformer decided to pass on to the next rule
[info]    |      + Attempting expansion of rule ImplicitConversion
[info]    |      + Rule ImplicitConversion decided to pass on to the next rule - some conditions were fulfilled but at least one failed: Implicit conversions are disabled
[info]    |      + Attempting expansion of rule Subtypes
[info]    |      + Rule Subtypes decided to pass on to the next rule
[info]    |      + Attempting expansion of rule TypeConstraint
[info]    |      + Rule TypeConstraint decided to pass on to the next rule - some conditions were fulfilled but at least one failed: <:< evidence is disabled
[info]    |      + Attempting expansion of rule ToSingleton
[info]    |      + Rule ToSingleton expanded successfully: pp2.Side.SELL
[info]    |    + Derived recursively total expression pp2.Side.SELL
[info]    |  + Rule SealedHierarchyToSealedHierarchy expanded successfully: (side: pp1.Side @scala.unchecked) match {
[info]    |  |   case buy @ pp1.Side.BUY =>
[info]    |  |     val buy: pp1.Side.BUY = buy
[info]    |  |     {
[info]    |  |       (buy: pp1.Side.BUY @scala.unchecked) match {
[info]    |  |         case _ =>
[info]    |  |           ()
[info]    |  |       }
[info]    |  |       ()
[info]    |  |     }
[info]    |  |     pp2.Side.BUY
[info]    |  |   case sell @ pp1.Side.SELL =>
[info]    |  |     val sell: pp1.Side.SELL = sell
[info]    |  |     {
[info]    |  |       (sell: pp1.Side.SELL @scala.unchecked) match {
[info]    |  |         case _ =>
[info]    |  |           ()
[info]    |  |       }
[info]    |  |       ()
[info]    |  |     }
[info]    |  |     pp2.Side.SELL
[info]    |  | }
[info]    |+ Derived final expression is:
[info]    || (side: pp1.Side @scala.unchecked) match {
[info]    ||   case buy @ pp1.Side.BUY =>
[info]    ||     val buy: pp1.Side.BUY = buy
[info]    ||     {
[info]    ||       (buy: pp1.Side.BUY @scala.unchecked) match {
[info]    ||         case _ =>
[info]    ||           ()
[info]    ||       }
[info]    ||       ()
[info]    ||     }
[info]    ||     pp2.Side.BUY
[info]    ||   case sell @ pp1.Side.SELL =>
[info]    ||     val sell: pp1.Side.SELL = sell
[info]    ||     {
[info]    ||       (sell: pp1.Side.SELL @scala.unchecked) match {
[info]    ||         case _ =>
[info]    ||           ()
[info]    ||       }
[info]    ||       ()
[info]    ||     }
[info]    ||     pp2.Side.SELL
[info]    || }
[info]    |+ Derivation took 0.077175000 s

For the failing scenario (Java enums in separate modules), it falls short at ToSingleton:

[info] -- Info: /Users/kamilk/github/kamilkloch/chimney_issues/module3/src/main/scala/TestTransformer.scala:7:12 
[info] 7 |    println(s1.into[p2.Side].enableMacrosLogging.transform)
[info]   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[info]   |
[info]   |+ Start derivation with context: ForTotal[From = p1.Side, To = p2.Side](src = side)(TransformerConfig(
[info]   ||   flags = TransformerFlags(displayMacrosLogging),
[info]   ||   localFlagsOverridden = false,
[info]   ||   runtimeOverrides = Vector(),
[info]   ||   preventImplicitSummoningForTypes = None
[info]   || ))
[info]   |+ Deriving Total Transformer expression from p1.Side to p2.Side with context:
[info]   || ForTotal[From = p1.Side, To = p2.Side](src = side)(TransformerConfig(
[info]   ||   flags = TransformerFlags(displayMacrosLogging),
[info]   ||   localFlagsOverridden = false,
[info]   ||   runtimeOverrides = Vector(),
[info]   ||   preventImplicitSummoningForTypes = None
[info]   || ))
[info]   |  + Attempting expansion of rule Implicit
[info]   |  + Rule Implicit decided to pass on to the next rule
[info]   |  + Attempting expansion of rule ImplicitOuterTransformer
[info]   |  + Rule ImplicitOuterTransformer decided to pass on to the next rule
[info]   |  + Attempting expansion of rule ImplicitConversion
[info]   |  + Rule ImplicitConversion decided to pass on to the next rule - some conditions were fulfilled but at least one failed: Implicit conversions are disabled
[info]   |  + Attempting expansion of rule Subtypes
[info]   |  + Rule Subtypes decided to pass on to the next rule
[info]   |  + Attempting expansion of rule TypeConstraint
[info]   |  + Rule TypeConstraint decided to pass on to the next rule - some conditions were fulfilled but at least one failed: <:< evidence is disabled
[info]   |+ Derivation took 0.034500000 s

[MateuszKubuszok]

It might be something about how macros sees them, maybe there is a difference between enums defined in the same module as macro expansion and a different one?

[kamilkloch]

Please take a look:

p2.Side.SELL // <- commenting out this line breaks compilation
println(s1.into[p2.Side].enableMacrosLogging.transform)

https://github.com/kamilkloch/chimney_issues/blob/d97946d9cf6309a7271cdc242966f3880c40b94c/module3/src/main/scala/TestTransformer.scala#L7

[MateuszKubuszok]

What I'm was trying to say, is that our macro logs are too high abstraction layer to debug this issue solely with them. They can only point to the last cross-platform implementation before the change in the implementation - which was also pointed out by the stack trace.

Beyond that we need to look at the platform-specific part and see what are the Symbols, their flags, what they resolve to and what is the difference on quotes.reflect API level:

// somewhere around here:
//...
        val A = TypeRepr.of[A]
        // Workaround for different Symbol used when we obtain things from SealedHierarchies and passed by user as
        // Enum.Value.type
        val sym = if A.termSymbol != Symbol.noSymbol then A.termSymbol else A.typeSymbol

        if isCaseVal[A] || isJavaEnumValue[A] then Some(Product.Constructor(ListMap.empty, _ => Ref(sym).asExprOf[A]))
        else Some(Product.Constructor(ListMap.empty, _ => Ref(sym.companionModule).asExprOf[A]))
// ...

I would start by checking why isJavaEnumValue[A] apparently returned false in this case, because if it was true the branch with

Ref(sym.companionModule) // assertion error is thrown at this line

would not be picked, in favor of a branch with

Ref(sym) // branch dedicated to e.g. Java enums

def isJavaEnumValue[A: Type]: Boolean =
  Type[A] <:< scala.quoted.Type.of[java.lang.Enum[?]] && !TypeRepr.of[A].typeSymbol.isAbstract

That might be a bug on Chimeny side or bug in Scala 3 macros API when it comes to parsing Java APIs (@jchyb knows more about that).

[zorba128]

This exactly where we dig into. The isJavaEnumValue is wrongly true, probably because of isAbstract. I added the case

      // to align API between Scala versions
      extension (sym: Symbol) {
        def isAbstract: Boolean = !sym.isNoSymbol && sym.flags.is(Flags.Abstract) || sym.flags.is(Flags.Trait) ||   
               (sym.flags.is(Flags.Enum) && sym.flags.is(Flags.JavaDefined) && !sym.flags.is(Flags.JavaStatic))

This fixed exception, but still I need to figure out why Type[JavaEnum] is not resolved to SealedHierarchy causing SealedHierarchyToSealedHierarchy rule not to fire.

Edit:
Added some logging when evaluating SealedHierarchy.isSealed - and got different set of flags for similar type when it was referenced (p1.Side)/not referenced (p2.Side). Namely Flags.Sealed and Flags.Abstract are missing...

isSealed(p1.Side)=true, flags=Flags.Abstract | Flags.Enum | Flags.Final | Flags.JavaDefined | Flags.Sealed, 
typeSymbol=TypeDef(Side,Template(DefDef(<init>,List(List()),TypeTree[TypeRef(ThisType(TypeRef(NoPrefix,module class p1)),class Side)],EmptyTree),List(TypeTree[AppliedType(TypeRef(ThisType(TypeRef(NoPrefix,module class lang)),class Enum),List(TypeRef(ThisType(TypeRef(NoPrefix,module class p1)),class Side)))]),ValDef(_,EmptyTree,EmptyTree),List()))

isSealed(p2.Side)=false, flags=Flags.Enum | Flags.Final | Flags.JavaDefined, 
typeSymbol=TypeDef(Side,Template(DefDef(<init>,List(List()),TypeTree[TypeRef(ThisType(TypeRef(NoPrefix,module class p2)),class Side)],EmptyTree),List(TypeTree[AppliedType(TypeRef(ThisType(TypeRef(NoPrefix,module class lang)),class Enum),List(TypeRef(ThisType(TypeRef(NoPrefix,module class p2)),class Side)))]),ValDef(_,EmptyTree,EmptyTree),List()))

[MateuszKubuszok]

It is possible that we use Flags incorrectly - I had to figure out it case by case, and it only works against the cases that I wrote code for - but there was already some issues with Scala 3 and Java sources - scala/scala3#20052. cc @jchyb