Cannot compile PureConfig with SHocon instead of typesafe config

[melrief]

I've tried to use shocon as replacement for typesafe config as backend of pureconfig. I've pushed the branch try_shocon_as_backend if you are interested. The project doesn't compile because shocon API is missing some parts of typesafe config API.

To reproduce, just run sbt "++ 2.12.1 core/compile". The error list is

[info] Compiling 9 Scala sources to /tmp/sbt/home/mariop/prog/pureconfig/core/scala-2.12/classes...
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/BasicConverters.scala:110: not found: type ConfigList
[error]   implicit val configListConfigConvert: ConfigConvert[ConfigList] = new ConfigConvert[ConfigList] {
[error]                                                       ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/error/ConfigReaderFailure.scala:8: object ConfigOrigin is not a member of package com.typesafe.config
[error] import com.typesafe.config.{ ConfigOrigin, ConfigValue }
[error]        ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/error/ConfigReaderFailure.scala:50: not found: type ConfigOrigin
[error]   def apply(co: ConfigOrigin): Option[ConfigValueLocation] =
[error]                 ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/BasicConverters.scala:93: value valueType is not a member of com.typesafe.config.ConfigValue
[error]       case other => fail(WrongType(other.valueType().toString, "ConfigObject", ConfigValueLocation(config), None))
[error]                                          ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/BasicConverters.scala:101: value valueType is not a member of com.typesafe.config.ConfigValue
[error]       case other => fail(WrongType(other.valueType().toString, "ConfigObject", ConfigValueLocation(config), None))
[error]                                          ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/BasicConverters.scala:110: not found: type ConfigList
[error]   implicit val configListConfigConvert: ConfigConvert[ConfigList] = new ConfigConvert[ConfigList] {
[error]                                                                                       ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/BasicConverters.scala:111: not found: type ConfigList
[error]     override def from(config: ConfigValue): Either[ConfigReaderFailures, ConfigList] = config match {
[error]                                                                          ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/BasicConverters.scala:115: not found: type ConfigList
[error]     override def to(t: ConfigList): ConfigValue = t
[error]                        ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/BasicConverters.scala:112: not found: type ConfigList
[error]       case c: ConfigList => Right(c)
[error]               ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/BasicConverters.scala:113: value valueType is not a member of com.typesafe.config.ConfigValue
[error]       case other => fail(WrongType(other.valueType().toString, "ConfigList", ConfigValueLocation(config), None))
[error]                                          ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/ConvertHelpers.scala:46: value valueType is not a member of com.typesafe.config.ConfigValue
[error]         val string = config.valueType match {
[error]                             ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/ConvertHelpers.scala:47: not found: value ConfigValueType
[error]           case ConfigValueType.STRING => config.unwrapped.toString
[error]                ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/ConvertHelpers.scala:47: value unwrapped is not a member of com.typesafe.config.ConfigValue
[error]           case ConfigValueType.STRING => config.unwrapped.toString
[error]                                                 ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/ConvertHelpers.scala:48: no arguments allowed for nullary method render: ()String
[error]           case _ => config.render(ConfigRenderOptions.concise)
[error]                                                       ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/ConvertHelpers.scala:94: not found: value ConfigValueFactory
[error]     override def to(t: T): ConfigValue = ConfigValueFactory.fromAnyRef(t)
[error]                                          ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/ConvertHelpers.scala:99: not found: value ConfigValueFactory
[error]     override def to(t: T): ConfigValue = ConfigValueFactory.fromAnyRef(t)
[error]                                          ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/ConvertHelpers.scala:133: not found: value ConfigValueFactory
[error]     override def to(t: T): ConfigValue = ConfigValueFactory.fromAnyRef(toF(t))
[error]                                          ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/CoproductHint.scala:72: value unwrapped is not a member of com.typesafe.config.ConfigValue
[error]         case Some(fv) => fv.unwrapped match {
[error]                             ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/CoproductHint.scala:75: value valueType is not a member of com.typesafe.config.ConfigValue
[error]           case _ => Left(ConfigReaderFailures(WrongType(fv.valueType.toString, expectedType = "String", ConfigValueLocation(fv), Some(key))))
[error]                                                            ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/CoproductHint.scala:79: value valueType is not a member of com.typesafe.config.ConfigValue
[error]     case _ => Left(ConfigReaderFailures(WrongType(cv.valueType.toString, expectedType = "ConfigObject", ConfigValueLocation(cv), None)))
[error]                                                      ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:178: not found: type ConfigList
[error]         case co: ConfigList =>
[error]                  ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:182: value asScala is not a member of com.typesafe.config.ConfigValue
[error]  Note: implicit method deriveTraversable is not applicable here because it comes after the application point and it lacks an explicit result type
[error]           co.asScala.foldLeft(z) {
[error]              ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:184: type mismatch;
[error]  found   : Any
[error]  required: Either[pureconfig.error.ConfigReaderFailures,?]
[error]  Note: implicit method deriveTraversable is not applicable here because it comes after the application point and it lacks an explicit result type
[error]               combineResults(acc, configConvert.value.from(value))(_ += _)
[error]                              ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:184: type mismatch;
[error]  found   : Any
[error]  required: com.typesafe.config.ConfigValue
[error]  Note: implicit method deriveTraversable is not applicable here because it comes after the application point and it lacks an explicit result type
[error]               combineResults(acc, configConvert.value.from(value))(_ += _)
[error]                                                            ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:189: ambiguous reference to overloaded definition,
[error] both method apply in object ConfigValueLocation of type (co: <error>)Option[pureconfig.error.ConfigValueLocation]
[error] and  method apply in object ConfigValueLocation of type (cv: com.typesafe.config.ConfigValue)Option[pureconfig.error.ConfigValueLocation]
[error] match argument types (com.typesafe.config.ConfigValue) and expected result type Option[pureconfig.error.ConfigValueLocation]
[error]             val keyResult = catchReadError(_.toInt)(implicitly)(key)(ConfigValueLocation(value)).left.flatMap(t => fail(CannotConvert(key, "Int",
[error]                                                                      ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:206: value valueType is not a member of com.typesafe.config.ConfigValue
[error]  Note: implicit method deriveTraversable is not applicable here because it comes after the application point and it lacks an explicit result type
[error]           fail(WrongType(other.valueType().toString, "ConfigList or ConfigObject", ConfigValueLocation(other), None))
[error]                                ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:211: not found: value ConfigValueFactory
[error]       ConfigValueFactory.fromIterable(ts.toList.map(configConvert.value.to).asJava)
[error]       ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:228: value valueType is not a member of com.typesafe.config.ConfigValue
[error]  Note: implicit method deriveMap is not applicable here because it comes after the application point and it lacks an explicit result type
[error]           fail(WrongType(other.valueType().toString, "ConfigObject", ConfigValueLocation(other), None))
[error]                                ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:233: not found: value ConfigValueFactory
[error]       ConfigValueFactory.fromMap(keyVals.mapValues(configConvert.value.to).asJava)
[error]       ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/CoproductHint.scala:85: type mismatch;
[error]  found   : com.typesafe.config.Config
[error]  required: com.typesafe.config.ConfigMergeable
[error]       else Right(Map(key -> fieldValue(name)).toConfig.withFallback(co.toConfig))
[error]                                                                        ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/CoproductHint.scala:88: value valueType is not a member of com.typesafe.config.ConfigValue
[error]       Left(ConfigReaderFailures(WrongType(cv.valueType.toString, "ConfigObject", ConfigValueLocation(cv), None)))
[error]                                              ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:31: value valueType is not a member of com.typesafe.config.ConfigValue
[error]       case other => fail(WrongType(foundType = other.valueType().toString, expectedType = "ConfigObject", ConfigValueLocation(other), None))
[error]                                                      ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:85: value withoutKey is not a member of com.typesafe.config.ConfigObject
[error]       val tailCo = if (hint.allowUnknownKeys) co else co.withoutKey(keyStr)
[error]                                                          ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:98: value withValue is not a member of com.typesafe.config.ConfigObject
[error]               rem.asInstanceOf[ConfigObject].withValue(keyStr, v)
[error]                                              ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:104: value withValue is not a member of com.typesafe.config.ConfigObject
[error]           rem.asInstanceOf[ConfigObject].withValue(keyStr, fieldEntry)
[error]                                          ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:156: value unwrapped is not a member of com.typesafe.config.ConfigValue
[error]       if (config == null || config.unwrapped() == null)
[error]                                    ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/DerivedConverters.scala:164: not found: value ConfigValueFactory
[error]       case None => ConfigValueFactory.fromAnyRef(null)
[error]                    ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/backend/ConfigFactoryWrapper.scala:18: value invalidateCaches is not a member of object com.typesafe.config.ConfigFactory
[error]     unsafeToEither(ConfigFactory.invalidateCaches())
[error]                                  ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/backend/ConfigFactoryWrapper.scala:22: shocon - statically reading configuration from file:/tmp/sbt/home/mariop/prog/pureconfig/core/scala-2.12/classes/application.conf
[error]     unsafeToEither(ConfigFactory.load())
[error]                                      ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/backend/ConfigFactoryWrapper.scala:26: value parseFile is not a member of object com.typesafe.config.ConfigFactory
[error]     unsafeToEither(ConfigFactory.parseFile(path.toFile))
[error]                                  ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/backend/ConfigFactoryWrapper.scala:34: type Parse is not a member of object com.typesafe.config.ConfigException
[error]       case e: ConfigException.Parse =>
[error]                               ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/backend/ConfigFactoryWrapper.scala:35: value origin is not a member of Throwable
[error]         Left(ConfigReaderFailures(CannotParse(e.getLocalizedMessage, ConfigValueLocation(e.origin()))))
[error]                                                                                            ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/backend/ConfigFactoryWrapper.scala:37: value origin is not a member of com.typesafe.config.ConfigException
[error]         Left(ConfigReaderFailures(ThrowableFailure(e, ConfigValueLocation(e.origin()), None)))
[error]                                                                             ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/error/ConfigReaderFailure.scala:36: value origin is not a member of com.typesafe.config.ConfigValue
[error]     Option(cv).flatMap(v => apply(v.origin()))
[error]                                     ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/error/ConfigReaderFailure.scala:8: Unused import
[error] import com.typesafe.config.{ ConfigOrigin, ConfigValue }
[error]                              ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/package.scala:46: ambiguous reference to overloaded definition,
[error] both method apply in object ConfigValueLocation of type (co: <error>)Option[pureconfig.error.ConfigValueLocation]
[error] and  method apply in object ConfigValueLocation of type (cv: com.typesafe.config.ConfigValue)Option[pureconfig.error.ConfigValueLocation]
[error] match argument types (java.util.AbstractMap[String,com.typesafe.config.ConfigValue] with com.typesafe.config.ConfigObject) and expected result type Option[pureconfig.error.ConfigValueLocation]
[error]       config <- improveFailures[Config](loadConfig[Config](rawConfig)(conv), namespace, ConfigValueLocation(rawConfig.root())).right
[error]                                                                                         ^
[error] /home/mariop/prog/pureconfig/core/src/main/scala/pureconfig/package.scala:79: ambiguous reference to overloaded definition,
[error] both method apply in object ConfigValueLocation of type (co: <error>)Option[pureconfig.error.ConfigValueLocation]
[error] and  method apply in object ConfigValueLocation of type (cv: com.typesafe.config.ConfigValue)Option[pureconfig.error.ConfigValueLocation]
[error] match argument types (java.util.AbstractMap[String,com.typesafe.config.ConfigValue] with com.typesafe.config.ConfigObject) and expected result type Option[pureconfig.error.ConfigValueLocation]
[error]       config <- improveFailures[Config](loadConfig[Config](rawConfig)(conv), namespace, ConfigValueLocation(rawConfig.root())).right
[error]                                                                                         ^
[error] 47 errors found
[error] (core/compile:compileIncremental) Compilation failed
[error] Total time: 15 s, completed Mar 23, 2017 9:31:51 PM

[andreaTP]

Thanks a lot for this report!
From this trace I can figure out that there are at least 3 kind of errors we can classify and try to manage:

• Trivial implementation -> such as unwrapped (just need to be implemented) or ConfigFactory.load() just need to remove -X:fatalWarnings
• Semantics differences, since Shocon is basically developed to be cross-platform we haven't intentionally implemented things like invalidateCaches() is it possible on your side to move this kind of calls to a separate modue that will be "JVM specific"?
• In between, need to understand better how things like valueType could work here.

[mfirry]

@andreaTP did you find 10 minutes to have a look at this?

[evacchi]

TBF I'm not sure it makes sense to implement the missing Java shims so that PureConfig can then turn them back into its own internal AST repr :P it would make sense to target SHocon's own native API or to agree upon a common repr we can use both :-)

[nightscape]

I'm trying to pull off the same thing for zio-config and running into the same problems.
I'm also not sure about how to proceed here. I could use SHocon's native API in one module and the Typesafe API in another module, but I'd either have to duplicate quite a bit of code or complicate the architecture by finding ways to deal with the stuff that is in Typesafe but not in SHocon.
Does somebody have an intuition how complicated it would be to port the following classes/methods?

• ConfigList
• ConfigValueFactory
• withValue
• unwrapped
• atKey

[andreaTP]

Hi @nightscape !
Thanks for the interest!
Let me try to have a first peek:

• ConfigList should just be a matter of wrapping values accordingly in the facade since shocon is already able to parse List of Configs: https://github.com/akka-js/shocon/blob/master/facade/shared/src/main/scala/com/typesafe/config/Config.scala#L186
• ConfigValueFactory again looks like is just a facade https://lightbend.github.io/config/latest/api/com/typesafe/config/ConfigValueFactory.html should be implemented, requires solely someone's time IIUC
• withValue / atKey to be implemented in the facade since they are helper methods to insert/mangle the Config Tree (note that using them we are possibly going to invalidate most of the macro based optimizations)
• unwrapped is already implemented and should work (of course, modulo bugs)

  shocon/facade/shared/src/test/scala/SHoconGenericSpec.scala

  Lines 280 to 293 in 29bcb17

  	'unwrappedNumber - {
  	val map = ConfigFactory.parseString(""" a=2 """).root.unwrapped
  	assert { 2 == map.get("a") }
  	}
  	'unwrappedDuration - {
  	val map = ConfigFactory.parseString(""" a=2ns """).root.unwrapped
  	assert { "2ns" == map.get("a") } // Duration is not automatically unwrapped.
  	}
  	'unwrappedBoolean - {
  	val map = ConfigFactory.parseString(""" a=true """).root.unwrapped
  	assert { true == map.get("a") }
  	}

All in all looks pretty doable to me having time to spend on the subject(e.g. carefully reading the specification and the code of the original implementation, checking edge cases etc.. ).

I'm not really focused on improving this library by myself at this point (and have other priorities), but I'm available for guidance/merging PRs/releasing accordingly if anyone wants to pick up the challenge!

[nightscape]

@andreaTP thanks for the estimation & tips!
I need to finish another part of the project I'm working on, but I'll try to create a PR once I'm in cleanup mode 😉