Running updatecli against YAML containing escaped strings fails

[jcmcken]

For example, use this config and replace '<{{ source "currentTraefikVersion" }}' with "<{{ source \"currentTraefikVersion\" }}". According to every other YAML parser I've tried, these are equivalent strings.

But when I updatecli apply the config with the escaped quotes, I get an error:

template: cfg:17: unexpected "\\" in operand

This appears to be related to the same problem from this issue. However my problem is I'm generating the YAML using jsonnet, which doesn't let me influence how the strings get rendered (jsonnet always renders with the escaped double quotes). So I can't just use the single-quote method

As the person in the Helm issue above complains, this appears to be a bug in Helm's YAML parser, yet they have seemingly refused to fix it. I'm unsure how exactly Helm's code comes into updatecli in this case, but I'm wondering what workarounds there might be. Or if this bug could somehow be fixed in updatecli even if it's still broken in Helm.

[jcmcken]

I found a solution in jsonnet but it's a little messy and difficult for someone to understand at a glance why it's needed.

Before:

{
  foo: '<{{ source "blah" }}',
}

After:

{
  foo: |||
    <{{ source "blah" }}
  |||,
}

This hints to jsonnet that it should put the string in a YAML block notation like:

foo: |
  <{{ source "blah" }}

[jcmcken]

In other cases, this workaround doesn't actually work. For example, if I use a yamlpath in a key, the presence of the newline due to the YAML block notation seems to cause it to error. If I change the YAML from | to |-, then this mechanism works fine.

E.g. this does not work:

engine: yamlpath
key: |
  $.spec.foo[*][?(@name=='{{ source "blah" }}')].version

And this does not work:

engine: yamlpath
key: "$.spec.foo[*][?(@name=='{{ source \"blah\" }}')].version"

But this does work:

engine: yamlpath
key: |-
  $.spec.foo[*][?(@name=='{{ source "blah" }}')].version

Unfortunately, there's no way to get jsonnet to output the "chomped" block style |-

Again, from other YAML parsers I've used, the last two examples should be identical strings.

[olblak]

Thanks for the detailed case.

In YAML '<{{ source "currentTraefikVersion" }}' and "<{{ source "currentTraefikVersion" }}" are not equivalent

=> https://www.kcl-lang.io/docs/user_docs/support/faq-yaml#1-what-is-the-difference-between-single-and-double-quote-yaml-strings

As the person in the Helm issue above complains, this appears to be a bug in Helm's YAML parser,

I am not sure if you shared the correct issue 😄

• Release.Revision is empty :( helm/helm#1944 I see it closed

Since you mentioned Helm, is this a technique that would work for Helm?
On the Updatecli project side, we ended up taking the same approach as the Helm project.

We run one Go templating at start time and then after each resource we run a second one to retrieve information for {{ source "xxx" }} which we only know at runtime and only then we execute the YAML parser, so the generated manifest must be YAML compliant

When the generated value contains specials characters, then we often use:

key: |-
  $.spec.foo[*][?(@name=='{{ source "blah" }}')].version

If that interest you, a while ago, I experimented with Cuelang.
#2008

I don't see much interest, so I stopped investigating more, but I think It could be useful for more complex pipelines

[jcmcken]

Thanks for the detailed case.

In YAML '<{{ source "currentTraefikVersion" }}' and "<{{ source "currentTraefikVersion" }}" are not equivalent

=> https://www.kcl-lang.io/docs/user_docs/support/faq-yaml#1-what-is-the-difference-between-single-and-double-quote-yaml-strings

Those strings are not equivalent, true. In fact, I don't think the second one isn't even valid YAML. The equivalent string would have escaped double quotes, e.g.:

somekey: "<{{ source \"currentTraefikVerison\" }}"

There's an example of this usage in the YAML spec

I am not sure if you shared the correct issue

Yes you're right. URL is wrong. Not immediately finding the issue I was looking at. Let me look for it

We run one Go templating at start time and then after each resource we run a second one to retrieve information for {{ source "xxx" }} which we only know at runtime and only then we execute the YAML parser, so the generated manifest must be YAML compliant

I'm thinking in terms of something like Ansible YAML. There, I believe the YAML is parsed first, and then the template values are later replaced. One reason for that is files which you call "whatever.yaml", you would expect to first and foremost be legitimate YAML. Other templating/behavior within each string value would be "in-band signalling" that gets resolved later.

The approach you take makes sense if you want to provide dynamism over the entire document vs dynamism only within a string value. But in that case, my strategy of using jsonnet (or even really any other tool that generates YAML) won't work in general. Since it's not really YAML you're working with, but a go template file (that happens to later resolve into a YAML string).

I guess if there was a way to configure the ordering, that would help in my case, i.e. instead of go -> go -> yaml, resolve the templates like yaml -> go -> go. But I understand that's not a trivial thing to make configurable