multi-inputs file name collisions cause silent information loss - prev.: QUESTION: When/How is `/valohai/config/inputs.json` created ?

[Kenneth-Schroeder]

When/How is /valohai/config/inputs.json created?
I noticed that when starting a valohai step (in my case adhoc), in valohai-utils, load_global_state() updates the inputs through the load_inputs_from_config() function - which accesses /valohai/config/inputs.json.
Unfortunately, I can't figure out how, when or through which code this file is created.

The issue I want to solve is that multiple inputs with the same name (and parent directories) are overwritten by each other.
The (magically-created) /valohai/config/inputs.json file however, is already removing items with duplicate names.

I have already made a few contributions to valohai, but I can't find the source of /valohai/config/inputs.json this time...

@akx @tokkoro

[ruksi]

yes, passing multiple files with the same name to a step input keep their names and overwrite each other

not a bug per se, but an implementation where we've followed what one might expect, although sometimes unwanted

changes to this would need a bit of design 🤔

let say I get two files, data.tar.gz and data.tar.gz, should they become:

• data.tar.gz and data.2.tar.gz (⬅️ probably?)
• data.tar.gz and data-2.tar.gz (dash alternative)
• data.tar.gz and data.tar.gz.2 (would break the meaning of file extension)
• what if the filenames don't have an extension? rare, but possible
• possibly an opt-in flag to the input so existing pipelines wouldn't break as some flows might rely on this behavior?
• should we allow users to specify on the step definition how the duplicate names are generated?

and ofc this new behavior should be documented and highlighted to happen in the execution logs

Valohai executions do warn about a input file name clash with an execution alert, this should be shown in both execution details and executions table if alert column is visible.

the /valohai/config/inputs.json is generated by a Valohai worker initialization process so valohai-utils is only a reader of the file but does not generate it, it's effectively written after all the files have been resolved and downloaded (and overridden 😥)

the same filename on different Valohai inputs do not clash however, but if the number of files you will be passing is "dynamic", that ain't a really scaleable solution either 😅

[ruksi]

that is to say, the solving of this issue would be on our end, but let me know if I can be of any help regarding your use case in the meantime 😅

[Kenneth-Schroeder]

Hi ruksi, thank you for the fast response.
I would strongly argue for an opt-in option with dash or underscore name extension.
So data.tar.gz becomes data-2.tar.gz (or data_2.tar.gz).
This should not break any extensions or file name parsing logic, should work for files without extensions as well and should be compatible with existing deployments.

Not sure about the implementation details or your private repos, but I think it would be nice to be able to enable this feature for individual inputs through the yaml or the valohai.inputs() function.

We indeed have a use-case with a dynamic number of inputs, so I guess we will stick to local execution with a separate download script for the time being 😄
It would be great if you could introduce this feature though!

[ruksi]

This should not break any extensions or file name parsing logic, should work for files without extensions as well and should be compatible with existing deployments.

yeah, I doubt that anyone is really relying on this duplicates-override-each-other behavior because the overriding order is undetermined as far as I can see (depends in the order the downloads run), although didn't dig too deep

It would be great if you could introduce this feature though!

a reasonable feature request, I'll +1 the related issue we have internally and link to this issue!

[Kenneth-Schroeder]

It might also make sense to add directories instead of appending to the file names.
So data.tar.gz could become input1/data.tar.gz.
This way, logic based on file names would not be affected and there might be fewer issues with unzipping.

[Kenneth-Schroeder]

FYI / small update: We stumbled across one of our critical executions where we missed the fact that same-name files are overwritten atm. I believe this could easily happen to many users and in many use cases/executions (even though there might be the warning in the logs).
In addition to the fact that the overwriting order is random, we would rate this as high priority issue for us now.

[Kenneth-Schroeder]

Hey @ruksi, any updates on this? I am tempted to created issues on every major repo to increase visibility :)

[ruksi]

hey @Kenneth-Schroeder

any updates on this?

I checked the internal ticket about "Two files with the same name inside a single input" but it hasn't been prioritized on the feature roadmap as of yet; it's a low priority item

I do still agree on my previous statement of

a reasonable feature request

has there been any updates from your side? is this behavior causing you a lot of issues?

[Kenneth-Schroeder]

We are using workarounds like generating unique file names, using file-renaming steps etc. to prevent this issue and I hope that we have succeeded in that so far.
However, the semi-silent nature of this issue and the fact that this could happen with any step and any execution, any of your customers just with one little inattention would make me rate this issue almost up there with critical bugs.

So it's indirectly costing a lot of engineering time (and money) over time to prevent the issue, it's not well documented (only on two pipeline-related pages of your documentation, I believe, parallel steps in pipelines and tasks in pipelines), it can have absolutely devastating effects and it should be relatively easy to fix.

Honestly, I can hardly think of anything that should have a higher priority than this 😅

[ruksi]

I'll bring it up in the next dev meeting and see if we can do something!

[ruksi]

we discussed the issue with the devs and agreed that this should be fixed in a timely fashion; at least one other user has had this minor issues with this in the past

no specific date was decided though

[leoyala]

I am unfortunately having the same issue.

[mbu93]

I also realised this issue, lucky enough, just before starting the actually time-consuming training step I was about to run. Would have cost me days of processing time (as only a fifth of the data would have been used). I politely would suggest prioritising this issue higher.