Landing zone move iRODS connection failure after v1.1.3 upgrade

[mikkonie]

It's been reported that moving certain landing zones started failing after a v1.1.3 upgrade in production with a Could not receive server response error. I've been told rolling back to v1.1.2 may have fixed this problem (waiting for confirmation on that).

Looking closer into the logs I can see the following:

sodar-celeryd-default-1  | [2025-07-22 10:05:19,076: CRITICAL/ForkPoolWorker-1] Read 0 bytes from socket instead of expected 4 bytes
sodar-celeryd-default-1  | [2025-07-22 10:05:19,076: ERROR/ForkPoolWorker-1] Could not receive server response

This seems suspiciously like the kind of iRODS server error one can get from a failed ACL request. That in turn would indicate something has gone horribly wrong in implementing #2217. That means my bad, sorry.

I'll see about reproducing this in dev to figure out the root cause and a fix. First thing is to check the timeline of affected landing zones to see if some zone setting (e.g. restrict collections) may affect this.

Looks like we'll have a v1.1.4 release after all.

[mikkonie]

I'm looking at a single zone which failed with this error on v1.1.3 but worked after the v1.1.2 rollback.

The zone was created at 2025-07-18 12:06:14, after v1.1.3 was deployed. So this was not a legacy zone.

Settings used to create the zone:

{
  config_data: {
  },
  description: "",
  create_colls: True,
  user_message: "",
  configuration: null,
  restrict_colls: True
}

The zone only contained 4 files, all in one collection structure 3 directories deep. So I doubt this was due to e.g. a timeout from too many ACL requests.

The project has 4 owners, 3 of which have been inherited, as well as 6 contributors. The zone in question was created by a contributor using a non-admin account.

[mikkonie]

I'm unable to reproduce this issue in dev with a similar project/zone/user setup.

There's still a possibility that this was somehow data and file system related and it started working after SODAR rollback by chance. The zone did have some unusually large files in it.

We should try out reuploading similar zones in staging and see if we can reproduce this there.

[mikkonie]

One more thing which speaks in favour of the changes in #2217 being the culprit: in the zone under investigation, the flow was first triggered as "validate only" which succeeded. It only crashed once called as "validate and move". The code with the changes in question is only called if in "validate and move" mode..

Maybe there is something in the project memberships which causes a failure, which I'm just not seeing. E.g. an inherited owner also having a local role and that somehow messing up the ACL changes?

[mikkonie]

For now, I'll add a temporary site level app setting to disable the enforcement. After deploying the latest dev build, we can then verify if this bit of code is indeed involved if we reach similar errors, by simply disabling the setting.

It's the best idea I have right now, since I've been unable to reproduce this in dev.

If the changes in #2217 do end up being the cause, the proper fix is to do #2215 in v1.2.0.

[mikkonie]

With the disabling of the ACL code in place, I'm setting this as ongoing and monitoring whether we can spot similar problems in production.

[mikkonie]

Ok, it is now confirmed that this is caused by the implementation for #2217 with some specific parameters. Since the feature deals with ACLs, I still believe this is somehow related to the member roles in a project. Some conflict between inherited roles? A member's project access not properly set in iRODS?

I'll need to continue trying to reproduce this. However, I will not spend too much time for it because, as mentioned before, the proper fix will be doing #2215 in v1.2 anyway.

For now, the workaround is for an admin to uncheck "Landing zone access cleanup" in site app settings.