Support Scenario where S3 Path is both a File and a Directory

[aaronkanzer]

While running:

s3invsync --allow-new-nonempty --ok-errors all s3://linc-brain-mit-staging-us-east-2/linc-brain-mit-staging-us-east-2/staging-configuration /orcd/data/linc/001/s3lincbrain/

against s3invsync --version 0.2.0 the following error occurred:

Error: path "zarr/7723d02f-1f71-4553-a7b0-47bda1ae8b42/0/0/0/0" is used as both a file and a directory

and confirmed within the AWS console:

This issue captures the need to handle this scenario.

Thanks in advance @jwodder

[jwodder]

@aaronkanzer I would argue that the S3 bucket is in the wrong here, and the proper fix is to correct it on that end.

[aaronkanzer]

@aaronkanzer I would argue that the S3 bucket is in the wrong here, and the proper fix is to correct it on that end.

In what way is the S3 bucket in the wrong here? The structure of the zarr or something else?

I will say that the LINC project accepts a myriad of file types where structure is not 100% predictable, thus I'm inclined to say that this use case could happen again somewhere else, if not zarr-based.

Cc @kabilar -- any chance you could confirm if this zarr structure is misaligned somehow?

[jwodder]

@aaronkanzer Because a path being both a file and a directory is impossible on real, non-S3 filesystems, and thus no on-disk Zarr could possibly look like that. How would s3invsync even handle this?

[kabilar]

@aaronkanzer On first glance, I think that the Zarr archive is incorrect. Looking more into it.

[jwodder]

Related bug in dandi-archive: dandi/dandi-archive#1834

[kabilar]

Hi @aaronkanzer, could you please point me to this asset (zarr/7723d02f-1f71-4553-a7b0-47bda1ae8b42) on the staging LINC Archive?

[aaronkanzer]

Hi @aaronkanzer, could you please point me to this asset (zarr/7723d02f-1f71-4553-a7b0-47bda1ae8b42) on the staging LINC Archive?

Oddly enough, I can't seem to find anything related to that UUID: https://staging-api.lincbrain.org/admin/zarr/zarrarchive/

[kabilar]

Hi @aaronkanzer, I'm having difficulty logging into the staging site - lincbrain/linc-cloud#15

[yarikoptic]

@jwodder couldn't we just ignore such "folders/"? for the sake of backup they AFAIK do not mean anything

[jwodder]

@yarikoptic That won't work if there's anything inside the folders, which I believe is the case if you're seeing the error message in the initial post.

[yarikoptic]

why "won't work"? I think it is our logic ATM which prevents it

src/syncer/treetracker/mod.rs-    /// A path was registered as both a file and a directory
src/syncer/treetracker/mod.rs:    #[error("path {0:?} is used as both a file and a directory")]
src/syncer/treetracker/mod.rs-    Conflict(String),

couldn't we just ignore completely folder/ prefixes while going through the inventory?

[jwodder]

@yarikoptic Paths that end in trailing slashes are already ignored, but later on in the code, if two different paths like 0/1 and 0 are encountered, you'll get "path ... is used as both a file and a directory", so ignoring the folder-ness of 0 here would mean ignoring all the files inside it, which doesn't seem like a good idea.

[aaronkanzer]

@jwodder @yarikoptic @kabilar

One of my concerns here is that we can't 100% enforce file structure with file types that still don't have 100% defined schema (albeit zarr might have more structure, but the LINC project has many unknown file formats exported by certain instruments)

I'm still a bit confused why the file path/name matters -- shouldn't the nature of the s3 sync/backup be embedded in exploring S3 in a tree-traversal-like nature, rather than naming conventions?

[jwodder]

@aaronkanzer

One of my concerns here is that we can't 100% enforce file structure with file types that still don't have 100% defined schema

I think "Zarrs aren't allowed to contain a path that's both a file and directory" is a perfectly reasonable preliminary requirement, not least because it's impossible to violate on a POSIX (and probably also Windows) filesystem.

I'm still a bit confused why the file path/name matters

The problem is that s3invsync can't back up both a file named foo and a file named foo/bar at the same time, as it's impossible under POSIX (and Windows?) to create a thing that's both a file and a directory at once.

[satra]

it seems we are running into mismatches in concepts of different object systems. a posix filesystem also doesn't have a concept of versions and there is a way this is supporting it.

@yarikoptic - perhaps it would be good to revisit the use cases (archiving, computing at mit, etc.,.) that we are hoping this backup will support, what layers (indexes, etc.,.), and what operations (CRUD) are needed to support them. i was thinking back about our original conversations of supporting similar layout as s3 or simply creating an index of objects.

alternatively, we could use s3invsync to sync to a local minio server that uses the filesystem as a backend, which would allow for a more one to one mapping.