Files are getting rejected from NEARLINE

[msmdcom]

Describe the bug
DCM4CHEE 5.32.1-secure has fs1 and nearline storages configured, received studies are stored in fs1. After study expiration DCM4CHEE copies the study files from fs1 to nearline storage, after copying it marks study for deletion from fs1, deletes from fs1, but it also rejects/deletes the study from nearline.

To Reproduce

1. Create new: Storage, Queue(or use exising), Exporter Descriptor, change fs1 storage Storage Duration: CACHE. As described in Vrinda post: https://groups.google.com/g/dcm4che/c/C5ZT_Z0e5Hw/m/EKppUXp9EAAJ

2. Create new Study Retention Policy:
   Name: StudyRetPolicyCopyToNearline
   Study Retention Period: P30D
   Export expired Study: CopyToNearlineStorage

3. Send a study to DCM4CHEE.

4. From Navigation > Study change Study expiration date to the past date.

5. Observe logs and see that new export task is SCHEDULED, then COMPLETED and study files are copied to nearline storage, files of the study in fs1 is marked for deletion, files from fs1 are deleted.

6. Problem, that the files in nearline are getting rejected and you end up with study deleted from fs1 and rejected from nearline and not visible in Navigation > Study

Expected behavior
After moving files from fs1 to nearline, the files should be kept in nearline storage and not get rejected/deleted.

Additional context
The exact same configuration works as expected with DCM4CHEE 5.31.2-secure.
Study files are copiend to nearline, deleted from fs1, files stays in nearline and are accessible.

[vrindanayak]

@msmdcom : Please share your configurations for fs1 storage, nearline storage, study retention policy StudyRetPolicyCopyToNearline, exporter CopyToNearlineStorage as ldif files

[msmdcom]

@vrindanayak please see below, as .ldif files not allowed to upload here:

dn: dcmStorageID=fs1,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
dcmCheckMountFilePath: NO_MOUNT
dcmDigestAlgorithm: MD5
objectClass: dcmStorage
dcmStorageID: fs1
dcmURI: file:///storage/fs1/
dcmStoragePathFormat: {now,date,yyyy/MM/dd}/{0020000D,hash}/{0020000E,hash}/{00080018,hash}
dcmStorageDuration: CACHE

dn: dcmStorageID=nearline,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
dcmInstanceAvailability: NEARLINE
dcmCheckMountFilePath: NO_MOUNT
dcmStoragePathFormat: {now,date,yyyy/MM/dd}/{0020000D,hash}/{0020000E,hash}/{00080018,hash}
dcmDigestAlgorithm: MD5
objectClass: dcmStorage
dcmRetrieveCacheStorageID: fs1
dcmStorageID: nearline
dcmURI: file:///storage/longterm/

dn: dcmExporterID=CopyToNearlineStorage,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
dcmExporterID: CopyToNearlineStorage
dcmDeleteStudyFromStorageID: fs1
dicomDescription: Copy To Nearline Storage
dcmQueueName: Nearline Storage Export
objectClass: dcmExporter
dicomAETitle: DCM4CHEE
dcmURI: storage:nearline

dn: cn=StudyRetPolicyCopyToNearline,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
dcmExporterID: CopyToNearlineStorage
dcmRetentionPeriod: P30D
objectClass: dcmStudyRetentionPolicy
cn: StudyRetPolicyCopyToNearline

dn: dcmQueueName=Nearline Storage Export,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
dicomDescription: Nearline Storage Export Tasks
dcmQueueName: Nearline Storage Export
objectClass: dcmQueue
dcmMaxRetries: 10
dcmRetryDelay: PT25S
dcmMaxRetryDelay: PT8M
dcmPurgeQueueMessageCompletedDelay: P1D

[vrindanayak]

@msmdcom : To us, the configs you have looks like a mixed up combination of configs for 2 different features of archive

• Studies on ONLINE CACHE storage -> export to PERMANENT NEARLINE storage -> delete studies from ONLINE CACHE storage
• Studies to be auto rejected for Data Retention Policy Expired reason after the retention period is over

It is not clear from why such a mixup of configs is done.

As for

Additional context
Study files are copiend to nearline, deleted from fs1, files stays in nearline and are accessible.

corresponds to

• Studies on ONLINE CACHE storage -> export to PERMANENT NEARLINE storage -> delete studies from ONLINE CACHE storage

A Study Retention Policy is normally configured when the study should no longer be available or referenced in the archive at all. Could you please explain the use case or workflow for configuration of StudyRetPolicyCopyToNearline?

If the only requirement is to retain studies on ONLINE CACHE fs1 for 30 days and then delete them from fs1 only, you just need to :

• configure Export Storage ID on fs1 pointing to nearline storage
• configure Delete Studies Received Before(s) as P30D
• remove dcmDeleteStudyFromStorageID: fs1 from CopyToNearlineStorage exporter
• remove StudyRetPolicyCopyToNearline study retention policy

[vrindanayak]

To auto export studies from fs1 to nearline when they are received in archive, additionally configure an Export Rule referencing Exporter ID(s) as CopyToNearlineStorage.

The configs for

Studies on ONLINE CACHE storage -> auto export to PERMANENT NEARLINE storage

can be found in sample-config.ldif

dcmExportRule : Copy to NEARLINE Storage
dcmExporter : CopyToNearlineStorage
dcmQueue : Nearline Storage Export
dcmStorage : nearline

[msmdcom]

Hello @vrindanayak , thanks for your reply. I was following your recommendations from this 2019 thread (https://groups.google.com/g/dcm4che/c/C5ZT_Z0e5Hw/m/EKppUXp9EAAJ). The idea that studies received to DCM4CHEE would be stored in fs1 for 30 days, and only after 30 days study automatically moved to nearline storage and deleted from fs1. Study should be accessible from nearline as well.
So in this thread you recommended to create Study Retention Policy which would trigger the export to nearline:

• Add a Study Retention Policy : Ensure to set Export expired Study to Exporter ID of exporter created above.

For testing we could set the Study Expiration Date to past date and it will trigger moving the study from fs1 to nearline.
And this approach is working fine in DCM4CHEE 5.31.2-secure version. However in newer version (5.32.1, 5.33.2) it does not work.

You had this bug (#4408) which was fixed in 5.32.0
In the Export Descriptor we have setting Reject Entity for Data Retention Expiry: FALSE
Could it be that fixing this bug now ignores this setting in 5.32.0 and newer version?
Because when Reject Entity for Data Retention Expiry: FALSE (default) in my understanding it should not reject study. It should reject study only this is set to TRUE.

Please check the attached PDF (Configuration_example.pdf) with current settings and the process (dcm4chee_fs1_to_nearline_steps_with_logs.pdf) how study is being sent to DCM4CHEE 5.32.1 fs1, study moved to nearline, study deleted from fs1, study rejected from nearline.
Please note that Reject Entity for Data Retention Expiry is set to FALSE in Exporter Descriptor.

The exact same configuration on 5.31.2 is working fine. After study is exported from fs1 to nearline it is not being rejected.

Configuration_example.pdf
dcm4chee_fs1_to_nearline_steps_with_logs.pdf

[vrindanayak]

When the user (Thiago Henrique Busarello) posted the first message in the dcm4che forum thread you're referring to, the archive did not have support for Deletion (of objects) of studies from storage system based on Study Age, Receive or LRU time. In the same dcm4che forum thread in the later posts, you will see that Gunter has referenced to this issue (#1998) opened as an Enhancement issue for the archive. The bug #4408 has nothing to do with the specific use case you're trying to achieve.

To fulfill your use-case, please refer

• Files are getting rejected from NEARLINE #4737 (comment)
• Files are getting rejected from NEARLINE #4737 (comment)