S3 as storage: Working example

[proteeti13]

Hello! I am trying to integrate S3 bucket as storage in my dcm4chee. I am now using the version 5.23.2 of dcm4chee.

I did the following steps:

1. Installed dcm4chee from here. Got it running at http://localhost:8080/dcm4chee-arc/ui2.
2. Created an s3 bucket, made it public
3. In the dcm4chee ui, did these from here

• Went to Menu --> Configuration --> dcm4chee-arc --> Extensions --> Edit Extensions --> Edit Extensions of Archive Device --> Deleted [X] fs1 from the Storage Descriptor list --> Save
• Went to Menu --> Configuration --> dcm4chee-arc --> Extensions --> Edit Extensions --> Edit Extensions of Archive Device --> Add new Storage Descriptor
• Added the following attributes:
  Storage ID: proteeti1
  Storage URI : jclouds:s3:
  Digest Algorithm: MD5
• In storage Property added the following from here
  identity=
  credential=
  container=
  containerExists=true
  jclouds.trust-all-certs=true
  jclouds.relax-hostname=true
  jclouds.s3.virtual-host-buckets=true
  jclouds.strip-expect-header=true

• Associate the new Storage with the Archive Application Entity:
  Configuration --> dcm4chee-arc --> Child Objects -> Network AEs -> DCM4CHEE -> Network AE Extension --> Archive Network AE -> Attributes -> Object Storage ID
• Send CT images to the Archive with docker run --rm --network=dcm4chee_default dcm4che/dcm4che-tools storescu -cDCM4CHEE@arc:11112 /opt/dcm4che/etc/testdata/dicom

Result:

E"ERROR: Received C-STORE-RSP with Status 0110H for /opt/dcm4che/etc/testdata/dicom/MR15.dcm"
(0000,0100) US [32769] CommandField
(0000,0120) US [15] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [272] Status
(0000,0902) LO [java.io.IOException: Upload failed] ErrorComment

09:42:39,163 INFO  - STORESCU->DCM4CHEE(1) >> 16:C-STORE-RSP[pcid=5, status=110H, errorComment=java.io.IOException: Upload failed
  tsuid=1.2.840.10008.1.2 - Implicit VR Little Endian]
09:42:39,164 DEBUG - STORESCU->DCM4CHEE(1) >> 16:C-STORE-RSP Command:
(0000,0100) US [32769] CommandField
(0000,0120) US [16] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [272] Status
(0000,0902) LO [java.io.IOException: Upload failed] ErrorComment

E"ERROR: Received C-STORE-RSP with Status 0110H for /opt/dcm4che/etc/testdata/dicom/MR03.dcm"
(0000,0100) US [32769] CommandField
(0000,0120) US [16] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [272] Status
(0000,0902) LO [java.io.IOException: Upload failed] ErrorComment

09:42:39,166 INFO  - STORESCU->DCM4CHEE(1) << A-RELEASE-RQ
09:42:39,166 DEBUG - STORESCU->DCM4CHEE(1): enter state: Sta7 - Awaiting A-RELEASE-RP PDU
09:42:39,167 INFO  - STORESCU->DCM4CHEE(1) >> A-RELEASE-RP
09:42:39,167 INFO  - STORESCU->DCM4CHEE(1): close Socket[addr=arc/172.19.0.4,port=11112,localport=52241]
09:42:39,169 DEBUG - STORESCU->DCM4CHEE(1): enter state: Sta1 - Idle
Sent 0 objects (=0MB) in 6.235s (=0MB/s)

[vrindanayak]

What do you see in the archive server.log? Can you post your storage descriptor configuration here?

[proteeti13]

Storage Descriptor config;
Storage ID: proteeti1
Storage URI : jclouds:s3:< s3 bucket-endpoint-url >
Digest Algorithm: MD5
container=< s3-bucket-bucket-name >
identity= < access-key >
credential= < secret-key >
containerExists=true
jclouds.trust-all-certs=true
jclouds.relax-hostname=true
jclouds.s3.virtual-host-buckets=true
jclouds.strip-expect-header=true

1. How can I see the archive server.log? Can you provide me some help/
2. Should I put the s3 bucket endppint url or the vpc enpoint in the "Storage URI"

[vrindanayak]

Archive server log should be in /var/local/dcm4chee-arc/wildfly/log. Does your start with http[s]?

[proteeti13]

My s3 bucket url starts with http.
Here is the server log
server log

[proteeti13]

Also I am configuring this on https://localhost:8443/dcm4chee-arc/ui2/#/

[vrindanayak]

Caused by: java.io.IOException: Upload failed
	at deployment.dcm4chee-arc-ear-5.23.2-psql.ear//org.dcm4chee.arc.storage.cloud.CloudStorage.afterOutputStreamClosed(CloudStorage.java:179)
	at deployment.dcm4chee-arc-ear-5.23.2-psql.ear//org.dcm4chee.arc.storage.AbstractStorage$1.close(AbstractStorage.java:175)
	at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:188)
	at org.dcm4che.core@5.23.2//org.dcm4che3.io.DicomOutputStream.close(DicomOutputStream.java:244)
	at org.dcm4che.imageio@5.23.2//org.dcm4che3.imageio.codec.Transcoder.close(Transcoder.java:422)
	at deployment.dcm4chee-arc-ear-5.23.2-psql.ear.dcm4chee-arc-store-5.23.2.jar//org.dcm4chee.arc.store.impl.StoreServiceImpl.writeToStorage(StoreServiceImpl.java:238)
	... 28 more
Caused by: org.jclouds.blobstore.KeyNotFoundException: /2021/04/20/366D49C5/60AB0084/625FEBA5 not found in container proteeti-test-for-dcm4chee.proteeti-test-for-dcm4chee.s3-website-ap-southeast-1.amazonaws.com: <html>
<head><title>404 Not Found</title></head>
<body>
<h1>404 Not Found</h1>
<ul>
<li>Code: NoSuchBucket</li>
<li>Message: The specified bucket does not exist</li>
<li>BucketName: proteeti-test-for-dcm4chee.proteeti-test-for-dcm4chee</li>
<li>RequestId: 7V8P45NBZ71NQM01</li>
<li>HostId: 8KO+LnkxMxLz3/u4wBuv4s9K7mNaDACmRMuFeGVtpIHfkaLcA82NQQFrLFXbBpNwl4QUZjllEM4=</li>
</ul>
<hr/>
</body>
</html>

[proteeti13]

Are there some specific policies that I need to attach to the bucket? I have made the bucket public already. Also, should I use the endpoint url or the vpc endpoint?

[vrindanayak]

Here's how I've configured for the first time and tested out exporting objects to Amazon S3

Configuration

Storage Descriptor

dn: dcmStorageID=amazon-s3,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
objectClass: dcmStorage
dcmStorageID: amazon-s3
dcmURI: jclouds:aws-s3:https://<bucket-name>.s3.eu-central-1.amazonaws.com
dcmDigestAlgorithm: MD5
dcmProperty: container=<bucket-name>
dcmProperty: jclouds.s3.virtual-host-buckets=true
dcmProperty: credential=<secret-key>
dcmProperty: identity=<access-key>
dcmProperty: jclouds.strip-expect-header=true
dcmProperty: containerExists=true
dcmProperty: jclouds.relax-hostname=true
dcmProperty: jclouds.trust-all-certs=true

Exporter configuration

dn: dcmExporterID=StoreToS3,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOMConfiguration,dc=dcm4che,dc=org
objectClass: dcmExporter
dcmExporterID: StoreToS3
dcmQueueName: Export6
dcmURI: storage:amazon-s3
dicomAETitle: DCM4CHEE

Settings in AWS :

While creating the bucket, I did the following

• unchecked Block Public Access settings for this bucket : By default, this setting was checked (ON)
• enabled Bucket Versioning : By default, this setting was Disable
• other parameters unchanged

Next, I created the Access Key and used it in the storage descriptor configuration above.

Verification

• Export study using Queued Exporter to S3 Storage

• Verify in Monitoring - Export

• Verify in Amazon AWS

Tested also by enabling the Block public access (bucket settings) setting to block all public access to this bucket and objects within it, had no issues on exporting objects to S3

Exporting existing study from archive to Amazon S3

Using Amazon S3 as Object Storage for studies received by archive

• Configured amazon-s3 as Object Storage ID for DCM4CHEE archive AE

• Stored study to archive. Verify in Archive UI and AWS S3

I'll add these configs / verification settings in a wiki document

[proteeti13]

I followed the following steps you mentioned. I exported and a success message was showed. When I got to the monitoring section to verify, it shows the status is "Completed".

Then I checked the S3 bucket, and couldn't find any study. So I expanded the study details to match the messages with your screenshots. I noticed that, in the "outcome message", your one's show "completed:1", while mine shows "completed:0". I am attaching a screenshot here.

I've followed all the config details as you mentioned. But got confused on the first lines of the Storage configuration:

dn: dcmStorageID=amazon-s3,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
objectClass: dcmStorage

And in the Exporter Configuration, you have mentioned this:

dn: dcmExporterID=StoreToS3,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOMConfiguration,dc=dcm4che,dc=org
objectClass: dcmExporter

What are you denoting by these? I couldn't find these properties while configuring. But was able to found and configure the rest of the mentioned. By any chance are you exporting the config from terminal?

Also, I couldn't understand the section "Using Amazon S3 as Object Storage for studies received by archive". What is the difference with this from using the "Export Study with the Queued exporter" option?

[vrindanayak]

I had tried two test cases to test storing objects to Amazon S3.

1. Using Amazon S3 as ONLINE storage for studies received by archive. By default, it is always fs1 referencing the filesystem where the archive is installed/deployed. For testing, I changed this from fs1 to amazon-s3. So this is denoted by Storage configuration above.
2. I kept ONLINE storage of studies received by archive as default i.e. fs1 referencing the filesystem where the archive is installed/deployed. There are normally use cases where an user wants to store archived studies to a NEARLINE storage. To do this with minimal configuration, I configured an Exporter and used archive UI to export studies to Amazon S3. This is denoted by Exporter configuration above.

Please check if your bucket settings if it can be accessed by the application - as in the server logs you had attached previously, it mentioned No such bucket found

[proteeti13]

I have checked the bucket settings and it is I've unchecked Block Public Access settings for this bucket, as you have mentioned. I've also checked my server log. It is showing this:

Caused by: io.jaegertracing.internal.exceptions.SenderException: Could not send 1 spans
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:85)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:113)
	... 3 more
Caused by: org.apache.thrift.transport.TTransportException: Cannot flush closed transport
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:148)
	at org.apache.thrift@0.13.0//org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
	at org.apache.thrift@0.13.0//org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:83)
	... 4 more
Caused by: java.net.PortUnreachableException: ICMP Port Unreachable
	at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
	at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:146)
	... 9 more

Also, previously I was uploading new studies via dcmsnd command and it worked fine. Now it is throwing this error:

./dcmsnd DCM4CHEE@127.0.0.1:11112 ../../dcm

Scanning files to send
.
Scanned 1 files in 0.052s (=52ms/file)
log4j:WARN No appenders could be found for logger (org.dcm4che2.net.NetworkConnection).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to DCM4CHEE@127.0.0.1:11112 in 0.042s
ERROR: Received RSP with Status 0110H for ../../dcm/0012.dcm, cuid=1.2.840.10008.5.1.4.1.1.12.1, tsuid=1.2.840.10008.1.2.4.50
(0000,0100) US #2 [32769] Command Field
(0000,0120) US #2 [1] Message ID Being Responded To
(0000,0800) US #2 [257] Command Data Set Type
(0000,0900) US #2 [272] Status
(0000,0902) LO #64 [java.io.IOException: No space left on configured storage systems] Error Comment

F
Sent 0 objects (=0.0KB) in 0.057s (=0.0KB/s)
Released connection to DCM4CHEE@127.0.0.1:1111

Also, does it mean doing ONLY the Storage Configuration that you've mentioned is enough to export files in the S3 as well as using it as the Online storage? Configuring the Object Storage ID for DCM4CHEE archive AE and the Exporter is for using the Nearline storage?

[gunterze]

Caused by: java.net.PortUnreachableException: ICMP Port Unreachable

is an unrelated warning caused by WFLY-14625. s.o. #3105

[vrindanayak]

Also, previously I was uploading new studies via dcmsnd command and it worked fine. Now it is throwing this error:

./dcmsnd DCM4CHEE@127.0.0.1:11112 ../../dcm

Scanning files to send
.
Scanned 1 files in 0.052s (=52ms/file)
log4j:WARN No appenders could be found for logger (org.dcm4che2.net.NetworkConnection).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to DCM4CHEE@127.0.0.1:11112 in 0.042s
ERROR: Received RSP with Status 0110H for ../../dcm/0012.dcm, cuid=1.2.840.10008.5.1.4.1.1.12.1, tsuid=1.2.840.10008.1.2.4.50
(0000,0100) US #2 [32769] Command Field
(0000,0120) US #2 [1] Message ID Being Responded To
(0000,0800) US #2 [257] Command Data Set Type
(0000,0900) US #2 [272] Status
(0000,0902) LO #64 [java.io.IOException: No space left on configured storage systems] Error Comment

F
Sent 0 objects (=0.0KB) in 0.057s (=0.0KB/s)
Released connection to DCM4CHEE@127.0.0.1:1111

Also, does it mean doing ONLY the Storage Configuration that you've mentioned is enough to export files in the S3 as well as using it as the Online storage? Configuring the Object Storage ID for DCM4CHEE archive AE and the Exporter is for using the Nearline storage?

If you just want to use Amazon S3 as your ONLINE storage for studies received by archive, then Storage Configuration denoted by dcmStorageID=amazon-s3 should be sufficient. Additionally, ensure on DCM4CHEE Archive AE Extension level to change the Object Storage ID from fs1 to amazon-s3. You don't need an Exporter for this, as I mentioned those were just the two test cases I tried to test Amazon S3 as storage system.

Regarding No space left on configured storage systems, which storage have you configured as ONLINE. What is the value of
Object Storage ID on DCM4CHEE Archive AE Extension level? You may choose to configure multiple storages - See https://groups.google.com/g/dcm4che/c/L0zQVesgMIU/m/kuNFMonFCQAJ

In dcm4chee 5.x, you may configure multiple Object Storage ID(s) of Archive Network AE and specify a Storage Threshold for each referred Storage. If you want to fill Storage Systems sequential one by one, keep Object Storage Count = 1. The Storage which Storage ID comes first in literal order will be filled first. So you may choose a naming pattern like fs001, fs002, fs003, ....