DemoStorage: Add support for deleteObject (IExternalGC)

So that it is generally possible to zodbrestore[1,2] zodbdumped[3]
database/transactions into DemoStorage. Because without this patch, when dump
contains deletion records, e.g.

    txn 0285cbacc06d3a4c " "
    user ""
    description ""
    extension ""
    obj 0000000000000007 delete

it fails like this:

    Traceback (most recent call last):
      ...
      File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbrestore.py", line 94, in main
        zodbrestore(stor, asbinstream(sys.stdin), _)
      File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbrestore.py", line 43, in zodbrestore
        zodbcommit(stor, at, txn)
      File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbcommit.py", line 131, in zodbcommit
        _()
      File "/home/kirr/src/wendelin/z/zodbtools/zodbtools/zodbcommit.py", line 118, in _
        stor.deleteObject(obj.oid, current_serial(obj.oid), txn)
    AttributeError: 'DemoStorage' object has no attribute 'deleteObject'
        demo_test.go:105: demo:(file:///tmp/demo470143633/δ0285cbac258bf266/base.fs)/(file:///tmp/demo470143633/δ0285cbac258bf266/δ.fs): zpyrestore: exit status 1

To be able to implement DemoStorage.deleteObject, we have to fix
FileStorage.deleteObject first: currently FileStorage.deleteObject raises
POSKeyError if an object is not present in the database, but for situation
where

- demo=base+δ,
- object is present in base, and
- object is not present in δ

it creates a problem because there is no way to add whiteout record for the
object into δ.

This change should be generally good; let me quote
https://lab.nexedi.com/kirr/neo/commit/543041a3 for why:

---- 8< ----
Even though currently it is not possible to create such data record via
FileStorage(py).deleteObject (because it raises POSKeyError if there is
no previous object revision), being able to use such data records is
semantically useful in overlayed DemoStorage settings, where δ part
marks an object that exists only in base with delete record whiteout.

It is also generally meaningful to be able to create "delete" record
even if object was not previously existing: "deleteObject" is actually
similar to "store" (and so should be better named as "storeDelete"). If
one wants to store deletion, there should not be a reason to reject it,
because deleteObject already has seatbelt in the form of oldserial, and
if the user calls deleteObject(oid, oldserial=z64), he/she is already
telling that "I know that this object does not exist in this storage
(oldserial=z64), but still please create a deletion record for it". Once
again this is useful in overlayed DemoStorage settings described above.

For the reference, such whiteout deletion records pass ZODB/scripts/fstest just fine.
---- 8< ----

DemoStorage.deleteObject implementation is straightforward, but builds on
loadAt[4] as precondition.

/cc @jimfulton, @jamadden, @perrinjerome

[1] https://lab.nexedi.com/nexedi/zodbtools/blob/129afa67/zodbtools/zodbrestore.py
[2] https://lab.nexedi.com/nexedi/zodbtools/merge_requests/19
[3] https://lab.nexedi.com/nexedi/zodbtools/blob/129afa67/zodbtools/zodbdump.py
[4] zopefoundation#323

Author: navytux

src/ZODB/DemoStorage.py

@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 ##############################################################################
 #
 # Copyright (c) Zope Corporation and Contributors.
@@ -108,11 +109,16 @@ def __init__(self, name=None, base=None, changes=None,
             if close_changes_on_close is None:
                 close_changes_on_close = False
         else:
+            self._temporary_changes = False
             if ZODB.interfaces.IBlobStorage.providedBy(changes):
                 zope.interface.alsoProvides(self, ZODB.interfaces.IBlobStorage)
             if close_changes_on_close is None:
                 close_changes_on_close = True
 
+        if ZODB.interfaces.IExternalGC.providedBy(changes):
+            zope.interface.alsoProvides(self, ZODB.interfaces.IExternalGC)
+            self.deleteObject = self._storeDelete
+
         self.changes = changes
         self.close_changes_on_close = close_changes_on_close
 
@@ -376,6 +382,40 @@ def store(self, oid, serial, data, version, transaction):
         else:
             self.changes.store(oid, serial, data, '', transaction)
 
+    # _storeDelete serves deleteObject when .changes implements IExternalGC.
+    def _storeDelete(self, oid, oldserial, transaction):
+        if transaction is not self._transaction:
+            raise ZODB.POSException.StorageTransactionError(self, transaction)
+
+        # oldserial ∈ changes  -> changes.deleteObject
+        baseHead = self.base.lastTransaction()
+        if oldserial > baseHead:
+            self.changes.deleteObject(oid, oldserial, transaction)
+            return
+
+        # oldserial ∈ base  -> find out it is indeed the latest there and then
+        # call changes.deleteObject(oldserial=z64)
+
+        _, serial = ZODB.utils.loadAt(self.changes, oid, self.changes.lastTransaction())
+        if serial != ZODB.utils.z64:
+            # object has data or deletion record in changes
+            raise ZODB.POSException.ConflictError(oid=oid, serials=(serial, oldserial))
+
+        _, serial = ZODB.utils.loadAt(self.base, oid, baseHead)
+        if serial != oldserial:
+            raise ZODB.POSException.ConflictError(oid=oid, serials=(serial, oldserial))
+
+        # object has no data/deletion record in changes and its latest revision in base == oldserial
+        # -> changes.deleteObject(oldserial=z64) + correct oldserial back on conflict.
+        try:
+            self.changes.deleteObject(oid, ZODB.utils.z64, transaction)
+        except ZODB.POSException.ConflictError as e:
+            assert len(e.serials) == 2
+            assert e.serials[1] == ZODB.utils.z64
+            e.serials = (e.serials[0], oldserial)
+            raise
+
+
     def storeBlob(self, oid, oldserial, data, blobfilename, version,
                   transaction):
         assert version=='', "versions aren't supported"

src/ZODB/FileStorage/FileStorage.py

@@ -661,9 +661,10 @@ def deleteObject(self, oid, oldserial, transaction):
         with self._lock:
             old = self._index_get(oid, 0)
             if not old:
-                raise POSKeyError(oid)
-            h = self._read_data_header(old, oid)
-            committed_tid = h.tid
+                committed_tid = z64
+            else:
+                h = self._read_data_header(old, oid)
+                committed_tid = h.tid
 
             if oldserial != committed_tid:
                 raise ConflictError(

src/ZODB/tests/testDemoStorage.py

@@ -384,4 +384,12 @@ def test_suite():
     suite.addTest(unittest.makeSuite(DemoStorageWrappedAroundHexMappingStorage,
                                      'check'))
     suite.addTest(unittest.makeSuite(DemoStorageTests2, 'check'))
+
+    def demo_for_gctest():
+        from ZODB.FileStorage import FileStorage
+        base = ZODB.FileStorage.FileStorage('data.fs', blob_dir="data_blobs")
+        changes = ZODB.FileStorage.FileStorage('changes.fs', blob_dir="changes_blobs", pack_gc=False)
+        return ZODB.DemoStorage.DemoStorage(base=base, changes=changes)
+    suite.addTest(PackableStorage.IExternalGC_suite(demo_for_gctest))
+
     return suite