Skip to content

Add new MemoryStatus interface to Theta. #589

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions src/main/java/org/apache/datasketches/common/MemoryStatus.java
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the new MemoryStatus interface that can be used eventually for the whole library. So far, this has only been applied to the Theta tree.

Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.datasketches.common;

import org.apache.datasketches.memory.Memory;

/**
* Methods for inquiring the status of a backing Memory object.
*/
public interface MemoryStatus {

/**
* Returns true if this object's internal data is backed by a Memory object,
* which may be on-heap or off-heap.
* @return true if this object's internal data is backed by a Memory object.
*/
default boolean hasMemory() { return false; }

/**
* Returns true if this object's internal data is backed by direct (off-heap) Memory.
* @return true if this object's internal data is backed by direct (off-heap) Memory.
*/
default boolean isDirect() { return false; }

/**
* Returns true if the backing resource of <i>this</i> is identical with the backing resource
* of <i>that</i>. The capacities must be the same. If <i>this</i> is a region,
* the region offset must also be the same.
*
* @param that A different non-null and alive Memory object.
* @return true if the backing resource of <i>this</i> is identical with the backing resource
* of <i>that</i>.
* @throws SketchesArgumentException if <i>that</i> is not alive (already closed).
*/
default boolean isSameResource(final Memory that) { return false; }

}
6 changes: 0 additions & 6 deletions src/main/java/org/apache/datasketches/theta/AnotBimpl.java
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory not used as a backing store

  • Removed import to Memory
  • Removed isSameResource

Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@
import java.util.Arrays;

import org.apache.datasketches.common.SketchesArgumentException;
import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.memory.WritableMemory;
import org.apache.datasketches.thetacommon.ThetaUtil;

Expand Down Expand Up @@ -148,11 +147,6 @@ int getRetainedEntries() {
return curCount_;
}

@Override
public boolean isSameResource(final Memory that) {
return false;
}

//restricted

private static long[] getHashArrA(final Sketch skA) { //returns a new array
Expand Down
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isSameResource(Memory that) was missing.

Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import java.util.concurrent.atomic.AtomicBoolean;

import org.apache.datasketches.common.ResizeFactor;
import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.thetacommon.HashOperations;

/**
Expand Down Expand Up @@ -167,6 +168,11 @@ public boolean isEstimationMode() {
return shared.isEstimationMode();
}

@Override
public boolean isSameResource(final Memory that) {
return shared.isSameResource(that);
}

//End of proxies

@Override
Expand Down
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a Base Interface

  • added extends MemoryStatus & import - default is OK here
  • removed hasMemory(), isDirect(), did not have isSameResource()

Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

import java.util.concurrent.atomic.AtomicBoolean;

import org.apache.datasketches.common.MemoryStatus;
import org.apache.datasketches.memory.WritableMemory;

/**
Expand All @@ -30,7 +31,7 @@
*
* @author eshcar
*/
interface ConcurrentSharedThetaSketch {
interface ConcurrentSharedThetaSketch extends MemoryStatus {

long NOT_SINGLE_HASH = -1L;
double MIN_ERROR = 0.0000001;
Expand Down Expand Up @@ -127,7 +128,7 @@ boolean propagate(final AtomicBoolean localPropagationInProgress, final Sketch s
//attempt to access these methods from the local buffer will be diverted to the shared
//sketch.

//From Sketch
//From Sketch and MemoryStatus

int getCompactBytes();

Expand All @@ -139,10 +140,6 @@ boolean propagate(final AtomicBoolean localPropagationInProgress, final Sketch s

double getUpperBound(int numStdDev);

boolean hasMemory();

boolean isDirect();

boolean isEmpty();

boolean isEstimationMode();
Expand Down
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory used as backing store

  • Updated hasMemory(), isDirect(), isSameResource() to check if mem != null

Original file line number Diff line number Diff line change
Expand Up @@ -109,12 +109,12 @@ public long getThetaLong() {

@Override
public boolean hasMemory() {
return true;
return mem_ != null;
}

@Override
public boolean isDirect() {
return mem_.isDirect();
return hasMemory() ? mem_.isDirect() : false;
}

@Override
Expand All @@ -132,7 +132,7 @@ public boolean isOrdered() {

@Override
public boolean isSameResource(final Memory that) {
return mem_.isSameResource(that);
return hasMemory() ? mem_.isSameResource(that) : false;
}

@Override
Expand Down
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory used as backing store

  • Updated hasMemory(), isDirect(), isSameResource() to check if mem != null

Original file line number Diff line number Diff line change
Expand Up @@ -144,12 +144,12 @@ public long getThetaLong() {

@Override
public boolean hasMemory() {
return true;
return wmem_ != null;
}

@Override
public boolean isDirect() {
return wmem_.isDirect();
return hasMemory() ? wmem_.isDirect() : false;
}

@Override
Expand All @@ -159,7 +159,7 @@ public boolean isEmpty() {

@Override
public boolean isSameResource(final Memory that) {
return wmem_.isSameResource(that);
return hasMemory() ? wmem_.isSameResource(that) : false;
}

@Override
Expand Down
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory not used as a backing store

  • Removed hasMemory(), isDirect(), (did not have isSameResource())

Original file line number Diff line number Diff line change
Expand Up @@ -91,16 +91,6 @@ public long getThetaLong() {
return Long.MAX_VALUE;
}

@Override
public boolean hasMemory() {
return false;
}

@Override
public boolean isDirect() {
return false;
}

@Override
public boolean isEmpty() {
return true;
Expand Down
10 changes: 0 additions & 10 deletions src/main/java/org/apache/datasketches/theta/HeapCompactSketch.java
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory not used as a backing store

  • Removed hasMemory(), isDirect(), did not have isSameResource()
  • Note: need to remove getMemory()

Original file line number Diff line number Diff line change
Expand Up @@ -102,16 +102,6 @@ public long getThetaLong() {
return thetaLong_;
}

@Override
public boolean hasMemory() {
return false;
}

@Override
public boolean isDirect() {
return false;
}

@Override
public boolean isEmpty() {
return empty_;
Expand Down
10 changes: 0 additions & 10 deletions src/main/java/org/apache/datasketches/theta/HeapUpdateSketch.java
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory not used as a backing store

  • Removed hasMemory(), isDirect(), did not have isSameResource()

Original file line number Diff line number Diff line change
Expand Up @@ -66,16 +66,6 @@ public int getCurrentBytes() {
return (preLongs + dataLongs) << 3;
}

@Override
public boolean isDirect() {
return false;
}

@Override
public boolean hasMemory() {
return false;
}

//UpdateSketch

@Override
Expand Down
14 changes: 12 additions & 2 deletions src/main/java/org/apache/datasketches/theta/IntersectionImpl.java
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory used as backing store

  • Added hasMemory(), isDirect(), updated isSameResource()

Original file line number Diff line number Diff line change
Expand Up @@ -336,14 +336,24 @@ public CompactSketch getResult(final boolean dstOrdered, final WritableMemory ds
dstMem, compactCache);
}

@Override
public boolean hasMemory() {
return wmem_ != null;
}

@Override
public boolean hasResult() {
return wmem_ != null ? wmem_.getInt(RETAINED_ENTRIES_INT) >= 0 : curCount_ >= 0;
return hasMemory() ? wmem_.getInt(RETAINED_ENTRIES_INT) >= 0 : curCount_ >= 0;
}

@Override
public boolean isDirect() {
return hasMemory() ? wmem_.isDirect() : false;
}

@Override
public boolean isSameResource(final Memory that) {
return wmem_ != null ? wmem_.isSameResource(that) : false;
return hasMemory() ? wmem_.isSameResource(that) : false;
}

@Override
Expand Down
17 changes: 2 additions & 15 deletions src/main/java/org/apache/datasketches/theta/SetOperation.java
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Base abstract class

  • Added implements MemoryStatus and import MemoryStatus
  • Removed abstract isSameResource()

Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import static org.apache.datasketches.theta.PreambleUtil.SER_VER_BYTE;

import org.apache.datasketches.common.Family;
import org.apache.datasketches.common.MemoryStatus;
import org.apache.datasketches.common.SketchesArgumentException;
import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.memory.WritableMemory;
Expand All @@ -35,7 +36,7 @@
*
* @author Lee Rhodes
*/
public abstract class SetOperation {
public abstract class SetOperation implements MemoryStatus {
static final int CONST_PREAMBLE_LONGS = 3;

SetOperation() {}
Expand Down Expand Up @@ -237,20 +238,6 @@ public static int getMaxAnotBResultBytes(final int nomEntries) {
*/
public abstract Family getFamily();

/**
* Returns true if the backing resource of <i>this</i> is identical with the backing resource
* of <i>that</i>. The capacities must be the same. If <i>this</i> is a region,
* the region offset must also be the same.
*
* <p>Note: Only certain set operators during stateful operations can be serialized.
* Only when they are stored into Memory will this be relevant.</p>
*
* @param that A different non-null and alive object
* @return true if the backing resource of <i>this</i> is the same as the backing resource
* of <i>that</i>.
*/
public abstract boolean isSameResource(Memory that);

//restricted

/**
Expand Down
10 changes: 0 additions & 10 deletions src/main/java/org/apache/datasketches/theta/SingleItemSketch.java
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory not used as a backing store

  • Removed hasMemory(), isDirect(), did not have isSameResource()

Original file line number Diff line number Diff line change
Expand Up @@ -343,16 +343,6 @@ public double getUpperBound(final int numStdDev) {
return 1.0;
}

@Override
public boolean hasMemory() {
return false;
}

@Override
public boolean isDirect() {
return false;
}

@Override
public boolean isEmpty() {
return false;
Expand Down
29 changes: 2 additions & 27 deletions src/main/java/org/apache/datasketches/theta/Sketch.java
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Base abstract class

  • Added implements MemoryStatus and import MemoryStatus
  • Removed hasMemory(), isDirect(), isSameResource()

Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
import static org.apache.datasketches.thetacommon.HashOperations.count;

import org.apache.datasketches.common.Family;
import org.apache.datasketches.common.MemoryStatus;
import org.apache.datasketches.common.SketchesArgumentException;
import org.apache.datasketches.memory.Memory;
import org.apache.datasketches.memory.WritableMemory;
Expand All @@ -44,7 +45,7 @@
*
* @author Lee Rhodes
*/
public abstract class Sketch {
public abstract class Sketch implements MemoryStatus {
static final int DEFAULT_LG_RESIZE_FACTOR = 3; //Unique to Heap

Sketch() {}
Expand Down Expand Up @@ -383,26 +384,12 @@ public double getUpperBound(final int numStdDev) {
: getRetainedEntries(true);
}

/**
* Returns true if this sketch's data structure is backed by Memory or WritableMemory.
* @return true if this sketch's data structure is backed by Memory or WritableMemory.
*/
public abstract boolean hasMemory();

/**
* Returns true if this sketch is in compact form.
* @return true if this sketch is in compact form.
*/
public abstract boolean isCompact();

/**
* Returns true if the this sketch's internal data structure is backed by direct (off-heap)
* Memory.
* @return true if the this sketch's internal data structure is backed by direct (off-heap)
* Memory.
*/
public abstract boolean isDirect();

/**
* <a href="{@docRoot}/resources/dictionary.html#empty">See Empty</a>
* @return true if empty.
Expand All @@ -424,18 +411,6 @@ public boolean isEstimationMode() {
*/
public abstract boolean isOrdered();

/**
* Returns true if the backing resource of <i>this</i> is identical with the backing resource
* of <i>that</i>. The capacities must be the same. If <i>this</i> is a region,
* the region offset must also be the same.
* @param that A different non-null object
* @return true if the backing resource of <i>this</i> is the same as the backing resource
* of <i>that</i>.
*/
public boolean isSameResource(final Memory that) {
return false;
}

/**
* Returns a HashIterator that can be used to iterate over the retained hash values of the
* Theta sketch.
Expand Down
14 changes: 13 additions & 1 deletion src/main/java/org/apache/datasketches/theta/UnionImpl.java
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory used as backing store via gadget

  • Added hasMemory(), isDirect(), updated isSameResource()

Original file line number Diff line number Diff line change
Expand Up @@ -262,10 +262,22 @@ public CompactSketch getResult(final boolean dstOrdered, final WritableMemory ds
minThetaLong, curCountOut, seedHash, empty, true, dstOrdered, dstOrdered, dstMem, compactCacheOut);
}

@Override
public boolean hasMemory() {
return gadget_ instanceof DirectQuickSelectSketchR
? gadget_.hasMemory() : false;
}

@Override
public boolean isDirect() {
return gadget_ instanceof DirectQuickSelectSketchR
? gadget_.isDirect() : false;
}

@Override
public boolean isSameResource(final Memory that) {
return gadget_ instanceof DirectQuickSelectSketchR
? gadget_.getMemory().isSameResource(that) : false;
? gadget_.isSameResource(that) : false;
}

@Override
Expand Down
Loading
Loading