Understanding result of array_equal with nested arrays

[alexandertuna]

Hi awkward. Thanks for this great codebase, and I'm sorry if I'm asking a question which has been answered elsewhere. I looked around and couldn't find anything.

Do you understand why array_equal gives True in this case? The nested arrays are different.

a = ak.Array([
    [ [], [1] ],
    [ [], [0] ]
])
b = ak.Array([
    [ [], [1], [] ],
    [ [0] ]
])
print(ak.array_equal(a, b))

ak.__version__ = 2.7.4

[ianna]

Hi @alexandertuna .

Thanks for brining it up. We try to follow the Numpy behaviour in what the functions do. However, it may not always be possible given that we deal with the rugged structures. For example, https://numpy.org/doc/stable/reference/generated/numpy.equal.html#numpy.equal:

>>> a = np.array([2, 4, 6])
>>> b = np.array([2, 4, 2])
>>> a == b
array([ True,  True, False])
>>> a = np.array([[2, 4, 6], [2, 4, 6]])
>>> b = np.array([[2, 4, 2], [2, 4, 2]])
>>> a == b
array([[ True,  True, False],
       [ True,  True, False]]) 

If we do the same == operation on the awkward arrays we get a

ValueError: cannot broadcast nested list

This error occurred while calling

    numpy.equal.__call__(
        <Array [[[1], [1], [1]], [[0]]] type='2 * var * var * int64'>
        <Array [[[1], [1]], [[1], [0]]] type='2 * var * var * int64'>
    )

So we have to remove the structure:

>>> ak.flatten(b)==ak.flatten(a)
<Array [[True], [True], [True], [True]] type='4 * var * bool'>

I think, array_equal behaviour should follow the == operation on the awkward arrays.

[alexandertuna]

Hi @ianna, thanks for explaining! I agree, ragged arrays make life much more complicated than regular arrays.

However, I think this result is confusing and a "gotcha". It seems to violate the definition in the documentation: "True if two arrays have the same shape and elements", since the arrays in my example have different shapes (depending on how "shape" is defined).

I think this result also violates my expectation of how "equal" arrays should behave. If a and b are "equal", I expect to be able to treat them as equal, but I can't:

a = ak.Array([
    [ [], [1] ],
    [ [], [0] ]
])
b = ak.Array([
    [ [], [1], [] ],
    [ [0] ]
])
print(ak.array_equal(a, b))
print(a + a)
print(a + b)

gives

True
[[[], [2]], [[], [0]]]
ValueError: cannot broadcast nested list

Is it possible to enforce structure-equality before checking element-equality? In my example, the two arrays have different ak.num:

print(ak.num(a))
print(ak.num(b))

gives

[2, 2]
[3, 1]

Just a thought. I don't know how well this scales to arbitrary depths of nesting

[ikrommyd]

The problem is that the two arrays have the same content and different index. That's easy to see if you look at the layouts.

In [4]: a = ak.Array([
   ...:     [ [], [1] ],
   ...:     [ [], [0] ]
   ...: ])
   ...: b = ak.Array([
   ...:     [ [], [1], [] ],
   ...:     [ [0] ]
   ...: ])

In [5]: a.layout
Out[5]:
<ListOffsetArray len='2'>
    <offsets><Index dtype='int64' len='3'>
        [0 2 4]
    </Index></offsets>
    <content><ListOffsetArray len='4'>
        <offsets><Index dtype='int64' len='5'>[0 0 1 1 2]</Index></offsets>
        <content><NumpyArray dtype='int64' len='2'>[1 0]</NumpyArray></content>
    </ListOffsetArray></content>
</ListOffsetArray>

In [6]: b.layout
Out[6]:
<ListOffsetArray len='2'>
    <offsets><Index dtype='int64' len='3'>
        [0 3 4]
    </Index></offsets>
    <content><ListOffsetArray len='4'>
        <offsets><Index dtype='int64' len='5'>[0 0 1 1 2]</Index></offsets>
        <content><NumpyArray dtype='int64' len='2'>[1 0]</NumpyArray></content>
    </ListOffsetArray></content>
</ListOffsetArray>

a's outer Index has [0 2 4] data while b's has [0 3 4] data. The contents are the same. Both have

    <content><ListOffsetArray len='4'>
        <offsets><Index dtype='int64' len='5'>[0 0 1 1 2]</Index></offsets>
        <content><NumpyArray dtype='int64' len='2'>[1 0]</NumpyArray></content>
    </ListOffsetArray></content>

They both have the same "shape" which is '2 * var * var * int64' but not the same inner lengths across all dimensions. When checking for equality of jagged arrays (that is either ListArray or ListOffsetArray) which only check for equality of their contents and not their Indexs. In my opinion, that should change and I agree with @alexandertuna that it's counter-intuitive.

[ikrommyd]

@alexandertuna Fixed by #3425. Feel free to close the discussion.

[alexandertuna]

Thanks @ianna @ikrommyd !!