Const generics support

[donkeyteethUX]

I am working on a library for creating a bag-of-words representation of a corpus of images, for use with image/place recognition or SLAM algorithms. There are various types of image keypoints/descriptors, but generally a descriptor is a bit array of 32, 64, or 128 bytes. It would be nice to use the new min-const-generics to cleanly support descriptors of any size, and use serde to save/load the resulting descriptor vocabulary.

Are there plans to support (de)serialization of generic sized arrays? Is there a decent workaround in the meantime?

Edit: I saw the discussion here: #1860 Is this being worked on?

[MikailBag]

AFAIK there is no way to directly support arrays of arbitrary ways without breaking backcompat.

As a workaround, one can make following:

struct Arr<T, const N: usize>(pub [T;N]);
impl<T: Serialize, const N: usize> Serialize for Arr<T, N> {}
impl<'de, T: Deserialize, const N: usize> Deserialize for Arr<T, N> {}

and also provide some with helpers.

[sdd]

This is also something that I am having problems with. I'm trying to work with both const generic arrays and Vecs of const generic arrays.

@MikailBag can you point me in the right place for how to do what you suggest above in a little more detail? It would be much appreciated. Thanks!

[jonasbb]

@sdd If you need support for nesting you either need wrapper types or you try serde_with. This will work for any number, you just need to make sure the value in the type and the annotation is equal.

#[serde_with::serde_as]
#[derive(Serialize)]
struct Foo {
    #[serde_as(as = "[_; 2]")]
    bar: [String; 2],
    #[serde_as(as = "Vec<[_; 5]>")]
    foobar: Vec<[u8; 5]>,
}

[MikailBag]

here is more verbose (but untested) code

mod arrays {
    use std::{convert::TryInto, marker::PhantomData};

    use serde::{
        de::{SeqAccess, Visitor},
        ser::SerializeTuple,
        Deserialize, Deserializer, Serialize, Serializer,
    };
    pub fn serialize<S: Serializer, T: Serialize, const N: usize>(
        data: &[T; N],
        ser: S,
    ) -> Result<S::Ok, S::Error> {
        let mut s = ser.serialize_tuple(N)?;
        for item in data {
            s.serialize_element(item)?;
        }
        s.end()
    }

    struct ArrayVisitor<T, const N: usize>(PhantomData<T>);

    impl<'de, T, const N: usize> Visitor<'de> for ArrayVisitor<T, N>
    where
        T: Deserialize<'de>,
    {
        type Value = [T; N];

        fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
            formatter.write_str(&format!("an array of length {}", N))
        }

        #[inline]
        fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
        where
            A: SeqAccess<'de>,
        {
            // can be optimized using MaybeUninit
            let mut data = Vec::with_capacity(N);
            for _ in 0..N {
                match (seq.next_element())? {
                    Some(val) => data.push(val),
                    None => return Err(serde::de::Error::invalid_length(N, &self)),
                }
            }
            match data.try_into() {
                Ok(arr) => Ok(arr),
                Err(_) => unreachable!(),
            }
        }
    }
    pub fn deserialize<'de, D, T, const N: usize>(deserializer: D) -> Result<[T; N], D::Error>
    where
        D: Deserializer<'de>,
        T: Deserialize<'de>,
    {
        deserializer.deserialize_tuple(N, ArrayVisitor::<T, N>(PhantomData))
    }
}

use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct TypeWithArray {
    #[serde(with = "arrays")]
    arr: [u8; 64],
}

[sdd]

I wasn't sure if the above was going to work for a type defined like this:

#[derive(Serialize, Deserialize, Debug)]
struct TypeWithConstGenericArray<const N: usize> {
    #[serde(with = "arrays")]
    arr: [u8; N],
}

But it does! See here: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=8b514073821e558a5ce862f64361492e

Thanks so much for your help @MikailBag and @jonasbb

[simias]

I apologize in advance if this is not the right place to voice this opinion but I really find that making it a pain to serialize arbitrarily large arrays in order to support the (in my experience) rare [T; 0] case was a questionable decision.

Now, hindsight is 20/20 and I understand that this issue was probably not obvious at the time, and now it would break backcompat, but IMO it's important enough that I think a serde 2.0 wouldn't necessarily be unwarranted here.

I don't know if it's because of the way I code but I encounter this limitation all the time and it's always frustrating to work around, especially now that Rust officially supports const generics.

I realize that you shouldn't break backcompat willy-nilly and I appreciate the commitment to stability, but that introduces a pretty huge usability issue for the sake of preserving some (IMO) very niche [T; 0] behavior.

Anyway, that's my two cents, I enjoy using serde a whole lot so I hope I don't come off as too entitled, I complain because I care!

[noonien]

I keep bumping into this issue on almost every project that I need to use serde on.

I agree with @simias. I don't think support for [T; 0], which makes no sense to (de)serialize, is worth not supporting arrays of any size.

[fadedbee]

Likewise, I also keep hitting this issue. Upon searching, I find my own Stack Overflow question.

[Kromey]

Since I needed this too, I began working on my own solution before being pointed to this issue. I wound up with something similar to @MikailBag 's code above, which I've just finished adding tests and documentation to. If anyone wants to take a look at my code I'd appreciate any feedback. If you're just looking for a fix, you can grab it from Crates.io.

I also happened to stumble upon another crate, serde-big-array, which can provide similar functionality; it relies on unsafe code, which frankly is beyond my ability to fully understand its safety or possible repercussions, hence continuing on to publish my own (though mine was basically done by the time I found it anyway).

[vadixidav]

Ran into this for (almost) the same reason as @donkeyteethUX. Working on computer vision stuff (binary feature descriptors from AKAZE, and other algorithms) and I have things like this and this (this one actually had a bug in it that broke bincode but not serde_json in my testing!). Official implementations would help a lot, as I made a subtle error here (in the second example).