feat(format): format does not specify an element type for list

[SemyonSinchenko]

Describe the enhancement requested

While LIST itself is specified in format spec (protobuf), the element type does not specified anyhow and serialization the protobuf will result in just "list". But in C++ implementation it is list<element_type>, for example, list<float>.

Component(s)

Format

[SemyonSinchenko]

It cannot be just a enum, except the case we want to have something like this:

enum DataType {
    BOOL = 0;
    INT32 = 1;
    INT64 = 2;
    FLOAT = 3;
    DOUBLE = 4;
    STRING = 5;
    DATE = 6;
    TIMESTAMP = 7;
    TIME = 8;
    LIST_BOOL = 9;
    LIST_INT32 = 10;
    LIST_INT64 = 11;
    ...
};

but in my understanding it will be impossible to maintain.

[SemyonSinchenko]

My proposal is to make each possible data type as a message, like this:

message Bool {
   string name = 1;
}

message Int32 {
   string name = 1;
}

...

message List {
   string name = 1;
   onof element_type {
      Bool bool_type = 2;
      Int32 int32_type = 3;
      ...
   }
}

but it is a breaking change.

[SemyonSinchenko]

@Thespica Hi! What do you think about it? As I remember, at the moment proto-specs are used inside Java only

[yangxk1]

@SemyonSinchenko Hi! List may look like List<Object> in java, but isn't ideal in terms of type safety and clarity. Maybe we can refer to the design of com.sun.org.apache.bcel.internal.generic.Type in Spark.

[SemyonSinchenko]

@SemyonSinchenko Hi! List may look like List<Object> in java, but isn't ideal in terms of type safety and clarity. Maybe we can refer to the design of com.sun.org.apache.bcel.internal.generic.Type in Spark.

But why we do not want to specify the element type in format spec? At the moment even cpp implementation does not follow the spec because it specifies the element type

[SemyonSinchenko]

Here (https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample%2Fparquet%2Fperson_with_feature.vertex.yml) it is a "typed" list, not just list. Should we specify it in format spec?

[yangxk1]

You're right.
The current implementation in Java cannot handle List<float>
The C++ implementation is similar to your first proposal.

incubator-graphar/cpp/src/graphar/types.cc

Line 119 in 712b204

std::shared_ptr<DataType> DataType::TypeNameToDataType(const std::string& str) {

we can use it If we don’t need to support nested types (e.g., List<List<float>>).