How to cast Decimal128 to a supported type?

[Hasham04]

I am reading a parquet file and some of the columns in that file are of type decimal128. I know from one of the issues that decimal is not supported in vaex #1605. But is there a workaround or a hack that I can use to convert it to a supported data type and then to any data type that I want? Thank you for your help!

[Hasham04]

My current approach

import pyarrow as pa
import vaex

df= vaex.open('file.parquet')
coloumns=df.get_column_names()

schema={}
for coloumn in coloumns:
    if df[coloumn].dtype ==pa.decimal128(29, 6):
        schema[coloumn]=pa.float64()
    else:
        schema[coloumn]=pa.string()

df = df.to_arrow_table()
fields = [pa.field(x, y) for x, y in schema.items()]
field=pa.schema(fields)
df = df.cast(field)
df=vaex.from_arrow_table(df)
print(df.dtypes)
# this works but looking for a better way to do it. i appreciate your help :)

[JovanVeljanoski]

Just cast those types to float.
so if x is of that decimal type you can do df['x'] = df.x.astype('float').

But we should support this i suppose.

[Hasham04]

Thanks for the quick reply.

df['x'] = df.x.astype('float') # works perfectly

[bls-lehoai]

@JovanVeljanoski
as I know the float type makes the calculation of the huge number incorrect. Is there another way to get the exact calculation on the huge number (like 30 digits)?

[JovanVeljanoski]

I don't know. Vaex is not special in this regard.