LSTM with DIST Keras :
the problem is with preprocessing the sparkDF for the model :
schema of the DF : each line have two matrix : FALSE and TRUE ( autocorrection )
schema = StructType([StructField('TRUE',ArrayType(ArrayType(IntegerType()), True)) , StructField('FALSE',ArrayType(ArrayType(IntegerType()), True))])**
reshape_transformer = ReshapeTransformer("TRUE", "TRUE_", (MAX_LEN , LEN))
reshape_transformer_ = ReshapeTransformer("FALSE", "FALSE_", (MAX_LEN , LEN))
trainer = AEASGD(model, optimizer, loss, metrics=["accuracy"], num_workers=num_workers, batch_size=120,features_col="FALSE_", label_col="TRUE_", num_epoch=2,communication_window=32, rho=5.0, learning_rate=0.01)
weights doesn't get updated
HELP