ResidualLFQ was successful, but ResidualVQ failed severely! #73
Description
When training the MeshAutoencoder, I compared ResidualLFQ and ResidualVQ.
ResidualLFQ is your default option, which can rebuild a reasonable structure.
However, when I use ResidualVQ (without changing any of your default parameters), the reconstruction results in significant errors, the validation set loss gradually increases, and the reconstruction only yields a few faces(e.g. 3 faces).
I'm not quite sure what the reason is.
if use_residual_lfq: self.quantizer = ResidualLFQ( dim = dim_codebook, num_quantizers = num_quantizers, codebook_size = codebook_size, commitment_loss_weight = 1., **rlfq_kwargs, **rq_kwargs ) else: self.quantizer = ResidualVQ( dim = dim_codebook, num_quantizers = num_quantizers, codebook_size = codebook_size, shared_codebook = True, commitment_weight = 1., stochastic_sample_codes = rvq_stochastic_sample_codes, # sample_codebook_temp = 0.1, # temperature for stochastically sampling codes, 0 would be equivalent to non-stochastic **rvq_kwargs, **rq_kwargs )
some defalt parameters:
use_residual_lfq = True, # whether to use the latest lookup-free quantization
rq_kwargs: dict = dict(
quantize_dropout = True,
quantize_dropout_cutoff_index = 1,
quantize_dropout_multiple_of = 1,
),
rvq_kwargs: dict = dict(
kmeans_init = True,
threshold_ema_dead_code = 2,
),
rlfq_kwargs: dict = dict(
frac_per_sample_entropy = 1.
),
rvq_stochastic_sample_codes = True,
loss curve:
red curve is ResidualVQ, and grey curve is ResidualLFQ.
