pretraining raises error

[moghadas76]

Unexpected key(s) in state_dict: "layers.0.estimation_gate.FC1.weight", "layers.0.estimation_gate.FC1.bias", "layers.0.estimation_gate.FC2.weight", "layers.0.estimation_gate.FC2.bias", "layers.0.inh_layer.sub_and_norm.ln.weight", "layers.0.inh_layer.sub_and_norm.ln.bias", "layers.1.estimation_gate.FC1.weight", "layers.1.estimation_gate.FC1.bias", "layers.1.estimation_gate.FC2.weight", "layers.1.estimation_gate.FC2.bias", "layers.1.inh_layer.sub_and_norm.ln.weight", "layers.1.inh_layer.sub_and_norm.ln.bias", "layers.2.estimation_gate.FC1.weight", "layers.2.estimation_gate.FC1.bias", "layers.2.estimation_gate.FC2.weight", "layers.2.estimation_gate.FC2.bias", "layers.2.inh_layer.sub_and_norm.ln.weight", "layers.2.inh_layer.sub_and_norm.ln.bias", "layers.3.estimation_gate.FC1.weight", "layers.3.estimation_gate.FC1.bias", "layers.3.estimation_gate.FC2.weight", "layers.3.estimation_gate.FC2.bias", "layers.3.inh_layer.sub_and_norm.ln.weight", "layers.3.inh_layer.sub_and_norm.ln.bias", "layers.4.estimation_gate.FC1.weight", "layers.4.estimation_gate.FC1.bias", "layers.4.estimation_gate.FC2.weight", "layers.4.estimation_gate.FC2.bias", "layers.4.inh_layer.sub_and_norm.ln.weight", "layers.4.inh_layer.sub_and_norm.ln.bias".

[zezhishao]

Thanks for your reporting. I will update ckpts later.