In the forward pass at the inference time, why don't we add output multiplier (1/N) like the training time?
Inference time:
|
logits = self.lm_head(x[:, [-1], :]) # note: using list [-1] to preserve the time dim |
Training time:
|
x *= self.config.mup_output_alpha / self.config.mup_width_multiplier |
