Use flatness to improve SSL for LMs #1
Description
Hi, thanks for the amazing work! I'm wondering if you've ever tried to leverage this flatness (Trace of Hessian) as a regularization to improve LM pretraining?
{{ message }}
Hi, thanks for the amazing work! I'm wondering if you've ever tried to leverage this flatness (Trace of Hessian) as a regularization to improve LM pretraining?