The "empirical Fisher" is not a well-motivated approximation of the Fisher (or the Hessian). It can be a comically bad preconditioner on simple regression problems (see plot). New preprint: