Weighting of Loss Function

[facusapienza21]

How to pick the hyperparameters of the regularized regression is quite a problem. One option is just to run many values, but this is not very efficient.

The following blog post has some good ideas and references that may be useful here. Of particular interest is the automatic re-weighting of the loss function based on a de-attach loss function that doesn't propagates the gradient. Not sure about the advantages of something like this, but worth trying.