[python-package] support sub-classing scikit-learn estimators

[jameslamb]

I recently saw a Stack Overflow post ("Why can't I wrap LGBM?") expressing the same concerns from #4426 ... it's difficult to sub-class lightgbm's scikit-learn estimators.

It doesn't have to be! Look how minimal the code is for XGBRFRegressor:

https://github.com/dmlc/xgboost/blob/45009413ce9f0d2bdfcd0c9ea8af1e71e3c0a191/python-package/xgboost/sklearn.py#L1869

This PR proposes borrowing some patterns I learned while working on xgboost's scikit-learn estimators to make it easier to sub-class lightgbm estimators. This also has the nice side effect of simplifying the lightgbm.dask code 😁

Notes for Reviewers

Why make the breaking change of requiring keyword args?

As part of this PR, I'm proposing immediately switching the constructors for scikit-learn estimators here (including those in lightgbm.dask) to only supporting keyword arguments.

Why I'm proposing this instead of a deprecation cycle:

• scikit-learn itself does this (HistGradientBoostingClassifier example)
  • so all of its machinery passing parameters around as keyword arguments
  • keyword arguments are recommended throughout https://scikit-learn.org/stable/developers/develop.html
• I strongly suspect that using positional arguments for these constructors is rare
• anyone relying on positional arguments will get a loud and easy-to-diagnose-and-fix error, so the effort to adjust should be minimal

import lightgbm as lgb
lgb.LGBMClassifier("gbdt")
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: LGBMClassifier.__init__() takes 1 positional argument but 2 were given

I posted a related answer to that Stack Overflow question

https://stackoverflow.com/a/79344862/3986677

[jameslamb]

Made these changes based on this test failure:

>       assert dask_spec.kwonlyargs == sklearn_spec.kwonlyargs
E       AssertionError: assert ['client'] == []
E         
E         Left contains one more item: 'client'
E         Use -v to get more diff

(build link)

But also... if the changes I'm proposing in dask.py are accepted, we wouldn't even need to have this test any more, in my opinion. It was just here to ensure the 2 lists of keyword args (one in LGBMModel and one in the Dask estimators) was consistent.

I'd like to discuss removing this test as part of the other review conversation on this PR.