[cmake] Simplify and improve CMake configuration

[jameslamb]

Summary

Opening this tracking issue to collect links and discussion for proposed work to improve LightGBM's CMake configuration.

LightGBM is on a very new version of CMake (3.28), thanks to #6550 ... let's take advantage of that newer version's powerful features.

Motivation

• make find_package(lightgbm) "just work" for projects depending on LightGBM
• make life easier for repackagers (like conan, conda-forge, nix, macports, etc.)
• make C/C++ examples easier to write (ref: Direct linking from C #4609, Can you provide documentation on C API best practices? #6261)

Description

Non-exhaustive list of potential changes to explore:

• setting VERSION on the project
• setting VERSION and SOVERSION on the shared library
• replacing overly-broad functions with target-specific equivalents:
  • include_directories() -> target_include_directories()
  • add_definitions() -> target_compile_definitions()
• moving some things out of the main CMakeLists.txt
  • e.g., adding a separate CMakeLists.txt in tests/cpp_tests
• making R CMD INSTALL R-package/ or pip install python-package/ "just work"
  • e.g. as XGBoost does (xgboost/R-package/CMakeLists.txt)
• using the CMake of third-party libraries (like fmt and Eigen) instead of just copying them and manually adding them to -I via include_directories() and similar

References

Some references informing this:

• https://gist.github.com/mbinna/c61dbb39bca0e4fb7d1f73b0d66a4fd1
• https://cliutils.gitlab.io/modern-cmake/README.html
• https://cmake.org/cmake/help/v3.31/guide/tutorial/index.html

Notes for Readers

Please comment with other ideas and I'd be happy to add them to the list here if we agree.