Merged master and resolved conflicts.

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
ray-project · Jan 20, 2025 · 1f2cc24 · 1f2cc24
2 parents 931b93c + 243927c
commit 1f2cc24
Show file tree

Hide file tree

Showing 36 changed files with 183 additions and 128 deletions.
diff --git a/doc/source/rllib/algorithm-config.rst b/doc/source/rllib/algorithm-config.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rllib-algo-configuration-docs:
 
 AlgorithmConfig API
 ===================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 RLlib's :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` API is
 the auto-validated and type-safe gateway into configuring and building an RLlib
 :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`.

diff --git a/doc/source/rllib/checkpoints.rst b/doc/source/rllib/checkpoints.rst
@@ -1,14 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
-
 .. _rllib-checkpoints-docs:
 
 Checkpointing
 =============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 RLlib offers a powerful checkpointing system for all its major classes, allowing you to save the
 states of :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` instances and their subcomponents
 to local disk or cloud storage, and restore previously run experiment states and individual subcomponents.

diff --git a/doc/source/rllib/external-envs.rst b/doc/source/rllib/external-envs.rst
@@ -1,13 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rllib-external-env-setups-doc:
 
-
 External Environments and Applications
 ======================================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 In many situations, it doesn't make sense for an RL environment to be "stepped" by RLlib.
 For example, if you train a policy inside a complex simulator that operates its own execution loop,
 like a game engine or a robotics simulation. A natural and user friendly approach is to flip this setup around

diff --git a/doc/source/rllib/hierarchical-envs.rst b/doc/source/rllib/hierarchical-envs.rst
@@ -1,13 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rllib-hierarchical-environments-doc:
 
-
 Hierarchical Environments
 =========================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 You can implement hierarchical training as a special case of multi-agent RL. For example, consider a two-level hierarchy of policies,
 where a top-level policy issues high level tasks that are executed at a finer timescale by one or more low-level policies.
 The following timeline shows one step of the top-level policy, which corresponds to four low-level actions:

diff --git a/doc/source/rllib/key-concepts.rst b/doc/source/rllib/key-concepts.rst
@@ -1,14 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
-
 .. _rllib-key-concepts:
 
 Key concepts
 ============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 To help you get a high-level understanding of how the library works, on this page, you learn about the
 key concepts and general architecture of RLlib.
 

diff --git a/doc/source/rllib/multi-agent-envs.rst b/doc/source/rllib/multi-agent-envs.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rllib-multi-agent-environments-doc:
 
 Multi-Agent Environments
 ========================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 In a multi-agent environment, multiple "agents" act simultaneously, in a turn-based
 sequence, or through an arbitrary combination of both.
 

diff --git a/doc/source/rllib/new-api-stack-migration-guide.rst b/doc/source/rllib/new-api-stack-migration-guide.rst
@@ -1,8 +1,5 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
-
 .. _rllib-new-api-stack-migration-guide:
 
 .. testcode::
@@ -15,6 +12,8 @@
 New API stack migration guide
 =============================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 This page explains, step by step, how to convert and translate your existing old API stack
 RLlib classes and code to RLlib's new API stack.
 

diff --git a/doc/source/rllib/package_ref/algorithm-config.rst b/doc/source/rllib/package_ref/algorithm-config.rst
@@ -1,13 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _algorithm-config-reference-docs:
 
-
 Algorithm Configuration API
 ===========================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 .. currentmodule:: ray.rllib.algorithms.algorithm_config
 
 Constructor

diff --git a/doc/source/rllib/package_ref/algorithm.rst b/doc/source/rllib/package_ref/algorithm.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _algorithm-reference-docs:
 
 Algorithms
 ==========
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 The :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` class is the highest-level API in RLlib responsible for **WHEN** and **WHAT** of RL algorithms.
 Things like **WHEN** should we sample the algorithm, **WHEN** should we perform a neural network update, and so on.
 The **HOW** will be delegated to components such as ``RolloutWorker``, etc..

diff --git a/doc/source/rllib/package_ref/callback.rst b/doc/source/rllib/package_ref/callback.rst
@@ -1,10 +1,12 @@
-.. include:: /_includes/rllib/new_api_stack.rst
+.. include:: /_includes/rllib/we_are_hiring.rst
 
 .. _rllib-callback-reference-docs:
 
 Callback APIs
 =============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 Callback APIs enable you to inject code into an experiment, an Algorithm,
 and the subcomponents of an Algorithm.
 

diff --git a/doc/source/rllib/package_ref/distributions.rst b/doc/source/rllib/package_ref/distributions.rst
@@ -1,13 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
-
 .. _rllib-distributions-reference-docs:
 
 Distribution API
 ================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 .. currentmodule:: ray.rllib.models.distributions
 
 Base Distribution class

diff --git a/doc/source/rllib/package_ref/env.rst b/doc/source/rllib/package_ref/env.rst
@@ -1,13 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _env-reference-docs:
 
 Environments
 ============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 RLlib mainly supports the `Farama gymnasium API <https://gymnasium.farama.org/>`__ for
 single-agent environments, and RLlib's own :py:class:`~ray.rllib.env.multi_agent_env.MultiAgentEnv`
 API for multi-agent setups.

diff --git a/doc/source/rllib/package_ref/env/env_runner.rst b/doc/source/rllib/package_ref/env/env_runner.rst
@@ -1,13 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _env-runner-reference-docs:
 
 EnvRunner API
 =============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.env_runner.EnvRunner
 ------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/multi_agent_env.rst b/doc/source/rllib/package_ref/env/multi_agent_env.rst
@@ -1,13 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _multi-agent-env-reference-docs:
 
 MultiAgentEnv API
 =================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.multi_agent_env.MultiAgentEnv
 ---------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/multi_agent_env_runner.rst b/doc/source/rllib/package_ref/env/multi_agent_env_runner.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _multi-agent-env-runner-reference-docs:
 
 MultiAgentEnvRunner API
 =======================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.multi_agent_env_runner.MultiAgentEnvRunner
 ----------------------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/multi_agent_episode.rst b/doc/source/rllib/package_ref/env/multi_agent_episode.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _multi-agent-episode-reference-docs:
 
 MultiAgentEpisode API
 =====================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.multi_agent_episode.MultiAgentEpisode
 -----------------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/single_agent_env_runner.rst b/doc/source/rllib/package_ref/env/single_agent_env_runner.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _single-agent-env-runner-reference-docs:
 
 SingleAgentEnvRunner API
 ========================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.single_agent_env_runner.SingleAgentEnvRunner
 ------------------------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/single_agent_episode.rst b/doc/source/rllib/package_ref/env/single_agent_episode.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _single-agent-episode-reference-docs:
 
 SingleAgentEpisode API
 ======================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.single_agent_episode.SingleAgentEpisode
 -------------------------------------------------
 

diff --git a/doc/source/rllib/package_ref/env/utils.rst b/doc/source/rllib/package_ref/env/utils.rst
@@ -1,12 +1,12 @@
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _env-utils-reference-docs:
 
 Env Utils
 =========
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 rllib.env.utils
 ---------------
 

diff --git a/doc/source/rllib/package_ref/index.rst b/doc/source/rllib/package_ref/index.rst
@@ -1,13 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rllib-reference-docs:
 
 Ray RLlib API
 =============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 .. tip:: We'd love to hear your feedback on using RLlib - `sign up to our forum and start asking questions <https://discuss.ray.io>`_!
 
 This section contains an overview of RLlib's package- and API reference.

diff --git a/doc/source/rllib/package_ref/learner.rst b/doc/source/rllib/package_ref/learner.rst
@@ -1,14 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _learner-reference-docs:
 
-
 LearnerGroup API
 ================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 Configuring a LearnerGroup and Learner actors
 ---------------------------------------------
 

diff --git a/doc/source/rllib/package_ref/offline.rst b/doc/source/rllib/package_ref/offline.rst
@@ -1,14 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _new-api-offline-reference-docs:
 
-
 Offline RL API
 ==============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 Configuring Offline RL
 ----------------------
 

diff --git a/doc/source/rllib/package_ref/replay-buffers.rst b/doc/source/rllib/package_ref/replay-buffers.rst
@@ -1,13 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _replay-buffer-api-reference-docs:
 
 Replay Buffer API
 =================
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 The following classes don't take into account the separation of experiences from different policies, multi-agent replay buffers will be explained further below.
 
 Replay Buffer Base Classes

diff --git a/doc/source/rllib/package_ref/rl_modules.rst b/doc/source/rllib/package_ref/rl_modules.rst
@@ -1,13 +1,11 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _rlmodule-reference-docs:
 
 RLModule APIs
 =============
 
+.. include:: /_includes/rllib/new_api_stack.rst
 
 RLModule specifications and configurations
 -------------------------------------------

diff --git a/doc/source/rllib/package_ref/utils.rst b/doc/source/rllib/package_ref/utils.rst
@@ -1,13 +1,12 @@
-
 .. include:: /_includes/rllib/we_are_hiring.rst
 
-.. include:: /_includes/rllib/new_api_stack.rst
-
 .. _utils-reference-docs:
 
 RLlib Utilities
 ===============
 
+.. include:: /_includes/rllib/new_api_stack.rst
+
 Here is a list of all the utilities available in RLlib.
 
 Scheduler API