Reinforcement Learning for Orbital Transfers at the 2026 AI Winter School (Brown University)

Originally published January 9, 2026 Updated September 6, 2026 6 min read

Training PPO policies for orbital transfers and comparing their trajectories and delta-v with a Hohmann baseline.

At the 2026 AI Winter School, hosted by the Center for the Fundamental Physics of the Universe at Brown University, I led a 2.5-hour hands-on workshop on reinforcement learning for orbital transfers.

I used a two-body transfer with a known analytic solution so we could compare learned policies with a baseline. This article is a workshop guide to training and inspecting policies, rather than a performance report. The notebook contains the environment, training procedure, and saved example outputs for running that comparison.

Control Problem

The notebook used nondimensional two-body dynamics: unit gravitational parameter, an initial circular orbit at radius 1, and a target circular orbit at radius 1.6. I call those radii $r_{1}$ and $r_{2}$ below. The model omitted drag, finite-duration thrust, J2 perturbations, third bodies, attitude dynamics, and mass depletion. Control was a tangential impulse applied once per simulation step.

Here the arrowed r denotes the spacecraft position vector; the plain r denotes its scalar radius. The state evolves under central gravity:

\begin{aligned}\frac{d\vec{r}}{dt} &= \vec{v} \\ \frac{d\vec{v}}{dt} &= -\frac{\mu\vec{r}}{r^3} \\ r &= |\vec{r}|\end{aligned}

For a circular target orbit, the target specific energy and angular momentum are:

\begin{aligned}E^* &= -\frac{\mu}{2r_2} \\ L^* &= \sqrt{\mu r_2}\end{aligned}

The RL environment did not need to know the absolute orbital angle. The observation vector used normalized radius, radial velocity, tangential velocity, angular-momentum error, energy error, and previous action. Removing angle makes the policy rotationally symmetric: the same local orbital state should produce the same control decision anywhere around the planet.

Hohmann Benchmark

Before training PPO, the notebook computed the Hohmann transfer. For circular, coplanar orbits with two impulsive burns, the transfer semi-major axis is:

a_T = \frac{r_1 + r_2}{2}

The two burns and transfer time are:

\begin{aligned}\Delta v_1 &= \sqrt{\mu\left(\frac{2}{r_1} - \frac{1}{a_T}\right)} - \sqrt{\frac{\mu}{r_1}} \\ \Delta v_2 &= \sqrt{\frac{\mu}{r_2}} - \sqrt{\mu\left(\frac{2}{r_2} - \frac{1}{a_T}\right)} \\ T &= \pi\sqrt{\frac{a_T^3}{\mu}}\end{aligned}

The policy comparison uses total Δv, circularization error, and burn history against this baseline.

Hohmann transfer trajectory: the transfer ellipse touching the inner start orbit and the outer target orbit

The transfer ellipse touches the initial circular orbit at r1 and the target orbit at r2.

Verification plots for the simulated Hohmann transfer: radius versus time rising to the target, two thrust impulses showing the burn-coast-burn structure, and cumulative delta-v matching the ideal total

Radius history, the two impulses, and accumulated Δv for the simulated Hohmann transfer.

The finite-timestep simulation does not land exactly on r2: the second burn fires on the first step at or after the computed transfer time. In the notebook’s saved baseline, the final radius is 1.5999 against the theoretical 1.6000, with a timestep of 0.00050; total Δv agrees at the displayed precision of 0.2066. This residual precedes policy training and reflects the numerical implementation of the analytic plan.

RL Formulation

The Gymnasium environment held the physics fixed and varied the control interface:

Component	Implementation
Observation	Normalized `r`, `vr`, `vt`, target angular-momentum error, target energy error, previous action
Discrete action	`coast`, full prograde impulse, full retrograde impulse
Continuous action	throttle in `[-1, 1]`, mapped to a signed tangential Δv impulse
Success criteria	tolerances on $\| r - r_{2} \|$ , $\| v_{r} \|$ , and $\| L - L^{*} \|$
Failure criteria	crash/escape radius or episode timeout

The dense reward used a combined energy/angular-momentum error:

\mathrm{err} = \frac{|E - E^*|}{|E^*|} + \frac{|L - L^*|}{|L^*|}

with shaping approximately proportional to:

\mathrm{err}_{previous} - \gamma\,\mathrm{err}_{current}

Then the environment subtracted fuel and ignition/switching penalties, added a one-time success bonus on first entry into the tolerance region, and added a holding reward for staying there. PPO was trained with observation/reward normalization during training, frozen normalization statistics during evaluation, and deterministic policy rollout for diagnostics.

Failure modes to inspect

The notebook compared policies using trajectory, radius history, radial velocity, thrust impulses, cumulative Δv, number of burns or active-thrust steps, closest-to-target statistics, and the mission report against the Hohmann ideal.

When inspecting a run, check for these possible failure modes:

Discrete control: small fixed impulses can reach the target with many prograde/retrograde corrections. The orbit may satisfy the tolerance band while wasting Δv.
Continuous control: throttle control is more expressive, but it can learn micro-thrusting: almost continuous small corrections that keep the error low while hiding poor fuel efficiency.
Tolerance exploitation: a policy that stops inside a loose tolerance band on an elliptical orbit has not met the same endpoint conditions as the Hohmann transfer. Its Δv is therefore not directly comparable to the ideal circular-to-circular transfer.
Final-state ambiguity: final radius alone is misleading for eccentric orbits. Closest approach, radial-velocity history, angular momentum, and thrust history are needed to interpret what the policy actually learned.

Experiment Loop

The final section exposes these parameters through ModeConfig:

dv_mag: control authority per step
fuel_cost_penalty: cost of using Δv
ignition_penalty: cost of turning on or changing thrust
reward_shaping_scale: strength of dense energy/angular-momentum shaping
training_timesteps, learning_rate, and ent_coef: PPO optimization and exploration behavior

For each comparison, record the configuration, seed, training budget, and success tolerances, then:

train a policy
inspect trajectory, thrust, and Δv
compare against Hohmann
explain the failure mode
change one parameter or design choice
rerun

The saved notebook includes discrete and continuous policy reports, but does not fix a training seed. Its final custom experiment also has a saved configuration printout that differs from the displayed setup. Those outputs illustrate the diagnostics; they do not supply a reproducible comparison of parameter choices.

Materials

Category
AI and Machine Learning 8