Reinforcement learning (RL) is attractive in healthcare because many clinical decisions are sequential. A treatment changes the patient's state, that new state shapes the next decision, and the consequences may appear much later. The same structure also makes clinical RL difficult to evaluate safely.

Sepsis management is a useful example. A patient's history can include vital signs, laboratory results, fluids, vasopressors, antibiotics, and other interventions over time. An RL formulation turns those observations into states, treatment choices into actions, and patient outcomes into rewards. But there is no simple short-term reward that captures the full clinical objective.

My reading of Gottesman et al. (2019) reduces the deployment problem to three questions.

Does the model observe the variables that shaped each decision?

An RL agent can use only what was recorded. Clinical decisions, however, can depend on context that is absent or poorly represented in the dataset. This creates a confounding problem.

For example, severely ill patients may receive fluids earlier and still have worse outcomes. A model can mistake the association between fluid administration and mortality for a harmful treatment effect, when illness severity influenced both the treatment and the outcome. Because actions and patient states unfold over time, missing variables can distort both immediate and delayed effects.

The first question is therefore not whether the dataset is large. It is whether the recorded state is rich enough to support the causal interpretation the policy needs.

How large is the effective dataset?

Offline policy evaluation asks how a new policy would perform using trajectories generated by an older clinical policy. The more the new policy differs from the observed decisions, the less evidence the dataset contains for that policy.

Gottesman et al. illustrate how quickly this support can collapse: in one sepsis example, a cohort of 3,855 patients produced an effective sample size of only a few dozen for evaluating a target policy. A large row count is not the same as broad coverage of long treatment trajectories.

This limits what observational data can justify. It may support careful refinement near existing practice, but it is weak evidence for a policy that repeatedly chooses actions rarely seen in the data.

Will the policy behave as intended in practice?

An RL system optimises the reward it is given, not the clinical intention behind it. Errors in reward design, data processing, or deployment context can therefore produce a policy that scores well in evaluation while making poor decisions.

A reward centred on short-term blood-pressure targets, for example, may encourage excessive vasopressor use while ignoring longer-term harm. The policy can also become stale as treatment standards, patient populations, or data collection practices change.

Prospective use therefore requires more than an offline performance estimate. Clinicians and technical teams need to inspect learned policies, stress-test unusual states, monitor drift, and establish clear limits on autonomous action.

Conclusion

RL may help with sequential treatment decisions, but the evidence burden is higher than for a conventional prediction model. Before deployment, I would ask whether the state captures the real decision context, whether the data support the proposed policy, and whether the reward and monitoring process constrain harmful behaviour. If any answer is unclear, the system is not ready for prospective clinical use.

Reference

• Gottesman, O., Johansson, F., Komorowski, M., Faisal, A., Sontag, D., Doshi-Velez, F., & Celi, L. A. (2019). Guidelines for reinforcement learning in healthcare. Nature Medicine, 25(1), 16–18. https://doi.org/10.1038/s41591-018-0310-5