Connecting Java Reinforcement Learning to Python Gymnasium
The first step toward an autonomous car
Gymnasium in the Java ecosystem
One of the limitations in the Java ecosystem is the lack of training environments for reinforcement learning, unlike Python which has Isaac Lab (Robotics mainly), OpenAI Gym (not maintained) and the fork Gymnasium, which is an ecosystem on its own with many environments.
Example of the two environments that are covered in Video Games and Reinforcement Learning and How to Never Forget Deep Q-Networks: Memory Palaces Meet Reinforcement Learning articles:
Being the first article of three, this article details the reason why the project java-rl-dqn-to-rainbow starts with the integration of Gymnasium. The main reason is testability. As CARLA1 is a very large and complex environment, simple environments such as CartPole2 or Mountain Car3 must first be made available to the algorithm, allowing shorter testing cycles. The implementation starts with DQN4, one of the simplest algorithms using Reinforcement Learning and Deep Neural Networks. Once this step is finished, the algorithm is refined and upgraded step by step until the complete Rainbow DQN implementation is reached by the end of the second article. In the last article, an integration with CARLA is made and the agent is trained with Rainbow DQN5.
Evaluating Integration Options
Two integration candidates were evaluated: Py4J (used in PySpark) and ZeroMQ, where ZeroMQ proved to be the better choice as it’s a brokerless messaging library with zero-copy optimizations, instead of Py4J which uses a gateway (broker) that translates the calls between the languages.
The initial trial was to start a Python process from Java and make the communication using the Request and Response pattern, but the problem was synchronizing the two platforms, as if there were any issue with the ordering, the socket would be in an invalid state.
For example, if two Java requests are made, the socket would be "broken", and the same occurs if a response is sent twice in Python, the following order must be respected: req→resp→req … and so on6.
Even if the socket is restarted, the server or client must also be restarted. The protocol was then changed to use the Pirate Patterns, to make the communication reliable7. Even so, the goal was a seamless integration between Java and Python, so the options evaluated were the FFM API (Foreign Function and Memory API) or JavaCPP, where both have the advantage of making an efficient integration with native code.
In the end, JavaCPP was chosen because it has the CPython integration and integrates easily with C++ code8, a plus to make a C++ integration with CARLA too, but in the FFM API, it's necessary to first create a C-API then create the bindings using jextract if the library is not written in pure C9.
JavaCPP
This library was used to call native Python functions, as it implements most of the CPython API, with some macro exceptions due to parser limitations10. In javacpp-presets , the Gym 0.26.2 library and ALE 0.8.0 (Arcade Learning Environment) are already implemented, and the gym module was used successfully. Inside the library, an embedded Python version is bundled within the jar for each platform to execute the gym code. Example:
// omitted code
public class gym {
private static File packageFile;
public static synchronized File cachePackage() throws IOException {
if (packageFile != null) {
return packageFile;
} else {
// This is the location of the jar that will be loaded
packageFile = Loader.cacheResource("/org/bytedeco/gym/python/");
return packageFile;
}
}
// omitted codeThe physical location11:
Similarly, this can be observed with CPython, but separated by platform:
To control the version and libraries used, a gymnasium folder was created in the root of the project with the following content:
README.md
requirements.txtREADME.md contains the instructions to install uv and the correct Python version, along with the environment variables to be exported for use by Java:
// omitted
pip install uv
uv python install cpython-3.12.1-windows-x86_64-none
uv venv --python 3.12.1
.venv/Scripts/activate
uv pip install -r requirements.txt
// omitted
- `export JAVA_RL_SITE_PACKAGES=/path/to/java-rl-dqn-to-rainbow/gymnasium/.venv/include/site/python3.12`
- `export JAVA_RL_SITE_PACKAGES=/path/to/java-rl-dqn-to-rainbow/gymnasium/.venv/Lib/site-packages`The requirements.txt contains the libraries to be installed:
numpy==2.3.5
pygame==2.6.1
matplotlib==3.10.8
gymnasium[classic-control,box2d]==1.2.2With the integration options evaluated, the implementation planning can proceed:
Python is started as a fully managed embedded process.
Code is executed inside this process and the output is retrieved.
The object is created inside Python and its methods are invoked in Java.
Python arrays are read inside Java with zero-copy or memory copy depending on the scenario.
env.render()is invoked to display the environment output in Java using JFrame.Actions are executed in the environment using
env.step(action)and the method output is retrieved.The output of the
stepmethod is converted to DJL12.Multiple tests are created for different environments to validate its usefulness.
During development, many segfaults occurred, as the lifecycle of Python must be managed manually.
Embed Python Lifecycle
Calling Py_Initialize() and Py_FinalizeEx() more than once is not guaranteed to work properly due to external modules, as is the case of this project. That is why Py_FinalizeEx() is not used, as gymnasium depends mainly on NumPy and PyGame13, and if called, a segmentation fault can occur.
To work safely with Python in a multi-thread environment where Java has its own thread state, the methods PyGILState_Ensure() and PyGILState_Release() must be used, e.g.:
// insideGil() wraps this block, accepting a Runnable or Function
var gstate = PyGILState_Ensure();
try {
result = callSomeFunction();
} finally {
PyGILState_Release(gstate);
}Without this, a segmentation fault can occur and the JVM will crash, as race conditions can leave objects in an invalid state, such as an object with a reference count of 0 instead of 1 when multiple threads decrement the reference count variable simultaneously, causing a double-free.
However, if only the thread that called Py_Initialize() is used, in most cases the code block above is not needed, which was encapsulated in the method insideGil() to avoid this boilerplate.
To understand why this matters, it helps to know about the GIL (Global Interpreter Lock) used in the Python environment. It is usually shared between all threads when using Py_Initialize(), which is a limiting factor. NumPy mitigates this by releasing the GIL during intensive C operations, which is why it remains fast.
To circumvent this limitation and better utilize the machine's multiprocessing, the vectorization from the gymnasium library can be used, which runs environments in parallel and returns multiple values at once. If necessary, these features will be implemented in Gymnasium4j in the future.
Multiple Python Interpreters
An alternative is Py_NewInterpreterFromConfig() which allows its own GIL per interpreter, introduced in Python 3.12 as shown in the link.
Another useful method is PyGILState_Check(), which returns 1 if the thread has the GIL locked and can be used for debugging the application14.
Another thing to pay attention to is memory management15 outside Python, as we embedded it in Java, we must take care of its reference counting, below a list for the methods that we can use:
Returns a new reference (you own it, must decrement reference):
PyObject_CallObject()PyObject_Call()PyObject_GetAttrString()PyImport_AddModule()Usually the new module is created and forgotten, as it is released when the Python process finishes, which is the case here, used in
initPython(), explained later.
PyUnicode_FromString()PyLong_FromLong()PyFloat_FromDouble()PyBool_FromLong()PyBytes_FromStringAndSize()PyByteArray_FromStringAndSize()PyDict_New()PyTuple_New()PyList_New()PyObject_Str()PyUnicode_AsUTF8String()PyDict_Items()
Returns a borrowed reference (you don’t own it, do not decrement reference):
PyTuple_GetItem()PyList_GetItem()PyDict_GetItem()PyModule_GetDict()PyErr_Occurred()
Steals a reference (takes ownership from you):
PyTuple_SetItem()PyList_SetItem()
PythonRuntime
Proceeding to the PythonRuntime, the main methods are initPython(), exec(), and eval():
The initPython() method is called only once when the Java process starts. It is synchronized to ensure the code is initialized only once. cachePackages() returns the location of the javacpp CPython and custom libraries installed using uv (the environment/property JAVA_RL_SITE_PACKAGES must be filled). The globals variable holds all variables created in Python by the integration:
@SneakyThrows
public static synchronized void initPython() {
if (initialized) return;
if (Boolean.getBoolean("python.initialized")) {
initialized = true;
return;
}
initialized = Py_Initialize(cachePackages());
globals = PyModule_GetDict(PyImport_AddModule("__main__"));
if (!initialized) {
throw new IllegalArgumentException("PythonRuntime is not initialized!");
}
System.setProperty("python.initialized", "true");
}The exec() method is where Python functions are executed in the interpreter:
public static void exec(String code) {
PyErr_Clear();
try (var _ = PyRun_StringFlags(
code,
Py_file_input,
globals,
globals,
null
)) {
checkError();
}
}Every variable is saved inside globals. For isolated executions, where variables do not live outside the scope, execIsolated() can be used as an alternative. Below an example of its usage:
// Code inside unit test
exec("""
import numpy as np
arr_little = np.array([1, 2, 3], dtype='<f4') # explicit Little-endian
arr_big = np.array([1, 2, 3], dtype='>f4') # explicit Big-endian
""");The eval() is very similar to exec(), the difference lies in the input parameter:
Py_eval_input expects an expression like (1 + 1) or NewInstanceExample(), while Py_file_input executes any code without returning a value:
public static PyObject eval(String expression) {
PyErr_Clear();
try (var result = PyRun_StringFlags(
expression,
Py_eval_input,
globals,
globals,
null
)) {
checkError();
return result;
}
}The PyErr_Clear() clears any non-fatal error that was not cleared and could interfere with PyErr_Occurred(). The checkError() method checks if any error occurred after execution, using PyErr_Occurred(). The pattern is to clear possible old errors and check again at the end.
The other utility classes are PythonDataStructures and PythonTypeChecks, explained as needed, but the names are intuitive.
Integration between NumPy with copy and zero-copy
The NumPyByteBuffer class is used to make an efficient transfer between the ndarray data structure in Python and Java's ByteBuffer (or other languages that support the Buffer Protocol). The NumPy array implements the Buffer Protocol16 , allowing communication without requiring a copy of its byte array, as its memory is contiguous (a block of memory, like a primitive array)17.
The static initializer is responsible for ensuring endianness alignment (the way bits are ordered in the hardware)18 between Java and NumPy. It is implemented this way to perform the check only once:
static {
initPython();
BYTE_ORDER = insideGil(() -> {
exec("import numpy as np");
exec("_test_arr = np.array([1], dtype=np.float32)");
try (var testArr = eval("_test_arr")) {
String byteOrder = attrStr(attr(testArr, "dtype"), "byteorder");
return switch (byteOrder) {
case ">" -> ByteOrder.BIG_ENDIAN;
case "<" -> ByteOrder.LITTLE_ENDIAN;
default -> ByteOrder.nativeOrder();
};
} finally {
exec("del _test_arr");
}
});
}The fillFromNumpy() method handles this communication by making a copy of the Python ndarray. If receives a native array, it reuses it, avoiding array copying. However, since no efficient cycle management is yet implemented in this project (such as an Object Pool pattern), a temporary copy is made, as incorrect deallocation can also cause segmentation faults.
public static void fillFromNumpy(PyObject ndarray, ByteBuffer buffer) {
buffer.clear();
try (var view = new NumPyBufferView(ndarray)) {
int size = view.capacity();
if (size > buffer.capacity()) {
throw new IllegalArgumentException(
"Buffer too small: capacity=" + buffer.capacity() + ", required=" + size
);
}
buffer.put(view.buffer());
}
buffer.flip();
}The NumPyBufferView uses the native Python C-API to bridge this communication. @Delegate from Lombok19 is used to expose the exact same public API of ByteBuffer without requiring a manual implementation, establishing a standard to use the Python object as closely as possible to a ByteBuffer. AutoCloseable is implemented to ensure the view is deallocated by the caller using the correct C-API:
public class NumPyBufferView implements AutoCloseable {
private final Py_buffer view;
@Delegate
private final ByteBuffer buffer;
public NumPyBufferView(@NonNull PyObject ndarray) {
this.view = new Py_buffer();
int rc = PyObject_GetBuffer(ndarray, view, PyBUF_SIMPLE);
if (rc != 0) {
throw new IllegalStateException("PyObject_GetBuffer failed (array not contiguous?), return code: " + rc);
}
long size = view.len();
this.buffer = view.buf().capacity(size).asByteBuffer();
}
/**
* Used internally!
*/
ByteBuffer buffer() {
return buffer;
}
@Override
public void close() {
PyBuffer_Release(view);
}
}The method PyBuffer_Release() deallocates only the view. The original PyObject must be deallocated separately.
When the ByteBuffer is modified, the changes are reflected in the Python object:
exec("import numpy as np; arr = np.array([1.0, 2.0, 3.0], dtype=np.float32)");
try (var pyArr = eval("arr");
var view = new NumPyBufferView(pyArr)) {
var buffer = view.asFloatBuffer();
assertEquals(1.0f, buffer.get(0), 0.0001f);
assertEquals(2.0f, buffer.get(1), 0.0001f);
assertEquals(3.0f, buffer.get(2), 0.0001f);
buffer.put(0, 4f);
buffer.put(1, 5f);
buffer.put(2, 6f);
assertArrayEquals(new double[]{4., 5., 6.}, toDoubleArray(pyArr), 0.0001);
IO.println(toStr(pyArr)); // Output: [4. 5. 6.]
}Gym Class: Building Gymnasium Environments in Java
To manage the environments, the Gym class is provided, making reference to the gymnasium as gym common import standard, and the Env class, the actual result of invoking the make() method, inspired by the original Python method:
@Slf4j
public class Gym {
@SneakyThrows
public static IEnv make(String name,
NDManager ndManager) {
// omitted code
}
public static final class EnvBuilder {
// omitted code
String generatePyEnvScript() {
var makeCall = generateMakeCall();
var importLibsPy = generateImportLibsPy();
if (wrappers.isEmpty()) {
return """
%s
env_%s = %s
""".formatted(importLibsPy, varEnvCode, makeCall);
}
var importPy = generateImportFromPy();
var wrappedEnvPy = generateWrappedEnvPy();
return """
%s
%s
env_%s = %s
%s
""".formatted(importLibsPy, importPy, varEnvCode, makeCall, wrappedEnvPy);
}
public EnvBuilder add(@NonNull IWrapper wrapper) {
// omitted code
}
public Env build() {
return new Env(varEnvCode, envName, generatePyEnvScript(), ndManager);
}
// other methods omitted
}
public static final class PyMap {
// omitted code
public String toPyDict() {
return params.entrySet()
.stream()
.map(entry -> "'" + entry.getKey() + "': " + entry.getValue())
.collect(Collectors.joining(", ", "{", "}"));
}
// omitted code
}
}The EnvBuilder is responsible for customizing the gymnasium environment, such as using ALE and other libraries. Wrappers can be applied to modify the environment's step and observation space, such as skipping frames, converting to grayscale, resizing and concatenating (also known as stacking) the states: MaxAndSkipObservation, GrayscaleObservation, ResizeObservation and FrameStackObservation20, which are the classes passed to the add method from EnvBuilder.
The generatePyEnvScript() method from EnvBuilder is the bridge to build the Python code, returning the configuration as a String. PyMap is a parameter configuration dictionary, equivalent to a HashMap in Java. Below are some examples of usage presented in the unit tests using JUnit 6:
// omitted code
class GymTest {
@Test
void shouldTestGeneratedWrappers() {
var envId = "CarRacing-v3";
var script = Gym.builder()
.envName(envId)
.importLib("ale_py")
.params(Gym.builderMap()
.put("domain_randomize", true)
.put("continuous", true))
.add(new DelayObservation(1),
new GrayscaleObservation(false),
new NormalizeObservation(),
new MaxAndSkipObservation(4),
new FrameStackObservation(4),
new ReshapeObservation(new int[] {1, 84, 84}),
new ResizeObservation(new int[] {50, 50, 1}))
.generatePyEnvScript();
// omitted code
}
@Nested
@DisplayName("PyMap Tests")
class PyMapTest {
// omitted code
@Test
void shouldConvertToPyDictFormat() {
var pyMap = Gym.builderMap()
.put("seed", 42)
.put("render_fps", 60);
assertEquals("{'seed': 42, 'render_fps': 60}", pyMap.toPyDict());
}
// omitted code
}
}Possible output of generatePyEnvScript():
import gymnasium as gym, ale_py
from gymnasium.wrappers import DelayObservation, GrayscaleObservation, NormalizeObservation, MaxAndSkipObservation, FrameStackObservation, ReshapeObservation, ResizeObservation
env_2931ac287e0f4fdfae8a4ed7b75347dc = gym.make('CarRacing-v3', render_mode='rgb_array', domain_randomize=True, continuous=True)
env_2931ac287e0f4fdfae8a4ed7b75347dc = DelayObservation(env_2931ac287e0f4fdfae8a4ed7b75347dc, delay=1)
# omitted codeEnv Class: Managing Gymnasium in Java
The render_mode='rgb_array' is a default value, as the main purpose is to display the environment visually when env.render() is called.
Below the main methods of the IEnv interface, implemented by the Env class:
public interface IEnv extends AutoCloseable {
boolean closed();
boolean scalarObservation();
ActionSpaceType actionSpaceType();
String actionSpaceStr();
String observationSpaceStr();
ActionSpaceType.ActionResult actionSpaceSample();
Pair<NDArray, Map<Object, Object>> reset();
EnvStepResult step(ActionSpaceType.ActionResult action);
EnvStepResult step(ActionSpaceType.ActionResult action, NDManager manager);
BufferedImage render();
NDManager manager();
@Override
void close();
}The Env constructor contains the native PyObject wrapper, used to call methods in Python. NDManager is used in DJL to create the architecture, tensors and numerous Deep Learning utilities. manager.newSubManager() is called to ensure that when the environment is closed on the Java side, the objects allocated inside DJL are deallocated. ActionSpaceType is a type-safe wrapper of Python's ActionSpace, providing an encapsulated execution of the main environments. varEnvCode is a mechanism to avoid overwriting Python's global variables (e.g.: env_cf59c3da7e24499fa9f1d6860a534cd7), as the Python interpreter uses the globals21 for each process. Another reason is that if execIsolated were used, the same variable would need to be recreated every time, requiring a scoped design that would differ significantly from the original API.
public final class Env implements IEnv {
// omitted code
private final NDManager manager;
@Getter
private final String varEnvCode;
@Getter
private final String envName;
private final PyObject pyEnv;
private final PyObject pyActionSpace;
private final PyObject pyObservationSpace;
private final PyObject pyRender;
private final PyObject pyStep;
private final PyObject pyReset;
private final ActionSpaceType actionSpaceType;
// omitted code
Env(@NonNull String varEnvCode,
@NonNull String envName,
@NonNull String generatedScript,
@NonNull NDManager manager) {
initPython();
this.varEnvCode = varEnvCode;
this.envName = envName;
this.manager = manager.newSubManager();
exec(generatedScript);
this.pyEnv = eval("env_" + varEnvCode);
this.pyActionSpace = attr(pyEnv, "action_space");
this.actionSpaceType = detectActionSpaceType(pyActionSpace);
this.pyObservationSpace = attr(pyEnv, "observation_space");
this.pyRender = attr(pyEnv, "render");
this.pyStep = attr(pyEnv, "step");
this.pyReset = attr(pyEnv, "reset");
}
// omitted code
}The methods eval and attr are from the PythonRuntime class. eval evaluates and executes Python's interpreter, returning a PyObject, an object that contains a pointer to the native Python object, without passing through any gateway. attr follows the same idea, but for accessing attributes and methods, since in Python both are objects and can be retrieved by name from any PyObject.
// PythonRuntime.class
// omitted code
public static PyObject attr(PyObject obj, String attr) {
var result = PyObject_GetAttrString(obj, attr);
if (isPyNull(result)) {
PyErr_Print();
throw new IllegalArgumentException("Attribute not found: " + attr);
}
return result;
}
// omitted codeAn equivalent example in Python, for the Env’s constructor, would be:
pyActionSpace = env_cf59c3da7e24499fa9f1d6860a534cd7.action_space
pyRender = env_cf59c3da7e24499fa9f1d6860a534cd7.render
# omitted codeEnv.render(): Visualizing Gymnasium in Java
The Env.render() method is used to get the visual state of the environment, enabling the possibility to debug and verify each step taken as the state changes, from s0 to s1, s1 to s2 and so on. The PythonRuntime.callFunction() executes the render method from the Python runtime’s env variable. The fillFromNumpy() method transfers bytes to the native ByteBuffer, which as shown above, is created only once since it holds an image.
// Env.java
public BufferedImage render() {
try (var ndarray = callFunction(pyRender)) {
if (renderMetadata == null) {
renderMetadata = EnvRenderMetadata.fromNumpy(ndarray);
imageBuffer = ByteBuffer
.allocateDirect(renderMetadata.size())
.order(ByteOrder.nativeOrder());
}
fillFromNumpy(ndarray, imageBuffer);
return ImageFromByteBuffer.byteBufferToImage(
imageBuffer,
renderMetadata.width(),
renderMetadata.height(),
renderMetadata.channels() == 4
);
}
}
EnvRenderMetadata.fromNumpy() handles the conversion of the NumPy array to EnvRenderMetadata, applying the appropriate treatment to produce the correct type and shape for DJL's NDArray.
public class EnvRenderMetadata extends EnvStateMetadata {
// omitted code
static EnvRenderMetadata fromNumpy(PyObject arr) {
var base = EnvStateMetadata.fromNumpy(arr);
return new EnvRenderMetadata(
base.shape,
base.dtype,
base.djlShape,
base.djlType,
base.size
);
}
// omitted code
}Env.reset(): Starting a Gymnasium Episode
Returning to the Env class, reset() must be called before invoking render() and step(), and is required after each finished episode22. The code handles scalar and array observations, as each environment has its own rules, covering most cases.
// Env.java
@Override
public Pair<NDArray, Map<Object, Object>> reset() {
this.stateMetadata = null;
this.stateBuffer = null;
try (var result = callFunction(pyReset)) {
var pyState = getItem(result, 0);
var infoMap = getItemMap(result, 1);
if (!hasAttr(pyState, "shape")) {
this.scalarObservation = true;
long observationValue = toLong(pyState);
var state = manager.create(observationValue);
log.debug("Discrete observation: {}", observationValue);
return new Pair<>(state, infoMap);
}
this.scalarObservation = false;
this.stateMetadata = EnvStateMetadata.fromNumpy(pyState);
this.stateBuffer = onHeapBufferNumpy(stateMetadata.size());
fillFromNumpy(pyState, stateBuffer);
var state = manager.create(
stateBuffer,
stateMetadata.djlShape,
stateMetadata.djlType
);
return new Pair<>(state, infoMap);
}
}Env.step(): Executing Actions in Gymnasium
The Env.step() is responsible for advancing the state (e.g.: s1 to s2) and executing the action in the environment. Note that each value must be retrieved separately from the result using the getItem* methods, as the result is a Python tuple:
// Env.java
@Override
public EnvStepResult step(ActionResult action, NDManager manager) {
try (var result = callFunction(pyStep, action.pyObj)) {
NDArray state;
if (scalarObservation) {
var pyState = getItem(result, 0);
long observationValue = toLong(pyState);
state = manager.create(observationValue);
log.debug("Discrete observation after step: {}", observationValue);
} else {
if (stateBuffer == null) {
throw new IllegalStateException("You should call reset() first!");
}
fillFromNumpy(getItem(result, 0), stateBuffer);
state = manager.create(
stateBuffer,
stateMetadata.djlShape,
stateMetadata.djlType
);
}
double reward = getItemDouble(result, 1);
boolean terminated = getItemBool(result, 2);
boolean truncated = getItemBool(result, 3);
var infoMap = getItemMap(result, 4);
return new EnvStepResult(reward, terminated, truncated, infoMap)
.state(state);
}
}The following unit tests demonstrate that Env can handle multiple types of environments:
// GymActionSpaceTest.java
@Test
@DisplayName("MountainCar-v0 should have Discrete(3) action space")
void testMountainCarActionSpace() {
try (var env = Gym.make("MountainCar-v0", ndManager)) {
// omitted code
try (var action = env.actionSpaceSample()) {
assertEquals(DISCRETE, action.spaceType());
var result = env.step(action);
assertNotNull(result);
assertNotNull(result.state());
assertFalse(result.state().isReleased());
// omitted code
}
}
}
@Test
@DisplayName("LunarLanderContinuous-v3 should have Box(2,) action space")
void testLunarLanderContinuousActionSpace() {
try (var env = Gym.make("LunarLanderContinuous-v3", ndManager)) {
// omitted code
try (var action = env.actionSpaceSample()) {
assertEquals(BOX, action.spaceType());
var result = env.step(action);
assertNotNull(result);
assertNotNull(result.state());
assertFalse(result.state().isReleased());
// omitted code
}
// omitted code
}
}
@Test
@DisplayName("FrozenLake-v1 should have Discrete(4) action space")
void testFrozenLakeActionSpace() {
try (var env = Gym.make("FrozenLake-v1", ndManager)) {
// omitted code
try (var action = env.actionSpaceSample()) {
assertEquals(DISCRETE, action.spaceType());
var result = env.step(action);
assertNotNull(result);
assertNotNull(result.state());
assertFalse(result.state().isReleased());
// omitted code
}
// omitted code
}
}With the Gymnasium integration established and validated, the Rainbow DQN algorithm will be implemented in the next article, using these simple environments as a stepping stone before training the agent in the CARLA environment. The complete code is available at: java-rl-dqn-to-rainbow.
DQN is Deep Q-Network.
ZeroMQ is excellent, but you have to consider the trade-off between each scenario.
The FFM API tends to be faster because it doesn’t have the overhead of JNI, usually not relevant, but depends on your needs.
Today you can use AI to generate this C-API code.
The gym library is platform independent, unlike CPython which is distributed as OS-dependent binaries.
DJL, acronym of Deep Java Library, a framework used in Deep Learning, with PyTorch, TensorFlow and JAX bindings, found at: link.
Further evidence can be found at the links from official documentation and real-python site.
An episode is the period from when the environment starts until a terminated or truncated state is returned. After that, reset() must be called to restart from the beginning.