I like to play video games. One reason I got into reinforcement learning was the idea you could teach a machine to learn in a virtual world, then port that knowledge to the real one. Games and simulations let you train a machine on how to behave before you risk anything physical.
We’re starting simple. Q-Learning uses a table to store what the agent learns. No CNNs, no Transformers. Just a straightforward algorithm that works.
Thanks for reading! Subscribe for free to receive new posts and support my work.
Here’s what we’re building: an agent that solves Frozen-Lake. You’ll see it figure out the safest path across the ice without falling through.
Don’t you worry, i’ll explain step by step how this algorithm works.
First, you’ll need Python 3.9 or later. I recommend setting up a virtual environment with venv or virtual env. That keeps your libraries isolated and prevents version conflicts with your system Python.
import time
import numpy as np
import gymnasium as gym
import matplotlib.pyplot as plt
from gymnasium import logger as gymlogger
from gymnasium.envs.toy_text.frozen_lake import generate_random_map
gymlogger.set_level(40) # show errors only
%matplotlib inline
To render the generated environment, we use the code below:
from IPython.display import clear_output
def show_environment(array):
plt.axis("off")
plt.imshow(array)
plt.show()
time.sleep(0.05)
Testing Random Actions to Get a Feel for the Environment
Action Space
Actions are formatted as (1,) in the range {0, 3}, telling the player which direction to move.
0: Move left
1: Move down
2: Move right
3: Move up
Observation Space
The observation is a single value representing the player’s current position as current_row * nrows + current_column (both row and column start at 0).
For example, the goal position on a 4x4 map calculates to 3 * 4 + 3 = 15. The number of possible observations depends on map size.
The observation is an int.
The “agent” moves randomly. The sample() method picks actions at random, so there’s no strategy yet, just exploration.
env_name = "FrozenLake-v1"
env = gym.make(env_name, render_mode="rgb_array",
desc=generate_random_map(size=4),
is_slippery=False)
env.reset()
for step in range(30):
environment = env.render()
show_environment(environment)
action = env.action_space.sample()
state, reward, term, trunc, info = env.step(action)
time.sleep(0.2)
clear_output(wait=True)
if term or trunc:
env.reset()
break
env.close()
Manual Game Execution
Note: If this doesn’t work, try running the notebook with python -m notebook or use Anaconda instead.
def walk():
"""
Function to get keyboard input for movement.
"""
key = input("Select wasd (walk) or q (quit):")
if 'a' == key:
action = 0
elif 's' == key:
action = 1
elif 'd' == key:
action = 2
elif 'w' == key:
action = 3
elif 'q' == key:
print("Quitting game!")
raise StopIteration()
return action
With the code below, you can use WASD keys to move the character through the environment:
env.reset()
try:
for step in range(30):
environment = env.render()
show_environment(environment)
action = walk()
_, _, term, trunc, _ = env.step(action)
clear_output(wait=True)
except StopIteration as stop:
pass
env.close()
Implementing the Q-Table for Q-Learning
In Gym, state and observation get used interchangeably. Technically, state is the full picture of the environment while observation is partial information. We’re using state in the code to match the math notation in Q-Learning formulas.
Exploitation means when you find a high reward, you milk it for all it’s worth. Exploration means trying new possibilities, like wandering around to discover new areas. Example:
def epsilon_greedy_action_selection(env, epsilon, q_table, discrete_state):
random_number = np.random.random()
if random_number > epsilon: # This is Exploitation
state_row = q_table[discrete_state,:]
action = np.argmax(state_row)
else: # This is Exploration (selects a random action)
action = env.action_space.sample()
return action
Q-Learning Equation:
\(Q(s, a) = Q(s, a) + \alpha[R(s, a) + \gamma maxQ(s', a') - Q(s, a)]\)
Q(s,a) is the q_table we built earlier. The s is the state, a is the action. In our code, that’s old_q_value.
α controls how much we update each q-value. Higher values speed up learning but make training unstable. Lower values are safer but slower. We’re using ALPHA = 0.8 because it learns fast without overshooting on a simple 4x4 grid.
R(s, a) is the reward we just earned by taking that action. That’s the reward variable from env.step().
Now γ gets interesting. It’s the discount factor, and it stops future rewards from overwhelming the current state. Set γ=0.99 and your agent dreams about the future. Set γ=0.01 and it grabs whatever’s in front of it right now. We’re using 0.95, which means “I care about what’s ahead, but I’ll take a good reward today.”, If γ equals 1, them it would make every path look equally valuable, and the algorithm would get stuck.
maxQ(s′,a′) is the best possible reward from the next state. It’s like playing hot-cold: you update your current position based on whether the next step gets you warmer. In the code, that’s next_optimal_q_value, which we calculate with np.max(q_table[new_state,:]).
The difference maxQ(s′,a′) − Q(s,a) is your prediction error. You thought the current state was worth X, but now you’ve learned it’s actually worth Y. That gap drives the update, the same way squared error drives linear regression.
The equation above is one option. A simpler approach is reducing epsilon by a fixed value like 0.00001 or 0.00005, but you’ll need to tune that based on your EPOCHS count. The exponential version handles epochs automatically.
Here’s what reduce_epsilon does: as the agent learns more about the environment, it doesn’t need to explore as much. We gradually shift from exploration to exploitation.
Note: Remember the code with random_number > epsilon means exploitation, otherwise exploration.
Agent training phases 🤖:
🤖 → I don’t know anything about this environment, so I’ll explore heavily (epsilon = 0.9 = 90%).
🤖 → I’m understanding things better now, so less exploring but I’ll still check new possibilities (epsilon = 0.5 = 50%).
🤖 → I know enough. Every once in a while, I’ll try something new (epsilon = 0.1 = 10%).
min_epsilon keeps epsilon from hitting 0.
np.exp(-DECAY_RATE*epoch) creates a smooth curve that gradually reduces epsilon over time.
The t+1 represents the next position relative to the current position t.
rewards = []
log_interval = 1000
fig = plt.figure()
ax = fig.add_subplot(111)
plt.ion()
fig.canvas.draw()
epoch_plot_tracker = []
total_reward_plot_tracker = []
# Agent plays the game
for epoch in range(EPOCHS):
state, info = env.reset()
done = False
total_rewards = 0
while not done:
# Action
action = epsilon_greedy_action_selection(env, epsilon, q_table, state)
new_state, reward, term, trunc, info = env.step(action)
# Old (current) Q-Value
old_q_value = q_table[state, action]
# Get the optimal next Q-Value -> max Q(s’, a’)
next_optimal_q_value = np.max(q_table[new_state,:])
# Calculate the "balanced" Q-Value we can use to update the Q-Value
next_q_value = compute_next_q_value(old_q_value, reward, next_optimal_q_value)
# Update the table, remember: Q(s, a) = Q(s, a) + alpha(R(s, a) + max Q(s’, a’) - Q(s, a))
q_table[state, action] = next_q_value
# Sum rewards to plot on the graph
total_rewards += reward
# Now we move to the next state, like walking on the map as the character navigates the scenario
state = new_state
# If the character fell through the ice or reached the treasure, done becomes True.
done = term or trunc
epsilon = reduce_epsilon(epoch)
# The agent finished this game round, meaning every time done is True, it hit "Game Over" or "Beat the level"
rewards.append(total_rewards)
# Used for each game to show the reward increase over time, the y-axis on the graph
total_reward_plot_tracker.append(np.sum(rewards))
# The epoch that had this reward, the x-axis on the graph
epoch_plot_tracker.append(epoch)
if epoch % log_interval == 0:
print("Rewards (Wins):", int(np.sum(rewards)))
ax.clear()
ax.plot(epoch_plot_tracker, total_reward_plot_tracker)
fig.canvas.draw()
env.close()
Output:
The q_table is filled in below. Heuristically, the optimal function (or close to it) has been recorded in the table:
Now we do the inference on the trained environment:
state, info = env.reset()
for steps in range(50):
environment = env.render()
show_environment(environment)
# Select the best option since we’re not training the algorithm
action = np.argmax(q_table[state,:])
state, reward, term, trunc, info = env.step(action)
time.sleep(0.2)
clear_output(wait=True)
if term or trunc:
break
env.close()
In the next article, we’ll solve this exercise using a CNN so we can tackle problems that use images as input. You’ll understand the limitations of classic Q-Learning with the Q-Table.
Flashcards
If you use Anki, I created some flashcards for this content:
Card 1
Front:
Reinforcement Learning [Q-Learning] - Given player X with possible rewards WIN=+10, LOSE=-10, and TIE=0, what’s the result of this function for the X player?
Back:
Explanation:
Player X moves to position 9 and creates the next state below. O sees this as -10 (a loss). X sees it as +10 (a win).
Reinforcement Learning [Q-Learning] - Given player O with possible rewards WIN=+10, LOSE=-10, and TIE=0, what’s the result of this function for the O player?
Back:
Explanation:
Player O moves to position 2 and creates the next state below. X sees this as -10 (a loss). O sees it as +10 (a win).
Reinforcement Learning [Q-Learning] - Given player O with possible rewards WIN=+10, LOSE=-10, and TIE=0, what’s the result of this function for the O player?
Back:
Explanation:
Player O moves to position 7 and creates the next state below. Both X and O see this as 0 (a tie).
Excellent content. Keep writing articles like this.