Finite State Machines for Enemy AI

Step 1: Define Your States

Start by listing every distinct behavior mode your enemy needs. Each state should represent one clear, self-contained activity. For a typical stealth game guard, the states might be:

Idle: The guard stands in place, occasionally looking around. This is the resting state when no patrol route is assigned or the guard has reached a post.

Patrol: The guard follows a predefined waypoint path, walking at a normal pace and periodically pausing at designated look-around points. This is usually the default active state.

Alert: The guard has detected something suspicious but has not confirmed a threat. It moves toward the disturbance location, moves more slowly, looks around more carefully, and might call out a warning. This is the transitional state between peaceful and hostile behavior.

Chase: The guard has confirmed the player's presence and pursues them at full speed. It follows the player's last known position and attempts to maintain line of sight. Other guards may be notified during this state.

Attack: The guard is within engagement range and actively fighting the player. The specific attack behavior depends on the enemy type, ranging from melee strikes to ranged weapon fire to ability usage.

Flee: The guard's health has dropped below a threshold or some other condition has triggered retreat. It moves away from the player, seeks cover or an exit, and may call for reinforcements while fleeing.

Keep the number of states as small as possible. Every state you add increases the number of transitions you need to manage. If a behavior feels like it needs its own state but only differs slightly from an existing state, consider using a parameter within the existing state instead of creating a new one.

Step 2: Map Transitions and Triggers

For each state, define every condition that causes a transition to another state. Be explicit and exhaustive because missing a transition is one of the most common FSM bugs, it causes NPCs to get stuck in a state with no way out.

Common trigger types include perception events (player spotted, sound heard, ally killed), distance thresholds (player within melee range, player escaped beyond chase range), health checks (health below 25%), timers (alert state expires after 30 seconds without confirmation), and external signals (alarm triggered, all-clear given by another NPC).

Draw the full transition diagram on paper or in a tool before writing any code. Each state is a node, each transition is a directed arrow labeled with its trigger condition. This diagram becomes your specification and your primary debugging reference. If an NPC behaves unexpectedly, you can trace through the diagram step by step to find where the actual behavior diverges from the intended behavior.

Pay attention to transition conflicts. What happens if two transitions are valid simultaneously? For example, if the guard is in Chase state and the player enters attack range at the same moment the guard's health drops below the flee threshold, which transition fires? Establish a priority order for conflicting transitions and document it clearly.

Step 3: Implement State Logic

Each state needs three functions: enter, execute, and exit. The enter function runs once when the NPC transitions into the state. It handles setup like starting an animation, setting movement speed, or initializing a timer. The execute function runs every tick (or at a configured update frequency) while the NPC remains in the state. It performs the ongoing behavior like following waypoints, aiming at the player, or scanning for threats. The exit function runs once when the NPC leaves the state, cleaning up anything the enter function set up.

This enter/execute/exit pattern prevents a whole class of bugs. Without explicit enter and exit functions, developers tend to scatter state initialization and cleanup throughout transition code, which leads to inconsistent behavior when the same state is entered from different previous states. With the pattern, each state is fully self-contained, initialization happens in enter regardless of where the transition came from, and cleanup happens in exit regardless of where the transition goes.

Keep execute functions focused. A common mistake is loading too much logic into the execute function, making it handle both the core behavior and the transition checking. Separating the behavior logic (what the NPC does in this state) from the transition logic (when the NPC should leave this state) makes both easier to maintain and test.

Step 4: Build the Transition Table

The transition table is the core data structure of your FSM. It maps a current state plus a trigger condition to a next state. This can be implemented as a two-dimensional lookup (a dictionary keyed by state and event), a switch statement, or a data-driven table loaded from a configuration file.

Data-driven transition tables are strongly recommended for production games. When transitions are defined in code, changing NPC behavior requires recompiling. When transitions are defined in data (JSON, XML, or a custom format), designers can modify and tune enemy behavior without programmer involvement. This is especially valuable during the iteration-heavy later stages of development when gameplay tuning happens rapidly.

A data-driven approach also makes it straightforward to create NPC variants. A basic guard and an elite guard can share the same FSM code but use different transition tables, with the elite guard having tighter detection thresholds, shorter alert-to-chase times, and no flee state.

Step 5: Add Perception and Memory

An FSM is only as good as the information it receives. Connect your state machine to perception systems that provide awareness of the player's position, visibility, sound events, ally states, and environmental conditions.

A basic perception layer for an FSM-based enemy includes a vision cone check (is the player within the field of view and unobstructed by geometry), a hearing radius (did a sound event occur within range), and a communication channel (did another NPC send an alert). Each perception event generates a signal that the FSM evaluates against its transition table.

Add a short-term memory system so the NPC does not instantly forget the player when line of sight is broken. Store the player's last known position and a timestamp. The alert and chase states can use this memory to investigate the last known position, and a timer determines how long the NPC remembers before giving up and returning to patrol. This single addition dramatically improves the believability of enemy behavior because it eliminates the jarring instant-amnesia problem that plagues naive FSM implementations.

Step 6: Test and Tune Transitions

Testing FSM-based AI requires systematically verifying every state and every transition. For each state, confirm that the NPC behaves correctly when entering from every possible previous state. For each transition, confirm that the trigger conditions fire reliably and that no edge case causes the NPC to skip a transition or get stuck.

The most common tuning issues are transition oscillation (the NPC rapidly switches between two states because both transition conditions are met alternately) and transition deadlocks (the NPC reaches a state combination where no outgoing transition is valid). Hysteresis solves oscillation by requiring different thresholds for entering and leaving a state. For example, the chase state activates when the player is within 20 meters but does not deactivate until the player is beyond 30 meters, creating a buffer zone that prevents flickering. Cooldown timers achieve a similar effect by preventing a transition from firing again within a minimum interval.

Build debug visualization that displays the current state, recent transitions, and active perception data directly in the game viewport. This visualization is invaluable during playtesting because it lets you see exactly what the NPC thinks is happening, which is usually the key to understanding why it is behaving unexpectedly.