Bezgelor

Why Elixir & OTP

Bezgelor's server is written in Elixir. Most game servers, including NexusForever, the C# server Bezgelor grew out of, are written in languages like C#, C++ or Java, so this choice surprises people. This page explains it for readers who have never seen Elixir: the handful of ideas the language is built on, why they fit an MMO server so well, where each one shows up in Bezgelor, and what we give up.

A game server is many small conversations at once

Think of what a server for an online game does all day. Hundreds of players are each connected, each sending a steady trickle of small messages: I moved, I cast this, I picked that up. Dozens of zones are each running their own little world, with creatures thinking and spells landing. Most of the time each of these is waiting for something. The hard parts are doing all of it at once without one slow thing holding up the rest, and keeping one broken thing from taking down everything else.

That is close to the problem Erlang was made for. Ericsson built it in the late 1980s to run telephone exchanges: huge numbers of independent calls at once, and a system that must keep running while parts of it fail. Erlang runs on a virtual machine called the BEAM, and comes with a set of libraries and conventions for building such systems called OTP. Elixir is a newer language, from 2012, that runs on the same virtual machine and uses OTP unchanged, with a friendlier syntax and better tools. Some of the best-known chat services have been built on the same foundation.

Processes: one for each thing that happens at once

The central idea is the process. A BEAM process is not an operating-system process or thread: it is much smaller, a few kilobytes when it starts, and the virtual machine can run millions of them. Each has its own memory, which no other process can touch, and the BEAM shares the computer's cores among them, switching between processes on its own so that no single one can hog a core.

So instead of a few threads juggling everything, you give each independent activity its own process. In Bezgelor:

  • every connected player has a connection process (BezgelorProtocol.Connection) for as long as they are connected;
  • every running zone has a zone instance process (BezgelorWorld.Zone.Instance), one per open-world zone and one per copy of a dungeon;
  • every character online has a progress tracker process for achievements, titles and reputation;
  • every trade between two players, every group, the mail service, the chat channels and the event buffer are processes too.

In our 50-player combat soak test, the whole server ran under 300 processes (soak test report). A busy zone on one core doesn't slow a quiet zone on another, and the virtual machine keeps both responsive.

Message passing: no shared memory, no locks

Processes don't share memory, so they cooperate by sending each other messages. Each process has a mailbox; messages wait there in order, and the process takes them one at a time.

Processes and their mailboxes Three connection processes, for players A, B and C, each send messages to one zone instance process. The messages wait in the zone's mailbox, here move, cast, move and leave, and the zone takes them one at a time and updates its own state: players, creatures and the spatial grid. No other process can touch that state directly. Connection player A message Connection player B Connection player C Zone instance (one process) mailbox move cast move leave one at a time its own state players, creatures, spatial grid

This is why a zone in Bezgelor needs no locks. A zone instance owns its players, creatures and spatial grid outright. When a player moves, their connection process sends the zone a message (Zone.Instance.player_moved/4); when they cast, another (Zone.Instance.cast_spell/3). The zone handles one message completely before the next, so a spell and a movement can never trample each other's changes, and there is no class of "two threads changed the same creature" bugs to hunt.

Most of these processes are written as a GenServer, OTP's ready-made shape for a process that holds some state and answers messages. A message can be a call, which waits for an answer (entering a zone does this, with a ten-second limit), or a cast, which doesn't (moving and casting do this, so a player's connection never waits for a busy zone).

The discipline this asks for: a process must never call itself, and two processes must not wait on each other in a circle, or they deadlock until a timeout. Bezgelor's rule is that code running inside a zone never makes a call to its own zone or to a process that may be waiting on it.

Supervision and "let it crash"

Every process in Bezgelor has a parent, a supervisor, whose only job is to watch its children and restart one when it fails. Supervisors have supervisors, up to each app's top supervisor: the supervision tree. Into the world shows the world server's.

That makes a different style of error handling possible, known as "let it crash". Instead of guarding every line against every possible failure, code handles the errors it expects (a player asks for something they aren't allowed) and lets truly unexpected ones crash the process. The supervisor then starts it again from a known-good state. A crash is contained to the one process: a bug in the mail service doesn't touch combat, and a bug triggered by one player's connection ends that connection, not the server.

A crash stays where it happens Two worlds, each with its own WorldSupervisor. World A's supervisor has three zone instances; instance 2 crashes and the supervisor starts it again, while instances 1 and 3 keep running. World B and its instance are not affected. Only if an instance keeps crashing does world A's supervisor give up, and even then world B carries on. World A World B WorldSupervisor world A Instance 1 running Instance 2 crashed → restarted Instance 3 running WorldSupervisor world B Instance 1 not affected If one keeps crashing, only world A's supervisor gives up.

The supervision tree is drawn to match how things can fail. Each world has its own supervisor (BezgelorWorld.Zone.WorldSupervisor) for its zone instances. A zone instance that crashes is restarted; one that crashes more than three times in five seconds makes its world's supervisor give up, and that world stops, but no other world is affected, and the next player to enter starts it again. The zone also watches (monitors) each player's connection process, so when a client vanishes without logging out, the zone hears about it and cleans up.

To be honest about the limits: a restarted process starts from scratch. A zone that restarts comes back with its creatures from the game data, but without the players who were in it, and today those players have to re-enter the zone. Anything that must survive is therefore written to the database as it changes: quests, experience and currency are, and so is a player's position, every few seconds.

Pattern matching, especially on binaries

Elixir lets you describe the shape of the data you expect and pull it apart in one step: pattern matching. For a network server the most useful form is matching on raw bytes. This is the function in Bezgelor that reads the header of every packet the client sends (apps/bezgelor_protocol/lib/bezgelor_protocol/packet.ex):

def parse_header(<<size::little-32, opcode::little-16>>) do
  {:ok, size, opcode}
end

def parse_header(_), do: {:error, :incomplete}

The first version says: if what we were given is a 32-bit little-endian number followed by a 16-bit one, call them size and opcode and return them. The second says: for anything else, report that the header isn't complete. There is no shifting, masking or index arithmetic in sight: the layout of the bytes is written down as a picture of the bytes, and the language checks that the data matches it.

The same works for single bits. WildStar packs its packet fields bit by bit, with no padding between them, and the BEAM's binaries can hold and match any number of bits, not just whole bytes. BezgelorProtocol.PacketReader and PacketWriter build on that. The wire shows a whole packet being read.

Pattern matching is used everywhere else too. A function can have several versions, each for a different shape of input, and the right one is picked when it is called. Handlers return {:ok, state} or {:reply, opcode, payload, state}, and the connection matches on which it got.

ETS: shared tables for data everyone reads

Not everything should be a message. The game's static data (every creature, spell, item and quest, from the client's own tables) is read all the time by every zone and connection and changes only when the server starts. For that, the BEAM has ETS: in-memory tables that any process can read directly.

BezgelorData.Store loads about two hundred of them at start-up, set up for many readers at once. After that, a zone looking up a creature's template or a spell's effects reads the table itself, without asking any process and without waiting. A handful of values read even more often and changed almost never, such as the table of packet handlers and the GUID counter, live in :persistent_term, which is faster still for reading.

Changing code while it runs

The BEAM can load a new version of a module into a running system: processes pick up the new code the next time they call it, and keep their state. This is hot code loading, and telephone exchanges were built around it. For a game server it means something players can feel: a fix can reach the running server while everyone stays logged in and playing.

That is how Bezgelor delivers changes. We do not wait for a quiet hour or a maintenance window for most fixes and improvements; we hot-deploy them as soon as they are tested:

  1. Classify the change. A small tool (tools/deploy/hot_check.py) reads the difference between the build the server is running and the one we want to deliver, and answers either hot (with the list of modules that change) or restart (with the reasons). Pure logic is hot: spell and combat rules, quest rules, packet handlers, the website's pages. A restart is needed when the change touches something a running node cannot take in by loading modules: configuration, dependencies, database migrations, the static game data (loaded into memory once, at start), the supervision tree, or the shape of the state that running processes hold.
  2. Load it. The deploy script compiles the new build on the server and loads every changed module in one step over the running node (BezgelorCore.HotLoad, BezgelorWorld.HotDeploy). All the modules are checked first, so a bad file stops the load before anything changes. A zone or a player's connection moves to the new code at its next message, with its players, creatures and state exactly as they were.
  3. Check it. The script waits until the server's health report shows the new build. If a change turns out to need a restart after all, the script says so and changes nothing, or falls back to the ordinary restart deploy.

Our testing server is delivered to this way, regularly. For example, a fix to how melee cones reach a target went out as a hot deploy of one module (BezgelorWorld.Spell.Caster), and a fix to a quest item that never credited its objective went out the same way, with nobody disconnected. The tooling also has a mode for release-based production servers (deploy/deploy.sh hot, not yet in live use), with the same rule: logic-only changes load any time, and anything that needs a restart waits for a maintenance window.

To be honest about the limits: hot loading is not magic. In a count over 300 recent commits (October 2026), 128 changed the running server; 56 of those could be hot-loaded and 72 needed a restart, and 42 of the 72 only because they changed static game data. The classifier is a check over the source, not a proof, so changes it flags for review are read by a person before they are loaded. A process that is in the middle of running old code can delay a load until it returns. We don't use OTP's full release upgrades (versioned upgrade scripts for state changes); a change to the shape of process state still means a restart.

Scripts have always had a quicker path: the admin pages' "reload scripts" button (BezgelorWorld.Script.Registry.reload/0) loads the new version of every changed boss or zone script into the running server and tells every zone. During development, recompile in the server's console does the same for any module.

One project, many apps

An Elixir project can be an umbrella of several apps, each with its own code, tests and dependencies, built and started together. Bezgelor has a dozen (the architecture overview lists them). The boundaries are real: the protocol app can't call the world app's code, because it doesn't depend on it. The world app registers its packet handlers with the protocol app when it starts (BezgelorWorld.HandlerRegistration) instead, which keeps the dependency pointing one way.

The rest of the stack, in the same language

The database layer is Ecto and the website is Phoenix with LiveView, all Elixir. Because the website runs in the same virtual machine as the game, an admin page that kicks a player or grants an item calls the world server's functions directly (BezgelorWorld.Portal); there is no internal API to build, secure and keep in step. LiveView pages update in the browser as the game changes, over the same kind of lightweight processes.

What we give up

No choice is free. These are the costs we know about:

  • Raw number crunching is slower. The BEAM is built for many concurrent, mostly-waiting activities, not for heavy arithmetic. Code that loops over thousands of creatures each tick has to be written with care. A big continent's zone tick once took hundreds of milliseconds; letting creatures far from every player sleep brought it down to about ten (zone tick report).
  • A process is sequential. One zone instance runs on one core at a time. That is what makes it lock-free, but it means a single very crowded zone is limited by one core; the soak test measures how far that goes.
  • Messages and table reads copy data. Sending a message or reading from ETS copies the data into the receiving process. It keeps processes independent, but large values need care; the zone encodes each packet once and sends the same bytes to everyone, rather than sending the data to every connection to encode.
  • Fewer people know it. Elixir is less familiar than C#, and its syntax looks unusual at first. We try to make up for it with plain moduledocs, these pages, and a glossary.
  • The compiler catches less. Elixir is dynamically typed: a wrong kind of value shows up as a test failure or a crash rather than a compile error. Function specs and tests carry much of that load; the client oracle and the headless-client tests check our packets against the real client.

We think the trade is clearly worth it for this kind of server. Processes, messages and supervisors map directly onto players, zones and the things that go wrong between them, and the result is a server that is easier to reason about and harder to knock over.

Coming from NexusForever

If you know NexusForever's C# code, most of it maps across. Its managers (one object per system, called from many threads) become processes that own their state and answer messages. Its game entities stay data, owned by the zone instance they are in. Its packet models become Elixir modules with a read and a write function, written with the same bit-packed reader and writer. Many Bezgelor modules name the NexusForever class they follow, and NexusForever's years of protocol work are the foundation all of this stands on.

See also: the architecture walkthrough and the glossary.