Bezgelor

Behind the scenes: data, database, events and the website

Architecture walkthrough, part 5 of 5

Walkthrough contents Part 5 of 5
  1. Overview
  2. Logging in: from the launcher to character select
  3. The wire: from bytes to packets to actions
  4. Into the world: zones, entities and movement
  5. Fighting: spells, creatures, loot and quests
  6. Behind the scenes: data, database, events and the website

The last part of the walkthrough is about what the game stands on: the game's own data, held in memory; the database, which remembers players between sessions; the event system, which records what happens; and the website, where players manage their accounts and the admins look after the server.

The game data, in memory

WildStar's client ships its game data as tables: every creature, spell, item, zone, quest and much more, in files ending in .tbl. Bezgelor reads its game data from the same tables, so a creature's name, level range and abilities on our server are the client's own.

From the client's files to memory The client's .tbl files in its game folder are read by the tbl_extractor tools, Python run by developers, into JSON files with provenance in apps/bezgelor_data/priv/data, which are committed to the repository. When the server starts, BezgelorData.Store loads everything into ETS tables, about two hundred tables plus indexes. Zone instances, connections and handlers then look data up by id, for example Store.get(:creatures, id), and many processes can read at once. .tbl files in the client's game folder tbl_extractor Python tools, run by developers JSON + provenance apps/bezgelor_data/ priv/data BezgelorData.Store loads everything at start-up ETS tables 203 tables + 28 indexes, many readers at once Zones, connections, handlers Store.get(:creatures, id) lookups Once, by developers: extract and commit. Every start: load. Then: read, read, read.
  • Extraction, once. Python tools in tools/tbl_extractor/ read the client's .tbl files (tbl_extractor.py) and write JSON into apps/bezgelor_data/priv/data/ (extract_game_data.py, which knows how to turn about 180 client tables into the files the server uses). The JSON is committed, so nobody needs the client to run the server.
  • Provenance. Not everything comes from the client: spawn positions come from NexusForever's world database, and some values we had to design. Generated records carry a provenance label, reference, derived or guess, and the reason for it (docs/design/provenance.md). The Where we differ page is built from those labels.
  • Loading, at every start. BezgelorData.Store (apps/bezgelor_data/lib/bezgelor_data/store.ex) is a process that, when the server starts, reads the files into ETS tables: about two hundred tables, one per kind of data, plus about thirty index tables such as "spells by base spell" or "quests by zone". The biggest files are cached in a compact binary form (BezgelorData.Compiler) so start-up is quicker.
  • Reading, all the time. Any process can look data up directly, without asking the Store: BezgelorData.Store.get(:creatures, id) returns {:ok, creature} or :error. The tables are set up for many readers at once (read_concurrency), which suits data that is written once at start-up and read millions of times after. Modules such as BezgelorData.Housing, BezgelorData.Paths and BezgelorData.Loot wrap the Store with lookups for one area of the game.

The database

Everything that changes as people play (accounts, characters, inventories, quests, guilds, mail, the marketplace, housing) is stored in PostgreSQL, through Ecto, Elixir's database library.

Saving, events and the website Top row: connections, zones and the website call context modules such as BezgelorDb.Characters, Inventory and Quests, which use the Ecto repository BezgelorDb.Repo to read and write PostgreSQL. Middle row: any part of the server calls Bezgelor.Events.emit; events are batched by a buffer, 500 events or one second at a time, and written by Sink.Postgres to the game_events table. Each event is also a telemetry event, feeding metrics, alerts and the live dashboard. Bottom right: the website and admin pages read the database (activity pages, the event stream, the audit log) and the telemetry metrics. Connections, zones and the website Context modules BezgelorDb.Characters, Inventory, Quests... Ecto Repo BezgelorDb.Repo PostgreSQL port 5433 Events.emit from any part Buffer batches: 500 events or one second Sink.Postgres game_events :telemetry metrics, alerts, live dashboard Website and admin pages activity, event stream, audit log, player and server tools reads
  • BezgelorDb.Repo (apps/bezgelor_db/lib/bezgelor_db/repo.ex) is the connection to PostgreSQL (port 5433 by default).
  • Contexts. The rest of the server never writes database queries itself. It calls a context module, one per area of the game, with functions named for what they do: BezgelorDb.Characters.update_position, BezgelorDb.Accounts.consume_game_token, BezgelorDb.Inventory, BezgelorDb.Quests, BezgelorDb.Guilds and about forty more, all in apps/bezgelor_db/lib/bezgelor_db/.
  • Schemas (apps/bezgelor_db/lib/bezgelor_db/schema/, about eighty) describe each table and check changes before they are written.
  • Migrations (apps/bezgelor_db/priv/repo/migrations/, about seventy) create and change the tables, in order, so any copy of the database can be brought up to date with mix ecto.migrate.
  • Two-factor secrets are stored encrypted (BezgelorDb.Vault).

When a world system saves depends on the system: quests, experience and currencies are written as they change, a player's position at most every five seconds and on logout.

Saving player progress

A disk can be slow. On a busy day a single write took up to two and a half seconds while reads took a few milliseconds, and when the zone waited for each write, loot reached the player seven seconds after they picked it up. So loot, experience, credits and quest progress are written behind: the zone tells the client at once, and the database catches up a moment later (BezgelorDb.WriteBehind, apps/bezgelor_db/lib/bezgelor_db/write_behind.ex).

  • A queue per character. The zone puts the write in a queue, an ETS table, and each character has a writer process that works through its queue in order. The queue doesn't belong to the writer, so a writer that crashes is restarted and picks up where it stopped.
  • What the client was told. Until a write lands, the state the client was told is kept next to the queue (the overlay), so a zone that reloads the character's bags sees them as they will be, not as the database has them yet. Anything that reads a character from the database first waits for that character's queue.
  • Batches. A writer takes what is queued, up to 50 writes or a quarter of a second of work, and saves it in one database transaction: one wait for the disk instead of fifty. If one write fails, the batch is undone and its writes are tried again one at a time, so one bad write never holds up the others. A write that still fails after five tries is dropped, logged and shown to the admins.
  • Trades, mail and the rest wait. Anything that moves items or credits between characters or out of the game (trades, mail, the marketplace, vendors, salvage, crafting, the guild bank, housing plugs, quest turn-ins) first waits for the queues of the characters involved, then saves at once and for certain. That is why a crash can roll a player back a little but can never copy an item or credits.

What a crash costs:

What happens What is lost
A clean stop (systemctl stop bezgelor) Nothing: players are logged out and every queued write is saved before the database connection closes
The server program is killed (kill -9, out of memory) What was still queued, normally less than a second of each player's loot, experience and quest progress
The database or the machine crashes Also up to about the last half second of saved progress (the database confirms these writes before they reach the disk, which is what makes them fast); never trades or mail, and nothing is corrupted

The admin metrics page shows how much is queued, how old the oldest write is and how far behind the writers are; docs/architecture.md has the details, and the self-hosting guide has what a server operator needs to know.

The event system

bezgelor_events keeps one record of what happens in the game: logins, kills, quest completions, trades, admin actions and about two hundred other kinds of event. There is one way in, Bezgelor.Events.emit(name, attributes) (apps/bezgelor_events/lib/bezgelor/events.ex), and it never blocks the caller and never fails:

  1. The name must be in the list of known events (Bezgelor.Events.Taxonomy).
  2. Who and where (the character, the connection, the zone) are filled in from the calling process's log metadata, so a zone only passes what is specific to the event.
  3. The event also fires as a telemetry event, which feeds the server's metrics and alerts.
  4. Ordinary events go to Bezgelor.Events.Buffer, which writes them in batches (500 events or one second, whichever comes first) through Bezgelor.Events.Sink.Postgres to the game_events table. If the database falls behind, the buffer drops and counts events past a limit rather than grow without end. High-volume events are summed per minute instead (Rollups), and a detailed debug record is kept only for characters an admin is investigating (Debug).

Chat text and secrets never go into the stream. Players can see and download their own events on the website's Activity page (Bezgelor.Events.PlayerView), and the admins explore all of them on the event stream pages. The design is in docs/plans/2026-10-11-event-stream.md.

Metrics and logs

BezgelorWorld.Telemetry keeps the server's metrics in memory (zone tick times, packets sent, players online), samples them every ten seconds and raises alerts. They show on the admin pages and on the live dashboard (/admin/dashboard, Phoenix LiveDashboard). Log lines start with a tag for the system they come from, such as [Loot], and carry the account, character, connection and zone as metadata, so one player's session can be followed through the logs. docs/observability.md lists the metrics.

The website and the admin pages

bezgelor_portal is the website, on port 4000, written with Phoenix and LiveView. It has three parts (apps/bezgelor_portal/lib/bezgelor_portal_web/router.ex):

  • Public pages: the home page, features, news, download, help, and these Workshop pages, which are Markdown files from docs/community/ turned into HTML when the site is built (BezgelorPortalWeb.CommunityDocs).
  • Account pages for players: their characters, their activity, their settings and two-factor login.
  • Admin pages (live/admin/, about forty LiveViews): accounts, roles, characters, guilds, the economy, the marketplace, mail, chat, loot, instances, housing, zone scripts, metrics, logging, the event stream, bug reports and more. Each needs a permission (BezgelorDb.Authorization, roles made of permissions), every admin page needs two-factor login, and admin actions are written to an audit log, which also feeds the event stream.

Because the website runs in the same node as the game, it talks to the world server by calling functions in BezgelorWorld.Portal (apps/bezgelor_world/lib/bezgelor_world/portal.ex): who is online, kick, broadcast a message, grant an item, reload zone scripts. LiveView pages update in the browser as things change, without the page being reloaded.

bezgelor_api is a much smaller web service on port 4002 (apps/bezgelor_api/lib/bezgelor_api/router.ex): the server's status, its zones and the players online, and health checks (/health) that check the three game listeners, the database, the zone tick, saving player progress and the active alerts.

Tests

Every app has its tests next to its code, in apps/<app>/test/. scripts/test_all.sh runs them all and prints a table of results. Tests that need the database get a private, rolled-back transaction each (BezgelorDb.DataCase), so they can run side by side. The slowest and most thorough are the end-to-end tests in apps/bezgelor_world/test/e2e/, where headless players log in, fight, loot and quest through the real protocol while the client oracle checks every packet they receive. Interesting runs are published as test reports.

Where to look

  • apps/bezgelor_data/lib/bezgelor_data/store.ex and tools/tbl_extractor/: the game data.
  • apps/bezgelor_db/lib/bezgelor_db/: contexts; schema/: tables; priv/repo/migrations/: changes; write_behind.ex and write_behind/: saving player progress.
  • apps/bezgelor_events/lib/bezgelor/events.ex and events/: the event system.
  • apps/bezgelor_world/lib/bezgelor_world/telemetry.ex: metrics and alerts.
  • apps/bezgelor_portal/lib/bezgelor_portal_web/: the website (router.ex, live/admin/).
  • apps/bezgelor_world/lib/bezgelor_world/portal.ex: what the website can ask of the world server.

That is the end of the walkthrough. Why Elixir & OTP explains why the server is built this way, and the glossary collects the words used along the way. Back to the overview.