Bezgelor

The wire: from bytes to packets to actions

Architecture walkthrough, part 2 of 5

Walkthrough contents Part 2 of 5
  1. Overview
  2. Logging in: from the launcher to character select
  3. The wire: from bytes to packets to actions
  4. Into the world: zones, entities and movement
  5. Fighting: spells, creatures, loot and quests
  6. Behind the scenes: data, database, events and the website

The realm and world servers speak WildStar's binary protocol. This page follows one message from the moment its bytes arrive to the moment the server acts on it and answers, and then shows how we check our answers with the real client's own code. All of it is in bezgelor_protocol.

A packet's way in and out Top: one frame on the wire is a 4-byte size, a 2-byte opcode and the payload, the packet's fields packed bit by bit. Middle row, client to server: the connection process buffers the bytes; framing cuts them into frames; the outer ClientEncrypted frame goes to EncryptedHandler, which decrypts it with PacketCrypt; the inner opcode is looked up in the PacketRegistry to find its handler; the handler reads the fields and acts. Bottom row, server to client: the reply is a packet struct written with PacketWriter, encrypted with PacketCrypt, framed with size and opcode, and sent on the socket. One frame on the wire size uint32 · 4 bytes opcode uint16 · 2 bytes payload the packet's fields, packed bit by bit Client → server Connection buffers the bytes Framing cuts out frames Decrypt EncryptedHandler PacketCrypt PacketRegistry opcode → handler Handler reads the fields, acts, replies Server → client Packet struct written with PacketWriter Encrypt PacketCrypt Frame size + opcode Socket back to the client

One process per connection

Both the realm and the world server start a BezgelorProtocol.TcpListener (apps/bezgelor_protocol/lib/bezgelor_protocol/tcp_listener.ex), a thin wrapper around Ranch. For every client that connects, Ranch starts a new BezgelorProtocol.Connection process (connection.ex). A player therefore has one connection process on the realm server, briefly, and then one on the world server for as long as they play.

The connection process owns everything about that one client: the socket, the bytes received but not yet used, the encryption state, and a session_data map with the account, the character, the player's GUID, the zone they are in and so on. No other process touches it. When the client disconnects, the process ends, and on the way out (terminate/2) it takes the player out of their zone and saves their position.

Frames

TCP delivers a stream of bytes, not messages, so the client wraps every message in a frame: a 4-byte size (counting the whole frame), a 2-byte opcode that says which message it is, and the payload. BezgelorProtocol.Framing (framing.ex) cuts complete frames out of the buffer and leaves a half-received frame there until the rest arrives.

Encryption

Almost everything is encrypted with WildStar's own packet cipher, BezgelorCrypto.PacketCrypt (apps/bezgelor_crypto/lib/bezgelor_crypto/packet_crypt.ex). The client sends its messages inside an outer ClientEncrypted frame (opcode 0x0244). Its handler, BezgelorProtocol.Handler.EncryptedHandler, decrypts the payload, reads the inner opcode and passes the inner message on as if it had arrived on its own. The server does the same in reverse, wrapping its messages in ServerRealmEncrypted on the world server and ServerAuthEncrypted on the realm server.

The key starts as one derived from the client build, which both sides know. After ClientHelloRealm, the world server switches to a key made from the session key the realm server handed out (WorldAuthHandler).

Reading a packet: bits, not bytes

Inside the payload, WildStar packs fields bit by bit, with no padding between them: a 5-bit number can be followed directly by a 13-bit one, straddling byte boundaries. BezgelorProtocol.PacketReader and PacketWriter keep track of the position to the bit. Each message has a module under apps/bezgelor_protocol/lib/bezgelor_protocol/packets/, with a read/1 for messages the client sends and a write/2 for messages the server sends. Here is a whole one, the message the client sends when a player splits a stack of items (shortened a little):

defmodule BezgelorProtocol.Packets.World.ClientItemSplit do
  @behaviour BezgelorProtocol.Packet.Readable

  defstruct [:guid, :location, :count]

  def opcode, do: :client_item_split

  def read(reader) do
    with {:ok, guid, reader} <- PacketReader.read_uint64(reader),
         {:ok, location, reader} <- ClientItemMove.read_location(reader),
         {:ok, count, reader} <- PacketReader.read_uint32(reader) do
      {:ok, %__MODULE__{guid: guid, location: location, count: count}, reader}
    end
  end
end

If you don't read Elixir: defstruct declares the three fields the message has. read reads them in order; each step hands the reader, moved past what it read, to the next step. with runs the steps one after another and stops at the first one that doesn't return {:ok, ...}, for example when the packet is too short, and that failure becomes the result.

Lower down, the reader takes bytes apart with binary pattern matching. This is the function that reads a frame's header (packet.ex, written here in its short form):

def parse_header(<<size::little-32, opcode::little-16>>), do: {:ok, size, opcode}
def parse_header(_), do: {:error, :incomplete}

The first line says: if the six bytes are a 32-bit little-endian number followed by a 16-bit one, call them size and opcode and return them. The second line catches everything else. The layout of the bytes is written as a picture of the bytes, and the language does the shifting and masking. Why Elixir & OTP says more about this.

There are about 200 packet files for the world server (packets/world/), many holding several message modules, and five for the realm server (packets/realm/).

Opcodes and handlers

BezgelorProtocol.Opcode (opcode.ex) gives every opcode a name, such as :client_cast_spell. The table itself, opcode/table.ex, is generated by tools/gen_opcodes.py from NexusForever's list of message opcodes, plus a few we found in the client.

BezgelorProtocol.PacketRegistry (packet_registry.ex) maps each opcode to the handler module that deals with it. The map is kept in :persistent_term, so looking a handler up costs no message and no lock. About forty handlers for the login stages and the most common world messages are built in; the world server registers about two hundred more when it starts (BezgelorWorld.HandlerRegistration, apps/bezgelor_world/lib/bezgelor_world/handler_registration.ex). Registering at start-up, rather than listing them in the protocol app, means the protocol app doesn't need to know the world app exists, which keeps the dependency between the two one-way.

A handler is a module with one function, handle(payload, state). It reads its packet, does its work, and returns what should happen next: {:ok, state} to carry on, or one of a few {:reply, ...} forms to send packets back. It runs inside the player's connection process, so it can change that player's session data directly, but anything that concerns the zone (moving, casting, picking up loot) is sent to the zone as a message. Into the world explains why.

A message nobody handles yet is not an error: the connection records a :packet_unhandled event, once per opcode per connection (BezgelorProtocol.PacketEvents), so we can see which parts of the protocol players reach that we haven't built.

Sending

To send, code builds a packet struct and passes it to the connection. The connection calls the struct's write/2, encrypts the result, frames it and writes it to the socket. Code outside the connection, such as a zone, uses Connection.send_packet/3, which is a cast: it puts a message in the connection's mailbox and returns at once. A zone that tells fifty players about one spell encodes the packet once and sends the same bytes to all fifty.

BezgelorProtocol.PacketStats counts packets and bytes sent, and times one send in 64 for the metrics ([:bezgelor, :packet, :encode] in telemetry).

Checking our packets with the real client: the oracle

A mistake in a server packet is usually silent: a field one bit too wide, and the client misreads everything after it, or quietly drops the message. We can't ask Carbine how the client reads each message, but we can ask the client itself.

How the real client checks our packets Our packet bytes, from a test or from the headless TestClient, go to ClientOracle, test code with a pool of workers. Each worker is a Python program, oracle.py, that runs the Unicorn emulator and inside it the real client's own unpack function for that message, taken from WildStar64.exe. The verdict comes back to the test: the client read the whole packet, so it passes, or it stopped early or failed, so it fails. The TestClient is a headless player that logs in through the real protocol and sends every packet it receives through the oracle. Our packet bytes from a test or the headless client ClientOracle test code, a pool of workers oracle.py Python, Unicorn CPU emulator WildStar64.exe the client's own unpack function Verdict read it all: pass stopped early: fail TestClient a headless player, logs in for real every packet it receives

The client oracle loads the real WildStar64.exe into a CPU emulator (Unicorn) and runs the client's own unpack function for a message on the bytes our server produced. A packet passes when the client reads all of it and returns success, and the same packet with its last byte cut off must fail. BezgelorProtocol.ClientOracle (apps/bezgelor_protocol/test/support/client_oracle.ex) is the Elixir side: a small pool of worker processes, each driving one tools/community/oracle/oracle.py program. Which function unpacks which message comes from message_table.csv, exported from a decompilation of the client.

Every server packet we write has an oracle test (apps/bezgelor_protocol/test/client_oracle/, about forty files). On top of that, BezgelorWorld.TestClient (apps/bezgelor_world/test/support/test_client.ex) is a headless player: a test logs it in through the real protocol, makes it walk, fight and loot, and checks every packet it receives with the oracle. About sixty test files use it, most of them under apps/bezgelor_world/test/e2e/.

The oracle needs the client binary and the reverse-engineering workspace, which are not part of the repository, so its tests are skipped on machines without them. The technique, and what it found while we ported NexusForever's messages, is written up for other projects in the client oracle guide; the message layouts it confirmed are in the protocol reference.

Where to look

  • apps/bezgelor_protocol/lib/bezgelor_protocol/connection.ex: the connection process.
  • framing.ex, packet.ex, packet_reader.ex, packet_writer.ex: frames and bits.
  • handler/encrypted_handler.ex: the encrypted wrapper.
  • opcode.ex, opcode/table.ex, packet_registry.ex: names and routing.
  • apps/bezgelor_world/lib/bezgelor_world/handler_registration.ex: the world server's handlers.
  • apps/bezgelor_protocol/test/client_oracle/: oracle tests.

Next: Into the world.