Integrate Godot Headless Server with Nakama

Introduction

"Nakama", aka 仲間, means "friend" in Japanese, which implies its multiplayer capability. Nakama is an open source BaaS server software. You can deploy it on your own server and get a free BaaS framework immediately. Besides, Nakama provides various client SDKs including Unity, Unreal, Godot, Cocos2d, Defold, etc. Using Nakama can save your time when it comes to multiplayer gamedev.

Different multiplayer games have different gameplays. Nakama allows developers to write server codes with its plugin system. At the moment it provides Lua runtime, Go runtime and JavaScript runtime. You can write plugins or modules and Nakama will load them at runtime. For example, you can write simple server logic inside matchLoop callback called by Nakama every server frame. The plugin system is enough for many light weight scenarios like turn-based card games, simple 2d games, chat rooms, etc. What about complex server logic like physics? What about combining Nakama with dedicated server exported by the game engine?

Well, Nakama allows you to do that.

For complex gameplay where you want the physics running server-side (e.g. Unity headless instances). Nakama can manage these headless instances, via an orchestration layer, and can be used for matchmaking, moving players on match completion, and reporting the match results.

However, the documentation doesn't cover this topic in detail. There's no example or guides on how to do it exactly.

This article talks about integrating dedicated server exported by Godot Engine with Nakama. All server codes are written in TypeScript. This article won't talk about the details of exporting dedicated server (You can get such information on Godot Docs), details of multiplayer game developing, or details of how to dockerize your Godot server or how to setup a Nakama server. These topics are beyond the range of this article. If you use another game engine, this article should help as well.

Communication Overview

We have two types of Nakama clients, the client on player's device and the client on server. The dedicated server is essentially also a client to Nakama. All of the communications are implemented by four ways:

RPC calls between Nakama client and Nakama server;
Match state message broadcast between Nakama client and Nakama server;
RPC calls between Godot client and Godot server;
Nakama HTTP API calls between Nakama client and Nakama server.

Create Match

This is done by calling Nakama RPC instead of a direct client HTTP API call, because we need to insert initial match data into storage like room status(Has game server already started? Are we waiting for players to be ready?), container host and port, player defined room name, etc. We insert the data, then create the match by API provided in server runtime, then return the match id to the client so player can join the match.

User ID is default system user ID 00000000-0000-0000-0000-000000000000 because these objects can never be modified by any player.

'room' is the module name, you may register a different module. Like I said above, details on how to setup Nakama server is not included in this article. You should read Nakama's doc if you are not familiar with this.

// server code
export const createMatch: nkruntime.RpcFunction = function (ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, payload: string) {
  try {
    const match_name = JSON.parse(payload).match_name
    const match_id = nk.matchCreate('room')

    const newMatchInfo: TrodMatchInfo = {
      host: '',
      port: '',
      matchId: match_id,
      containerId: '',
      status: 'preparing',
      kickIds: [],
      roomOwnerId: ctx.userId,
      roomName: match_name
    }

    nk.storageWrite([
      {
        collection: TrodMatchInfoCollectionKey,
        key: match_id,
        userId: SystemUserId,
        value: newMatchInfo,
      }
    ])

    logger.info('new match info written')

    return JSON.stringify({ match_id, match_name })
  } catch (err) {
    logger.error(err.message)
    throw err
  }
}

Use Match Loop as Room Handler

After joining the match, client and server can communicate through Match State Message.

Server code

// server code
export const matchLoop = function (ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, dispatcher: nkruntime.MatchDispatcher, tick: number, state: nkruntime.MatchState, messages: nkruntime.MatchMessage[]) : { state: nkruntime.MatchState } | null {
  logger.debug('Room match loop executed')

  messages.forEach(function (message) {
    logger.info('Received %v from %v', message.data, message.sender.userId)

    if (message.opCode === MatchOpCode.ReadyStateUpdatedYes) {
      // ...
    }
    // ...
  })
  // ...
}

Client code

# client code
# ...somewhere
Network.socket.connect("received_match_state", self, "_on_match_state")
# ...

func _on_match_state(match_state: NakamaRTAPI.MatchData):
    if match_state.op_code == GameData.OpCode.ReadyStateUpdatedYes:
    pass
    # ...handle match state message

Op codes must be defined both on client and server, values must be same. Below is what I defined in my case:

// server code
export enum MatchOpCode {
  StartGameServer = 0,
  GameServerInited,
  StartGameServerFailed,
  ReadyStateUpdatedYes,
  ReadyStateUpdatedNo,
  ReadyStatesBroadcast,
  Kicked,
  UsernameUpdated,
  RoomNameUpdated,
  GameServerStarting,
}

For example, when player hits ready button on the client, we emit a Match State Message.

func _on_ready_button_clicked():
    AudioManager.play_click_sound()
    imready = not imready
    var op_code = GameData.OpCode.ReadyStateUpdatedYes if imready else GameData.OpCode.ReadyStateUpdatedNo
    yield(Network.socket.send_match_state_async(match_id, op_code, JSON.print({})), "completed")

And server broadcast to other clients

// server code inside matchLoop
if (message.opCode === MatchOpCode.ReadyStateUpdatedYes) {
  state.readyStates[message.sender.userId] = true
  readyStatesUpdated = true
} else if (message.opCode === MatchOpCode.ReadyStateUpdatedNo) {
  state.readyStates[message.sender.userId] = false
  readyStatesUpdated = true
}
//...
if (readyStatesUpdated) {
  dispatcher.broadcastMessage(message.opCode, message.data, null, message.sender)
}

Then client can update the ready states at local or display a "I'm ready" on UI.

When the room owner (In my case only room owner can start the game) hits Start button, we also broadcast message to other clients like the example above. And more importantly, we need to create and start the game server container.

Spawn Server Container

I assume you already dockerize your game server, I'm afraid the only way to run game server is through Docker HTTP API. Because Nakama JavaScript runtime doesn't support running bash command. It may never support as Nakama is designed to be scalable and is typically run inside a container.

Find an Available Port

We actually don't know if a port can be used because of the limits of the TypeScript runtime. So I store the using ports in the storage. Every time we create a container, we only use the port that has not been stored, and has not been occupied by other services on server. We assign these ports before we start Nakama (retainedPorts below).

// server code
const canUsePort = function (port: string, logger: nkruntime.Logger, nk: nkruntime.Nakama): boolean {
  try {
    if (retainedPorts.indexOf(port) !== -1) {
      return false
    }

    const objects = nk.storageRead([
      { collection: TrodPortsKey, key: port, userId: SystemUserId }
    ])

    if (objects.length === 0) {
      return true
    }

    return false
  } catch (err) {
    logger.error(err.message)
    return false
  }
}

Get an available port with canUsePort above.

// server code inside startGameServer function
let port = getRandomPort()
let triedTimes = 0
const tryingCount = 20

while (!canUsePort(port, logger, nk)) {
  port = getRandomPort()
  triedTimes++

  if (triedTimes > tryingCount) {
    throw new Error('cannot allocate usable ports')
  }
}

This solution is not perfect, we may start other services after we start Nakama. Accidentally, we may fail to create container because the port is occupied and player will see error messages.

Pass Parameters to Godot server

Command lines of container creating API can be used to pass parameters from Nakama to Godot server. These parameters can be retrieved by Godot server through APIs provided by game engine, like OS.get_cmdline_args() in Godot. In my case, I pass values below:

// server code inside startGameServer function
const cmd = [
  `--server-port=${port}`,
  `--match-id=${matchId}`,
  `--max-players=${maxPlayers}`,
  `--imposters-count=${maxImposters}`,
]
const containerId = createContainer(ImageName, port, cmd, `game-server-${matchId}-${port}`, options)

Make HTTP Call through `nk` Object

// server code
export const createContainer = function (image: string, port: string, cmd: string[], name: string, options: InternalCallOptions) {
  const { nk, logger, ctx } = options
  
  const dockerAPIUrl = `${DockerAPI.scheme}${ctx.env['DOCKER_HOST']}:${DockerAPI.port}`
  const url = `${dockerAPIUrl}/v1.41/containers/create?name=${name}`
  const headers = { 'Content-Type': 'application/json' }
  const body = JSON.stringify({
    Image: image,
    Tty: true,
    ExposedPorts: {
        [`${port}/tcp`]: {},
        [`${port}/udp`]: {},
    },
    Cmd: cmd,
    HostConfig: {
        PortBindings: {
            [`${port}/tcp`]: [{
                "HostIP": "0.0.0.0",
                "HostPort": port
            }],
            [`${port}/udp`]: [{
                "HostIP": "0.0.0.0",
                "HostPort": port
            }],
        },
    },
  })

  try {
    const response = nk.httpRequest(url, 'post', headers, body)
    if (response.code === 201) {
      const dockerCreateResult: { Id: string, Warnings: Array<any> } = JSON.parse(response.body)
      logger.info('container %q created', dockerCreateResult.Id)
      return dockerCreateResult.Id
    }

    throw new Error(JSON.stringify(response.body))
  } catch (err) {
    logger.error('error occurred while creating container, %q', err.message)
    throw err
  }
}

Wait for the Server Initialization

Godot server may have time-consuming bootstrap to process like loading level resources. Godot server notifies Nakama when it is ready for players to join. After that can players connect to it.

# client code in godot server
func _on_server_init_completed():
    print("_on_server_init_completed")
    if GameData.server_match_id == "":
        push_error("server inited but no match id")
        return
    
    var result: NakamaAPI.ApiRpc = yield(Network.socket.rpc_async(
        "game_server_inited",
        JSON.print({ "match_id": GameData.server_match_id })
    ), "completed")
    
    print("game_server_inited rpc called")
    if result.is_exception():
        push_error(result.get_exception().message)

We changes the status of a match info in the storage, then broadcasts messages to the clients inside matchLoop, which is the only place we can emit Match State Message. We are transfering information from RPC function to matchLoop through the storage. Many features have to be implemented this way, including kicking players.

Notice: When you're developing on Nakama's JavaScript runtime, you must understand the limits brought by Goja and Nakama. Like, there's no event loop. JavaScript runtime is pooled by Nakama, so global variables can't be used.

// server code
export const gameServerInited: nkruntime.RpcFunction = function (ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, payload: string) {
  try {
    if (!isPrivateIP(ctx.clientIp)) {
      throw new Error('Headless server can only be run by Nakama')
    }

    const obj = JSON.parse(payload)
    const matchId = obj.match_id

    const objects: nkruntime.StorageObject[] = nk.storageRead([
      { collection: TrodMatchInfoCollectionKey, key: matchId, userId: SystemUserId }
    ])

    if (objects.length !== 1) {
      throw new Error('matchinfos read results not correct')
    }

    const value = objects[0].value as TrodMatchInfo
    value.status = 'inited'

    nk.storageWrite([
      { collection: TrodMatchInfoCollectionKey, key: matchId, userId: SystemUserId, value }
    ])

    logger.info('game server %q is inited', value.containerId)
  } catch (err) {
    logger.error(err.message)
    throw err
  }
}

Check status of match info in storage in matchLoop every server frame. Tell players' clients to connect to Godot server if needed.

// server code inside matchLoop
// get match info as value from storage...
if (value.status === 'inited' && value.host && value.port) {
  logger.info(JSON.stringify(value))
  dispatcher.broadcastMessage(MatchOpCode.GameServerInited, JSON.stringify({
    host: value.host,
    port: Number(value.port),
  }))

  value.status = 'broadcasted'
  nk.storageWrite([
    { collection: TrodMatchInfoCollectionKey, userId: SystemUserId, key: ctx.matchId, value }
  ])
  logger.info('gameserverinited broadcasted')
}

Connect to Godot server after being notified the server is ready.

# client code
# ...
elif match_state.op_code == GameData.OpCode.GameServerInited:
  var result := JSON.parse(match_state.data)
  if result.error_string == "":
    print(result.result)
    var host: String = result.result["host"]
    var port: int = result.result["port"]
    
    Network.room.connect_to_game_server(host, port)
  else:
    GameData.show_toast(result.error_string)
# ...

Validate Identity

When user logins, we store its token into storage using client HTTP API. When player connects to the server, the server calls RPC to Nakama with player's token, Nakama checks the token in the storage and returns the result to the server. Specifically:

Godot client connects to Godot server
Godot server requests user token through Godot RPC call
Client replies
Godot validate token through Nakama RPC call

# client code in godot server
func _verify_peer_id(id: int) -> bool:
    print("veryfing ", id)
    if not _cached_player_session.has(id):
        print("no session cached, verify fails")
        return false
    
    var result: NakamaAPI.ApiRpc = yield(Network.socket.rpc_async("validate_token", JSON.print(({
        "user_id": _cached_player_session[id]["user_id"],
        "token": _cached_player_session[id]["token"],
    }))), "completed")
    
    if result.is_exception():
        print("token not matched, verify fails")
        return false
    
    return true

If succeeds, add character into game world
If fails or timeouts, disconnect with the client peer. Any other RPCs between this client and server are not allowed meantime

One thing to note, we also need to validate that if the caller is Godot server. We don't allow malicious Godot clients to call RPCs which are only for Godot servers. How to do that? We only allow this RPC be called from LAN by checking ctx.clientIp:

// server code
export const validateToken: nkruntime.RpcFunction = function (ctx: nkruntime.Context, logger: nkruntime.Logger, nk: nkruntime.Nakama, payload: string) {
  try {
    if (!isPrivateIP(ctx.clientIp)) {
      throw new Error('Headless server can only be run by Nakama')
    }

    const obj = JSON.parse(payload)
    const userId = obj.user_id
    const token = obj.token

    if (!userId || !token) {
      throw new Error('validation fails')
    }

    const objects = nk.storageRead([
      { collection: TrodTokensKey, key: userId, userId }
    ])

    if (objects.length !== 1) {
      throw new Error('validation fails')
    }

    if (objects[0].value['value'] !== token) {
      throw new Error('validation fails')
    }

    logger.info('validate user %q', userId)
    return JSON.stringify({
      success: true
    })
  } catch (err) {
    logger.error(err)
    throw err
  }
}

Game Over and Collect Game Result

After players join the Nakama, all commnunications should be done between Godot client and Godot server. With or without Nakama, there should be no difference on this part.

When the game produces a result, Godot server can notify Nakama through a Nakama RPC call, as I've showed you many times. Nakama can save the game record into storage, depending on your usecase.

Most importantly, when it is appropriate, remove the container through Docker HTTP API and remove the port from the storage.

// server code
export const removeContainer = function (id: string, options: InternalCallOptions) {
  const { nk, logger, ctx } = options

  const dockerAPIUrl = `${DockerAPI.scheme}${ctx.env['DOCKER_HOST']}:${DockerAPI.port}`
  const url = `${dockerAPIUrl}/v1.41/containers/${id}?force=true`
  const headers = { 'Content-Type': 'application/json' }

  try {
    logger.info(`removeContainer ${id}`)
    const response = nk.httpRequest(url, 'delete', headers)

    if (response.code !== 204 && response.code !== 404) {
      const body: { message: string } = JSON.parse(response.body)
      throw new Error(body.message)
    }

    logger.info('container %q removed', id)
  } catch (err) {
    logger.error('error occurred while removing container, %q', err.message)
    throw err
  }
}

Attention: Nakama supports HTTP delete method from 3.15.0 on

@2023-01-16 21:18