Dev Log #2 – Network Architecture, Part 1

Broadly speaking, there are two types of games: singleplayer games and multiplayer games. Multiplayer games are far more complicated to develop than single player games. In order to build out the back end we need to understand what the network architecture looks like.

Introduction

This is the first part of a series that will explore building a multiplayer game using Unity and a few third party products/libraries. This entry will be more of a high-level outline than a how-to guide or progress report. At this point I am still exploring libraries, frameworks, and methodologies.

There’s a saying I heard once that went something like this: “you either build multiplayer from the start, or you do it at the end, but if you do it at the end then you have to start over again.” The reason for this is that the way you structure the logic and code of a multiplayer game is very important; there any many things that need to be synchronized across the network. If you wait until you’ve nearly finished development to introduce multiplayer, you might have to rewrite a significant chunk of the codebase for your game.

If you have built a single player game and want to make it multiplayer, all of a sudden you have to figure out how to synchronize the position and state of every movable or interactable object in the game. Is a door open or closed? Is a switch on or off? How much health does this enemy have remaining? Where is this moving platform currently? What is inside this chest? All these are questions that need to be answered by some authority.

In a single player game, the client is always the authority. The game itself running locally on your device keeps track of all that for you. In multiplayer, however, what if your device says one thing but another player’s says something else? If this conflict isn’t handled well, it can lead to all sorts of unexpected behaviours.

I love multiplayer games, and I don’t want to limit myself to singleplayer games. In order to achieve that goal I need a rough idea of what a multiplayer “game” actually looks like in terms of network architecture. Even if I end up deciding to scale back the scope and release a single player game, if I build the foundations right, it would be more straightforwards to add multiplayer as a later update.

Back-end as a Service (BaaS)

Regarding “multiplayer games”, it’s worth noting that “multiplayer” doesn’t mean the same thing in every game. Some games have what you might call “serverless multiplayer” in the sense that the game is an online game, has sign in, friends, leaderboards, etc., but you never actually get to play the game loop with another player in the same scene as you. Many “multiplayer” games, especially mobile games, follow this pattern.

One telltale sign of this is an asynchronous PvP system where you can fight other players in arena matches, but what you’re really fighting is just the AI controlling their character or making choices for them without the other player even knowing you are fighting them at all. This level of multiplayer can be accomplished using a “Back-end as a Service” (BaaS) solution such as Unity Gaming Services, Playfab, Nakama, or even Firebase. It’s basically a singleplayer game with leaderboards, essentially.

However, even for multiplayer games that are as interactive as possible (for example, MMORPGs), these are still features that play an important role in the client-server network architecture. As the saying goes, “don’t reinvent the wheel”. In that spirit, I am planning on selecting such a service to handle player authentication and other social features like friends lists, groups, etc.

I am currently evaluating Nakama as a BaaS solution for a game. PlayFab is also worth considering, but Nakama has the obvious appeal of being partially open source and self-hostable. I currently have the basic workflow down to add custom APIs/RPCs in TypeScript, then build a container with docker to run the Nakama image with my configuration and custom APIs.

One significant trade-off of Nakama is that their open-source, self-hostable version does not support clustering or horizontal scaling. This means that one Nakama server would be handling the entire load of all concurrently connected users (CCUs). However, this could still be in the thousands depending on what the Nakama server actually has to do. If all it is responsible for is authentication and some basic info about the character and their friends, then even one server could handle many users.

For the actual gameplay server, my plan is to use Mirror in Unity, and then create a dedicated Linux server build which I can host in a CPU-optimized droplet in Digital Ocean. I still need to figure out how to integrate the server build with the Nakama instance, although there seems to be good documentation and examples and SDKs for that.

Network Architecture Diagrams

Below is a rough network architecture diagram that I have in mind. PlayFab could be swapped for the Nakama/PostgreSQL database if desired. Unity is used to build the client and gameplay server builds as Mirror allows most of the code to be reused.

With this network architecture in mind, following swimlane diagram shows how a client might authenticate with Nakama before connecting to the “real” server that actually runs the game. Different features of the overall solution are handled by different servers, spreading and lightening the load.

Periodically, or when the player logs out, the Mirror server can update the Nakama server with information about the player’s last known position and progress. That way, they can pick up where they left off when they log in next time, even if it’s on a different server.

Asset Bundles

One more piece of the networking puzzle is content distribution. I have only just started learning about Unity’s Asset Bundles, but they seem like a good solution for this. Theoretically, I can pack chunks of the project as Asset Bundles to download at runtime. By doing this, the initial game download isn’t too large, and I I should be able to replace the Asset Bundles faster than I can get a modified build through various publishers’ approval pipelines. At least this seems to be how many other games do it. In practice, what this looks like is you download the game, and then on the first launch the game downloads the missing pieces.

One additional challenge introduced by VR is that users don’t have patience to keep the headset on while staring at a loading bar. This can be worked around by having the tutorial or starting area as the first item to download, and then download the rest of the assets while the user is playing through that content.

Voice Chat

Finally, I still need to find a solution for voice chat. PlayFab has Party and Unity Gaming Service (UGS) has Vivox, but Nakama does not have a voice chat solution. I am uncertain if it would be feasible to try and duct tape these solutions together as PlayFab, UGS, and Nakama all handle authentication in their own way.

It might make the network architecture a little more convoluted, but I believe it would be worthwhile to offload voice chat to a third party service if at all possible. For PC or mobile games, generally speaking it’s safe to leave this to users as most will just use Discord instead of in-game voice chat. However, this is where VR presents another challenge, because it is not so straightforwards for users to join a Discord call from a VR headset while still playing the game.

In the next dev log, I will discuss the different VR frameworks I have been reviewing and my experiences with them so far.