Splatoon 2’s netcode and matchmaking has been criticized for many different reasons. Some of the common complaints are frequent disconnections, low tick rate, lag, and that the netcode uses a peer-to-peer architecture. In this blog post, I will attempt to explain how the netcode works and address some people’s complaints. This post is the culmination of a year’s worth of research and collaboration with others. I hope that you will learn something from it!
If you are allergic to essays, I’ve included a list of key points at the end of the post. I still recommend to try and read the whole thing, however.
(The listing image was taken by Ethan. Thank you so much!)
Need-to-Know
Let’s go over some basic information first.
Basic Netcode Terminology
Please have a look over PCGamer’s excellent netcode overview to review basic information about what netcode is and its associated terminology. I will not be explaining every term or concept here otherwise this post would be a million words long. (As if it wasn’t already that long in the first place…)
Pia
At its core, Splatoon 2 uses a library called “pia”. Pia was developed by Nintendo in order to make creating networking features for games easier. It implements network features based on a “peer-to-peer” architecture, unlike the traditional “client-server” architecture. This means that instead of one server having a “one-on-one conversation” between each Nintendo Switch console to exchange information, all of the consoles communicate directly with each other to share information.
However, this doesn’t mean that there isn’t a “master console” similar to what a server would be. Pia assigns one console as the “host” of the session. This is usually the person who opened the session first, and people who join the session are considered to be regular clients. (Contrary to popular belief, it hasn’t been definitively proven that the person at the top of the matchmaking screen is the host of the session.) But how do other people find each other to join sessions?
NEX
NEX is a two-part system comprising of a game server and an API used to communicate with the servers. NEX handles matchmaking for peer-to-peer sessions, NAT traversal (helping people behind a firewall or router with getting connected to others), and rankings. Pia has built-in support for NEX’s matchmaking and NAT traversal features, making it easy for developers who are already using pia to implement online play. (NEX actually has its own peer-to-peer networking library independent of pia called VSocket, but Splatoon 2 does not use this. As far as I know, I don’t think any game does…)
NEX actually began its life as Quazal Technologies Inc.’s Rendez-Vous, which started development all the way back in 2003. Several games from the 2000s used technology from Quazal to implement multiplayer features. Ubisoft later acquired Quazal Technologies in 2010. Rendez-Vous lives on to at least 2015, as Ubisoft Montreal continues to add features to the library.
As for NEX, it appears Nintendo must have either bought or licensed Rendez-Vous from Quazal. They then proceeded to make various changes to the library and rebrand it as NEX.
“Libraries”?
In the previous sections, I referred to pia and NEX as “libraries”. If you don’t know, a library is a set of code meant to be shared between different programs. You might see where I’m going with this…
Pia and NEX are actually used by multiple games on the 3DS, Wii U, and Switch. They were not built specifically for Splatoon 2. Here’s a selection of software that uses them:
3DS
- Friends service (NEX)
Wii U
- Splatoon
- Mario Kart 8
- Friends service (NEX)
Switch
- Splatoon 2 (yes, this means that Splatoon 2’s netcode is an evolution of Splatoon 1’s)
- Mario Kart 8 Deluxe
- ARMS
- Nintendo Entertainment System – Nintendo Switch Online
As you can see, it’s all first party software. This isn’t an exhaustive list – I’m less familiar with 3DS games, though pia is definitely available for 3DS. Nintendo does provide the libraries to third-party developers but I haven’t seen any third-party games using pia or NEX yet.
Splatoon 2
Now that we know the basics, let’s apply our knowledge to Splatoon 2!
Clones
Splatoon 2’s netcode is built entirely using a feature of pia called “clones”. A clone is created by a console, and other consoles can “subscribe” to the clone. If the owning console makes changes to the clone, the subscribing consoles are notified of any changes. For example, a clone can be made using a variable called “X”. If the owning console changes X to 5, then all of the subscribers will be notified that the value of X was changed to 5. Splatoon 2 does not use any other method of data synchronization other than clones, though pia supports sending messages directly between consoles without using clones.
There are three types of clones:
- Unreliable – changing the clone will cause a notification to be sent to subscribing consoles, but the notification may or may not arrive at the subscribers because of packet loss
- Reliable – a notification will be sent to subscribing consoles, with more being sent if they are not received, guaranteeing synchronization
- Event – clone synchronization is guaranteed, and any data changes in rapid succession are guaranteed to be processed in the order that they were made
Here are some examples of where the various clone types are used:
- Unreliable – player position, as the game can interpolate (guess) where the player is heading towards based off their current speed and position if there is data loss
- Reliable – basic player information like their name and equipment, as this information is required to be synchronized between consoles, but the order in which it is received does not matter
- Event – a boss Salmonid spawning in Salmon Run, as other consoles need to know that this happened and the order in which bosses spawn in
In addition, clones have various “access permissions” that can be set:
- Send – values can be only set and sent to subscribers with an associated Receive clone
- Receive – values can be only received from an associated Send clone
- Sequential – all consoles can set the values, though because of this an Event clone cannot also be a Sequential clone (impossible to preserve the order of values set)
- Atomic – all consoles can set the values, but only one console can set the value at a time
To keep synchronization, each console starts up a “clone clock”. For Splatoon 2, this clock goes up 120 units per second and continues until the session closes. (For the nerds, an integer overflow wouldn’t happen for a very, very long time. It would be over a year before it overflows assuming that the data type is an unsigned integer.) If the clocks become out of sync, the clocks are resynchronized. This can occur if, for example, there is a sudden spike in CPU load and the game can’t keep up with its refresh rate. This clock value is also occasionally used as a “seed” for the random number generator (RNG), which is used in various in-game actions that involve randomness.
Tick rate
The tick rate situation in Splatoon 2 is complicated and marred with controversy. The controversy part mainly comes from a post by OctoBbte, who published an article saying that Splatoon 2 runs at 15.75Hz. This isn’t entirely true.
To start off, the game ticks at 60Hz. (Fun fact – when the game enters the plaza and halves the frame rate, it accomplishes this by lowering the entire tick rate of the game to 30Hz.) Each tick, the game tells pia to send packets, receive packets, and process any events. This means that the netcode also processes at 60Hz. However, Nintendo set a configuration flag that says clone data should only be sent every 4 ticks. This means that the tick rate for sending data is artificially constrained to 15Hz. Normal processing of packets occurs at 60Hz, so receiving and processing clone data runs at “normal speed”.
What does this 15Hz send rate mean?
The pia library will combine packets during the 4 tick waiting period into one big packet at the end. (This coalescing of data explains why Octobyte found an increase of average packet size compared to Splatoon 1.) This waiting period also means that data may arrive at minimum 4 ticks later. However, remember that event clones will always be processed in the order that they occur. Event clones are used for things like player damage, inking the map, and bullet spawning. This may mitigate some of the effects caused by the lower tick rate.
Why is Nintendo artificially constraining the tick rate?
This is likely for bandwidth reasons, as Octobyte stated. According to their data, there was a 45% reduction in bandwidth going from Splatoon 1 to Splatoon 2. It could also be for stability reasons. By using less data, it can allow players with less capable Internet connections to be able to play online. This is especially beneficial for mobile hotspots.
How does this compare to Splatoon 1?
Unfortunately, it’s very hard to analyze Splatoon 1’s netcode. Nintendo made it easy for me to analyze Splatoon 2 since they accidentally left debugging information for many versions. Splatoon 1, on the other hand, is essentially a black box. I have found a configuration setting that may be similar to the “waiting period” setting in Splatoon 2, but I have been unable to confirm this. In addition, Splatoon 1 uses a very old version of pia, and there have been several major changes to the library since (mostly relating to how packets are serialized). It would be unfair to do a “direct” comparison between Splatoon 1 and Splatoon 2.
However, I do want to say that I believe Octobyte’s claim that Splatoon 1 had trouble keeping up at 25Hz is incorrect. The fluctuation in data size could be there for other reasons. For example, the waiting period may be set lower, causing clone data to not be coalesced as often. This would result in lower overall packet sizes and data to be sent much more frequently.
Disconnects
The game has three utilities in order to detect connection issues in-game. First off, a message that you’re probably all familiar with.
screenshot taken by Ethan
How does the game decide when to show this message? On every frame, the game checks if there is a “period of silence”. A period of silence is an unusual gap in the usual communications where nothing is being received from other consoles. If one is detected, the game considers the connection to be “unstable”. After around 5 seconds, the game gives up and brings up the “communication error has occurred” message. (If packets from other consoles are received again, the countdown is reset.) Pia could also choose to disconnect from the session if it considers the connection to be unrecoverable. There can be many reasons for a period of silence. It could be a momentary burst of lag on a person’s Switch. There could be packet loss somewhere along the connection path. Someone could accidentally open the HOME menu and cause the game to go into the background. Because of this, it is difficult to know what exactly is the cause of every “the connection is unstable” message.
The second utility that the game uses is slightly more technical. Remember that there is a “clone clock” that is synchronized between all consoles. The clone clock is used with Event clones to ensure that they are processed in the correct order. Event clones will also wait for the receiving console to acknowledge that they have received the values before proceeding with sending additional updates. If there is a backlog of Event clones waiting for acknowledgements, the game will check each Event clone’s attached clock value to see when the data was initially sent. If the clock is 4 seconds or more in the past, the game will designate the receiving console to be a “bottleneck”. If an Event clone is stuck waiting for an acknowledgement for that long, it is likely that the receiving console has high latency or high packet loss on their connection. The console is then at risk of being kicked out of the session by force in order to maintain responsiveness, stability, and synchronization.
The last utility is one you might already be familiar with. The game will check on every frame if you are idle using based on various factors. These include your current velocity, gyroscope movement, right stick movement, and more. If you are found to be idle, a 60-second countdown is started. The countdown is reset to 0 if the game detects movement. If you are idle for 60 seconds, then the game will bring up the “communication error has occurred” message and kick you out from the session. A penalty (temporary ban) will then be assigned to you.
What happens if you lose connection to the host, however? Pia has a feature called “host migration” to solve that problem. When the session loses connection to the host, pia will consider all of the remaining consoles in the session as a candidate for a new host. The highest ranking console will be selected as the new host and the session continues as normal. If the session is unable to migrate to the new host, then it will go to the next candidate in the ranking. This will continue until a host is chosen.
BitRateMeasureRate
As of the time of writing (Tuesday, October 30th, 2018), Nintendo has enabled a parameter called BitRateMeasureRate for all regions. This parameter allows them to remotely activate a connection quality test on a specific amount of sessions. This connection test measures the uplink bitrate of the current session and sends it back to Nintendo once it finishes. Right now, 1 in every 1000 matches will have their connection quality measured. It is unknown why exactly Nintendo is doing this. It could be because they want to make improvements to the netcode, and are using trends in connection quality to help them decide where to improve. They could also be gathering data for statistics purposes for reference in future games. I’m not Nintendo, so I can’t really know for sure what they are doing.
As of now (Oct. 30th), here are the past BitRateMeasureRate periods:
- June 6th: The parameter was added but not activated
- Oct. 24th from approximately 1:30 am to 2:30 am Eastern: Every 1 in 100 matches was measured
- Oct. 26th from approximately 3:30 am to 7:30 am Eastern: Every 1 in 100 matches was measured
- Oct. 29th from approximately 1:30 am Eastern and ongoing: Every 1 in 1000 matches is being measured
Matchmaking
To start, you should have some basic knowledge of how the Power and Rank system works in Splatoon 2. If you haven’t read my previous blog post about that, click here.
Some of you may remember an erroneous tweet that I made earlier this month. In case you missed it, here it is:
I was looking at a configuration file which seemed to indicate “lobby ID” numbers. For example, Regular Battle in Japan is set to ID 1. From the game’s launch date to September, Regular Battle in North America and Europe was set to ID 1, but Nintendo changed it to 51 earlier this month. This led me to believe that the lobbies were segregated and people from Japan couldn’t be randomly matched with people in North America or Europe. I was then corrected by my followers on Twitter, as they were still getting matched with players in Japan. (Thank you for that!) Further reverse engineering of the game revealed that my “lobby IDs” interpretation was incorrect. However, I was at least semi-correct when I assumed they were ID numbers.
Each VS multiplayer mode in Splatoon 2 has a specific “index number” assigned to it. These indices differ between regions.
Each number represents a specific matchmaking configuration. While everyone is put into the same matchmaking pool (per mode), the NEX game server will attempt to match a player using the configuration for their region and mode. To determine which session someone should be matched with, each individual criteria in the configuration is assigned a score value, and the session with the highest total score is chosen.
Here is the list of criteria that Nintendo can assign scores to:
- The difference in Power between the session owner and the joining player
- The difference in the rate of disconnection between the owner and joining player
- How far the distance between the owner and the joining player is (calculated using GeoIP)
- If the country of origin matches
- How long the session has been waiting for players
- and more…
Because configuration data differs between regions and modes, Nintendo can adjust matchmaking in various ways. For example, for Japanese copies of the game, Nintendo might adjust Turf War matchmaking to prefer players within Japan. However, they might increase the threshold of the distance criteria when playing in Rank X lobbies to prevent long matchmaking times, as only 1% of the player base is in Rank X. In North America and Europe, Nintendo would also increase the threshold for distance in all modes as the distance between players can be higher overall compared to Japan.
Unfortunately, we are unable to retrieve the specific values used for the matchmaking criteria. These are only visible to Nintendo.
Key Points
TL;DR:
- Nintendo has a peer-to-peer networking library called “pia”. They also have a matchmaking library called NEX. These libraries are used in many first-party games.
- Splatoon 2 works off “clones”. These are objects which other consoles can subscribe to for notifications if the associated value changes.
- The game and netcode ticks at 60Hz. However, packets can only be sent every 4 ticks. This results in an artificially constrained packet send tick rate of 15Hz. This may not be as bad as it seems, as various other factors could help mitigate it.
- The infamous “connection is unstable” message appears when there is no incoming data.
- If a console takes too long to respond to a clone data notification, it is at risk of being kicked from the session.
- Idling for 60 seconds will cause you to be disconnected and receive a temporary ban.
- Nintendo is gathering connection quality data from every region.
- Nintendo can configure matchmaking per region and per mode using various criteria like power, distance, country, etc.
Conclusion
I hope this clears up some misconceptions and confusion with the netcode and matchmaking. I might have missed a few things or may need to correct some details, so I may publish some follow-up posts either on my blog or on my Twitter account. Stay tuned and thank you very much for reading!
Follow me and Simon on Twitter:
Join my Discord server:
Credits
- Simonx22, for their love and support
- Shadów Research University
- Wii Sports Discord Server
- Inkademy
- everyone who joined OatmealDome’s home
- Ethan for the listing image