Issues with hit validation have been a big topic in Hunt lately, and so we wanted to address all of your concerns by taking a moment to explain where we are in the process right now, and what we are planning to do going forward.

Hunt's Hit Validation System: What It Does and Why We Need It

Since this blog post will get pretty technical, I'm going to start with an overview of Hunt's networking model. If you already know the basics, just skip ahead to the next section for the specifics.

Hunt's networking model—like those of many games—is built around the concept of a dedicated server. When you matchmake into a game you are being connected to a dedicated Hunt server. This server is running on a machine in a datacenter near you (hopefully) and is running a special version of the game. This version runs much of the same logic as that is running on your own computer, skipping the bits it doesn't need to do its job.

The dedicated server is important because it is basically in charge of the game world. It has authority over player positions, physics, and other related aspects of the game.

The version of Hunt running on your own computer is called the client. The dedicated server sends your client a copy of the game world. This copy is updated regularly via the network. Meanwhile, the client acts on your input, and updates the dedicated server on what you are doing in-game.

Though the internet is fast, it isn't instant, so when your client sends the server a message telling it that you've just pulled the trigger on your Caldwell, that message isn't instant either. This delay is typically called network latency, and it is measured in milliseconds (1 second=1000milliseconds=1000ms).

Your ping, which you can measure via your game performance stats, is the amount of time a message takes to go from your client to the server, and then back to your client.

This latency—which is caused by ISP infrastructure, distance, hardware connection quality, and other factors—is not a problem that any one game can solve. It's something that all game development teams have to account for, and we all have strategies for minimizing its impact. Hunt is no different.

When it comes to the issue of correctly registering gunshots and hits (player vs player), latency affects your game as follows:

1. Because of network latency on your local machine, you never see the current state of the world. If a message from the server takes 40 milliseconds to arrive, you are seeing what the game world looked like 40 milliseconds ago.
2. When you move or press the trigger of your gun, a message from your client has to be sent to the server, then the server has to process your input, apply it to your character, and send it back to you before you can see the result of that shot. Factoring in the delay on both sides, this could result in an up to 80 millisecond delay.

If we didn't do anything to mitigate this, the game would feel like it had input lag: the bigger the ping, the bigger the lag.

To keep the game responsive in spite of this, we use a client-side prediction system. This system allows you to take action in your local game client and see the results immediately by making a prediction about what happened. When the server response arrives, the system checks whether what you did locally matches what happened on the server. If it doesn't match, the system reconciles your local client to the server's version of events.

This system creates an interesting problem: You are being allowed to move and act instantly, while seeing a “past" representation of the game world. Yet, it is as if your character is acting in the “future", since the server has not yet received and integrated your response into the big picture.

At this point, you are probably wondering what any of this has to do with shooting Hunters…

Well, the problem is that when you are aiming your gun at a Hunter, they are, technically, not actually there. You are seeing where that Hunter was 40 milliseconds ago, in relation to the server. For the person playing that other Hunter—since his client is allowing him to predicatively move ahead of the server too—the Hunter might be farther away from the spot where you are aiming.

If we hadn't done anything about this, then it would be very hard to hit moving targets because you would have to account not only for the bullet's travel time, but also for the lag between your computer and the server. And since bullets travel quite fast, a difference of milliseconds would be enough to throw off your aim.

This is quite a complicated conundrum to solve because each client has a different delay-to-world state, as well as a different head start when it comes to the individual character. For example, on my screen, I might be seeing a Hunter walking straight ahead and have them in my sights. But on the server, that same Hunter may already be further along. And the player themselves, on their screen, might already have their Hunter behind a wall.

Due to network latency, no client will ever agree on the world state, and neither will the server. This is why we have a hit validation system in place. This system, which runs on the dedicated server, acts as a referee whenever a client shoots a target. The hit validation system always has to take a side when making a decision. Our system, like those in many other games, takes the “Favor the Shooter" approach.

So if a client claims that he has hit a shot, the hit validation system will run a number of validation strategies to confirm what the client claimed. If it is deemed legitimate, the shot will be considered a hit.

The validation strategies involve looking at the past history of the world state (matching the delay at which that client is seeing the world) and figuring out if a gunshot—from the client's position, at that point in the map, with that aiming direction, using that weapon, with that ammunition—should have hit the target, according to the records the server had on each player involved in the damage contention.

In some ways, this means that we allow you to slightly re-write the servers' world state history if your client's claim is deemed valid. The bigger your latency to the server, the further back in time those rewrites can go—though we do have a hard cap set at 800 milliseconds.

The unfortunate side effect of this system is that the Hunter that was shot might appear to already be behind cover when the server decides that a shot was valid, making it seem like you were shot out of nowhere, or that bullets are going through the walls.

At this point you might be thinking, this is terrible. But consider the alternatives:

- We trust the server over the client. If we did not mitigate latency, we would need to wait for the server to simulate the shot to see if it hits. You would have to take aim based on your server latency, which is a pain, especially if you have variable latency.

- We trust the client of the Hunter being shot at. This would mean if you didn't appear to be in the bullet's trajectory, it couldn't hit you. But because that player is moving predicatively, you would have to account for your server latency and the latency between the server and that player in order to take aim at all. This would make hits extremely inconsistent, since each player has different latency to the server.

As you can see, network latency problems are complex beasts to tackle. But that's not all…

Latency is rarely stable. One message might take 20ms to reach the server, while the next might take 25ms, and the following message might take 10ms. Messages can also be lost in transit, which means they have to be re-sent, causing an even bigger delay. This uncertainty means you can never truly know how far ahead/behind the clients and server are, which can result in de-syncs.

Imagine the last message you sent to the server was that you were pressing W (moving forward). The server simulates your character moving forward. Then you release W, stopping in place. But due to a variation in latency, the message to stop takes longer to be delivered. If that were to happen, the server will think you were pressing W for longer than you were, meaning that your final position on the server will be different than on your client. This situation is a de-sync.

Because latency is rarely stable, things will always be slightly desynced, which in turn means that soft-reconciliation is happening all the time via ever-so-slight modifications to your movement velocity. This shouldn't normally be noticeable. But if your desync is too big, or has been going on for too long, you would be hard-reconciled in the form of a very visible teleport.

De-syncs complicate the process of hit validation further, because if the client claims that the player landed a shot during a heavy desync period, the system has to decide if it's still a valid shot from a truly innocent client, or if it is an invalid shot from a lag switching player or cheater. Most of the time, this is a really hard distinction to make.

The hit validation system operates based on a set of tolerances. Maybe your shot would have been valid, but is deemed invalid because of network de-sync. (If you are getting worried about being banned because of a bad connection right now, don't worry: we never ban players based only on invalidated shots.) If the game is having performance issues, on the client or the server side for example, this can also increase desync and impact the predictability of the simulation.

TL;DR

Hunt allows clients to predicatively interact with their own view of the server state and uses a hit validation system to verify that shots and melee attacks are valid. This system takes the “favor the shooter" approach, meaning that a shot is considered valid as long as it is deemed legitimate according to the server's game state history, though this process is complicated by the nature of latency and the existence of de-synchronization between client and server.

The crux of our work is in improving the hit validation system so that it is lenient enough about desyncs that it doesn't invalidate legitimate shots, but strict enough that it doesn't open up the game up to exploitation.

What's Going Wrong

In the following video MrSpwn analyses the same engagement from two perspectives and identifies some issues. I'm going to go through his points to provide more context about what is happening. First, the video:

So. Starting at 2:00, MrSpwn describes a moment when his shot, which you can see on his screen, is not visible on the other player's screen. This was most likely caused by a mechanism of the hit validation system known as projectile launch validation. This step ensures that a player's gunshot came from a valid position, that is to say, matches up with the server's history of the event. If he was desynced—but not yet reconciled—at that moment, his shot could have failed the projectile launch minimum tolerance levels test.

You can see in the video that, for the most part, he maintains a ping of around 70ms, which indicates a somewhat stable connection. That means it's unlikely that it was network instability that caused the invalidation to happen.

As you know, the game has had some performance issues in the past few weeks that mostly affected your client frame rate. But because our game servers run a lot of the same code as the client, they are also vulnerable to the same performance fluctuations. This can, of course, cause desyncs, and the hit validation telemetry data we collect from the live game suggests that this is one reason for the recent spike in invalidation rates.

The other problem MrSpwn describes in his video, and one you can also observe in these two clips, is a more typical problem that stems from the nature of the “favor the shooter" approach. First, the videos:

In these situations, the server deemed that when the opponent shot you on his screen, you were still visible, but that due to the time the messages took to propagate, your local client had allowed you to move behind a wall.

What We're Doing to Fix It

With the next performance patch , we expect hit invalidation to decrease because of the game's increased stability. But our work doesn't stop there.

We are also working on further changes and adjustments to improve your experience, from tweaks to the validation system, better network synchronization algorithms, and improved hitboxes.

To help us in our efforts, we are always collecting data from the live game to measure performance. Based on this data, we make adjustments to the existing systems. Here you can see a summary of 200 matches, from not too long ago:

This isn't all we look at, though. Your feedback is essential to us during this process—especially in the form of video clips like the ones I referenced above. We will continue to collect data from the community and from our live servers to aid in our investigation and improvement efforts. If you'd like to let us know what you think, head over to the Steam forums or our Discord and let us know. We're always listening.

---