Misfires and failure to aim

[Cairnsy44]

Since Sunday, I have had a very high number of misfires after a full reload is complete. I also have had a high number of times when I try to aim and it just fails to do so. Either toggling between at the ready and shoulder arms a few times or nothing at all. It’s particularly gamebreaking on maps like the cornfield.

[John Jones]

Yep, had the fail to aim issue myself on Sunday. The animation would go from 'Ready' half into the aim and then recover the arms, without ever actually aiming. Happened on the Bloody lane map and I think Hooker's Push. Had to re-start the game to get it corrected. Error report sent.

[Charles Caldwell]

Ditto, on Sunday also.... restart always seems to fix the issue.

[Fancy Sweetroll]

We've been talking to G-Portal about the issue. It turns out the cause is fairly simple. We've always known that if the servers are running really really slowly, like 3 fps or less, this reload behavior starts occuring, and the lower the framerate is on the server, the more noticeable the issue becomes. Players also start lagging somewhat when moving, though our heavy interpolation compensates for it quite a bit.

Initially when we made 150 player servers available, the servers were running at about 3-4 fps where the reload issue could sometimes be experienced. But with various optimizations to the game, a 150 player server is now normally running at about 10-15 fps, consuming 70% or so of a regular 3.6 ish ghz quad core cpu.

From the beginning with G-Portal, we made it clear to them that we could only vouch for one 150 players server per quad core machine. We assumed that this policy would then be applied to all the hosted servers, but it turns out that it is only our official servers that have the benefit of being the single game server on a machine. Community hosted servers are running game servers for multiple other games as well, which ends up leaving very little CPU performance headroom for your War of Rights server. The results experienced so far may have varied a bit as the load on the CPU depends on how many players are active on the other servers. Today we discovered that this has resulted in most of the community hosted War of Rights servers running at just 1 fps, which is definately not enough for a game like War of Rights. It is also possible that this terrible performance have been the main cause of all the server crashing and clients disconnecting issues we've been experiencing lately. The cpu on the server being busy with many other things increases the risks of clients timing out or some of the various threads the server is running to malfunction as the hardware is too stressed.

We are still talking to G-Portal about how best to resolve this issue, but currently, it'll be something like, moving the War of Rights servers to different hardware where less game servers are sharing the particular hardware. This hardware has the same CPU, but less RAM. I currently dont think the War of Rights server will be the only server running on this hardware, it will still be shared, but to a much more reasonable extent. This however also means less customers for G-Portal for their hardware, which results in them unfortunately having to raise the prices. We currently don't know by how much as all of this have first been discovered and set in motion today, but from the way we see it, better pay a little more for something that actually works, than paying for something that doesnt really work. As this is harming our game quite a bit and us not earning anything from you hosting servers through G-Portal, we are hoping to have this resolved as quickly as possible.

But due to this, we will probably also soon make it possible for you to host your own dedicated servers manually from whatever dedicated server provider you prefer where you will have direct access to it via Windows Remote Desktop, prices and services can however vary a lot with these types of dedicated servers.

These issues also only apply to community hosted servers. It gives me a bad taste in my mouth to say this, but our official War of Rights servers do have exclusivity to the hardware and does not share the hardware with other games and does function properly. But even when this have been fully resolved, the servers can of course still crash on occasion and clients can be randomly disconnected for various reason, but it should happen a lot less frequently.

We do apologize for the inconvenience.

[Oleander]

I've heard that some games have built in settings, or presets or something like that, that is essentially a server build that uses a lower amount of resources to improve performance. One that comes to mind is Run 8 which is an extreme CPU hog, but they built in a radio check box in the graphics setting which put the game into an ultra low res setting that is basically unplayable for the average user, but is still able to keep track of movements and player positions. Is that something that might be possible for WoR?

[TrustyJam]

I've heard that some games have built in settings, or presets or something like that, that is essentially a server build that uses a lower amount of resources to improve performance. One that comes to mind is Run 8 which is an extreme CPU hog, but they built in a radio check box in the graphics setting which put the game into an ultra low res setting that is basically unplayable for the average user, but is still able to keep track of movements and player positions. Is that something that might be possible for WoR?

There are no graphics involved in the server performance so not sure I understand your suggestion.

The dedicated server window is entirely text based.

The main issue is not bad server optimization - it’s the fact that G-Portal ran 16 game servers (15 from other games such as Ark, Atlas and Conan) instead of 1 on a single machine (again, the official WoR servers are the exception to this).

- Trusty

[Fancy Sweetroll]

I've heard that some games have built in settings, or presets or something like that, that is essentially a server build that uses a lower amount of resources to improve performance. One that comes to mind is Run 8 which is an extreme CPU hog, but they built in a radio check box in the graphics setting which put the game into an ultra low res setting that is basically unplayable for the average user, but is still able to keep track of movements and player positions. Is that something that might be possible for WoR?

The server doesnt have a renderer so there are no graphics, or sound. The server is really just a console application. Very similar to using command prompt in Windows. A lot of the client specific functionality is also not being compiled for the server (the triggering of footsteps, feet on characters aligning to the terrain, turning dead players into corpses and many other things)

[Redleader]

So the server got demanded to much on the resource department.
I always found it funny you could mess around with virtual ram/diskspace on a server, it never bode well with me.

The community will be very interested in finding a solution and/or a reliable(affordable) partner and a E.T.A.

[Oleander]

The server doesnt have a renderer so there are no graphics, or sound. The server is really just a console application. Very similar to using command prompt in Windows. A lot of the client specific functionality is also not being compiled for the server (the triggering of footsteps, feet on characters aligning to the terrain, turning dead players into corpses and many other things)
That's what I was wondering. I missed the part about community servers sharing machines with other games on the first read through, that's why I was scratching my head about CPU headroom.

[Drazion]

Good to hear that you're considering releasing the server packages, unfortunate it's windows based, was hoping there'd be Unix server support, but happy to hear it either way.

[TimTracey]

But due to this, we will probably also soon make it possible for you to host your own dedicated servers manually from whatever dedicated server provider you prefer where you will have direct access to it via Windows Remote Desktop, prices and services can however vary a lot with these types of dedicated servers.


This is a Happy New Year.
I look forward to self hosting!

Cheers!

[Poorlaggedman]

Is the server processor burnout mostly as a result of player counts alone? I'm just trying to gauge from this how realistic even larger servers will be one day or whether the costs will get truly prohibitive without radical optimization.

[Drazion]

Is the server processor burnout mostly as a result of player counts alone? I'm just trying to gauge from this how realistic even larger servers will be one day or whether the costs will get truly prohibitive without radical optimization.

It's likely due to the processor/cache and/or RAM being maxed out because it's being shared by who knows how many servers on one machine. I've hosted several multiplayer servers on DigitalOcean's Unix servers, and even with high (relatively) player counts and their $20/month server (4GB RAM 2CPUs) there were no performance issues when we were pushing 50 players. Having said that, if you look at articles it's a lot of "it depends". I'll use an example from a Conan Exiles server I hosted, for 75 simulatenous players they suggested

A 64 bit Windows 7 or higher operating system
Intel Corei7 6700 3.4 to 4 GHz or higher processor.
Minimum 12GB RAM and 35GB of Hard drive

But I would imagine that Conan Exiles has a higher "tick" rate than a relatively slow paced game like WoR.

All of that to say that a moderately powerful server with a decent amount of RAM and 1-2 good processors should be able to handle a 150 player server if it's hosted all by itself and is not competing with who knows how many other server instances on the same machine.

**the digital ocean server was 7 Days to Die not Conan Exiles

[LaBelle]

Could we feasibly up the player count to 200 if the server were hosted on a sufficient box?

[Fancy Sweetroll]

Good to hear that you're considering releasing the server packages, unfortunate it's windows based, was hoping there'd be Unix server support, but happy to hear it either way.

We are considering making the server available on Linux systems as well, though I wouldnt count on it being available anytime soon.


Could we feasibly up the player count to 200 if the server were hosted on a sufficient box?

When a War of Rights server is the only intense process running on a quad core 3.8 ish ghz cpu (Intel Xeon 1270v6 in our case), the server would probably be able to handle it. It would of course run slower than with 150 players, but the performance might be acceptable enough. It is of course untested so there is no way to say for sure. The primary issue we are facing right now is the performance on some clients probably not being up to par. If you have a 7700K CPU clocked to about 4.5 ghz on all cores or higher, it probably wont be an issue at all, but for players running with slower CPUs, their GPU might end up being utilized only by 25% as the CPU cant keep it fed, significantly hurting the framerate. We are still optimizing the game.


Is the server processor burnout mostly as a result of player counts alone? I'm just trying to gauge from this how realistic even larger servers will be one day or whether the costs will get truly prohibitive without radical optimization.

When the server is empty, the server is mostly idling at 0% cpu utilization. As the 150 player server starts filling up, some of the cores starts being fully utilized, resulting in more or less 70% cpu utilization. The server uses less than 4 gb of ram. The systems that were running the servers previously had 64 GB of ram with 85% of it being used more or less, by the various other game servers running on the system, in this particular case, there were 15 other game servers.

According to G-Portal they have actually already moved the larger 100+ player servers onto their own system where they are the only server running. The smaller servers might still be sharing hardware with other small War of Rights servers. We are yet to find out what the prices will be.

[One_Man_army]

We are considering making the server available on Linux systems as well, though I wouldnt count on it being available anytime soon.



When a War of Rights server is the only intense process running on a quad core 3.8 ish ghz cpu (Intel Xeon 1270v6 in our case), the server would probably be able to handle it. It would of course run slower than with 150 players, but the performance might be acceptable enough. It is of course untested so there is no way to say for sure. The primary issue we are facing right now is the performance on some clients probably not being up to par. If you have a 7700K CPU clocked to about 4.5 ghz on all cores or higher, it probably wont be an issue at all, but for players running with slower CPUs, their GPU might end up being utilized only by 25% as the CPU cant keep it fed, significantly hurting the framerate. We are still optimizing the game.



When the server is empty, the server is mostly idling at 0% cpu utilization. As the 150 player server starts filling up, some of the cores starts being fully utilized, resulting in more or less 70% cpu utilization. The server uses less than 4 gb of ram. The systems that were running the servers previously had 64 GB of ram with 85% of it being used more or less, by the various other game servers running on the system, in this particular case, there were 15 other game servers.

According to G-Portal they have actually already moved the larger 100+ player servers onto their own system where they are the only server running. The smaller servers might still be sharing hardware with other small War of Rights servers. We are yet to find out what the prices will be.



This sounds like the hosting provider is over-utilizing CPU and possibly disk I/O on the servers. They should have monitoring in place for this over-utilization and in turn bump up the specs to cope with the load or migrate to a more cpu heavy box. This sounds like the provider is cheaping out by trying to cram anything they can and not monitoring.

A quick google search shows people complaining for years about terrible performance with gportal in particular.. Here is just 1 example.
https://steamcommunity.com/app/440900/discussions/0/133257324792395435/

[Chris_CHA]

Not sure if there's any causation, but I've found there to be a correlation in server crashes on servers running the Antietam map. The servers - both official and community - will frequently crash during or between skirmish maps that have drastic time and weather changes.

I'll be making a new thread in Bug Reports with more info as I finish making my notes and see if there is any other relationship between these events other than what I've stated here. I'll also see if I can consistently reproduce the issue.