Help on Server Lag and Scripts

delpi

Well-Known Member
I setup a routine that logs my server FPS every 60 seconds and also some other values

What it shows is the fps starts at 50 with about 1 to 3 players on and then drops down to 3 by the time 16 players are on it. Ouch, or at least I think ouch. Anyone know if this is bad.

Looking at the other details.
  • 30 or so groups due mostly to my ai missions
  • 80 or so ai probably at peak time, but often less.
  • 800 or so buildings. Tracked this down and it is the serverside POI that have about 500 grass cutters in it. Going to modify that to not be so insane. I'll probably start tackling that tonight.
  • Already modified the Rocket Trash so I intercept the spawn and don't let the trash spawn, but do let the balota, stary, ect additions take place. This made a big jump in client fps
  • lowered zeds from 500 to 250. This made a small jump in client fps

Does the server fps impact the client fps?? I would think not, but not 100% on that.

Looking at the server control panel, the RAM and CPU never get loaded. I've run this on my test server at the house running on a beast of a computer and it looks like the same trend. I couldn't get 15 people on my test server, but got 5 and the trend looks similar.

Now for scripts. What script calls kill the server performance?
  • playableunits?
  • nearbyentities vs nearestobjects? On this, is there a way to detect a specific type of vehicle with nearbyentities. It always seems to fail for me unless I use generic group expressions like "Man", "Car"
  • others????
I've been going through my server routines and slowing down loops and lessening searches to make sure that isn't making an impact, but I'm not finding anything that looks bad here.


I would appreciate any help or advice.
 
something is definitely going on, when testing the repack with approx 5 heli patrols, 5 car patrols, 50-100 AI plus 5-10 missions active at any one time, and approx 300-400 buildings and 20+ players most of us sat on a nice 30-40fps.

i would start by disabling all scripts (using true/false statements and a file to control them is an easy way to turn them on/off, see the repack on how i set it up if needed) run the same fps checks with everything disabled then enable 1 by 1 to find the culprit.

i found that buildings spawned are the biggest cause of server lag for me.
 
Looking at the load, I had about 20 players, 60 ai, 800 buildings and the fps was down to 2.

Most of the 800 was part of the built in POI. I played with that last night and got it down to 300. I also did some thing with offloading my AI to get it down to 30. It seems to have made it better. However, the fps had dropped down into 20's with 8 players.

So, that still doesn't compare to yours.

I'm going to put some things in tonight to pause all loops and kill all AI separately to test. The only problem is I have to do this on my live server to have enough players to determine if it helped. With only a few people on the test server, things stay at good fps.

What did you see/do to determine that spawned buildings were the biggest source?

Now to be very clear, I'm talking Server FPS, not client FPS. Client FPS is staying good.
 
Looking at the load, I had about 20 players, 60 ai, 800 buildings and the fps was down to 2.

Most of the 800 was part of the built in POI. I played with that last night and got it down to 300. I also did some thing with offloading my AI to get it down to 30. It seems to have made it better. However, the fps had dropped down into 20's with 8 players.

So, that still doesn't compare to yours.

I'm going to put some things in tonight to pause all loops and kill all AI separately to test. The only problem is I have to do this on my live server to have enough players to determine if it helped. With only a few people on the test server, things stay at good fps.

What did you see/do to determine that spawned buildings were the biggest source?

Now to be very clear, I'm talking Server FPS, not client FPS. Client FPS is staying good.
yep server FPS stayed around 30-40 (should have mentioned that i had the debug show the server AND client FPS to all players), i disable all buildings and noticed the jump in server FPS straight away was more of a lucky find than anything else.

Did you make/place the buildings yourself? ive found that most of the premade ones contain a crap load of "clutter" thats just for looks and is not really needed.

Also ive found that AI can have a massive impact on the server performance.


so i would try the following to see if there is any improvement in server FPS

1. disable all custom buildings
2. disable all AI
3. disable any loop intensive scripts
 
what do you consider a script intensive loop?
basically anything that runs a constant check or multiple loops in a short amount of time continuously

examples of constant running loops, now depending on whats contained in them depends on there impact on performace
Code:
while {alive player} do {
....
};
Code:
while {true}do{
....
};
example of constant check loop the also builds an array and loops thru it constantly.
Code:
if (!_canDrink) then {
        _objectsWell = nearestObjects [_playerPos, [], 4];
        {
            //Check for Well
            _isWell = ["_well",str(_x),false] call fnc_inString;
            if (_isWell) then {_canDrink = true};
        } forEach _objectsWell;
    };


these are just some random examples but they can slow down the server

obviously you cant avoid these types of loops but a lot of them (especially the last example) can cause low fps or fps drops as the loop runs for every player so with 20 players you end up with 20 arrays being built, searched thru and updating constantly.
 
IMHO:
If your server is STARTING at 50fps its not the scripts and objects that are on the server.. What you have is something that is additive as time goes on. Objects being spawned as players are on that are not being deleted, self-referencing scripts that are spawning many copies ... etc.
Arma is a leaky program, Dayz makes it worse ... not sure how much you can do for it. My server would always start at 50fps and after hours of play with 10-20 players it would be down to single digits. Vehicles seemed to be one of the biggest hits for me. The Server FPS doesnt seem to have a direct correlation to the players experience though as most things are handled clientside. I think its just about normal to have single digits after a few hours which is why some servers restart every 3 hours.
 
I am starting at 50 fps and will be back up to it if players log out.

Generally the clients can maintain good FPS. A few complain about FPS in big cities, but that is sort of normal.

I build two admin scripts last night. One kills and unloads all current AI. The second pauses all scripts on the server.

I'm going to test to see what is the FPS change when fps drops next time.

I'm struggling with the spawned in buildings being the source. I lowered the count from 800 or so down to 300 by changing up the PoI vanilla code. It made Zero impact and I expected a huge one. I have one more trick that would take this to zero, but its a bit annoying to go program so I am saving that one to the bitter end.
 
I never spawned many buildings, but noticed that 200 vehicles would cause a major FPS drop. The difference between buildings and vehicles is probably the simulation .. I suspect there is very little simulation with buildings
 
There is a script for Wasteland that saves all vehicles to an array, then despawns them and respawns them when players are near. It was called something like vehicle fix ...
This could work for buildings too ... But one guy I worked on his server had about 10,000 buildings and items added to Chernarus and it didnt have a huge performance hit.
I had tested this script a while back and it did improve performance. I cant find it today, but there are a few methods it could have used:

saved vehicle data to array and removed the vehicle from map, replacing area with trigger. This would give you 200 triggers instead of vehicles which might offset any gains or actually make it slower.

Despawn vehicle, saving info to array then check player position with the array position every few seconds to see if they are within range of respawning that vehicle.

also needs a block to check if vehicle is outside of player bubble and put vehicle into array/despawn vehicle that runs every few minutes.

There is a caching system for AI that does something like above. saves teh info and respawns when they would be in player range.

Edit:, now that I am thinking about it. I believer it might have just run enablesimulation=false on every vehicle and set to true when the player was within range ....
 
Last edited:
Now i read this.

I set up an elaborate system for all my temp vehicles to spawn/despawn. Oh well, it works well.

I'll try the enablesimulation on the real stuff.
 
There is now a command "enableSimulationGlobal". Going to have to load this up and test it.
 
Nevermind, that is for arma 3. need to figure out side on own. Here is something for server/client to help others out.

Code:
// Simulation
CD_Vehicle_Simulation = {
    private ["_vehicle","_status"];
    _vehicle = _this select 0;
    _status = _this select 1;
    if (local _vehicle) then {
        _vehicle enableSimulation  _status;
    } else {
        CD_Vehicle_Simulation_Call = _this;
        if (isDedicated) then {
            (owner _vehicle) publicVariableClient "CD_Vehicle_Simulation_Call";
        } else {
            publicVariableServer "CD_Vehicle_Simulation_Call";
        };
    };
};
"CD_Vehicle_Simulation_Call" addPublicVariableEventHandler {
    (_this select 1) call CD_Vehicle_Simulation;
};
 
ShootingBlanks,
reading up it says it has no impact on static buildings. I'm assuming this is any buildings I've spawned in. It really says no network traffic. Is there a load from those spawned in static buildings?
 
As I mentioned, one guy gave me his server files to work on and it was absolutely FILLED with buildings and tank traps, bunkers, .. I really think he had like 10,000 items ... and it was a fairly popular server without significant lag. So I dont think the buildings have much impact.
for a refernce, I am working on a 3d space game right now in Unity. I have 1800 stars, 4800 planets, 1000 asteroid belts ... just a butt-ton of stuff and here is the profiler output. The green line on top is the CPU resources spent on rendering the stuff and the blue is the CPU resources spent on scripts such as making the planets orbit, asteroids move, So my point is that the scripts require significantly more than drawing ... so anything without simulation is not going to interact with the environment and therefore should have little impact.
VszUhAi.png
 
So, with a variety of people on from 10 to 20 people on the server i did a few test.

I suspended all server side script loops (done with client side variable pass to server in case you are wondering). This had no detectable impact on server fps. I didn't expect it to as I has optimized all my routines.

Then I killed all the AI. This shifted the server side fps by about 4. At the time there were around 50 ai active.

As i mentioned before by doing some magic I had dropped the vanilla loaded vehicles from 800 down to 300 with no impact.

I also put in code to delete all temporary vehicles unless a player is near, and I used your suspend trick for real vehicles. I also purged any non claimed real vehicles during the test time to be sure.

This had no impact.

I'm freaking stumped. There is no way with way I just talked about my server FPS should be that low on vanilla and certainly i should have been able to impact it with what i just did.
 
Now the main question is: Do the PLAYERS say its laggy? do they see AI running in place or not immediately able to enter vehicles?
If the answer is no, then ... maybe you just have to live with it. I recall having your same issues with FPS on one of my first servers and while it would start at 40-50fps it would drop to single digits after a short time but without any apparent effect on the players. As long as it stayed in the 6-10 fps it seemed okay. Of course maybe it was laggy and i just dont remember that .....

Possibly its a zombie issue. Zombies are spawned and deleted but some 'zombie processes' remain (yes, thats a pun). And not sure how to properly test that, disable zombie spawning and see what happens. FPS would skyrocket without the zombies anyways so a true measure might be hard to arrive at.
What about without any players, like testing locally? does the FPS drop over time or does it REQUIRE players to drop?

Why dont we ask @DangerRuss what his FPS is like at startup and after an hour or two for another reference?
 
I can leave it running on my test server with just me and my 2nd account for endless time and nothing changes. It stays at 50 fps.

When players join it starts to drop quickly.

I've got a zombie counter on my FPS log script. Even with the zombies down in the 50 to 100 range, its seeing the drop the same.

Good Pun
 
Back
Top