Memory Leak with latest update

Fox

Fox

Well-Known Member
Yesterday we tried out the new 1.5 after a few hours we get a out of memory since the new update...


Cheers fox

Galaxy S2 using tapatalk
 
Servers running through night wirh no out of memory !

Yeah might the fix did it for me ,thx sarge (1.5.2).
 
I experienced this last night. Downloaded the 1.5.1 files and updated the server. Setup the configuration again. Seemed to be going Ok. I logged off for the night after playing for about 4 hours. Woke up to a bunch of text messages and teamspeak pokes saying the server had died. Logged in and saw an out of memory error. I've downgraded to 1.1 again until this is resolved.
 
ok let me help u !

I loaded the 1.52. stuff and all the other files ,moved it to the mission.pbo and started the server .
That worked for me ,try it if u fail u can contact me.

btw map is chernarus

Currently trying to maintain some things over the sarge config over our cherno server ,seems my ControlCenter is buggy sometimes...


https://github.com/Swiss-Sarge/SAR_AI-1.5.0
 
I downloaded the files from the GitHub and installed them approximately 14 hours ago. When I did it, GitHub said the last updates were 8 hours prior.

I got the Memory Leak after the update. Server had only been running for about 4 hours when it crashed. Not using any absurd AI numbers or percentages. I can paste my spawn config if need be.

I have 8GB of RAM dedicated to this server, with a 4GB paging file. I could increase the page file size, but that's not the solution here.


@Sarge: Is there anything specific that I can provide you to help track this issue down? With this kind of issue I'm not sure where to look or what to provide.
 
Well, thats a weird one ...

my testserver runs 24/7 and i never ran into the oom error. How many players do you have in average on your server ? anything in the rpt before the crash that looks weird ?

I'd suggest turning on the debug levels, and let it run again. post the last 200 lines before the crash.

Is this only happening wit SARGE AI enabled, and only happening with the latest version ?

Thx for a feedback,

cheers, Sarge
 
Hey Sarge,

First I just want to say you're an awesome mod developer. Quick, responsive, polite, etc. Very much appreciated.

So the OOM error only happened after I upgraded to version 1.5.2, at least that's the first time I experienced it. I have a 50 person server, and there were between 8-12 players on last night when it happened. With version 1.1 I never had this issue, I could leave the server up for days and it would be fine.

I grabbed some stuff from my RPT file that occurred just before the OOM error. See quoted code below. There 36,436 "Unrecognized CfgNonAIVehicles simulation" lines that were spammed in 1 hour 38 minutes. During this time the server appeared to be up and running, but no one could join. Leading up to this players reported to me that if they died they could not spawn in again, and if they disconnected and tried reconnecting they'd just get stuck at the loading screen.


Leading up to the spamming and OOM...
Code: 
14:29:04 "KRON_UPS_AI_Debug: Nearest Enemy for O 1-1-L:2 is R 1-1-L:3, knows about value is 4"
14:29:04 "KRON_UPS_AI_Debug: Nearest Enemy for O 1-1-L:3 is R 1-1-L:3, knows about value is 4"
14:29:15 Server: Object 9:1136 not found (message 94)
14:29:26 Server: Object 22:398 not found (message 70)
14:29:58 Server: Object 22:400 not found (message 94)
14:30:15 Server: Object 22:401 not found (message 94)
14:30:31 Server: Object 22:402 not found (message 70)
14:30:36 Server: Object 22:403 not found (message 70)
14:30:42 Server: Object 22:404 not found (message 70)
14:30:46 Server: Object 22:405 not found (message 70)
14:30:50 Server: Object 22:406 not found (message 70)
14:30:56 Server: Object 22:407 not found (message 70)
14:31:05 Server: Object 22:408 not found (message 94)
14:31:12 Client: Remote object 29:0 not found
14:31:12 Client: Object 24:150 (type Type_149) not found.
14:31:12 Client: Object 27:227 (type Type_149) not found.
14:31:33 Server: Object 22:410 not found (message 70)
14:32:07 "DEBUG FPS: 9.50119"
14:32:08 Server: Object 22:414 not found (message 94)
14:32:18 "PLAYER: DISCONNECT: Synder (120955846) Object: B 1-1-H:1 (Synder) REMOTE, Character ID: 355"
14:32:24 Client: Remote object 29:3 not found
14:32:24 Client: Remote object 30:0 not found
14:33:26 Client: Remote object 2:12165 not found
14:35:42 Client: Remote object 31:3 not found
14:35:42 Client: Remote object 32:0 not found
14:35:50 Client: Remote object 2:12173 not found
14:35:50 Client: Remote object 2:12174 not found
14:35:50 Client: Remote object 2:12175 not found
14:35:50 Client: Remote object 31:0 not found
14:36:50 Client: Remote object 2:12185 not found
14:37:38 Client: Remote object 33:0 not found
14:37:38 Client: Remote object 32:5 not found
14:38:48 "PLAYER: SYNC FAILED: No Player ID"
14:39:10 Server: Object 22:498 not found (message 91)
14:39:10 Server: Object 22:497 not found (message 98)
14:39:10 Server: Object 22:499 not found (message 99)
14:39:10 Server: Object 22:500 not found (message 91)
14:39:10 Server: Object 22:502 not found (message 91)
14:39:10 Server: Object 22:501 not found (message 98)
14:41:09 Client: Remote object 2:12196 not found
14:41:09 Client: Remote object 35:0 not found
14:42:48 Client: Remote object 34:0 not found
14:42:48 Client: Remote object 35:3 not found
14:43:32 "PLAYER: DISCONNECT: patelbrudda (10971136) Object: 46264080# 1177899: man_survivor.p3d REMOTE, Character ID: 353"
14:43:32 Client: Remote object 27:0 not found
14:44:18 "PLAYER: DISCONNECT: Leif (122390918) Object: B 1-1-D:1 (Leif) REMOTE, Character ID: 354"
14:44:18 Client: Remote object 25:0 not found
14:44:18 Client: Remote object 25:6 not found
14:45:28 Warning: Cleanup player - person 25:5 not found
14:45:31 Server: Object 22:729 not found (message 99)
14:45:31 Server: Object 22:730 not found (message 91)
14:45:31 Server: Object 22:732 not found (message 91)
14:45:31 Server: Object 22:731 not found (message 99)
14:45:32 Warning: Cleanup player - person 25:5 not found
14:46:19 "PLAYER: DISCONNECT: Tpeso (121355910) Object: 3b9ae040# 1110028: man_hero.p3d REMOTE, Character ID: 320"
14:46:19 Client: Remote object 22:0 not found
14:46:19 Client: Remote object 22:753 not found
14:46:19 Client: Remote object 22:752 not found
14:47:01 Server: Object 9:1442 not found (message 98)
14:47:01 Server: Object 9:1443 not found (message 91)
14:47:02 Server: Object 9:1452 not found (message 98)
14:47:02 Server: Object 9:1453 not found (message 91)
14:47:28 Server: Object 22:6 not found (message 221)
14:47:31 Warning: Cleanup player - person 22:754 not found
14:48:33 "SAR_HITKILL_DEBUG: AI hit - 4b074080# 1181234: woman_bandit.p3d - Type: BanditW1_DZ Side: CIV Group Side: UNKNOWN"
14:48:33 "SAR_HITKILL_DEBUG: AI attacker - Type: Rocket_DZ Name: Sarge Side: GUER Group Side: GUER"
14:48:48 No player found for channel 573497472 - message ignored
14:48:48 No player found for channel 573497472 - message ignored
14:48:53 Client: Remote object 2:12204 not found
14:48:53 Client: Remote object 37:0 not found
14:48:53 Unit 2:12204 not found, cannot update
14:49:42 "SAR_HITKILL_DEBUG: AI hit - 4b074080# 1181234: woman_bandit.p3d - Type: BanditW1_DZ Side: CIV Group Side: UNKNOWN"
14:49:42 "SAR_HITKILL_DEBUG: AI attacker - Type: Rocket_DZ Name: Sarge Side: GUER Group Side: GUER"
14:50:18 "PLAYER: DISCONNECT: Tpeso (121355910) Object: 3b9ae040# 1110028: man_hero.p3d, Character ID: 320"
14:50:18 Client: Remote object 38:3 not found
14:50:18 Client: Remote object 39:0 not found
14:53:00 "SAR_HITKILL_DEBUG: AI hit - O 1-1-G:4 - Type: Bandit1_DZ Side: EAST Group Side: EAST"
14:53:00 "SAR_HITKILL_DEBUG: AI attacker - Type: Survivor2_DZ Name: Survivor2_DZ Side: CIV Group Side: UNKNOWN"
14:53:23 Client: Remote object 37:3 not found
14:53:23 Client: Remote object 38:0 not found
14:53:59 "PLAYER: DISCONNECT: Tpeso (121355910) Object: 3b9ae040# 1110028: man_hero.p3d, Character ID: 320"
14:53:59 Client: Remote object 38:5 not found
14:54:25 "SAR_HITKILL_DEBUG: AI hit - 4b074080# 1181234: woman_bandit.p3d - Type: BanditW1_DZ Side: CIV Group Side: UNKNOWN"
14:54:25 "SAR_HITKILL_DEBUG: AI attacker - Type: Rocket_DZ Name: Sarge Side: GUER Group Side: GUER"
14:54:26 "SAR_HITKILL_DEBUG: AI killed - Type: BanditW1_DZ Side: CIV Group Side: UNKNOWN"
14:54:26 "SAR_HITKILL_DEBUG: AI Killer - Type: Rocket_DZ Name: Sarge Side: GUER Group Side: GUER"
14:55:09 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_PowLines_ConcL/
14:55:29 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_lampa_ind/
14:55:33 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_PowLines_WoodL/


This is a sample of the 36,436 lines that were spammed. You can see it's the same stuff over and over and over...
Code: 
16:17:12 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/StreetLamp_BaseWeakYellow/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_PowLines_WoodL/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/StreetLamp_BaseWeakYellow/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/StreetLamp/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_lampa_ind_zebr/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/StreetLamp/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_PowLines_WoodL/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/StreetLamp_BaseWeakYellow/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_lampa_sidl/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_PowLines_ConcL/
16:17:13 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/StreetLamp_BaseMediumPale/
16:17:14 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_lampa_sidl/
16:17:14 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/StreetLamp/
16:17:14 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/Land_PowLines_WoodL/
16:17:14 Unrecognized CfgNonAIVehicles simulation thing in bin\config.bin/CfgNonAIVehicles/StreetLamp_BaseWeakYellow/


And finally here's the crash. The lines of spam suddenly stopped and after the last one this is what appeared...
Code: 
16:17:15 Virtual memory total 4095 MB (4294836224 B)
16:17:15 Virtual memory free 218 MB (229527552 B)
16:17:15 Physical memory free 1481 MB (1553854464 B)
16:17:15 Page file free 7945 MB (8331288576 B)
16:17:15 Process working set 3625 MB (3801694208 B)
16:17:15 Process page file used 3646 MB (3823697920 B)
16:17:15 Longest free VM region: 22646784 B
16:17:15 VM busy 4082409472 B (reserved 192532480 B, committed 3889876992 B, mapped 43622400 B), free 212426752 B
16:17:15 Small mapped regions: 12, size 57344 B
ErrorMessage: Out of memory (requested 25246 KB).
  footprint 3432513536 KB.
  pages 327680 KB.

If there's anything else you require or want to see just let me know.
 
thx a lot for these mate.

a) disable streetlights and houselights to see if the "unrecognized" messages go away.
b) there ARE issues with my AI, e.g. the side / group side values should never be "UNKNOWN". They may be "CIV" in the hit messages, thats when a unit is dead, but they shouldnt be UNKNOWN.
c) your server fps are very very low in that snippet - 9 FPS means severe lag.
d) which numbers of AI are you running ?
e) i'd recommend to restart a DayZ server every 8 -12 hours, it is well known that the Dayz Code and Arma 2 tend to bug out after a while.

Sarge
 
A) I'll Google how to do that, but I might come back asking how.

B) Is there anything you want me to do for this, or should I be adjusting something?

C) The low FPS might be from my Virtual Host. I'm running this server on VMWare ESXi 5.1. I have a funny feeling it's having slow HDD access speeds. I'll try it on a dedicated box without VMWare and see if performance increases.

D) I'm running low group and member counts per grid. Usually 2-3 groups of 2-4 AI per grid and at most a 50% chance to spawn the group. Nothing too crazy.

E) I usually restart my server once every 24-hours for fresh crashed heli spawns. Prior to 1.5, when I was running 1.1, I'd leave my server running for 24+ hours sometimes without any issues.

Just as a note, the only other MODs I have installed are the Refuel MOD and the BaseBuilding 1.2 MOD which I'll probably be taking off soon as it's really poorly coded, and causes all sorts of DB nightmares and the author has already indicated he won't be working on it anymore or supporting it.
 
to me it looks as if you have the streetlights mod enabled - can't explain the 30 k lines with "unrecognized" otherwise ... can you recheck that ?

You cant do anything about the "UNKNOWN" groups, something goes wrong while they are active ...

Sarge
 
The only mods I had installed at the time of that log dump was:
  • Sarge AI 1.5.2
  • Auto Refuel
  • BaseBuilding 1.2
I'm using DayZCC as my Private Hive server software.

I'm going to do a vanilla install of the software on a dedicated box (not a VM but a full physical system) tonight. I'll install only the Refuel mod and the latest SargeAI files from GitHub. I'll get my group of guys to play on it for a few hours and see what we come up with. I'll report any odd occurrences.

If after 12+ hours nothing happens then I'll assume something with one of the other mods has buggered up my Public Chernarus server and I'll re-install it from scratch and just add 1 mod at a time for 24 each till something breaks.

Do you have a "beta" channel for the SargeAI development. Me and my group of friends play at least 6+ hours nightly. We can beta test the crap out of anything you want.
 
Installed ARMAII & DayZCC on a dedicated system last night. Put the refuel mod and your 1.5.2 AI on the server. Ran it for 6+ hours with a minimum of 6 players on the server. Copied the AI config files from my primary server to the test one. Encountered no issues at all. Checked the RPT file in the morning and nothing spammed, no memory leak, and no errors of any kind. AI kills log properly to the DB tables, and humanity is affected by AI kills.

I may need to "reset" my primary server and move it off of a Virtual Host to a dedicated machine. I'll do more testing and see what happens.

Thanks for all your help and your amazing support.
 
Thought I would add. The Street lights, shown in the log, are never referenced. A lightpoint is created at the location of the lamp from a map of positions that has previously been made. It shouldn't be the lights due to the lack of referencing the object.

Those 'simulation things' entries are the usual ones when you fire a server up.

the 'server object not found' entries I get my log spammed when I spawn an apache in. .

Do you have people hacking stuff in. Have had server fps issues when logging too much for debug during development. Completely crashed my physical test server with a loop logging to the .rpt file. .
 
As far as I know, no one has hacked anything into the server. But some strange stuff has been happening vehicle wise recently on my primary server. Disappearing from the database, but still in game until the server restarts, then suddenly it's gone from in game too. I think my server is in need of a hard reset, and to take it off the VM Host it's currently running on and put it on a dedicated physical box.

I have a feeling the Basebuilding 1.2 mod has a lot to do with it. There was a lot of modifications to dayz_code files in order to get it to work, and it never seemed to work 100% of the time either.
 
If a vehicle goes from the database the game will not delete it until restart, is cached in game. You should see errors in your log as the game tries to update the vehicle as people add/remove from the inventory, if it no longer exists in the DB.

Gotta ask yourself why the entry has gone from the DB, something / someone has removed it..?
 
axeman said:
If a vehicle goes from the database the game will not delete it until restart, is cached in game. You should see errors in your log as the game tries to update the vehicle as people add/remove from the inventory, if it no longer exists in the DB.

Gotta ask yourself why the entry has gone from the DB, something / someone has removed it..?
Click to expand...

Ya, that's exactly what I'm asking myself.

It was funny, because I saw it on the map of my web panel for my server, but then it disappeared about 10 minutes later. Not sure why. I looked through the logs and saw the errors about not being able to update the inventory in the DB. But didn't see anything about it being deleted from the inventory. So I'm not sure what happened.
 
Seriosly since this night its back and we dont know why . Should we post our rpt.log ,sarge ?

Cheers

Galaxy S2 using tapatalk
 
You must log in or register to reply here.
Back
Top