Jump to content
ALTIS LIFE
...
GTA RP
...
TEAMSPEAK
...

We are currently undergoing maintenance

The forums are available for use, yet please do not expect full functionality.

Ciaran

Altis Life UK Server Development - Feedback

Recommended Posts

UndamagedVirus
6 hours ago, TinyBigJacko said:

Some feedback on the 'mass kicks' and 'random kicks' issues.

Firstly, be assured that we are working on this pretty much fulltime at the moment. I am doing virtually nothing else right now, and have so many windows open on my various development machines, that I am beginning to think I will never see sunlight again.

We have two ongoing problems, as far as we can tell.

The first, the 'kicks with no reason' or the 'random kicks' is the situation where a player gets kicked off, and is given the simple, rather blunt message "you were kicked off the game". No other information, no mention of admins, Battleye, or anything approximating a reason, just 'you were kicked off'. Really annoyingly, our RCON system logs this purely as a 'player disconnected' (which of course, he or she did, but not voluntarily!), and mentions nothing about it being a kick. Even the ARMA3 server log records the same - a 'player left' notification, with no hint of any kick or kick-reason. It is totally inseparable and indistinguishable from a normal 'player leaving' notification, which of course makes debugging virtually impossible (because players genuinely leave the game hundreds of times a session - people come and go).

Our only current angle with debugging these (from an 'after-the-fact' perspective) is the realisation that in-game, other players will see the message 'Player X was kicked off the game' (or something similar) and then the 'player X disconnected' message below it, all in the system-chat on the lower left-hand side. This is our only clue that this was a 'random kick' and not just a player leaving of their own volition.

What I'm trying to do currently is gather many of these incidents via a packet-sniffer running on the server, by re-running and recycling a capture-buffer, until one of these reports comes up in system-chat. Then I save the capture, take notes about the player's 'drop' time, their IP address, and suchlike, and then set out analysing their packet stream in the moments preceding their kick, to see if there is anything obvious.

The only fly in the ointment, of course, is the fact that Bohemia have encrypted the UDP packet-data in order to stop people injecting cheats into the data-stream, and this currently means that all the packets we capture are effectively indecipherable, in terms of working out exactly what the server and the player's client thought was being 'discussed', in game-data terms. We're currently trying to source a method of decoding these packets reliably, in case it sheds any light on the problem, but of course, finding this information out, and a Wireshark ARMA3 UDP packet-decoder (one which actually works and is current), involves visits to the 'dark side', which is of course not something we're very keen on. Steps are being taken, though...

Meanwhile we're still gathering data in the hope that it will give us something to go on, simply by virtue of the data-sequences that we see repeated, during and up to a random, no-message kick. Patterns still emerge, even if the precise detail is obscured, sometimes.

Then there is the 'mass-kick' problem... at first, we thought these were due to our own game script code, maybe something we'd added in a recent update. We have pulled out all sorts of features over recent days, in the hope of stopping the problem. We tuned down audio codecs (resulting in loads of people complaining about poor audio), we turned off weather, we tweaked vehicle management, altered loads of Battleye script detections, and, well, did too many things to list, really. Obviously this is an extremely slow and laborious process - turn off one thing in the development repo, wait for a server restart window, update the mission-code on the server, restart, sit back and wait, and hope that the kicks have gone. Rinse and repeat when we discover that they haven't gone, and came back when the server was next fully populated (which might be 24 hours away from the last update, depending on the time!) Then reset everything and try again. Mind-numbingly dull and frustrating, when it doesn't resolve the issue solidly. Especially when the mass-kicks sometimes seem to vanish for a couple of days; maybe due to population levels, maybe due to some person simpy hitting 'the magic trigger' in game (whatever it turns out to be). We don't know yet... but we will.

Worse, with the 'mass-kicks' we are unable to grab a complete packet-dump during the time of the kicks, because the server is flooding out many megabits of data per second, and it blows up the protocol analyser's capture-buffer, and leaves us with nothing. Totally frustrating. Many rude words have been said and invented over recent days, specifically due to this. Seeing the packet storms, we wondered if we were being DDOSed, but not getting reports from our server host. We actually changed to a different server host in order to run some comparative tests, and then we discovered (by some helpful input from our host) that the packet-storms were being generated internally, and not coming from outside at all! Basically our own server was going into 'scream mode' - and in the process, deafening itself, and shutting out Battleye. This in turn results in large chunks of the player-base losing contact with the server, which Battleye interprets as a 'no response'... and eventually, it's Battleye that does the kicking. Plonk! Anywhere from 3 to 100 players all get kicked with Battleye: Client Not Responding at exactly the same time - which itself, may actually be several minutes or seconds after the actual source of the problem (such is the nature of timeout periods triggering).

As you can imagine, we've tried a hundred different firewall configurations, server config tweaks, bandwidth filter modifications or removals, all to no avail, so far. I have a hunch that maybe the 'random no-message kicks' are possibly related (maybe piling up at certain points, and bottlenecking the server or something), but it is just a hunch, really. We'll know more when we are able to conduct some real in-depth packet-data analysis, I hope.

One thing I did discover is that this has been going on a long time. Much longer than I imagined, and the effect appears to have increased recently. I conducted some data-analysis on our ARMA3 logfiles, gathered since last August, and scoured every single one for all mentions of 'Battleye Client Not Responding' kicks. Obviously some of these will legitimately be just people getting kicked for having crummy network ping-rates or lossy streams... but it was still a valid exercise. I was able to pinpoint the first date where we saw multiple people all being kicked for 'Battleye Client Not Responding' at the exact same date, hour, minute, and second. And from that date, track an overall rise, to the current date.

The date it all seems to have started was 11th October 2016 - the date we released our big Altis update, which pulled in a great many features from the Tanoa build. Of course, our immediate thought was that it was some of the Tanoa core code, and we've set about disabling (temporarily) various features that might be an issue, or rewriting certain loops and routines where it's essential we keep that code in place, in such a way that they cannot 'lock' the server for any length of time. Not that we're certain this is happening, but it's good practice, and development is an iterative learning process, so we've learned plenty even since then, plus we've had new people like Rath join us, bringing ever more knowledge to the table.

However, it turns out it may not even be our code after all. The dice are still rolling. On the 11th October, Bohemia Interactive released a Hotfix to the game-engine. I remember it well; Wilco and myself laughing at the timing, because it happened literally minutes before we were holding down the servers in order to update the mission-file, and we knew full well that this was going to make for a muddied audit-trail in the future, if ever anything went wrong. We'd never know whether the fault was ours, or Bohemia's... and now here we are six months down the line, wincing and laughing wryly, as we look back at that date. 

I'm not going to say at this point, that this is an ARMA problem. It might be. It might not be. We simply do not know yet. But we do know (as of only a day or two ago) that we are not alone in suffering it, and this is a possibly crucial turning point. We have some friends in other large ARMA communities - some running Life games, some not, plus we have some not-so-friendlies in other communities, and they make server reports just like this one, from time to time. Turns out that some of the communities running larger-population servers are suffering the exact-same 'mass kick' Battleye Client Not Responding problem as we are. Identical. We're currently in the process of trying to establish whether certain specific, recent (as of October last year) enhancements to the ARMA scripting are being used on their games, as they are on ours - things like SimpleObjects, for example, and whether they are also suffering from the quiet-but-constant residual 'no-message random kicks' too. 

We're hopeful that we can find the root cause, and nail it. Maybe by information-sharing with a few other cool and clever people in the wider community, we can move faster, but in any event, we're still doing all we can to get to the bottom of it. Admin staff will, of course, just tell you that everything which can be done is being done, and they won't have any answers about dates of fixes, or reasons, nor will they want to hear you cussing them out, or cussing out the dev-team or management, for fairly obvious reasons. They're busy trying to hold it all together during this process, and continue their job of managing the players and ensuring fair play. Please support them in this objective, and realise that they are not being 'indifferent' or 'hostile' by not telling you exactly what's going on. They're doing their job as well as they can, in the circumstances.

However, I wanted to make this post just so that you knew a little bit more about what was going on down here in the BatCave, and what we're trying to do to resolve the problem. Of course I can empathise with you, if you're affected, and losing play progress or gear due to a random kick or being inconvenienced massively during a mass-kick and losing all your fellow roleplayers! What I cannot yet do, though, is give you promises of a date when it will stop. I can promise you that we're doing a great deal of work to get to the bottom of it - work which, if I'm honest, Bohemia really ought to be helping us (and others) to do more easily... but we're well-used to being left out here on a limb at the sharp-end by now, so I'm not going to complain about that too much! It achieves nothing, and we have some strategies in that 'direct-approach' department that we're about to use...

All I can say for now, is sit tight, weather the storm, and keep the faith. RPUK will always do its best for you, the player.

As Liam Neeson (almost) said: "Glitch, I don't know what you are. I don't know what you want. If you are looking for ransom, I can tell you I don't have money. But what I do have is a very particular set of skills; skills I have acquired over a very long career. Skills that make me a nightmare for glitches like you. I will look for you, I will find you, and I will kill you."

To all the devs. 

Take as long as you need to guys! I hope no one is blaming you guys!

 

Kepp up the good work <3

  • Like 2

Share this post


Link to post
FriendlyFireGuy
7 minutes ago, lorenzo12314 said:

im liking all the new updates, about time i guess

About time? What are you hoping to achieve from that comment? The devs here spend their free time making this place great, they don't get paid, they work their arses off and they do it to provide YOU and ME an environment to enjoy. Sometimes things go wrong, sometimes things happen, but LOOK, People like TBJ spend THEIR time to explain the situation rather than keep us all in the dark - The least you could say is thanks.

  • Like 15

Share this post


Link to post
Imstyle

@TinyBigJacko First and foremost I would like to say a big thank you to you for given such an informational explanation upon the servers issues as of late.

secondary I would like to remind everybody that no matter how frustrating the server issues are they are also frustrating to the management and the development Team we all know how frustrating Arma and it's game mechanics can be but  server hosting and coding of Arma can be 100 times more frustrating so please give the staff some thanks and appreciate how deep of an effort they are actually trying to make to resolve this issue

  • Like 7

Share this post


Link to post
Jordy
10 hours ago, TinyBigJacko said:

Some feedback on the 'mass kicks' and 'random kicks' issues.

Firstly, be assured that we are working on this pretty much fulltime at the moment. I am doing virtually nothing else right now, and have so many windows open on my various development machines, that I am beginning to think I will never see sunlight again.

We have two ongoing problems, as far as we can tell.

The first, the 'kicks with no reason' or the 'random kicks' is the situation where a player gets kicked off, and is given the simple, rather blunt message "you were kicked off the game". No other information, no mention of admins, Battleye, or anything approximating a reason, just 'you were kicked off'. Really annoyingly, our RCON system logs this purely as a 'player disconnected' (which of course, he or she did, but not voluntarily!), and mentions nothing about it being a kick. Even the ARMA3 server log records the same - a 'player left' notification, with no hint of any kick or kick-reason. It is totally inseparable and indistinguishable from a normal 'player leaving' notification, which of course makes debugging virtually impossible (because players genuinely leave the game hundreds of times a session - people come and go).

Our only current angle with debugging these (from an 'after-the-fact' perspective) is the realisation that in-game, other players will see the message 'Player X was kicked off the game' (or something similar) and then the 'player X disconnected' message below it, all in the system-chat on the lower left-hand side. This is our only clue that this was a 'random kick' and not just a player leaving of their own volition.

What I'm trying to do currently is gather many of these incidents via a packet-sniffer running on the server, by re-running and recycling a capture-buffer, until one of these reports comes up in system-chat. Then I save the capture, take notes about the player's 'drop' time, their IP address, and suchlike, and then set out analysing their packet stream in the moments preceding their kick, to see if there is anything obvious.

The only fly in the ointment, of course, is the fact that Bohemia have encrypted the UDP packet-data in order to stop people injecting cheats into the data-stream, and this currently means that all the packets we capture are effectively indecipherable, in terms of working out exactly what the server and the player's client thought was being 'discussed', in game-data terms. We're currently trying to source a method of decoding these packets reliably, in case it sheds any light on the problem, but of course, finding this information out, and a Wireshark ARMA3 UDP packet-decoder (one which actually works and is current), involves visits to the 'dark side', which is of course not something we're very keen on. Steps are being taken, though...

Meanwhile we're still gathering data in the hope that it will give us something to go on, simply by virtue of the data-sequences that we see repeated, during and up to a random, no-message kick. Patterns still emerge, even if the precise detail is obscured, sometimes.

Then there is the 'mass-kick' problem... at first, we thought these were due to our own game script code, maybe something we'd added in a recent update. We have pulled out all sorts of features over recent days, in the hope of stopping the problem. We tuned down audio codecs (resulting in loads of people complaining about poor audio), we turned off weather, we tweaked vehicle management, altered loads of Battleye script detections, and, well, did too many things to list, really. Obviously this is an extremely slow and laborious process - turn off one thing in the development repo, wait for a server restart window, update the mission-code on the server, restart, sit back and wait, and hope that the kicks have gone. Rinse and repeat when we discover that they haven't gone, and came back when the server was next fully populated (which might be 24 hours away from the last update, depending on the time!) Then reset everything and try again. Mind-numbingly dull and frustrating, when it doesn't resolve the issue solidly. Especially when the mass-kicks sometimes seem to vanish for a couple of days; maybe due to population levels, maybe due to some person simpy hitting 'the magic trigger' in game (whatever it turns out to be). We don't know yet... but we will.

Worse, with the 'mass-kicks' we are unable to grab a complete packet-dump during the time of the kicks, because the server is flooding out many megabits of data per second, and it blows up the protocol analyser's capture-buffer, and leaves us with nothing. Totally frustrating. Many rude words have been said and invented over recent days, specifically due to this. Seeing the packet storms, we wondered if we were being DDOSed, but not getting reports from our server host. We actually changed to a different server host in order to run some comparative tests, and then we discovered (by some helpful input from our host) that the packet-storms were being generated internally, and not coming from outside at all! Basically our own server was going into 'scream mode' - and in the process, deafening itself, and shutting out Battleye. This in turn results in large chunks of the player-base losing contact with the server, which Battleye interprets as a 'no response'... and eventually, it's Battleye that does the kicking. Plonk! Anywhere from 3 to 100 players all get kicked with Battleye: Client Not Responding at exactly the same time - which itself, may actually be several minutes or seconds after the actual source of the problem (such is the nature of timeout periods triggering).

As you can imagine, we've tried a hundred different firewall configurations, server config tweaks, bandwidth filter modifications or removals, all to no avail, so far. I have a hunch that maybe the 'random no-message kicks' are possibly related (maybe piling up at certain points, and bottlenecking the server or something), but it is just a hunch, really. We'll know more when we are able to conduct some real in-depth packet-data analysis, I hope.

One thing I did discover is that this has been going on a long time. Much longer than I imagined, and the effect appears to have increased recently. I conducted some data-analysis on our ARMA3 logfiles, gathered since last August, and scoured every single one for all mentions of 'Battleye Client Not Responding' kicks. Obviously some of these will legitimately be just people getting kicked for having crummy network ping-rates or lossy streams... but it was still a valid exercise. I was able to pinpoint the first date where we saw multiple people all being kicked for 'Battleye Client Not Responding' at the exact same date, hour, minute, and second. And from that date, track an overall rise, to the current date.

The date it all seems to have started was 11th October 2016 - the date we released our big Altis update, which pulled in a great many features from the Tanoa build. Of course, our immediate thought was that it was some of the Tanoa core code, and we've set about disabling (temporarily) various features that might be an issue, or rewriting certain loops and routines where it's essential we keep that code in place, in such a way that they cannot 'lock' the server for any length of time. Not that we're certain this is happening, but it's good practice, and development is an iterative learning process, so we've learned plenty even since then, plus we've had new people like Rath join us, bringing ever more knowledge to the table.

However, it turns out it may not even be our code after all. The dice are still rolling. On the 11th October, Bohemia Interactive released a Hotfix to the game-engine. I remember it well; Wilco and myself laughing at the timing, because it happened literally minutes before we were holding down the servers in order to update the mission-file, and we knew full well that this was going to make for a muddied audit-trail in the future, if ever anything went wrong. We'd never know whether the fault was ours, or Bohemia's... and now here we are six months down the line, wincing and laughing wryly, as we look back at that date. 

I'm not going to say at this point, that this is an ARMA problem. It might be. It might not be. We simply do not know yet. But we do know (as of only a day or two ago) that we are not alone in suffering it, and this is a possibly crucial turning point. We have some friends in other large ARMA communities - some running Life games, some not, plus we have some not-so-friendlies in other communities, and they make server reports just like this one, from time to time. Turns out that some of the communities running larger-population servers are suffering the exact-same 'mass kick' Battleye Client Not Responding problem as we are. Identical. We're currently in the process of trying to establish whether certain specific, recent (as of October last year) enhancements to the ARMA scripting are being used on their games, as they are on ours - things like SimpleObjects, for example, and whether they are also suffering from the quiet-but-constant residual 'no-message random kicks' too. 

We're hopeful that we can find the root cause, and nail it. Maybe by information-sharing with a few other cool and clever people in the wider community, we can move faster, but in any event, we're still doing all we can to get to the bottom of it. Admin staff will, of course, just tell you that everything which can be done is being done, and they won't have any answers about dates of fixes, or reasons, nor will they want to hear you cussing them out, or cussing out the dev-team or management, for fairly obvious reasons. They're busy trying to hold it all together during this process, and continue their job of managing the players and ensuring fair play. Please support them in this objective, and realise that they are not being 'indifferent' or 'hostile' by not telling you exactly what's going on. They're doing their job as well as they can, in the circumstances.

However, I wanted to make this post just so that you knew a little bit more about what was going on down here in the BatCave, and what we're trying to do to resolve the problem. Of course I can empathise with you, if you're affected, and losing play progress or gear due to a random kick or being inconvenienced massively during a mass-kick and losing all your fellow roleplayers! What I cannot yet do, though, is give you promises of a date when it will stop. I can promise you that we're doing a great deal of work to get to the bottom of it - work which, if I'm honest, Bohemia really ought to be helping us (and others) to do more easily... but we're well-used to being left out here on a limb at the sharp-end by now, so I'm not going to complain about that too much! It achieves nothing, and we have some strategies in that 'direct-approach' department that we're about to use...

All I can say for now, is sit tight, weather the storm, and keep the faith. RPUK will always do its best for you, the player.

As Liam Neeson (almost) said: "Glitch, I don't know what you are. I don't know what you want. If you are looking for ransom, I can tell you I don't have money. But what I do have is a very particular set of skills; skills I have acquired over a very long career. Skills that make me a nightmare for glitches like you. I will look for you, I will find you, and I will kill you."

Thanks alot for explaining this TBJ.

 Also thanks to all the devs who are working hard rolling out these new updates and tryna fix bugs like these.

Let's show some support guys and be more patient in terms of the new updates, the last thing these guys need our us lot breathing down their necks!

keep it up devs😁

  • Like 4

Share this post


Link to post
CSI JAEGARN
19 hours ago, TinyBigJacko said:

Some feedback on the 'mass kicks' and 'random kicks' issues.

Firstly, be assured that we are working on this pretty much fulltime at the moment. I am doing virtually nothing else right now, and have so many windows open on my various development machines, that I am beginning to think I will never see sunlight again.

We have two ongoing problems, as far as we can tell.

The first, the 'kicks with no reason' or the 'random kicks' is the situation where a player gets kicked off, and is given the simple, rather blunt message "you were kicked off the game". No other information, no mention of admins, Battleye, or anything approximating a reason, just 'you were kicked off'. Really annoyingly, our RCON system logs this purely as a 'player disconnected' (which of course, he or she did, but not voluntarily!), and mentions nothing about it being a kick. Even the ARMA3 server log records the same - a 'player left' notification, with no hint of any kick or kick-reason. It is totally inseparable and indistinguishable from a normal 'player leaving' notification, which of course makes debugging virtually impossible (because players genuinely leave the game hundreds of times a session - people come and go).

Our only current angle with debugging these (from an 'after-the-fact' perspective) is the realisation that in-game, other players will see the message 'Player X was kicked off the game' (or something similar) and then the 'player X disconnected' message below it, all in the system-chat on the lower left-hand side. This is our only clue that this was a 'random kick' and not just a player leaving of their own volition.

What I'm trying to do currently is gather many of these incidents via a packet-sniffer running on the server, by re-running and recycling a capture-buffer, until one of these reports comes up in system-chat. Then I save the capture, take notes about the player's 'drop' time, their IP address, and suchlike, and then set out analysing their packet stream in the moments preceding their kick, to see if there is anything obvious.

The only fly in the ointment, of course, is the fact that Bohemia have encrypted the UDP packet-data in order to stop people injecting cheats into the data-stream, and this currently means that all the packets we capture are effectively indecipherable, in terms of working out exactly what the server and the player's client thought was being 'discussed', in game-data terms. We're currently trying to source a method of decoding these packets reliably, in case it sheds any light on the problem, but of course, finding this information out, and a Wireshark ARMA3 UDP packet-decoder (one which actually works and is current), involves visits to the 'dark side', which is of course not something we're very keen on. Steps are being taken, though...

Meanwhile we're still gathering data in the hope that it will give us something to go on, simply by virtue of the data-sequences that we see repeated, during and up to a random, no-message kick. Patterns still emerge, even if the precise detail is obscured, sometimes.

Then there is the 'mass-kick' problem... at first, we thought these were due to our own game script code, maybe something we'd added in a recent update. We have pulled out all sorts of features over recent days, in the hope of stopping the problem. We tuned down audio codecs (resulting in loads of people complaining about poor audio), we turned off weather, we tweaked vehicle management, altered loads of Battleye script detections, and, well, did too many things to list, really. Obviously this is an extremely slow and laborious process - turn off one thing in the development repo, wait for a server restart window, update the mission-code on the server, restart, sit back and wait, and hope that the kicks have gone. Rinse and repeat when we discover that they haven't gone, and came back when the server was next fully populated (which might be 24 hours away from the last update, depending on the time!) Then reset everything and try again. Mind-numbingly dull and frustrating, when it doesn't resolve the issue solidly. Especially when the mass-kicks sometimes seem to vanish for a couple of days; maybe due to population levels, maybe due to some person simpy hitting 'the magic trigger' in game (whatever it turns out to be). We don't know yet... but we will.

Worse, with the 'mass-kicks' we are unable to grab a complete packet-dump during the time of the kicks, because the server is flooding out many megabits of data per second, and it blows up the protocol analyser's capture-buffer, and leaves us with nothing. Totally frustrating. Many rude words have been said and invented over recent days, specifically due to this. Seeing the packet storms, we wondered if we were being DDOSed, but not getting reports from our server host. We actually changed to a different server host in order to run some comparative tests, and then we discovered (by some helpful input from our host) that the packet-storms were being generated internally, and not coming from outside at all! Basically our own server was going into 'scream mode' - and in the process, deafening itself, and shutting out Battleye. This in turn results in large chunks of the player-base losing contact with the server, which Battleye interprets as a 'no response'... and eventually, it's Battleye that does the kicking. Plonk! Anywhere from 3 to 100 players all get kicked with Battleye: Client Not Responding at exactly the same time - which itself, may actually be several minutes or seconds after the actual source of the problem (such is the nature of timeout periods triggering).

As you can imagine, we've tried a hundred different firewall configurations, server config tweaks, bandwidth filter modifications or removals, all to no avail, so far. I have a hunch that maybe the 'random no-message kicks' are possibly related (maybe piling up at certain points, and bottlenecking the server or something), but it is just a hunch, really. We'll know more when we are able to conduct some real in-depth packet-data analysis, I hope.

One thing I did discover is that this has been going on a long time. Much longer than I imagined, and the effect appears to have increased recently. I conducted some data-analysis on our ARMA3 logfiles, gathered since last August, and scoured every single one for all mentions of 'Battleye Client Not Responding' kicks. Obviously some of these will legitimately be just people getting kicked for having crummy network ping-rates or lossy streams... but it was still a valid exercise. I was able to pinpoint the first date where we saw multiple people all being kicked for 'Battleye Client Not Responding' at the exact same date, hour, minute, and second. And from that date, track an overall rise, to the current date.

The date it all seems to have started was 11th October 2016 - the date we released our big Altis update, which pulled in a great many features from the Tanoa build. Of course, our immediate thought was that it was some of the Tanoa core code, and we've set about disabling (temporarily) various features that might be an issue, or rewriting certain loops and routines where it's essential we keep that code in place, in such a way that they cannot 'lock' the server for any length of time. Not that we're certain this is happening, but it's good practice, and development is an iterative learning process, so we've learned plenty even since then, plus we've had new people like Rath join us, bringing ever more knowledge to the table.

However, it turns out it may not even be our code after all. The dice are still rolling. On the 11th October, Bohemia Interactive released a Hotfix to the game-engine. I remember it well; Wilco and myself laughing at the timing, because it happened literally minutes before we were holding down the servers in order to update the mission-file, and we knew full well that this was going to make for a muddied audit-trail in the future, if ever anything went wrong. We'd never know whether the fault was ours, or Bohemia's... and now here we are six months down the line, wincing and laughing wryly, as we look back at that date. 

I'm not going to say at this point, that this is an ARMA problem. It might be. It might not be. We simply do not know yet. But we do know (as of only a day or two ago) that we are not alone in suffering it, and this is a possibly crucial turning point. We have some friends in other large ARMA communities - some running Life games, some not, plus we have some not-so-friendlies in other communities, and they make server reports just like this one, from time to time. Turns out that some of the communities running larger-population servers are suffering the exact-same 'mass kick' Battleye Client Not Responding problem as we are. Identical. We're currently in the process of trying to establish whether certain specific, recent (as of October last year) enhancements to the ARMA scripting are being used on their games, as they are on ours - things like SimpleObjects, for example, and whether they are also suffering from the quiet-but-constant residual 'no-message random kicks' too. 

We're hopeful that we can find the root cause, and nail it. Maybe by information-sharing with a few other cool and clever people in the wider community, we can move faster, but in any event, we're still doing all we can to get to the bottom of it. Admin staff will, of course, just tell you that everything which can be done is being done, and they won't have any answers about dates of fixes, or reasons, nor will they want to hear you cussing them out, or cussing out the dev-team or management, for fairly obvious reasons. They're busy trying to hold it all together during this process, and continue their job of managing the players and ensuring fair play. Please support them in this objective, and realise that they are not being 'indifferent' or 'hostile' by not telling you exactly what's going on. They're doing their job as well as they can, in the circumstances.

However, I wanted to make this post just so that you knew a little bit more about what was going on down here in the BatCave, and what we're trying to do to resolve the problem. Of course I can empathise with you, if you're affected, and losing play progress or gear due to a random kick or being inconvenienced massively during a mass-kick and losing all your fellow roleplayers! What I cannot yet do, though, is give you promises of a date when it will stop. I can promise you that we're doing a great deal of work to get to the bottom of it - work which, if I'm honest, Bohemia really ought to be helping us (and others) to do more easily... but we're well-used to being left out here on a limb at the sharp-end by now, so I'm not going to complain about that too much! It achieves nothing, and we have some strategies in that 'direct-approach' department that we're about to use...

All I can say for now, is sit tight, weather the storm, and keep the faith. RPUK will always do its best for you, the player.

As Liam Neeson (almost) said: "Glitch, I don't know what you are. I don't know what you want. If you are looking for ransom, I can tell you I don't have money. But what I do have is a very particular set of skills; skills I have acquired over a very long career. Skills that make me a nightmare for glitches like you. I will look for you, I will find you, and I will kill you."

The thing is, you are loosing players daily cuz of this. A month ago, both servers where full at 8 restart. Now none of them are. Why isnt bohemia willing to help you with this? If i where them, i would help the Arma 3 community as much as i could, so more ppl would by the game. I really hope you find the thing causing this, cuz me along with others are thinkin about stop playing cuz it ruins the experience ingame. Many of us are still playing cuz we are dedicated to RPUK, and dont want to play anywhere else, but when you cant play more than 30min without being kicked/masskicked, its not fun anymore.

Share this post


Link to post
TinyBigJacko
2 hours ago, Jaegarn said:

The thing is, you are loosing players daily cuz of this. A month ago, both servers where full at 8 restart. Now none of them are. Why isnt bohemia willing to help you with this? If i where them, i would help the Arma 3 community as much as i could, so more ppl would by the game. I really hope you find the thing causing this, cuz me along with others are thinkin about stop playing cuz it ruins the experience ingame. Many of us are still playing cuz we are dedicated to RPUK, and dont want to play anywhere else, but when you cant play more than 30min without being kicked/masskicked, its not fun anymore.

With respect, apart from yourself and maybe your friends, you really have no idea how many players we might or might not be losing. We're a big server; we've had situations worse than this before, and probably will again, but we've managed to last three years... so I'm hopeful we'll carry on. 

To answer your questions - Bohemia doesn't really help anyone. They never have. Not sure whether it's a language problem, or simply that they're too small and too busy. Who knows. We do have contact with a few individuals who work for them, and we're approaching one with some specific logs, in the hope that they might be able to shed some light, but we are only 50% hopeful. The reality is that Bohemia still likes to pretend that the 'Life' communities don't exist, or are some kind of aberrant behaviour, and we can all see that they never really intended their hardcore milsim game to be used in this fashion. So our types of community are always the last on their list to help, it seems. Never mind that the Life games probably sold as many copies of ARMA3 as the milsim games did - perhaps more. Much like I'm asking you to leave the over-arching concerns about player-numbers to us, we leave those sorts of assistance-decisions and worries to Bohemia. We usually get what we need, from other learned people in the wider community.

We appreciate that the experience of being 'random no-message' kicked is a royal pain in the ass. We are still working on this. We made some alterations last night which have had some positive effects... but I am not going to jinx anything by saying on what! Let's leave it a few more hours and see. If and when we have resolved the mass-kick problem, the 'random no-message' kicks are next for the chop, if we can find them. We are currently building our own protocol analyser in the hope that we can at least follow server-to-client conversations at the data level, and see what these random kicks are being caused by. That should help us isolate the fix.

As always, my apologies for any inconvenience, and of course, you're free to complain - but just bear in mind that it doesn't really help to move us forward any. I do understand your frustrations though. I also appreciate your support too, so many thanks for that. We'll do our best for you, rest assured.

 

  • Like 11

Share this post


Link to post
The Fiddler

Guys pls stop quoting the entirety of TBJ s post, my finger is starting to hurt from this scrolling pls

  • Like 8

Share this post


Link to post
Danish
15 minutes ago, TinyBigJacko said:

With respect, apart from yourself and maybe your friends, you really have no idea how many players we might or might not be losing. We're a big server; we've had situations worse than this before, and probably will again, but we've managed to last three years... so I'm hopeful we'll carry on. 

To answer your questions - Bohemia doesn't really help anyone. They never have. Not sure whether it's a language problem, or simply that they're too small and too busy. Who knows. We do have contact with a few individuals who work for them, and we're approaching one with some specific logs, in the hope that they might be able to shed some light, but we are only 50% hopeful. The reality is that Bohemia still likes to pretend that the 'Life' communities don't exist, or are some kind of aberrant behaviour, and we can all see that they never really intended their hardcore milsim game to be used in this fashion. So our types of community are always the last on their list to help, it seems. Never mind that the Life games probably sold as many copies of ARMA3 as the milsim games did - perhaps more. Much like I'm asking you to leave the over-arching concerns about player-numbers to us, we leave those sorts of assistance-decisions and worries to Bohemia. We usually get what we need, from other learned people in the wider community.

We appreciate that the experience of being 'random no-message' kicked is a royal pain in the ass. We are still working on this. We made some alterations last night which have had some positive effects... but I am not going to jinx anything by saying on what! Let's leave it a few more hours and see. If and when we have resolved the mass-kick problem, the 'random no-message' kicks are next for the chop, if we can find them. We are currently building our own protocol analyser in the hope that we can at least follow server-to-client conversations at the data level, and see what these random kicks are being caused by. That should help us isolate the fix.

As always, my apologies for any inconvenience, and of course, you're free to complain - but just bear in mind that it doesn't really help to move us forward any. I do understand your frustrations though. I also appreciate your support too, so many thanks for that. We'll do our best for you, rest assured.

 

Have you tried turning it off and on? 

No, but all jokes aside. I do appreciate the effort you're putting in, and I wish I would be able to help out, but I am merely a filthy potato farming peasant, who knows nothing of coding. 

I wish you the very best of luck and may a solution be found soon.

  • Like 1

Share this post


Link to post
Shambala12

I know there's a good chance that you've already tried it but have you been logging the physical server performance and network bandwidth to compare to the timeframe where people are being kicked to see if it's the actual hardware/datacenter which is causing the issue? @TinyBigJacko

Also massive thanks to all the devs and staff who are working hard to fix the servers and keep the community together.

  • Like 2

Share this post


Link to post
TinyBigJacko
30 minutes ago, Shambala12 said:

I know there's a good chance that you've already tried it but have you been logging the physical server performance and network bandwidth to compare to the timeframe where people are being kicked to see if it's the actual hardware/datacenter which is causing the issue? @TinyBigJacko

Also massive thanks to all the devs and staff who are working hard to fix the servers and keep the community together.

Yes.

And thanks.

  • Like 3

Share this post


Link to post
CSI JAEGARN
4 hours ago, TinyBigJacko said:

With respect, apart from yourself and maybe your friends, you really have no idea how many players we might or might not be losing. We're a big server; we've had situations worse than this before, and probably will again, but we've managed to last three years... so I'm hopeful we'll carry on. 

To answer your questions - Bohemia doesn't really help anyone. They never have. Not sure whether it's a language problem, or simply that they're too small and too busy. Who knows. We do have contact with a few individuals who work for them, and we're approaching one with some specific logs, in the hope that they might be able to shed some light, but we are only 50% hopeful. The reality is that Bohemia still likes to pretend that the 'Life' communities don't exist, or are some kind of aberrant behaviour, and we can all see that they never really intended their hardcore milsim game to be used in this fashion. So our types of community are always the last on their list to help, it seems. Never mind that the Life games probably sold as many copies of ARMA3 as the milsim games did - perhaps more. Much like I'm asking you to leave the over-arching concerns about player-numbers to us, we leave those sorts of assistance-decisions and worries to Bohemia. We usually get what we need, from other learned people in the wider community.

We appreciate that the experience of being 'random no-message' kicked is a royal pain in the ass. We are still working on this. We made some alterations last night which have had some positive effects... but I am not going to jinx anything by saying on what! Let's leave it a few more hours and see. If and when we have resolved the mass-kick problem, the 'random no-message' kicks are next for the chop, if we can find them. We are currently building our own protocol analyser in the hope that we can at least follow server-to-client conversations at the data level, and see what these random kicks are being caused by. That should help us isolate the fix.

As always, my apologies for any inconvenience, and of course, you're free to complain - but just bear in mind that it doesn't really help to move us forward any. I do understand your frustrations though. I also appreciate your support too, so many thanks for that. We'll do our best for you, rest assured.

 

Today has acutally been a REALLY good day. No masskicks/and no randomkicks for anyone in the UNMC. I think you sorted out the problem? :)

Just to clarify, im not trying to be rude or something. Just frustrated cuz i do wanna play on this server, as much as i can. But when this is happening all the day(not a single one today tho) it brings so much frustration. When someone drives a choppers/or you doing runs and getting kicked. But as i said, today has been a wonderful day so far

  • Like 2

Share this post


Link to post
Kai Barry
10 hours ago, Jaegarn said:

Today has acutally been a REALLY good day. No masskicks/and no randomkicks for anyone in the UNMC. I think you sorted out the problem? :)

Just to clarify, im not trying to be rude or something. Just frustrated cuz i do wanna play on this server, as much as i can. But when this is happening all the day(not a single one today tho) it brings so much frustration. When someone drives a choppers/or you doing runs and getting kicked. But as i said, today has been a wonderful day so far

There's been much worse situations bud, when you grab the loot from the cargo, and you get desync rammed because of mass kick. I heard this had happened.

Share this post


Link to post
Alex Daland
15 minutes ago, Kai Barry said:

There's been much worse situations bud, when you grab the loot from the cargo, and you get desync rammed because of mass kick. I heard this had happened.

At the end of the day, I have to put my sympathy to the dev guys,  it is fu**** annoying being kicked, and I swear and hate, but I'm sure all the kicks/mass-kicks on the server is something the dev guys swear hard(er) about, I do ofc agree that it is annoying, so I'm playing a bit less and reading the forums a bit more, seeing if I can help anywhere.

I'm sure it'll be fixed, let's "all" as a community ask what we can do to help the dev team? Would it help if all players kicked send a message somewhere with time ie...? 

Don't ask what the country can do for you, but what you can do for the country -JFK :) 

 

Edited by Alex Daland
bad formulation, same point
  • Like 2

Share this post


Link to post
killersky
On 2017/4/11 at 5:52 PM, Tuplajuusto said:

I think the airdrops do not have a place in a roleplaying server especially if you can get 7.62 LMG'S there

Just Telling my opinion

 

On 2017/4/11 at 6:27 PM, GHOSTK1LL3R said:

Regarding the airdrop I have a few statements i'd like to make:

Currently the airdrops spawn everywhere on the map. Personally, I feel they should only drop in redzone locations to keep them confined away from possible rp that could be occuring in other areas. Equally, 200M around the airdrop is too small. What it leads to is people staying outside the redzone area and running around it and only entering once they see someone they can easily frag etc. If you was to put it into redzone this wouldn't be the case

Airdrops are well to easy to swipe. Once it swaps you can literally just go to it, take the gun and be gone instantly. The idea of the airdrop is good however how it's been implemented I personally disagree with. 

My recommendations for this are: 

Make it so you need bolt cutters in order to get into it. This will stop people getting to it and instantly taking it without even any combat. They're dropping rare items and as a result should be harder to take. 

Personally, I think once it's been announced it should have a 5min delay timer. What I mean by this, is the announcement will appear in the top right saying 'An airdrop is imminent at xyz'. Once this is said it gives 5 minutes for players who want to get the airdrop to head over to the location. When the 5minuties is up, the airdrop will fly down. With the implementation of this, it will lead to there being more people actually contesting this air drop idea because as of right now you can just get really good loot super easily.

My final suggestion is to personally increase the spawn time of it. As it's a new feature in the game I have been waiting the whole server restart for these things to go off. Currently, when the server restarts nothing happens for 40mins, then when the 40mins are up something will spawn but it's not always the airdrop. This means I need to wait 30mins for this to despawn and another 30mins for a new objective to re-appear. Sometimes i've gone full restarts without even seeing one airdrop on S1 at peak time.

If you were to alter the system into what i've suggested it would be much more enjoyable as you'd have squads of gangs waiting for the airdrop to commence and once it does getting to the position and holding the area until the airdrop drops. 

I really hope you consider my ideas which should be very simple to implement.

@papa john i totally agree in my opinion airdrop does not have any roleplay value or potentiall

@GHOSTK1LL3R if its in redzone how does this contribute to roleplay? this is a roleplay server, in my opinion everything added shall be pioritized to improve roleplay or add more exotic roleplay to this server.

now airdrop it self is a great idea but not with what we have now, people go there for the guns are selfish and kill people to get the weapons.

and droppiung it in redzone? hell no!!! more people--->redzone=more KOTH and less RPUK.

Share this post


Link to post
Guest
12 minutes ago, killersky said:

 

@papa john i totally agree in my opinion airdrop does not have any roleplay value or potentiall

@GHOSTK1LL3R if its in redzone how does this contribute to roleplay? this is a roleplay server, in my opinion everything added shall be pioritized to improve roleplay or add more exotic roleplay to this server.

now airdrop it self is a great idea but not with what we have now, people go there for the guns are selfish and kill people to get the weapons.

and droppiung it in redzone? hell no!!! more people--->redzone=more KOTH and less RPUK.

I'd love you to explain to me how initiation will work when 3-4 gangs are all going for the airdrop and one gang initiates on the other. The other 2 gangs that turn up once guns are pulled will instantly get sprayed down due to it being an active gunfight.

Share this post


Link to post
SolomoNn
28 minutes ago, killersky said:

 

@papa john i totally agree in my opinion airdrop does not have any roleplay value or potentiall

@GHOSTK1LL3R if its in redzone how does this contribute to roleplay? this is a roleplay server, in my opinion everything added shall be pioritized to improve roleplay or add more exotic roleplay to this server.

now airdrop it self is a great idea but not with what we have now, people go there for the guns are selfish and kill people to get the weapons.

and droppiung it in redzone? hell no!!! more people--->redzone=more KOTH and less RPUK.

At the end of the day, it s an arma 3 server and gunfights will always happen unless all of the guns are removed. Airdrops are a way for rebels to have something to do apart from robbing all the hobos doing runs

  • Like 1

Share this post


Link to post
killersky
2 minutes ago, DSGT Solomon said:

At the end of the day, it s an arma 3 server and gunfights will always happen unless all of the guns are removed. Airdrops are a way for rebels to have something to do apart from robbing all the hobos doing runs

true but what does this at to roleplay?

Share this post


Link to post
SI Max ORyan
16 minutes ago, killersky said:

true but what does this at to roleplay?

Nothing, it adds something fun for rebels to do. It doesn't take anything away from roleplay, and simply adds something cool to try out, so I don't see what the issue is.

  • Like 1

Share this post


Link to post
Staticz
22 minutes ago, SGT Max ORyan said:

Nothing, it adds something fun for rebels to do. It doesn't take anything away from roleplay, and simply adds something cool to try out, so I don't see what the issue is.

I totally agree roleplay can happen alot but it is something fun to do and at the end of the day arma is a milsim shooter

Share this post


Link to post
killersky
32 minutes ago, SGT Max ORyan said:

Nothing, it adds something fun for rebels to do. It doesn't take anything away from roleplay, and simply adds something cool to try out, so I don't see what the issue is.

it doesnt? take stuff away? max i know you play alot coop too, and dont you dare tell me the mk200 is not overpowered........ it is a laser gun 

recoil is close to none if someone tells ou they have got friends in the area then you can expect 200 suppressed round coming at you from 1.2 km away, if it was only the rebel guns fine but more OP guns? ........

Share this post


Link to post
TinyBigJacko
57 minutes ago, DSGT Solomon said:

At the end of the day, it s an arma 3 server and gunfights will always happen unless all of the guns are removed. Airdrops are a way for rebels to have something to do apart from robbing all the hobos doing runs

Someone gets it!

It's a free-for-all, an opportunity to blag something cool, big surprise, Easter-egg hunt, KOS mini-redzone all rolled into one. Because people wanted new stuff to do, and this was a way to get even the lowest of the low hobos some funky stuff if they took a massive risk.

Of course, it's also a giant opportunity to get killed and lose some stuff that needs to be rebought - aka a money-sink... but one which is much more fun and exciting than, say, jacking up the price of houses or a cool gun. 

Is it RP? Well, clearly, not in the conventional 'speaking to each other about marmite and your-mum's-got-saggy-tits' highbrow way that we're used to elsewhere on the server, but it is an opportunity to roleplay an extremely high risk, high reward extraction mission. Obviously, it'll be a very, very brave single hobo that attempts this, but who knows - he might get lucky and have one land in his garden and get it before anyone else even turns up.

But probably, hobos and little people will steer clear, and it'll be a way for the gangs to be focussed on something other than... well, I think @DSGT Solomon said it spot on, tbh.

  • Like 7

Share this post


Link to post
Gurlanin
22 minutes ago, TinyBigJacko said:

Someone gets it!

It's a free-for-all, an opportunity to blag something cool, big surprise, Easter-egg hunt, KOS mini-redzone all rolled into one. Because people wanted new stuff to do, and this was a way to get even the lowest of the low hobos some funky stuff if they took a massive risk.

Of course, it's also a giant opportunity to get killed and lose some stuff that needs to be rebought - aka a money-sink... but one which is much more fun and exciting than, say, jacking up the price of houses or a cool gun. 

Is it RP? Well, clearly, not in the conventional 'speaking to each other about marmite and your-mum's-got-saggy-tits' highbrow way that we're used to elsewhere on the server, but it is an opportunity to roleplay an extremely high risk, high reward extraction mission. Obviously, it'll be a very, very brave single hobo that attempts this, but who knows - he might get lucky and have one land in his garden and get it before anyone else even turns up.

But probably, hobos and little people will steer clear, and it'll be a way for the gangs to be focussed on something other than... well, I think @DSGT Solomon said it spot on, tbh.

And if you do want an RP reason for the airdrops, then there's always the good ol' "foreign power/CIA/aliens/terrorists trying to encourage a civil war by arming rebels/gangs/criminals/other aliens/turtles with free weapons".

  • Like 1

Share this post


Link to post
Guest
24 minutes ago, TinyBigJacko said:

Someone gets it!

It's a free-for-all, an opportunity to blag something cool, big surprise, Easter-egg hunt, KOS mini-redzone all rolled into one. Because people wanted new stuff to do, and this was a way to get even the lowest of the low hobos some funky stuff if they took a massive risk.

Of course, it's also a giant opportunity to get killed and lose some stuff that needs to be rebought - aka a money-sink... but one which is much more fun and exciting than, say, jacking up the price of houses or a cool gun. 

Is it RP? Well, clearly, not in the conventional 'speaking to each other about marmite and your-mum's-got-saggy-tits' highbrow way that we're used to elsewhere on the server, but it is an opportunity to roleplay an extremely high risk, high reward extraction mission. Obviously, it'll be a very, very brave single hobo that attempts this, but who knows - he might get lucky and have one land in his garden and get it before anyone else even turns up.

But probably, hobos and little people will steer clear, and it'll be a way for the gangs to be focussed on something other than... well, I think @DSGT Solomon said it spot on, tbh.

Tiny, love the airdrops however I think they could be so much better and bigger, a suggestion I made regarding them on this thread? Any chance I could get your 2 cents on the suggestion?

'Regarding the airdrop I have a few statements i'd like to make:

Currently the airdrops spawn everywhere on the map. Personally, I feel they should only drop in redzone locations to keep them confined away from possible rp that could be occuring in other areas. Equally, 200M around the airdrop is too small. What it leads to is people staying outside the redzone area and running around it and only entering once they see someone they can easily frag etc. If you was to put it into redzone this wouldn't be the case

Airdrops are well to easy to swipe. Once it swaps you can literally just go to it, take the gun and be gone instantly. The idea of the airdrop is good however how it's been implemented I personally disagree with. 

My recommendations for this are: 

Make it so you need bolt cutters in order to get into it. This will stop people getting to it and instantly taking it without even any combat. They're dropping rare items and as a result should be harder to take. 

Personally, I think once it's been announced it should have a 5min delay timer. What I mean by this, is the announcement will appear in the top right saying 'An airdrop is imminent at xyz'. Once this is said it gives 5 minutes for players who want to get the airdrop to head over to the location. When the 5minuties is up, the airdrop will fly down. With the implementation of this, it will lead to there being more people actually contesting this air drop idea because as of right now you can just get really good loot super easily.

My final suggestion is to personally increase the spawn time of it. As it's a new feature in the game I have been waiting the whole server restart for these things to go off. Currently, when the server restarts nothing happens for 40mins, then when the 40mins are up something will spawn but it's not always the airdrop. This means I need to wait 30mins for this to despawn and another 30mins for a new objective to re-appear. Sometimes i've gone full restarts without even seeing one airdrop on S1 at peak time.

If you were to alter the system into what i've suggested it would be much more enjoyable as you'd have squads of gangs waiting for the airdrop to commence and once it does getting to the position and holding the area until the airdrop drops. 

I really hope you consider my ideas which should be very simple to implement.'

  • Like 1

Share this post


Link to post
CSI JAEGARN

This new maprework, with new runs and locations, i love it! 

What more that should be added is something that would bring more traffic to UNMC lands. If you where to swap Blackmarket with dodgey daves instead, and moved something else to the small redzone, or just remove that redzone and putting some mines or processors there instead. Sometimes you can stand by the border and nothing happends in 2 hours. Not a single one comes, or try to borderhop. This suggestion is so we have more reasons to stay in our lands, and have something to deal with, instead of going to queensland everytime we want some RP or something else to happend. 

 

I have faith in you guys :) Keep up the good work, and everybody seems so much more into it now after you fixed the random/masskicks :)

 

@TinyBigJacko @ArrogantBread @Wilco 

Couldnt tag more of you guys for some reason.

  • Like 3

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×