|
Author |
Thread Statistics | Show CCP posts - 24 post(s) |
|
CCP Dolan
C C P C C P Alliance
748
|
Posted - 2013.12.03 15:29:00 -
[1] - Quote
Check out some math and numbers that make me feel dumb, and make the server feel great with CCP Prism X's new Dev Blog. CCP Dolan | Community Representative
Twitter: @CCPDolan
Gooby pls |
|
|
CCP Prism X
C C P C C P Alliance
1386
|
Posted - 2013.12.03 15:36:00 -
[2] - Quote
I hear this guy listens to really stupid music (NSFW) which is probably indicative of his cognitive capacity. Probably not worth reading this! @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
CCP Explorer
C C P C C P Alliance
1872
|
Posted - 2013.12.03 15:40:00 -
[3] - Quote
I'm more scared and scarred of his pictures. Not the pictures in this devblog though, they are safe. Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Prism X
C C P C C P Alliance
1387
|
Posted - 2013.12.03 15:50:00 -
[4] - Quote
Weaselior wrote:I have a concern. As you point out, this system means that ajacent systems tend to share the same physical hardware.
Doesn't that massively increase the chances of spillover tidi in nullsec battles: i.e. our staging system is overloaded so every system next to it is overloaded, massively increasing the pain in the ass to leave? Or the system the fight is in is massively overloaded, causing all systems around it to be massively overloaded as well making it a giant pain in the ass to get there?
If your fighting system is reinforced that shouldn't be an issue. This is also no more of an issue than it used to be, and now the TiDi you create in your staging systems will at least not be affecting players on the other side of the universe who have nothing to do with your pew pew.
And of course more nullsec nodes mean smaller pockets grouped together. @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
CCP Prism X
C C P C C P Alliance
1387
|
Posted - 2013.12.03 15:57:00 -
[5] - Quote
There seems to be an image missing there right now and some problem with our CDN. Hopefully that will get resolved soon, but it's why there's no "visualization" for the splitting process. Just a table of stats. @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
CCP Prism X
C C P C C P Alliance
1388
|
Posted - 2013.12.03 16:10:00 -
[6] - Quote
Wormholes do not need to be split by proximity. There's no sense of locality or proximity in WH space so they just get a very dumb but efficient method applied to them.
However I reduced the number of nodes running WH space. We've added a few back, apparently I should look at adding more before the weekend.
Sorry
Edit: Yes this distribution is remade every startup. @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
CCP Explorer
C C P C C P Alliance
1873
|
Posted - 2013.12.03 16:35:00 -
[7] - Quote
Aquila Sagitta wrote:How is this being applied to wh systems? It isn't. Solarsystems that are not geographically connected (WH systems) and non-solarsystem services are loadbalanced differently. Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Explorer
C C P C C P Alliance
1873
|
Posted - 2013.12.03 16:43:00 -
[8] - Quote
GeeShizzle MacCloud wrote:mynnna wrote:Grarr Dexx wrote:Looks like a whole load of baloney! Us lowsec residents have been suffering from utterly ridiculous amounts of tidi, in sub-100 man fights or even moving thirty people ten gates. If you did do this rebalancing act, you sure seemed to have forgotten the systems between 0.5 and 0.0. So I'm going to quote you so you can't edit out the part where you look silly, and then quote the devblog that explains why you look silly. Quote:We're hoping to have this code out by tomorrow Wednesday, December 4th. also just saying that the phrase used being 'empire systems' would most likely refer to both high and lowsec systems, as im fairly certain all of the 4 main factions have lowsec areas. if you consider the area u fly in as being non-empire i suggest looking at the logos on the station toolbar next time you're docked up. Yes, empire refers to high and low security space together. Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Explorer
C C P C C P Alliance
1873
|
Posted - 2013.12.03 16:44:00 -
[9] - Quote
Berluth Luthian wrote:CCP Explorer wrote:Aquila Sagitta wrote:How is this being applied to wh systems? It isn't. Solarsystems that are not geographically connected (WH systems) and non-solarsystem services are loadbalanced differently. So does this mean that there are systems that are non-wh, non-solar systems? Polaris is such a system. Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Prism X
C C P C C P Alliance
1388
|
Posted - 2013.12.03 16:52:00 -
[10] - Quote
GeeShizzle MacCloud wrote:Prism is it possible to put hard barriers for the pre-loader to stop connecting low and null sec systems together on one node? does this happen already?
when remapping systems from an overloaded node to another can a low and null system end up on the same node after tomorrow?
Currently Nullsec is split off from Empire Space (High and Low). That's a change we did a long time ago and was covered in my first draft of this blog. That was however about three times the size of this one, without pictures. So yeah we could so that, but we don't.
There's probably a case to be made for their load fingerprint being different due to different player behaviour. But as it stands they get lumped in with high-sec. @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
|
CCP Explorer
C C P C C P Alliance
1876
|
Posted - 2013.12.03 21:32:00 -
[11] - Quote
Abdiel Kavash wrote:I am slightly confused. You started the devblog by asserting that the old system was bad, because it put clusters of connected systems on one node (and intra-node jumps are more expensive than inter-node jumps). Yet you end up with a system that puts even bigger clusters of connected systems on the same node? From the pictures it looks like instead of constellations, now huge chunks of regions share the same node. Wouldn't that lead to even more intra-node jumps? Before solarsystems were grouped into constellations for loadbalancing purposes and then constellations from very different parts of space were grouped together on nodes. But that provided a very strange TiDi experience and also the constellation-locality didn't match the load. Now we ignore the constellations and simply group the solarsystems together based on their load and actual-locality. That last part was needed to provide a sensible TiDi experience but since we ignore the constellation then the groups become much more sensible with even load. The final piece of this puzzle, the intra-node jumps vs. inter-node jumps we ultimately want to solve with Brain in a Box. Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Explorer
C C P C C P Alliance
1876
|
Posted - 2013.12.03 21:34:00 -
[12] - Quote
Setsune Rin wrote:when was this fixed deployed or is it still in the pipeline to release? We deployed the initial version a few weeks before Rubicon per https://twitter.com/erlendur/status/398211109922279425 and then an improved version will be deployed tomorrow. Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Explorer
C C P C C P Alliance
1876
|
Posted - 2013.12.03 21:38:00 -
[13] - Quote
One more thing to mention, as a part of this change then the cluster is starting up 2 minutes faster. See here https://forums.eveonline.com/default.aspx?g=posts&m=3897467#post3897467 and here https://forums.eveonline.com/default.aspx?g=posts&m=3899297#post3899297 for details. Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Explorer
C C P C C P Alliance
1878
|
Posted - 2013.12.04 08:17:00 -
[14] - Quote
Suomi Khan wrote:CCP Prism X wrote:There's no sense of locality or proximity in WH space so they just get a very dumb but efficient method applied to them.
Read the Devblog and must say, looks like you guys out in a lot of work to make TiDi less frustrating, thank you a lot for that :) Is it possible for you to make an addition to the Devblog explaining how the load is distributed in w-space by chance? We now know and understand in detail what is happening in known space, but it could be very cool to know how you solve w-space, even though it might be "dumb but efficient" :) Greedy minimal-first without locality.
Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Explorer
C C P C C P Alliance
1878
|
Posted - 2013.12.04 08:18:00 -
[15] - Quote
Max Kolonko wrote:So we have like, what? 6 - 8 minutes of sleep left daily? More during the week since deployment target is < 24 minutes, but on weekends you're down to 6-8, yes. Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Explorer
C C P C C P Alliance
1878
|
Posted - 2013.12.04 08:23:00 -
[16] - Quote
Kossaw wrote:CCP Explorer wrote:The final piece of this puzzle, the intra-node jumps vs. inter-node jumps we ultimately want to solve with Brain in a Box. We're still waiting for that dev blog mate I know The timeline really is that it's going to be released when ready Erlendur S. Thorsteinsson | Development Director | EVE Online // CCP Games | @erlendur |
|
|
CCP Prism X
C C P C C P Alliance
1402
|
Posted - 2013.12.04 15:23:00 -
[17] - Quote
I'M BACK!
I just want to clear up some confusion I'm seeing first: this is not meant to be the Holy Grail of lag reduction. This is a static load balancer. It is in no way the end-all solution to our load problems, it's an initial step that's required before anything further can be done. Optimizing underused resources is just wasted work. Well it's not wasted but it makes more sense to do it the other way around.
There is no load reduction going on here. Todays total load pressures with, or without, my code would be exactly the same. But with my code it will be more evenly distributed between nodes so that the probabily of a "wild" TiDi appearing have been reduced. TiDi in unreinforced systems with a massive fleet presence will still happen as it always has. We'll need to reduce the CPU footprint per user if we want to prevent Fleet TiDi (and my money is on that only increasing fleet sizes until TiDi becomes unbearable again).
So with that being said I'm going to try and answer some of the more frequent questions here.
"Adjacent systems" are bad for fleet movement / staging.
They always have been. They've actually been worse because the old system would be so aggressive on grouping systems of the same constellation together that it would, is so many cases that I was given time to work on this, chose to overload the node rather than split up the constellation.
If your Staging system is not reinforced, it's going to share its node with other systems. If the fleet is large enough to cause TiDi it will cause TiDi no matter what these other systems are. It's even possible that this staging system had enough load caused on it the day before to be reinforced on its own because its simply too loaded to share its node with any other system. But that will not help you with TiDi if your fleet is large enough to overload that node.
If you control the space around your staging system, you can now command people to stay out of those systems to avoid TiDi-ing your fleets staging system. I'd love to offer you a map of all node allocations so that you could discern wether or not that was needed.. but I'm certain people would metagame TiDi into existance through that.
I'm not sure what more to say. Nobody likes being TiDi'd. I'm not going to try to convince you to like it. But you'll be able to anticipate it now. And in case it's not clear: Nullsec and Empire do not run on the same nodes, and they have not for many years. Fleets in Nullsec do thus not cause TiDi in Empire, or vice versa. They cause TiDi in other Nullsec systems. This means that under the old system a staging system from the north could be allocated to a node already running a staging system from the south. That can no longer happen. That's something, yeah?
This does not help with sudden escalations.
Absolutely not. This is a static load balancer that balances system between nodes at server startup. I'm actually reading a paper from some people at the University of Bonn about predicting destinations based on previous system jumps. It's pretty interesting but my brain now hurts. But if we could hook something like that in we could detect staging systems forming. We'd still not detect sudden escalations. Sudden escalations need dynamic load balancing if we're to handle them gracefully under the current CPU per User fingerprint.
What about reinforced nodes?
Reinforcing nodes means that we, usually, move that system to a single node that is running nothing other than that system. We currently have three nodes on standby for the premapper to use according to reinforcement requests. Any system marked for reinforcement, at startup, is completely excluded from this premapping process. They're effectively marked as "Fleet Fight Systems" rather than "Null Sec Systems" and thus the "Null Sec System" load balancing method will not include them.
Dynamic V Static Load Balancing.
As I mentioned dynamic load balancing would solve a lot of our issues. But there are massive hurdles to that happening. I know that sounds weird to some people that work in an environment where it's easy as pie. But that's not our environment. We simply are not in the state where we can move a system between nodes without offlining everyone first. Would we like to be in that state: Ofcourse! But we're not. So we're stuck with the static approach until that changes.
So instead we run three other spare nodes (that are not the fleet fight nodes mentioned above) that we can allocate systems to if we need to separate them from a system with a sudden escalation fight in it.
Why is Empire split from Null?
Because Empire has a completely different load fingerprint than Nullsec (Crimewatch has been mentioned). Players in Empire also have a fairly different behaviour than players in Nullsec. Wormhole space is also seperated from these two groups of systems for the same reasons.
Sadly I think we have to few WH space nodes allocated now. If Empire runs smoothly today and tomorrow then I'm thinking I'll move one or two empire nodes into the WH space rotation for a better weekend experience.
And now I have to run to a meeting (probably already too late by the time I finish editing this on the forums). Sorry if these answers feel a bit abrupt, I was in a hurry. @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
CCP Prism X
C C P C C P Alliance
1405
|
Posted - 2013.12.04 16:33:00 -
[18] - Quote
Vincent Athena wrote:So what efforts are being done to reduce the CPU footprint per user? I'm not privy to the plans of others, and have never been one to promise work on behalf of anyone other than myself. But I can totally tell you that I and RESCINDED just joined Gridlock to contribute to the Brain in a Box project.
That's of course a project outside our feature expansion release cycle. It's done when it's done. So I can't tell you anything more concrete than that. But feel free to berate me about it until then. I've got a hide as thick as my head. But I don't want you to do that to RESCINDED because that would be promising stuff on their behalf. @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
CCP Prism X
C C P C C P Alliance
1405
|
Posted - 2013.12.04 16:41:00 -
[19] - Quote
Sentient Blade wrote:I've mentioned it elsewhere, but why are these machines not virtualised (or are they?) surely something like vMotion would be able to move high-use systems onto dedicated hardware without the need to pause anything.
Can I say "Because if it was that easy we'd have done it already" and leave it at that? I'd rather not try to elaborate on that very complex subject because I don't know everything about everything and I'd rather not accidentally lie to you. @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
CCP Prism X
C C P C C P Alliance
1405
|
Posted - 2013.12.04 16:49:00 -
[20] - Quote
Vincent Athena wrote:Ive heard the words "Brain in a Box" quite a bit and seen vague descriptions of it having to do with preparing session change data on a separate node. But is there a full description somewhere? What it does, how much load it will remove, and so on?
It's meant to offload work currently being done by solar system nodes onto a different node, as well as to reduce the total amount of work needing to be done over and over again at certain time. How much it will offload is wholly dependant on the end result and how much of the current code is refactored into C.
I'm not sure if there are external sources with information for you. If I were at the office I could look something up for you or ask people. But I'm not. Perhaps I'll remember tomorrow! @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
|
CCP Prism X
C C P C C P Alliance
1405
|
Posted - 2013.12.04 17:08:00 -
[21] - Quote
Mioelnir wrote:@ Prism X: Over what timeframe are the cpu metrics used for premapping collected? Could a single escalated fight in a usually empty system not skew the metrics, forcing the system to become reinforced the next day? Are outlier datapoints stripped?
It's really not as sophisticated as it should be. Essentially we do know the load from hardware metrics but have to split that load between the many different systems running on the node. To do that we store the time it takes one simulation loop (IIRC, I am not in the office and don't feel like VPNing) to finish for a given system. Using the ratios between that we can estimate the % load that belongs to that system and we use that to evolve the value.
Outliers are factored in here, I'm not sure if it would be a good idea not to as they can be indicative of staging systems. Some outliers are however ignored like any system that's been moved to the Incursion load balancing group will be ignored while it is there (again IIRC). So a single escelated fight will skew the system for a bit, but it will then start regressing again.
This code has pretty much not been touched. I did a minor change to it when we first started noticing empire going all whack. It used to assign load by number of jumps and docks in a system but that's in no way related to the load an empire system will sustain. It might have worked for nullsec, but nullsec runs either cool or burning hot so trying to find the perfect balance there is an exercise in futility. We need fleet fighter prediction tools for that and the ones we currently have are riddled with false positives and have been turned off.
But yeah, this is the first step of many we need to do just for the static balancer. Changing the load evolution will probably be the next one. BiB comes first tho. @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
CCP Phantom
C C P C C P Alliance
3768
|
Posted - 2013.12.04 20:41:00 -
[22] - Quote
Gilbaron wrote:is there any kind of support for university papers ? i might actually be interested (not on a technical level, but for markets or politics) Yes, there is! It depends a bit on the type of research etc., but yes, in theory we can support academical research and did that in the past already. If you have serious interest and some specifics already in place, please contact the Community team (just send a support ticket with some details).
CCP Phantom - Senior Community Representative - Volunteer Manager |
|
|
CCP Prism X
C C P C C P Alliance
1431
|
Posted - 2013.12.11 15:42:00 -
[23] - Quote
I dislike leaving things hanging.
Wormhole Mass reduction is well outside of the scope of my work. Obviously mass should not be decreased on a denied jump but that should be bug reported through the proper channels so that it receives proper due diligence. Apparently this has been BR'd already so YAY!
Libras are not trustworthy. We're extremely dodgy characters who seem to be willing to adopt any side of everything for the sake of whatever deviant impulses drive us at that given moment.
I do believe everything else has been covered already in previous posts.
Fly safe... ..ish! @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
CCP Prism X
C C P C C P Alliance
1432
|
Posted - 2013.12.13 08:19:00 -
[24] - Quote
No. Distributing connected solar systems between different nodes will not make any load dissapear. It will just move it elsewhere. Perhaps I wasn't clear enough when I said that intra-jumps were more expensive: It's more expensive for that specific node as it will have to do both the tasks. Jumping between nodes does not magic away some part of the work that needs to be done. It's just done by two nodes. Remember: We do not want a fleet in the south to cause TiDi in the north.
And to reiterate this point: This is not about reducing load, it's about using currently available resources more efficiently. I've already started on another project, that will take some time, that is aimed at actually reducing load generated per client. It will also most probably end up in offloading much work from the system nodes to other nodes thus making any work put into "no adjacent systems on the same node" redundant. I'd rather just start working on an actual performance boost rather than attempting to satisfy unpredictable spike load with a static load balancer. @CCP_PrismX EVE Database Developer and Expert Ranter Member of a Different Team, every day. |
|
|
|
|