Pages: 1 2 [3] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 10 post(s) |
Dersen Lowery
Scanners Live in Vain
1726
|
Posted - 2015.08.08 19:21:20 -
[61] - Quote
KenFlorian wrote:Marc Callan wrote:Illuminating. But worryingly, I got the distinct impression that CCP figured out what was causing the problem but not why - unless the underlying cause of the logging issues has since been determined? As a former software developer/IT guy this happens more often than most of us choose to publicly acknowledge. Hat off to CCP for telling us what happened as best they could sort it out. They, more than anybody, would like a perfectly coherent explanation...some of the time it's impossible.
Also, it's a question of priorities. If you're running a service, priority #1 is restoring the service. Once you've done that, you have the luxury of time to figure out the why of it.
Proud founder and member of the Belligerent Desirables.
I voted in CSM X!
|
GeeShizzle MacCloud
593
|
Posted - 2015.08.09 01:08:51 -
[62] - Quote
so TL;DR Tranq is more unique than any of us (players and devs alike) ever thought... |
Waffelweib
Hostile Erection
1
|
Posted - 2015.08.09 11:21:36 -
[63] - Quote
I loved that Devblog, thx a lot! It just left me wondering... what are these little white tables on the pizzas?
|
KenFlorian
Jednota Inc
28
|
Posted - 2015.08.09 12:10:27 -
[64] - Quote
Waffelweib wrote:I loved that Devblog, thx a lot! It just left me wondering... what are these little white tables on the pizzas?
the "tables" keep the top of the box from crushing the pizza...think of it as shield extender. |
KenFlorian
Jednota Inc
29
|
Posted - 2015.08.09 13:10:02 -
[65] - Quote
KenFlorian wrote:Waffelweib wrote:I loved that Devblog, thx a lot! It just left me wondering... what are these little white tables on the pizzas?
the "tables" keep the top of the box from crushing the pizza...think of it as shield extender. Note: a single pizza box won't require a shield extender (it's alone in the pizzaverse). When stacked, however, there is a "weight penalty" which requires the support of a shield extender "white table". |
unimatrix0030
Viperfleet Inc. Official Winners Of Takeshi's Castle
201
|
Posted - 2015.08.09 18:13:47 -
[66] - Quote
Thanks for the devblog! I love to read more of those! Thanks for all the work you guys put in that day to get the game back up and running!
No local in null sec would fix everything!
Fleet warp proposal = the rubix cube is back into eve especialy the second part of the saying.
Wh players need to adapt, null sec players get the rules changed.
|
Elegbara
White Wolf Enterprises Harmonious Ascent
612
|
Posted - 2015.08.09 18:28:31 -
[67] - Quote
Code formatting broken? Looks like there is some actual python code in the blog, but I only see numbered lines and comments.
Also - "Perforce sync hangs causing delays in our build system" - I thought I am the only one having problems with Perforce.
Open your eyes. And awaken.
|
Lord Battlestar
Faulcon de Lazy
215
|
Posted - 2015.08.10 03:42:06 -
[68] - Quote
Very interesting read, thanks for the info!
I once podded myself by blowing a huge fart.
|
Corraidhin Farsaidh
Farsaidh's Freeborn
1518
|
Posted - 2015.08.10 09:17:25 -
[69] - Quote
It's always interesting to see others peoples pain in these situations, mainly because misery loves company :D
I'd suggest replacing the databases with Oracle and hiring me to build them for you...and nuke the M$ stuff from orbit, it's the only way to be sure... |
|
CCP Goliath
C C P C C P Alliance
2709
|
Posted - 2015.08.10 10:01:39 -
[70] - Quote
Dradis Aulmais wrote:Sounds like Ghost in the machine.TQ is a very unique system. 12 years old, reborn several times. Code here code there, its own little ecosystem. Its like the ultimate Capsleer.
I kind of love this analogy
CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
|
|
|
Corraidhin Farsaidh
Farsaidh's Freeborn
1518
|
Posted - 2015.08.10 13:01:50 -
[71] - Quote
CCP Goliath wrote:Dradis Aulmais wrote:Sounds like Ghost in the machine.TQ is a very unique system. 12 years old, reborn several times. Code here code there, its own little ecosystem. Its like the ultimate Capsleer. I kind of love this analogy
I prefer my analogy of M$ software being a xenomorph that clamps itself to your face before infesting you with a parasite that eats you from the inside.
Then again my view may be somewhat tainted by having bought a laptop with Windows 8 pre-installed a few years back...someone pass me an m41a pulse rifle... |
Estella Osoka
Perkone Caldari State
681
|
Posted - 2015.08.10 19:54:25 -
[72] - Quote
If you can't fix it with a hammer, time to get a bigger hammer. |
Pestilen Ratte
Artimus Ratte
28
|
Posted - 2015.08.11 12:44:04 -
[73] - Quote
Vincent Athena wrote:Legacy code? I'll make a prediction. The channel you used for campaign logging was used in the past for doing something else. You thought that code was removed, but some part of it still remains. When you started campaign logging, some old code woke up, tried to do something related to that channel, and "bad things" resulted.
I agree with this reasoning, except that the nodes didn't fail until the python code went into a sustained loop. If the code in the log changed a binary setting for a channel, the single instance would also have caused the crash. The crash seems to be something to do with the loop.
Alternatively, there could very well be an if statement in the legacy code that triggers when any communication happens on this log channel. Even if that legacy code doesn't break the node when called once, it could disrupt the run loop by leaking memory when called over and over again. Then the node crashes.
Whatever it is, something in your run loop is freaking out when that channel gets used, so I would want to know why that channel was used in the past, meaning for what areas of functionality, and start looking in those areas of code for if channel statements.
|
KenFlorian
Jednota Inc
31
|
Posted - 2015.08.11 21:49:22 -
[74] - Quote
Pestilen Ratte wrote:
Whatever it is, something in your run loop is freaking out when that channel gets used, so I would want to know why that channel was used in the past, meaning for what areas of functionality, and start looking in those areas of code for if channel statements.
Which is why, when I was coding and leading coding teams, extra beer and pizza were the just rewards for in-line comments that explained WHY a given segment of code was written the WAY it was written.
Interpreting what a given line of code is doing is, usually, straightforward. Understanding why it was done this way and not that way or explaining why this will cause something later to perform in a given way....Priceless.
I loved documentation....in-line in the source code. External documentation? Usually worthless.
|
|
CCP DeNormalized
C C P C C P Alliance
298
|
Posted - 2015.08.11 22:18:37 -
[75] - Quote
Corraidhin Farsaidh wrote:It's always interesting to see others peoples pain in these situations, mainly because misery loves company :D
I'd suggest replacing the databases with Oracle and hiring me to build them for you...and nuke the M$ stuff from orbit, it's the only way to be sure...
hmmm, I'd rather we not go that route, thanks :)
CCP DeNormalized
DBA
Virtual World Operations
|
|
Corraidhin Farsaidh
Farsaidh's Freeborn
1528
|
Posted - 2015.08.11 22:51:06 -
[76] - Quote
CCP DeNormalized wrote:Corraidhin Farsaidh wrote:It's always interesting to see others peoples pain in these situations, mainly because misery loves company :D
I'd suggest replacing the databases with Oracle and hiring me to build them for you...and nuke the M$ stuff from orbit, it's the only way to be sure... hmmm, I'd rather we not go that route, thanks :) I'm sensing a vested interest... |
KenFlorian
Jednota Inc
31
|
Posted - 2015.08.11 23:22:55 -
[77] - Quote
Corraidhin Farsaidh wrote:I'm sensing a vested interest...
It's always good to know with what (or whom) one's SQL's are joined...outer but most especially the often problematic inner.
|
Joia Crenca
Science and Trade Institute Caldari State
227
|
Posted - 2015.08.12 05:55:37 -
[78] - Quote
Looked tasty: Pizza!
I'll have to re-read the rest NOT at midnight after a full day |
KenFlorian
Jednota Inc
32
|
Posted - 2015.08.12 20:44:08 -
[79] - Quote
CCP DeNormalized wrote:
hmmm, I'd rather we not go that route, thanks :)
What could possibly go wrong?
Ah! I know what you are. You're a carebear, aren't you? (Says the 6 month old capsuleer with nary a kill to his name).
I think I see Carebear Control Corporation on their way and, generally speaking, they know where you live.
|
|
CCP DeNormalized
C C P C C P Alliance
301
|
Posted - 2015.08.14 15:51:30 -
[80] - Quote
Actually, Oracle RAC is super sexy stuff and I'd love to get some experience with it!
it kicks the pants off SQL Server's offerings in this regard (although I've never used it and perhaps it's junk - but it sounds awesome!)
CCP DeNormalized
DBA
Virtual World Operations
|
|
|
KenFlorian
Jednota Inc
33
|
Posted - 2015.08.14 16:22:29 -
[81] - Quote
CCP DeNormalized wrote:Actually, Oracle RAC is super sexy stuff and I'd love to get some experience with it!
it kicks the pants off SQL Server's offerings in this regard (although I've never used it and perhaps it's junk - but it sounds awesome!) Actually, I am the HR guy for a company that works on Netezza.
6th normal form on many-terabyte sized data?
No problem. |
KenFlorian
Jednota Inc
33
|
Posted - 2015.08.14 16:34:07 -
[82] - Quote
I don't know anything about RAC but the companies that are choosing Netezza are not the small players in the world. If you need many petabytes and up, then you need to look somewhere else...Hadoop for instance.
But, the stuff our guys do with Netezza is very cool.
Did I mention 6th normal form?
Want a 300G data mart from N data sources that did not exist 10 minutes ago?
No problem.
It's all good fun until a naughty line of loggihg code completely spoils your day.
Better have been damn fine pizza. |
Corraidhin Farsaidh
Farsaidh's Freeborn
1553
|
Posted - 2015.08.17 08:41:25 -
[83] - Quote
CCP DeNormalized wrote:Cor'len wrote:
I expect it was also a case of "We can't reproduce this reliably on our test servers, so we have to debug it in production". I think it was somewhat unclear whether the DB was modified in a way which would've prevented the rollback, and as Masterplan said, a DB restore takes time - I seem to recall a figure of multiple hours.
CCP: Thanks for fixing it, for the skillpoints, and also for the well-written report on the pizza. <3
We take full backups prior to each DT, so it would of been a full backup in no recovery mode plus a few transaction log backups to bring us to just past DT. Roughly 3 hours for the 3 TB+ restore. Funny enough we can restore faster in our test env. due to having a massive pool of SAS disks (100's) on the SAN vs. the small pool of SSD disks that the TQ DB uses :)
Pssst...you know that Oracle can do incremental backups whilst the DB is still up and running...would take significantly less time that 3 hours..
On a completely unrelated note I know lots about Oracle RMAN backup and recovery... |
Corraidhin Farsaidh
Farsaidh's Freeborn
1553
|
Posted - 2015.08.17 08:44:15 -
[84] - Quote
KenFlorian wrote:CCP DeNormalized wrote:Actually, Oracle RAC is super sexy stuff and I'd love to get some experience with it!
it kicks the pants off SQL Server's offerings in this regard (although I've never used it and perhaps it's junk - but it sounds awesome!) Actually, I am the HR guy for a company that works on Netezza. 6th normal form on many-terabyte sized data? No problem.
Oracle RAC works very well indeed as long as it is configured correctly (just like anything else :D ). If you want a nice stable cluster with great backup options (RMAN is free btw) You should really look at Oracle
And nice Oracle DBA's.... |
Tex Bloodhunter
DEFCON. The Initiative.
12
|
Posted - 2015.08.26 23:15:58 -
[85] - Quote
That's a good read! Thanks for putting all the details out there! |
|
|
|
Pages: 1 2 [3] :: one page |
First page | Previous page | Next page | Last page |