Pages: [1] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 1 post(s) |
Kaitai Noctus
Imperial Academy Amarr Empire
0
|
Posted - 2015.04.19 19:41:06 -
[1] - Quote
There appears to be a gradually-getting-worse widespread problem with something on CCP's end (or the network very close to them -- it is impossible for end-users to tell**) that's affecting the following things intermittently:
http://community.eveonline.com/ and http://www.eveonline.com/ EVE Launcher XML API / CREST
The issue manifests itself either as very long timeouts (60-90 seconds) when trying to establish a TCP connection to these resources, or immediate errors (e.g. "Connection reset by peer" or similar). The latter is likely caused by TCP RST being sent back from the server to the client either during handshake or immediately after (and before any actual payload is sent). For EVE Launcher, another symptom is the launcher starting up in "Offline Mode".
Twitter account @EVE_status mentions nothing.
Some threads discussing all of these so far -- but this is not a full list:
https://forums.eveonline.com/default.aspx?g=posts&t=419294 (brief technical details in post #8) https://forums.eveonline.com/default.aspx?g=posts&t=419478 https://forums.eveonline.com/default.aspx?g=posts&t=419492 https://forums.eveonline.com/default.aspx?g=posts&t=419530 https://forums.eveonline.com/default.aspx?g=posts&t=419253 https://forums.eveonline.com/default.aspx?g=posts&t=419505
Is anyone at CCP on call? Work weekends? Is there a NOC? Is monitoring set up for these endpoints from external sources (i.e. Internet-bound) and monitoring tied to that? How can we reach out to CCP to inform them of this? Else my gut feeling is that it's just going to stay like this until Monday when someone rolls into work and finds something quite busted.
** -- Please do not bother with traceroutes or mtrs. Border routers and/or firewalls block ICMP echo and/or response time-exceeded messages, not to mention IP routing is asymmetric and we can only see one half of the path -- someone with access to the EVE servers would need to be doing the same back to you and then both results compared for it to be effective. |
Jack Morrison
Manson Family Advent of Fate
191
|
Posted - 2015.04.19 20:06:02 -
[2] - Quote
CCNA seems to be strong within you, any chance CCP did this on purpose as a measure to lower DDoS impacts ?
"This is nothing more than a rumor with no basis of fact. Hope that's enough of a confirmation for you guys. :)"
Coolstorybro
|
Kaitai Noctus
Imperial Academy Amarr Empire
0
|
Posted - 2015.04.19 21:13:42 -
[3] - Quote
Anything is possible, but I'd think the possibility of that would be unlikely -- who would implement that form of throttling on a Friday? :-)
I did some packet captures for the EVE Launcher issue and can confirm it's the same issue that's affecting the community.eveonline.com and www.eveonline.com websites (read: manifests itself the same way):
No. Time Source srcport Destination dstport Protocol Length Info 3 0.024649000 192.168.1.50 1758 87.237.39.208 443 TCP 78 1758 > 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=16 TSval=0 TSecr=0 SACK_PERM=1 4 0.044623000 87.237.39.208 443 192.168.1.50 1758 TCP 60 443 > 1758 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 5 0.044639000 192.168.1.50 1758 87.237.39.208 443 TCP 54 1758 > 443 [ACK] Seq=1 Ack=1 Win=65535 Len=0 6 0.064743000 87.237.39.208 443 192.168.1.50 1758 TCP 60 443 > 1758 [RST, ACK] Seq=1 Ack=1 Win=16384 Len=0
Here we can clearly see CCP's 87.237.39.208 server (a.k.a. client.eveonline.com (verified this per DNS capture)) and my workstation completing the full 3-way handshake (SYN, SYN+ACK, ACK) followed immediately by CCP's server abruptly closing the connection with TCP RST. This isn't a clean shutdown (FIN, FIN+ACK), this is a deliberate RST. (Yes, the above capture involves a machine behind NAT, but rest assured when I say the packet capture I did on the WAN interface of my router showed the same behaviour).
Things to take note of:
1) client.eveonline.com has an A record TTL of 300, so I imagine the IP may vary/change for people. No idea if the issue is specific to just 87.237.39.208 or not, but my money is on that it isn't (because I've seen this happen for different CCP destinations within different CIDR blocks (see aforementioned linked thread, re: post #8))).
2) There is no payload -- this happens immediately after TCP handshake. So this isn't a case of, say, an HTTPS webserver rejecting a request or throttling based on the HTTP request itself, or an expired SSL certificate. This instead would be something like a very broken webserver daemon or a load balancer misbehaving. The possibilities are endless though (layer 7 filtering device, something like a Sandvine unit in the network path, etc.), and impossible for an end-user to determine.
3) I've manually tried to reproduce this using openssl s_client -showcerts -servername client.eveonline.com -connect 87.237.39.208:443 -- as expected, this works most of the time but still intermittently, which further supports my theory that CCP may just have a one (or more?) broken server(s) behind a load balancer. But it could just as equally be something with the LB itself.
When things do work, I notice Microsoft IIS is involved (yes, this is across HTTPS/port 443, but webservers these days wait for the client to send data before determining if it should be SSL or plaintext -- this is me manually using plaintext)):
GET / HTTP/1.0 Host: client.eveonline.com
HTTP/1.1 200 OK Cache-Control: public, max-age=38 Content-Type: text/html; charset=utf-8 Expires: Sun, 19 Apr 2015 21:09:09 GMT Last-Modified: Sun, 19 Apr 2015 21:08:09 GMT Vary: * Server: Microsoft-IIS/7.5 X-AspNetMvc-Version: 5.0 X-AspNet-Version: 4.0.30319 X-Powered-By: ASP.NET Date: Sun, 19 Apr 2015 21:08:30 GMT Connection: close Content-Length: 40446
{snip HTML response}
I will note that in the HTML response, there is Javascript that references https://login.eveonline.com/ and https://crest.eveonline.com -- the latter is interesting because it could explain how the XML API / CREST is being impacted. I wouldn't have thought the Launcher used CREST, but chances are it does.
4) Other threads on the forum show people in the UK, and all throughout the US (different countries, states, and ISPs), and on different OSes (I'm on XP, others are on 7 or newer) are experiencing this. It's possible there's a commonality network-path-wise, but it's much more likely it's something on CCP's end. I just wish I knew who I could poke via Twitter etc. so they could engage their Systems folks to figure out what's up.
P.S. -- No CCNA, just a 20-years-in-the-field UNIX SA who's well-trained in networking as well. :-) |
Gunner GzR
Anomalous Existence Low-Class
87
|
Posted - 2015.04.19 23:09:23 -
[4] - Quote
i have developed this issue as well. Started Friday as well
Change For the Sake of Change is bad
Listen to you community CCP We are what make you Money.
Remove Local From 0.0 and Low Space Please
|
Terramort Suiadan
Hounds Of Haides
0
|
Posted - 2015.04.19 23:35:54 -
[5] - Quote
My launcher has evolved into an unplayable state, and download speeds are so slow it's impossible to patch. Started getting minor issues about Friday as well. |
FLaShGoRDoN Flashers
Tacere Sicarius
0
|
Posted - 2015.04.20 00:07:47 -
[6] - Quote
Starting Friday I noticed that my launcher showed TQ as being OFFLINE
When restarting the launcher It will load corrctly about half the time, but when i put in my login info and the launcher would normally say "Launch", its now a greyed out bar that wont respond to clicking at all and the launcher says TQ is offline
So i have been connecting via the exe. file in Bin folder
I log on ok and after jumping systems i begin to glitch, my ship mods wont appear, i freeze up and my ship warps like im in 100% TIDI
I have tried playing in all window modes, low graphic settings, and it always happens the same way. The game glitches out and i have to relog to get things unglitched again. This starts the process all over again
I have played eve without any issues for 3 years and this all started friday
I have run the repair tool 30 times trying to find corrupted files, still to no avail
I am fairly confident that this is a ccp issue and as someone that spends 250 dollars a month i find it very frustrating
is there anyone that has suggestions or ideas about how I can make EVE playable again?
|
vccv
39
|
Posted - 2015.04.20 00:54:47 -
[7] - Quote
Yea it would be nice if they would at least make a thread stating what is going on.. or at the very least that they know about it and are working out the problem. I don't much care for the silent treatment/avoidance.
Hey look.. there is an elephant in the room.... |
|
CCP FoxFour
C C P C C P Alliance
3983
|
Posted - 2015.04.20 08:06:45 -
[8] - Quote
Please see this thread for the API issues: https://forums.eveonline.com/default.aspx?g=posts&t=419546&find=unread
@CCP_FoxFour // Technical Designer // Team Size Matters
Third-party developer? Check out the official developers site for dev blogs, resources, and more.
|
|
Nagarythe Tinurandir
House of Freedom The Pursuit of Happiness
195
|
Posted - 2015.04.21 08:26:02 -
[9] - Quote
Having the Launcher problems (TQ OFFline) since the last patch and it is getting worse. Yesterday all my overview settings were gone and all notification settings, as well as password protected channels were resetted/ closed. That was kind of the switch between inconvenience to annoyance.
what kind of logs would help CCP in a bug report?
|
|
|
|
Pages: [1] :: one page |
First page | Previous page | Next page | Last page |