Pages: [1] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Zifrian
Licentia Ex Vereor Intrepid Crossing
433
|
Posted - 2012.12.04 02:08:00 -
[1] - Quote
Today (before Retribution patch) I'm constantly refreshing the data export site trying to find the data dump for tomorrow's patch. Why? Well I have a program (EVE IPH - link in sig) that a decent number of people use to build items and I really wanted to get a patch ready for tomorrow when I get home from work. This is when the servers should be back up and my users would have the latest data. Most times the patches aren't so game changing that they will have an effect on EVE IPH but ones like this have big changes. We have some new items (salvage drones, mining frigs, micro jump drives, and all the cruiser/destroyer/frig mineral changes) that EVE IPH users will want to see. But not just me, there are at least 20 sites off the top of my head that do something similar to what I have and are mainstays of the EVE industry community.
As I've continued to develop EVE IPH I've tried to add easy ways to update the data so that I can deliver an updated product as soon as I get the data dump. Awhile back, I wrote an updater that I run locally to build my database for EVE IPH and it works pretty well. Not elegant but gets the job done. Everything worked great for several months of patches but two things have changed since then.
1 - The patches seem to make more and more changes or add items that are huge in the industry world. As I said above, all the 3rd party programs we have built as a community are mainstays for industry players that work to get the newest items ready for market. These changes suggest that waiting several days after a patch is too late for the community we support.
2 - The database dump is now going to be switched to YAML but slowly over time when changes are made in house. This in itself is a pain, but I think we are fine with it overall. The problem though is that without knowing what changes will come (up until now there have been minor fields replaced that I just comment out of the update because I don't use them), we won't be able to prepare our programs for updates. For instance, if you intend on making a very large switch of data to YAML that affects say killboard data or industry data, those 3rd party programs will be down for some time until they can get the data fixed or some nice dev rebuilds the data for us and posts it for download.
These 2 changes really are making it harder to maintain my program. I'm sure I'm not the only one. Not to mention that sometimes the data dump comes out and it has missing data then takes a few days to get another update. One of the great things about EVE for me as been the ability to build a program that helps the players have an easier time doing industry, and thus add to their overall enjoyment. It's also been fun just to develop. But we are doing this in our spare time. Spending days trying to figure out why the data isn't working when it did last patch isn't really fun. It's also not fun for my users to wait a week till I get time to update my data. I know you really support the 3rd party community, but times I just wonder if this is a "get to it when you can" thing.
Two things I can think of to make this better - 1 - Support the data dump process more formally. Right now it seems like this is a secondary "oh go post this data when you get a chance" job. Outside of the YAML change post, there doesn't seem to be much discussion on it. Either it's me and my lack of familiarity with YAML or it's not as big a deal yet for most of us to care. Also not much real process to knowing when the data is posted or when/how it will be. 2 - Post the data dump 2-3 days before the patch. If the patches are on Tuesdays, any way to get it on the weekend? Or at least the Monday prior? This would go a long way for helping out getting the data out to our users on patch day. Not to mention we have some time to work the YAML changes when they come up.
I'm probably rambling a bit, but the TL;DR - Can we get more support to put up the data dump a day or two before the patch? Can this be a consistent thing that is part of every patch check list so we can count on it?
Thanks! Maximze your Industry Potential! - Get EVE Isk per Hour! |
Khorkrak
Reasonable People Of Sound Mind
32
|
Posted - 2012.12.04 02:41:00 -
[2] - Quote
It would be 500% less of a pita if they'd simply post the database patch scripts along with the static data dump. That way all we'd have to do is convert from the Microsoft Transact-SQL dialect to MySQL and PostgreSQL. All of the data inserted, updated etc would be apparent. We'd also have the code for doing these changes nearly ready to go - a few minor tweaks such as replacing the GO statements with semicolons and so on and that's that.
I am so not looking forward to weeding through the data - doing diffs etc and then updating my scaled down versions of the static data dumps in MySQL and PostgreSQL format. Huge waste of my time. But I will do it as I want my project and site to continue on. Developer of http://www.decloaked.com and http://sourceforge.net/projects/pykb/ |
Golden Gnu
EVE University Ivy League
68
|
Posted - 2012.12.04 11:18:00 -
[3] - Quote
Zifrian wrote: TL;DR - Can we get more support to put up the data dump a day or two before the patch?
+1 Download is the meaning of life, upload is the meaning of intelligent life http://eve.nikr.net - home of jEveAssets |
Desmont McCallock
222
|
Posted - 2012.12.04 12:08:00 -
[4] - Quote
Would you 3rd party app devs be interested in a tool that will export the SDE from the client's bulkdata folder? This tool will guarantee that you can have a semi-official SDE (after all bulkdata are the same data the client uses, if you notice the login screen), right after the client patch and you have logged in at least once (so for the client to fetch the bulkdata). Considering support for MSSQL and MySQL for the beginning.
Thoughts? |
Muscaat
EVE Markets
12
|
Posted - 2012.12.04 12:16:00 -
[5] - Quote
Desmont McCallock wrote:Would you 3rd party app devs be interested in a tool that will export the SDE from the client's bulkdata folder? This tool will guarantee that you can have a semi-official SDE (after all bulkdata are the same data the client uses, if you notice the login screen), right after the client patch and you have logged in at least once (so for the client to fetch the bulkdata). Considering support for MSSQL and MySQL for the beginning.
Thoughts?
Yes. Very interested indeed!
I did at one point have reverence compiled and working, but that was ages ago, and the database schema was different in annoying little ways (I think just denormalised a little). |
Muscaat
EVE Markets
12
|
Posted - 2012.12.04 12:30:00 -
[6] - Quote
Apparently the data dump is out, just not linked to from the Community page yet.
https://twitter.com/CCP_PrismX/status/275902721637380097 |
Steve Ronuken
Fuzzwork Enterprises
821
|
Posted - 2012.12.04 12:55:00 -
[7] - Quote
Khorkrak wrote:It would be 500% less of a pita if they'd simply post the database patch scripts along with the static data dump. That way all we'd have to do is convert from the Microsoft Transact-SQL dialect to MySQL and PostgreSQL. All of the data inserted, updated etc would be apparent. We'd also have the code for doing these changes nearly ready to go - a few minor tweaks such as replacing the GO statements with semicolons and so on and that's that.
I am so not looking forward to weeding through the data - doing diffs etc and then updating my scaled down versions of the static data dumps in MySQL and PostgreSQL format. Huge waste of my time. But I will do it as I want my project and site to continue on.
From the things that I've heard, what we get isn't just a dump of tables. Some of it is munged, through a process to create the tables we get, from different underlying ones.
Makes getting patch files a trifle more complex. FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator, invention chance calculator, isk/m3 Ore chart-á and other 'useful' utilities.As well as mysql and CSV/XLS conversions of the Static Data Extract. |
Zifrian
Licentia Ex Vereor Intrepid Crossing
433
|
Posted - 2012.12.04 13:23:00 -
[8] - Quote
Thanks but this is part of what I'm frustrated about. Why do I need to go anywhere else than this forum or the community page to get it? Maximze your Industry Potential! - Get EVE Isk per Hour! |
Cyerus
Galactic Dominion Eternal Strife
63
|
Posted - 2012.12.04 14:56:00 -
[9] - Quote
Zifrian wrote:Thanks but this is part of what I'm frustrated about. Why do I need to go anywhere else than this forum or the community page to get it? Simply because it is bigger than that. As explained in this post in a somewhat similar topic, the community is bigger than just the Tech Lab forums and Community page.
Quote:You can also join the irc channel '#eve-dev' on irc network 'coldfront' (irc.coldfront.net), note that this channel is not an official CCP channel, but a community driven channel helping out anyone with challenging technical questions regarding the toolkit and EVE in general. Even the Community Toolkit page itself holds an invitation to the IRC channel. It's your loss if you aren't using all means available. |
Zifrian
Licentia Ex Vereor Intrepid Crossing
433
|
Posted - 2012.12.04 15:11:00 -
[10] - Quote
Cyerus wrote:Zifrian wrote:Thanks but this is part of what I'm frustrated about. Why do I need to go anywhere else than this forum or the community page to get it? Simply because it is bigger than that. As explained in this post in a somewhat similar topic, the community is bigger than just the Tech Lab forums and Community page. Quote:You can also join the irc channel '#eve-dev' on irc network 'coldfront' (irc.coldfront.net), note that this channel is not an official CCP channel, but a community driven channel helping out anyone with challenging technical questions regarding the toolkit and EVE in general. Even the Community Toolkit page itself holds an invitation to the IRC channel. It's your loss if you aren't using all means available. Where do I go to download the data? The community page. Why is it posted through a twitter feed then? If it's available, why hasn't it been updated to the community page?
It's not standardized and that's the whole point of this post. Maximze your Industry Potential! - Get EVE Isk per Hour! |
|
Cyerus
Galactic Dominion Eternal Strife
63
|
Posted - 2012.12.04 15:35:00 -
[11] - Quote
And I'm saying you can't standardize (spelling? dunno) this much more. Anyways, Fuzzysteve on IRC just released his MySQL package, uploaded on eve-files.com.
Hugs to him.
http://dl.eve-files.com/media/1212/mysql-retribution-1.0-82596.dmp.zip |
Steve Ronuken
Fuzzwork Enterprises
822
|
Posted - 2012.12.04 15:50:00 -
[12] - Quote
I'll get it up on my site properly, when it's stopped being lazy and sleeping. Along with at least a sqllite conversion and xls/csv extracts. possibly a postgres conversion too, as I'm more comfortable there now. FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator, invention chance calculator, isk/m3 Ore chart-á and other 'useful' utilities.As well as mysql and CSV/XLS conversions of the Static Data Extract. |
Dei
Mamaleek
2
|
Posted - 2012.12.04 17:09:00 -
[13] - Quote
Cyerus wrote:And I'm saying you can't standardize (spelling? dunno) this much more.
I find that ridiculous. If one part of the process is left out then it's not standardised, is it? The part that is left out is made redundant and never used so it might as well not even be there in the first place. It might as well say, "Join IRC or wait until someone can be bothered to post it on the forums for the rest of you losers." |
Two step
Aperture Harmonics K162
2341
|
Posted - 2012.12.05 20:41:00 -
[14] - Quote
Desmont McCallock wrote:Would you 3rd party app devs be interested in a tool that will export the SDE from the client's bulkdata folder? This tool will guarantee that you can have a semi-official SDE (after all bulkdata are the same data the client uses, if you notice the login screen), right after the client patch and you have logged in at least once (so for the client to fetch the bulkdata). Considering support for MSSQL and MySQL for the beginning.
Thoughts?
This seems like a no-brainer to me, then we aren't reliant on CCP getting the proper SDE out, and can even have stuff prepped from the test server before launch. CSM 7 Secretary CSM 6 Alternate Delegate @two_step_eve on Twitter My Blog
|
Zifrian
Licentia Ex Vereor Intrepid Crossing
438
|
Posted - 2012.12.05 20:50:00 -
[15] - Quote
Desmont McCallock wrote:Would you 3rd party app devs be interested in a tool that will export the SDE from the client's bulkdata folder? This tool will guarantee that you can have a semi-official SDE (after all bulkdata are the same data the client uses, if you notice the login screen), right after the client patch and you have logged in at least once (so for the client to fetch the bulkdata). Considering support for MSSQL and MySQL for the beginning.
Thoughts? I'd be interested in this as long as it's not a heavy lift. You think it would work on the test server data if we were to log in there as well? I'm assuming it's all the same and we could potentially (assuming no last minute changes) get the data early. Maximze your Industry Potential! - Get EVE Isk per Hour! |
Dei
Mamaleek
3
|
Posted - 2012.12.05 21:19:00 -
[16] - Quote
Shouldn't this be a CSM issue to bring to CCP's doors? It doesn't seem right to be hacking away at the metadata in our own clients to get the data that CCP promised they would give us. |
Desmont McCallock
223
|
Posted - 2012.12.05 21:52:00 -
[17] - Quote
Zifrian wrote:Desmont McCallock wrote:Would you 3rd party app devs be interested in a tool that will export the SDE from the client's bulkdata folder? This tool will guarantee that you can have a semi-official SDE (after all bulkdata are the same data the client uses, if you notice the login screen), right after the client patch and you have logged in at least once (so for the client to fetch the bulkdata). Considering support for MSSQL and MySQL for the beginning.
Thoughts? I'd be interested in this as long as it's not a heavy lift. You think it would work on the test server data if we were to log in there as well? I'm assuming it's all the same and we could potentially (assuming no last minute changes) get the data early. This could be configurable. And yes, CSM should definitely have a saying over this. Without SDE -> no 3rd party apps -> EVE Armageddon.
What I have to point out here though is that this SDE extractor will be a Windows based app. I'll try to have it Mono compatible (I can't promise that though). I just hope that Wine will be cooperative to it. |
Two step
Aperture Harmonics K162
2342
|
Posted - 2012.12.05 22:22:00 -
[18] - Quote
Dei wrote:Shouldn't this be a CSM issue to bring to CCP's doors? It doesn't seem right to be hacking away at the metadata in our own clients to get the data that CCP promised they would give us.
Perhaps you didn't see that pretty little banner over there < -------------------------------------------------------------------------
I bring this sort of thing up with CCP pretty regularly. Producing the SDE should be part of the release process, and shouldn't be something added on at the end CSM 7 Secretary CSM 6 Alternate Delegate @two_step_eve on Twitter My Blog
|
Dei
Mamaleek
3
|
Posted - 2012.12.05 22:49:00 -
[19] - Quote
Hmm, what was their response? I don't see anything in the last minutes or the previous one. Considering there are a few key third party apps that are integral to the community, you would've thought CCP would prioritise keeping their unpaid developers happy.
I mean this is from the meetings in May/June:
Quote:CCP Explorer mentioned EVE Gate as a potential place to house these controls, but UAxDEATH was concerned with the fact that the corporate UI was already pretty poor, and that third party tools would allow players to come up with something better.
Two step explained that from what he has seen, the folks working on the CREST project are mostly preoccupied getting Dust 514 integration ready, and that this may be the reason we haven't seen any beta versions of the new CREST system for EVE Online itself.
CCP Unifex responded by saying that a dev blog was in the works regarding this matter.
Where the hell is this blog post? Why has the dev team been moved off Eve to a different project, and why are they using their dev time reinventing third party tools? You need to give them a giant kick up the arse! Nothing has changed for 6 months! You need to start shaking the tree until PrismX falls down so you can poke him with a stick until he gives in. |
Steve Ronuken
Fuzzwork Enterprises
838
|
Posted - 2012.12.05 23:13:00 -
[20] - Quote
Dei wrote:Hmm, what was their response? I don't see anything in the last minutes or the previous one. Considering there are a few key third party apps that are integral to the community, you would've thought CCP would prioritise keeping their unpaid developers happy. I mean this is from the meetings in May/June: Quote:CCP Explorer mentioned EVE Gate as a potential place to house these controls, but UAxDEATH was concerned with the fact that the corporate UI was already pretty poor, and that third party tools would allow players to come up with something better.
Two step explained that from what he has seen, the folks working on the CREST project are mostly preoccupied getting Dust 514 integration ready, and that this may be the reason we haven't seen any beta versions of the new CREST system for EVE Online itself.
CCP Unifex responded by saying that a dev blog was in the works regarding this matter. Where the hell is this blog post? Why has the dev team been moved off Eve to a different project, and why are they using their dev time reinventing third party tools? You need to give them a giant kick up the arse! Nothing has changed for 6 months! You need to start shaking the tree until PrismX falls down so you can poke him with a stick until he gives in.
I don't think you quite get it.
Dust and Eve, from a CREST perspective, aren't hugely different. They're the same backend. Stuff like mail is /identical/ (or near as damn) FuzzWork Enterprises http://www.fuzzwork.co.uk/ Blueprint calculator, invention chance calculator, isk/m3 Ore chart-á and other 'useful' utilities.As well as mysql and CSV/XLS conversions of the Static Data Extract. |
|
Zifrian
Licentia Ex Vereor Intrepid Crossing
438
|
Posted - 2012.12.05 23:14:00 -
[21] - Quote
Two step wrote:Dei wrote:Shouldn't this be a CSM issue to bring to CCP's doors? It doesn't seem right to be hacking away at the metadata in our own clients to get the data that CCP promised they would give us. Perhaps you didn't see that pretty little banner over there < ------------------------------------------------------------------------- I bring this sort of thing up with CCP pretty regularly. Producing the SDE should be part of the release process, and shouldn't be something added on at the end Thanks for the support.
Any insight on their take? Maximze your Industry Potential! - Get EVE Isk per Hour! |
Dei
Mamaleek
3
|
Posted - 2012.12.05 23:41:00 -
[22] - Quote
Steve Ronuken wrote: I don't think you quite get it.
Dust and Eve, from a CREST perspective, aren't hugely different. They're the same backend. Stuff like mail is /identical/ (or near as damn)
Yeah but according to those minutes CREST isn't even being worked on - the team is working on Dust integration with Eve. This means no API, this means PI and ingame developments. The problem I was pointing out was that nothing is actually happening from a third party developer perspective as the news about CREST and the EVE API has been a giant black hole for 6 months.
I do note a few dev posts that suggest a blog post is in the pipeline, but I still don't see a blog post...
|
Zifrian
Licentia Ex Vereor Intrepid Crossing
442
|
Posted - 2012.12.08 15:32:00 -
[23] - Quote
This SDE just supports the need for standardization.
Appreciate CCP Prizim X for his support. Maximze your Industry Potential! - Get EVE Isk per Hour! |
Nik W
Critical Mass Inc. Nexus Fleet
3
|
Posted - 2012.12.10 01:56:00 -
[24] - Quote
If you are going to switch to YAML then please just do it. This bits and pieces is frustrating.
Also, putting out a data dump to say "let us know where the problems are" is a tad crazy.
3rd party apps are a big part of Eve's bread and butter. They are used on every scale imaginable from fitting ships to tools used to run the largest coalitions in game. Yet here we are nearly a week after a major expansion, and we don't yet have a working dump file.
CCP: Make no mistake, when 3rd party apps don't work, a huge chunk of your player base gets annoyed. It is completely avoidable.
IMO dump files should be properly tested and released no later than 48 hours pre-patch. |
Two step
Aperture Harmonics K162
2348
|
Posted - 2012.12.10 17:22:00 -
[25] - Quote
Dei wrote:Steve Ronuken wrote: I don't think you quite get it.
Dust and Eve, from a CREST perspective, aren't hugely different. They're the same backend. Stuff like mail is /identical/ (or near as damn)
Yeah but according to those minutes CREST isn't even being worked on - the team is working on Dust integration with Eve. This means no API, this means PI and ingame developments. The problem I was pointing out was that nothing is actually happening from a third party developer perspective as the news about CREST and the EVE API has been a giant black hole for 6 months. I do note a few dev posts that suggest a blog post is in the pipeline, but I still don't see a blog post...
Perhaps you might want to check out this thread on the CREST session at the CSM summit this week
There haven't been many posts from 3rd party devs in there, and certainly not a whole lot of feedback. CSM 7 Secretary CSM 6 Alternate Delegate @two_step_eve on Twitter My Blog
|
Cyerus
Galactic Dominion Eternal Strife
64
|
Posted - 2012.12.10 17:49:00 -
[26] - Quote
Two step wrote:Dei wrote:Steve Ronuken wrote: I don't think you quite get it.
Dust and Eve, from a CREST perspective, aren't hugely different. They're the same backend. Stuff like mail is /identical/ (or near as damn)
Yeah but according to those minutes CREST isn't even being worked on - the team is working on Dust integration with Eve. This means no API, this means PI and ingame developments. The problem I was pointing out was that nothing is actually happening from a third party developer perspective as the news about CREST and the EVE API has been a giant black hole for 6 months. I do note a few dev posts that suggest a blog post is in the pipeline, but I still don't see a blog post... Perhaps you might want to check out this thread on the CREST session at the CSM summit this weekThere haven't been many posts from 3rd party devs in there, and certainly not a whole lot of feedback. If they wanted feedback from 3rd party devs, why not cross-post that topic into EVE Tech Lab? |
|
|
|
Pages: [1] :: one page |
First page | Previous page | Next page | Last page |