Pages: [1] 2 3 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Kaladr
Amarr Viziam
|
Posted - 2009.08.05 01:46:00 -
[1]
So, I'm going to post this officially, to attract more attention to the project, and to test the waters.
I present you with libevecache: http://gitorious.org/libevecache
Things it does so far:
* Structure dumping of the Market Orders cache file (others frankly untested). * RLE decompression of the actual market data rows (the zeros! we have zeros!) * Understands nested constructs, such as a second cache file stream embedded in it (and maybe even figures out when it ends ;)).
Things it doesn't do yet:
* Unpacking all the fields from the market database rows * Walking the cache structure into something more meaningful * Get the structure on the second half of the file right (database rows starts devolving randomly...)
Things it will never do:
* Reserializing of the cache file.
Have fun! ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
wxEswat
Amarr
|
Posted - 2009.08.06 09:03:00 -
[2]
Interesting.....haven't had time to look too much at the code yet though. Was happy to see GPL and from my initial look appears to be decently cross-platform?
I'll definitely take a longer look at this later this week for possible inclusion into my own Eve project.
wxEswat
|
Kaladr
Amarr Viziam
|
Posted - 2009.08.06 16:26:00 -
[3]
Yes, the library is GPLv3 (note, not LGPL, and yes, this is intentional, for now)
So far, it is entirely (mostly) ISO C++. It builds and runs correctly on Linux (32 and 64bit), though I haven't yet tried building it on Win32/Win64. The right DLL export bits are in place however. You can see some of the dumper output here.
The make system is qmake (from Qt), mainly designed to support a future product using that framework (the EVE-Central uploader 2.0). If someone would want to contribute waf/scons/cmake rules, I'd be willing to accept them. No pre-canned VS solution files or autoconf-based builders, please. ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
Kaladr
Amarr
|
Posted - 2009.08.09 06:45:00 -
[4]
Edited by: Kaladr on 09/08/2009 06:45:26 There are some small fixes available now which allow the library to correctly(?) understand the cache file structure nesting, with only some small hacking going on at the end of the file. If you know better, feel free to fix some of the logic :-)
Next task is the market decoder engine, which is not difficult. ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
darius mclever
|
Posted - 2009.08.09 14:21:00 -
[5]
gplv3 is one of the worst licenses to pick for a library. the adaption rate will be close to 0.
|
Xaroth Brook
Minmatar BIG Libertas Fidelitas
|
Posted - 2009.08.09 15:04:00 -
[6]
I have to agree on the GPLv3 thing.. it'd be a lot more useful if it were changed to GPLv2 (or 2+ for that matter) .. 3 is just .. wrong.
It was like a baby, it landed on my lap and was helpless and totally defenseless. Then I shot it and bragged about it on a killboard.
|
darius mclever
|
Posted - 2009.08.09 16:39:00 -
[7]
actually LGPL would be better.
|
Kaladr
Amarr
|
Posted - 2009.08.10 06:09:00 -
[8]
Edited by: Kaladr on 10/08/2009 06:12:26 Edited by: Kaladr on 10/08/2009 06:09:48 License adjusted to GPL v2.0+. I didn't go as far as doing what I normally do (BSD), but figure dropping 3.0 is a good step in the right direction. ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
darius mclever
|
Posted - 2009.08.10 11:40:00 -
[9]
Originally by: Kaladr Edited by: Kaladr on 10/08/2009 06:12:26 Edited by: Kaladr on 10/08/2009 06:09:48 License adjusted to GPL v2.0+. I didn't go as far as doing what I normally do (BSD), but figure dropping 3.0 is a good step in the right direction.
why not lgpl? this would allow people to link it also in non gpl programs.
|
Xenofur
Di-Tron Heavy Industries Atlas Alliance
|
Posted - 2009.08.11 13:45:00 -
[10]
Why not go the whole mile and call it PD? Do you really care about restricting how others use it?
|
|
Muscaat
ASCENTIS CORPORATION Free Worlds Alliance
|
Posted - 2009.08.11 13:50:00 -
[11]
Kaladr, this is brilliant. If I had any C++ knowledge I'd offer to help in any way I could - but my experience is in Java.
I'd very much like to use what you've learned here to help get my Java cache reader working properly. ---------- EVE Markets - global price trends at your fingertips |
Dragonaire
Caldari Corax. New Eden Retail Federation
|
Posted - 2009.08.11 13:59:00 -
[12]
Problem with PD is laws about it varies depending on country etc. but all the open source licenses avoid that problem by clearly stating how it can be used.
I'd also suggest using LGPL as it is a 'library' and that is what it was made for but GPL2 should be workable for most people as well. -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API.
|
Xenofur
Di-Tron Heavy Industries Atlas Alliance
|
Posted - 2009.08.11 14:56:00 -
[13]
Edited by: Xenofur on 11/08/2009 14:58:07
Originally by: Dragonaire Problem with PD is laws about it varies depending on country etc. but all the open source licenses avoid that problem by clearly stating how it can be used.
Not to get too far off-topic here, but: What the hell? How can laws have any effect on a license that consists of only the sentiment "Do whatever you want."? Got any links to back that up?
And hell, if you really do need an explicit license, there's still this: http://creativecommons.org/publicdomain/zero/1.0/
|
Dragonaire
Caldari Corax. New Eden Retail Federation
|
Posted - 2009.08.11 15:18:00 -
[14]
Let me wikipedia that for you http://en.wikipedia.org/wiki/Public_domain
Quote: Furthermore, the laws of various countries define the scope of the public domain differently, making it necessary to specify which jurisdiction's public domain is being discussed.
BTW that article was the first hit on Google also so I Googled it for you too
Quote: And hell, if you really do need an explicit license, there's still this: http://creativecommons.org/publicdomain/zero/1.0/
Which is a great license if it was a work of art or book etc but creative commons themselves say for software there are better ones to use but I'll leave that for those interested in doing their own research to find it. -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API.
|
Kaladr
Amarr
|
Posted - 2009.08.11 16:21:00 -
[15]
Originally by: Xenofur Edited by: Xenofur on 11/08/2009 14:58:07
Originally by: Dragonaire Problem with PD is laws about it varies depending on country etc. but all the open source licenses avoid that problem by clearly stating how it can be used.
Not to get too far off-topic here, but: What the hell? How can laws have any effect on a license that consists of only the sentiment "Do whatever you want."? Got any links to back that up?
And hell, if you really do need an explicit license, there's still this: http://creativecommons.org/publicdomain/zero/1.0/
The problem with public domain is that your are giving up all ownership of the code. It is not a license in which you retain ownership through copyright. Some countries aren't set up to deal with property in the commons.
And to answer questions, not using LGPL is an intended consequence. Call it my rebellion against closed source applications and hidden secrets. If you're developing an open source product not covered by the GPL (such as the BSD, Apache-derivative, etc) and would like to use the library, contact me. I will not allow any closed source applications to use the library however. ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.08.11 18:19:00 -
[16]
Keep in mind that this is a program that handles a proprietary dataformat that has been reverse engineered, so I'm not entirely sure where this stands, legally, in terms of licencing.
On a side note, did this project get approved by CCP? As last time I checked they were a bit tight on stuff pertaining their cache format :P _
Got Item? | EVE API? |
Xenofur
Di-Tron Heavy Industries Atlas Alliance
|
Posted - 2009.08.11 19:06:00 -
[17]
Edited by: Xenofur on 11/08/2009 19:07:28
Originally by: Kaladr The problem with public domain is that your are giving up all ownership of the code.
Well yeah, but why would you care abo... Originally by: Kaladr my rebellion against closed source applications
Oh. :v
Good to see that you're honest and straight-forward about it and don't try to call it "free". :)
Any idea if and when the API is stabilized? I'm considering writing a Perl module to access the library via that.
|
Muscaat
ASCENTIS CORPORATION Free Worlds Alliance
|
Posted - 2009.08.11 20:44:00 -
[18]
Originally by: Entity [D]id this project get approved by CCP? As last time I checked they were a bit tight on stuff pertaining their cache format :P
Canonical source of CCP's attitude towards cache readers ---------- EVE Markets - global price trends at your fingertips |
Entity
X-Factor Industries Synthetic Existence
|
Posted - 2009.08.11 21:00:00 -
[19]
Originally by: Muscaat
Originally by: Entity [D]id this project get approved by CCP? As last time I checked they were a bit tight on stuff pertaining their cache format :P
Canonical source of CCP's attitude towards cache readers
I'm well aware of that post. But the eve metrics uploader had to get approval, and my sexy python based cache toolkit was denied for some reason at the time ;-) _
Got Item? | EVE API? |
Kaladr
Amarr
|
Posted - 2009.08.12 01:01:00 -
[20]
Originally by: Xenofur
Any idea if and when the API is stabilized? I'm considering writing a Perl module to access the library via that.
Not really stable, and the direct access to market data methods isn't implemented yet.
You can always attempt a SWIG based wrapper for simplicity. ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
|
Kaladr
Amarr
|
Posted - 2009.08.12 01:06:00 -
[21]
Edited by: Kaladr on 12/08/2009 01:06:47
Originally by: Muscaat
Originally by: Entity [D]id this project get approved by CCP? As last time I checked they were a bit tight on stuff pertaining their cache format :P
Canonical source of CCP's attitude towards cache readers
This is the official direction I am taking with this in regards to CCP's attitude regarding cache reading, and talk about the format thereof. If they didn't want user contributed documentation regarding the cache, its very easy for them to make threads go *poof*. If this thread goes *poof*, or the next patch contains a radical cache format change, then we know what their stance is
libevecache was written without any inspection, debugging, disassembly, or other "bad stuff" of EVE binaries or DLLs. It only used intuition (spread over about a month part time), comparison with .CSV exports of the market data, and third party information, such as that above mentioned forum thread. ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
Xenofur
Di-Tron Heavy Industries Atlas Alliance
|
Posted - 2009.08.12 22:24:00 -
[22]
Originally by: Kaladr Not really stable, and the direct access to market data methods isn't implemented yet.
Alright, thanks. :)
Originally by: Kaladr You can always attempt a SWIG based wrapper for simplicity.
I don't really understand this however. Wrapping your library in perl pretty much only means to write a .xs file which defines all the functions that your library exports and doing some munging so they're a bit nicer to use data-wise. How would SWIG help there?
|
Kaladr
Amarr
|
Posted - 2009.08.13 16:11:00 -
[23]
SWIG would allow for the same thing, but is language agnostic, and understands how to deal with C++ STL types -> native type conversion as well. If Perl XS is what you want, I see no problem in writing a wrapper for Perl, and if you would want to contribute it, would be willing to accept it ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
Kaladr
Amarr
|
Posted - 2009.08.19 05:24:00 -
[24]
Made some updates regarding my current knowledge of each entry in the DBRow. There is no direct parser yet, but it can be hand decoded. Parser for DB row next. ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
Herio Mortis
Caldari Deep Core Mining Inc.
|
Posted - 2009.08.19 13:56:00 -
[25]
Edited by: Herio Mortis on 19/08/2009 13:56:37 In the type field in the parser, the one you do: check = iter.readChar() & 0x3f; // magic
Also check the 0x40 bit in that one. If it is set, store a reference to the decoded object starting the count from 0. The 0x1b (EStreamIdent) opcode refers to this, so when you encounter that one replace with the stored object matching the 0x1b data byte.
At least that is my understanding of how that works, makes sense in my own playing around at least.
|
Anders1
|
Posted - 2009.08.23 09:36:00 -
[26]
I tried to compile this, you seem to use qmake or so, but I don't have Qt installed on my Linux box. I don't want to install it either because it pulls in lots of other dependencies like X, gstreamer, etc.
for x in *.cpp; do g++ -fPIC -c $x -o $x.o -I. done
g++ -o libevecache.so *.o -shared
then in util, g++ -o dumper dumper.cpp -I../lib -L../lib -levecache
../lib/libevecache.so: undefined reference to `EveCache::SDBRow::setLast(bool)' ../lib/libevecache.so: undefined reference to `EveCache::SDBRow::isLast() const' ../lib/libevecache.so: undefined reference to `EveCache::ColumnLookup::isInit'
git a6a652b88ada9a9e1e4cacf16bbd5b651d30b423 I guess ?
|
Muscaat
ASCENTIS CORPORATION Free Worlds Alliance
|
Posted - 2009.08.24 08:59:00 -
[27]
FWIW I also get that compilation error (running KDE, I have much of Qt installed). ---------- EVE Markets - global price trends at your fingertips |
Herio Mortis
Caldari Dark Nebula Academy
|
Posted - 2009.08.24 13:06:00 -
[28]
Yep, current build is broken.
I used a quick scons to build it myself: --- env = Environment(CPPPATH = ['lib/'])
lib = env.SharedLibrary('evecache', Glob('lib/*cpp')) env.Program('util/dumper.cpp', LIBS=lib) ---
|
Kaladr
Amarr
|
Posted - 2009.08.25 03:00:00 -
[29]
Silly build errors fixed, and new SConstruct file added (for easier top level building) ---- EVE-Central.com - Your EVE economic and corporate site, featuring the inter-region market browser |
Skyvyr
Caldari House Of BlackStar Semper-Mortis
|
Posted - 2009.08.25 13:17:00 -
[30]
Originally by: Kaladr Also, a bunch of CachedMethodCalls cache files can be decoded now, beyond the GetOrders call.
I haven't looked close enough, but are the S&I windows cached in there?
I would love to be able to export a list of blueprints with their me/pe and on copies, run levels.
Keep up the good work! -- I often heard the Sky's the limit, sadly I'm broke fairly often. |
|
|
|
|
Pages: [1] 2 3 :: one page |
First page | Previous page | Next page | Last page |