Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 .. 24 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Dragonaire
Caldari Corax.
|
Posted - 2008.10.22 17:56:00 -
[1]
Hi all I've started a new SourceForge project for people to look at. It's based on some code I've been developing for a while and I decided to release it under GNU LPGL 3.0 in hopes others would find it useful.
http://sourceforge.net/projects/yapeal/ -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
Dragonaire
Caldari Corax.
|
Posted - 2008.10.22 18:00:00 -
[2]
Reserved for future posts -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
Dragonaire
Caldari Corax.
|
Posted - 2008.10.25 23:14:00 -
[3]
New version 0.1.0 available sorry for the broken one had up before. Not that this one might not have bugs but it should at least try to run ;) Also added MySQL files to build Database to db-tools directory. |
Rehtom Lamina
Coalition of Nations Free Trade Zone.
|
Posted - 2008.10.27 14:46:00 -
[4]
will download and have a play tonight.
|
Dragonaire
Caldari Corax.
|
Posted - 2008.10.29 00:11:00 -
[5]
Ok been doing a lot of bug hunting and seem to have eve and corp API stuff working but still working on the char part. Needed to make a couple changes to the membertracking table so you'll need to drop and re-create it for everything to work. I'm not doing a release at this point so if you want the most current code need to get it from the subversion.
svn co https://yapeal.svn.sourceforge.net/svnroot/yapeal/trunk/ -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
Dragonaire
Caldari Corax.
|
Posted - 2008.11.02 20:59:00 -
[6]
I've moved the project over to google code so it's easier for me to manage along with another project I'm working on now. -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
Dragonaire
Caldari Corax.
|
Posted - 2008.11.18 00:56:00 -
[7]
Ok after over 200 revisions according to subversion I think I've killed all the previous bugs that made it unusable. It's got a lot of fixes to database code since I've changed over to using ADOdb (included) instead of MDB2. Yapeal is now fully PEAR free so it should be much easier to install and use. |
Rehtom Lamina
ECM Space Patrol
|
Posted - 2008.11.19 10:49:00 -
[8]
Good work ------ Sig -------
Selling unique, one off web design, Link |
oUTSKIRTs
|
Posted - 2008.11.20 15:47:00 -
[9]
This is really cool, almost everything seems to be working now, the things that doesn't work right now, did work at some point during the revisions, so i'm excited to see how this project will turn out. Right now my cron job is doing exactly what it is supposed to (exept for at the moment 4 tables not being updated due to missing indexes or upper/lowercase issues that i noticed you're hard at work on fixing). Tables not updating is: fuel, starbasedetail, IndustryJobs and MarketOrders
I haven't checked this latest revision 228 out, yet, waiting for cache timer (so tired of that). Thank you for this, it is really really usefull to get rid of the xml problems and get right to work on a database instead. Awesome!
|
Dragonaire
Caldari Corax.
|
Posted - 2008.11.21 02:36:00 -
[10]
Thanks I'm killing bugs as fast as I find them but going to need some help from someone with some of the ones you talked about because the alt I'm using doesn't own any POS so that might make it hard testing
My original code that I extracted Yapeal from had all the eve/ and corp/ stuff I used working but with breaking it out into separate project, changing DB structure, and switching DB libraries many things ended up broken
I decided to work on the char/ stuff first this time since I'd never got around to making it all work before since my other project was mostly corp stuff and usually once it's working for one it's easy to move changes into the other code. That only leaves the few things that's unique to each to fix if they are still broken.
I'm up to revision 249 at this time so look for some more changes to be pushed back into trunk when I'm sure they're work. I'm trying to leave trunk as the more stable branch now that it's starting to work and I don't feel like I'm losing ground like before with it
Also a trick I use during debugging is to have the XML caching on and then delete the records in the CachedUntil table for main script 'eve-api-pull' and which even one I'm test like 'MarketOrder' then see if it works or doesn't It's absolute need when working with AssetList and that 23 hour timer As long as it has a good cache file it doesn't try grabbing one from API and you can cheat on that to for testing by just changing the time in XML. Saves me a lot of 101 errors from API during testing and the server from some of the worse of my typos |
|
oUTSKIRTs
|
Posted - 2008.11.21 12:40:00 -
[11]
It's looking good, nice work!
I am doing some tests here and there, and I like the cache feature a lot, i was trying to get the whole deal to connect through an api-proxy - so that other people still have their eve-mon when i pull the data to this db, but it seems to fail horribly right away. It is probably the proxy stuff not doing it just right. Not really important at this point.
A few more things I have noticed, that I forgot to mention:
There is a table in Yapeal.sql called 'logo' - but the program is calling it as 'corplogo' when trying to access it. It works fine by just renaming the table in the sql.
The starbasedetails (and i think 'fuel' too) table is actually gone from yapeal.sql None of them get any data imported any longer, it has been gone for a while, i think i still have the revision that was infact pulling these two tables allright a while back.. Somewhere.. :)
In the marketorders table, there is a field called "range", very nasty name for a database field - this is also the name of a mysql function, making it fail unless in backtics or maybe calling the field something else might be better... Some say they have tables called range without noticing the problem, really I think it would be better to call it something else. I can see how that might become slightly impractical to say the least, but looking at your code im pretty amazed, you know your stuff and im sure you'll think of something lol
Other than that -- my database is alive, it's
Really close, keep it up ..
|
Dragonaire
Caldari Corax.
|
Posted - 2008.11.21 17:50:00 -
[12]
Glad you like it and thanks for the catch on the logo table but it was the PHP that was wrong. All the tables and fields are directly named from the fields in the API so when there's a difference on something look at Eve-Dev or in the XML for what it should be. So in the case of 'logo' it's now logo everywhere like it was meant to be.
As to the 'range' field yes it sucks and there's a couple others but that what CCP used so that what I use and I just have to make sure I use the back-ticks or let ADOdb take care of any quoting for stuff like that.
I drop the other POS stuff for now because I felt it needed done differently and didn't have time to mess with it right now and no way to test it but I think I'll have some friends from another project to help me out with that now.
My naming for all the main tables should be in CamelCase just like the API calls are and the "sub" tables as I call them follow the XML which seems to be camelCase. The fields are also in camelCase so far anyway so that's what I'm using. For all the backend administration tables they should follow the same naming as well and as I am getting around to it I'm converting all the naming of class, functions, methods, and variables to follow PEAR conventions with a few mods here and there. Example with the function named 'getAPIinfo' I should uppercase the second 'I' and had the 'pi' lowercase but just didn't look right to me and didn't want to re-name it either
Hope this has been helpful on see where some of the changes I'm make are coming from when not smashing bugs as I find them
I've never tested it with any of the proxies out there but did make it so that any that act just like the API should work fine after all what we get from CCP is all through a proxy already on their end. Even if other proxies only follow the API naming and return XML and normal HTTP errors it should work but it would probably lose some of the special caching extras I've built in to work with the main API. -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
oUTSKIRTs
|
Posted - 2008.11.21 18:15:00 -
[13]
This is probably not the way to fix this, but..
Quote:
function makeMultiUpsert(array $data, array $params, $table, $dsn) { // Need this so we can do quoting. $con = connect($dsn); $needsQuote = array('C', 'X', 'D', 'T'); // Build query sections $insert = 'insert into ' . $table . ' (`'; $insert.= implode('`,`', array_keys($params)) . '`)';
... a little further down ...
// Loop thru and build update section. $updates = array(); foreach($params as $k => $v) { $updates[] = "`" . $k . "`" . '=values(' . "`" . $k . "`" . ')'; };
Function taken from the common_db file.
I just punched in some backticks around the fields, and well.. it worked.. I think it shouldn't break anything else :) The table is updated, and it looks kinda ok to me.
|
Dragonaire
Caldari Corax.
|
Posted - 2008.11.21 21:47:00 -
[14]
Never had any problem in that function with 'range' etc because MySQL is smart enough to know that you can't have it inside the insert (range) ... values() ... in duplicate key update range=values('range') statement that it's building and adding extra back-ticks might break it. What I need to do is change to using a prepare, execute instead but I haven't yet because I figured I'd run into trouble some where along the line between my code , ADOdb, and MySQL because of the on duplicate key update part. I didn't figure out a way to make it work with MDB2 but I should take another look at it now with ADOdb and see if it could work. It should be a bit faster to, although that doesn't seem to be much of a problem once I started using transactions for the larger updates of WalletJournal, etc. I've seen an up to 10x speed up since using the transactions. It has to do with DB not have to do as much locking it seems. There was even measurable differences down to 10 records level which is what others have posted on-line so that why you'll see I check for number of records in multipleUpsert() and use transactions for the larger ones now. |
oUTSKIRTs
|
Posted - 2008.11.22 00:11:00 -
[15]
Just checked out rev. 266 and it looks like it is doing the range thing correctly
Real good job
|
Dragonaire
Caldari Corax.
|
Posted - 2008.11.22 01:16:00 -
[16]
Ok transfered all the fixes from char/ to corp/ and everything seems to be working. Can't test StarbaseList at this time so it's only one I really has any questions about. Did a complete drop and re-create on DB and everything works so it seems the code and SQL are in sync now finally. Still might have a few bugs but I've began to test it in crontab off and on now that it seem to be stable enough. It's not like I've ever had it caused problems just wouldn't always run right and filling up it's logs much faster then I could use while fixing problems -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
oUTSKIRTs
|
Posted - 2008.11.22 13:09:00 -
[17]
StarbaseList is updating correctly, you also got the certificates right first time around. Man I totally like this stuff :)
Could you use my cache files to test with?
|
Dragonaire
Caldari Corax.
|
Posted - 2008.11.23 01:28:00 -
[18]
Yep certificates was an easy one to do and for now I'm going to keep try to fill in some of the missing sub-tables before I really try adding more to it. I've had it run now with crontab for over 12 hours and it's never glitch even though API did once or twice on me Been trying it on another hosted site with someone else's info and had a few problems but think it's something not configured right on site and we'll have to wait a couple days until owner is back in town to make some changes. Thanks for the offer on the cache files we'll see how it goes and if I decide to get the POS stuff wrote back in (it's commented out right now) I might contact you about them if friend doesn't get back where I can use him as guinea pig first -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
Dragonaire
Caldari Corax.
|
Posted - 2008.11.23 19:55:00 -
[19]
Ok did some looking around on prepare/execute in ADOdb and found that because mysql does things a little different they only emulate using prepare with it. Since that basically what I'm doing already I might as well use my code as their's While I was looking up stuff on prepare/execute in the MySQL manual I also decided to take another look at the stuff on insert and update about the quoting and decide even though I was right that it worked without problems as is, you were right oUTSKIRTs that the field names should be quoted with the back-ticks to insure no future problems so I've added them now in the latest versions. Glad you said something that made me take another look at it. Have them in there should help cut off some possible mystery bugs if MySQL purser changes or they add something to API in the future that could break it. -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
oUTSKIRTs
|
Posted - 2008.11.24 16:48:00 -
[20]
Thanks for continuing the work on this!
Just following up, the back-tick thing worked ok and look nice and solid too. However some of the changes from rev 274 and up must've somehow cracked the whole deal out, I'll leave my cron running hourly tho, incase it's another temp eve-api oddity, which i doubt actually, I think there is a little something somewhere...
What happens is this: Nothing - pretty much absolutely nothing that I can see
I cleared the cacheduntill table, but it doesn't even update/create the 'eve-api-pull' record, no logs created even with trace on and trace level above 0, trace set to 'all'. It'll just exit, after a short pause - like a second or so - it does SOMETHING that's for sure, and it doesn't exit with any error - no output at all. Rather odd.
The ./cache/* dirs are all writeable. I'll take a look later, will also 'restart' from the svn source with a fresh database and basically crawl the script a bit to see what's up incase you have no idea what this could be and that there could be something that didn't sync correctly (i *might* have a few local changes, but I'm almost 100% sure i have rolled all changes back each time before a sync)
I'm a little too busy the next few days to really go too much into it, but i'll be watching the svn closely :)
|
|
Dragonaire
Caldari Corax.
|
Posted - 2008.11.24 20:32:00 -
[21]
Yeah sorry about that broke it for a couple there but 280-281 works again I just committed what I was working on to test it and had to fix it only took a few minutes but seems you grabbed it while still broken Usually when I break stuff like that you'll start getting error and warning logs in cache/log/ as long is you have them turned on in ini which are on by default. -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
oUTSKIRTs
|
Posted - 2008.11.24 22:39:00 -
[22]
A fresh svn got it back up and running just now
|
Dragonaire
Caldari Corax.
|
Posted - 2008.11.29 20:09:00 -
[23]
Just in case you hadn't noticed Yapeal doesn't have anything implemented to deal with the generation change that has happened with the wallet journal transaction Id. You can read more about the problem in this thread:
Journal API RefID change
I'd kind of forgot about them doing this so hadn't done anything to work around the problem (really was hoping CCP would just move to a bigint) but since they don't seem to plan on do that I'll be looking at a way to work around their bug Since I'm already using bigint for all ID type fields in the database it shouldn't be hard to work something out but main problem will be dealing with php5's lack of an native bigint. I'll update here as I decided on how to deal with it. -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
Grim
Amarr Fallen Angel's Blade.
|
Posted - 2008.11.30 05:10:00 -
[24]
Very nice, major props. Coding an assets pricer and viewer. It is very interesting because I am getting 3 databases to work together as one, one from eve central, the mysql item dump, and the ui. So, I have to say it is quite a challenge.
- Grim -- Death By Titan :( |
Dragonaire
Caldari Corax.
|
Posted - 2008.12.02 23:44:00 -
[25]
Added a new incomplete RegisteredCharacterManagement class and some helper files. Plan on having a couple more for Corporation, and User plus maybe ones to manage groups/ collections of the above. Thought I'd get some feedback on wither someone out there thinks they might be useful and maybe be willing to do some of the testing for me when I finish it up
Still working on the WalletJournal generations stuff hope to have something to test in a couple days on that. -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
ThyShiva
|
Posted - 2008.12.16 23:19:00 -
[26]
Hi, I downloaded the Yapeal code on the 1st Dec so will still have to look at you newest release.
Out the box and after configuring the yapeal.ini file, the alliancelist, conquerablestationlist, errorlist, reftypes and serverstatus tables were populated.
At first I only populated the registereduser table with my Eve details but my char data did not download. Only after reading the code did I realise that registeredcharacter table must have data in it too. There seems to be a step missing because the avg user will not have the information you need to populate registeredcharacter table.
I see youÆre looking for a tester, put my name on that list as your project lends itself nicely to my current distraction.
-Shiva
|
Dragonaire
Caldari Corax.
|
Posted - 2008.12.17 16:52:00 -
[27]
Hi glad to hear someone else finds it useful
The basic information need for the RegisteredCharacter table can be found with a call to /account/Characters which only needs the info from RegisteredUser to get. Setting up and 'registering' characters, corporations, or users is a frontend application function so that why I really haven't added anything in Yapeal for doing it. I have been playing around with some ideas for managing those tables which you can check out in RegisteredAccountManagement.class.php and RegisteredCharacterManagement.class.php. They are untested and just some ideas I've have on how to do it. I've done a little more work on some of the ideas in a branch but it's tied in with some other changes I've been looking at making and I'm not ready show it to others until I've worked on it some more.
For something that's more of a application framework you should look at another project I'm working on with some others at EMPA. It mostly in the planning/outline stage without any real code but it was a large part of the driving force behind the changes to the design of Yapeal. -- Finds camping stations from the inside much easier. Been known to write some PHP code for API from time to time too. |
Tanja Torture
|
Posted - 2008.12.18 13:49:00 -
[28]
Hi Dragonaire
I've setup yapeal on my server. I use one database only (I only filled the first DNS and left the rest commented). After I inserted an entry for RegistredUser and RegistredCharacter. When I run the eve-api-pull.php through "php5 eve-api-pull.php", I get the following message:
Quote:
[2008-12-18 13:40:45.651] Tired of waiting! Getting eve-api-pull for 0 [2008-12-18 13:40:45.658] Tired of waiting! Getting CharacterSheet for 91xxxxx
But the charactersheet and all others tables except "ChachedUntil" stay empty. What I'm doing wrong? I use revision 312.
greets Tanja
|
ThyShiva
|
Posted - 2008.12.18 17:33:00 -
[29]
Tanja û have a look in the Yapeal_warning.log file in the \cache\log directory for any clues, also have a look inside the directory \cache\char for any xml files û they often have a clue to the error too.
Dragonaire û Thanks for the help and new classes û IÆll play with them over the next few days. IÆm looking into asset management as part on my project and I have noticed that you are storing the station location of the asset but not storing what container/ship the asset is in. You will need to store the container/ship ID the asset is in, either in a new field or by replacing locationID with the itemID of the container/ship. I would think a new field is the best way forward.
-Shiva
|
Dragonaire
Caldari Corax.
|
Posted - 2008.12.19 00:28:00 -
[30]
Tanja Torture - Make sure you set `isActive`=1 but from the error messages you give it seems you have. The 'Tired of waiting ...' can mean one of two things either you never tried get that API before so there's nothing in cachedUntil to compare with or it's now more than ~5 minutes pass when cachedUntil says that API expired. It's just a notice really that I setup to help keep track when the API is acting up and let you know it's doing something and follow ThyShiva advise if that still doesn't seem to let you figure it out try turning on some of the trace stuff and look in cache/log/Yapeal_trace.log and see if you can find where it's getting to. I'll need that information to help narrow it down more.
ThyShiva - You'll want to learn about 'nested sets' and how you use the `lft` and `rgt` fields to find what is inside what. Basic idea is if item is inside of something else the child's lft > parent's lft and child's rgt < parent's rgt. You can find more information about it in HierarchicalData which is from another project I'm working on with some other people.
If you or others are interested in help with that project you'd be more than welcome as we've seemed to have stalled a bit after the planning stage We basically want to make a GPL'd modular API application framework that will bring all our varies other API projects together into one and then expand out from there. Like to see all the people working on API applications not be reinventing the wheel so much and get out something we can all use and contribute to so it can become truly useful for everyone. -- Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. |
|
|
|
|
Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 .. 24 :: one page |
First page | Previous page | Next page | Last page |