PDA

View Full Version : ChessLib 0.0.5



skip to my lou
11-07-2004, 10:02 AM
Under Development. Please give feedback.

skip to my lou
11-07-2004, 11:43 AM
Engine code updated, neater layout, download module hooked. The entire row is highlighted when mouse is over it. Clicking a row automatically opens the pgn viewer.

Plans for 0.0.5: headers (fulltext?) search, rating module, member module

skip to my lou
11-07-2004, 05:17 PM
Result logic completed (currently searches movetext)

Edit: Now searches headers since there may be result for variations.

skip to my lou
11-07-2004, 06:50 PM
Paging module completed, (just 3 lines of code).

%perpage (How many games to list per page) and %page (Page number to view)

skip to my lou
11-07-2004, 07:09 PM
bye is excluded from display.

Game is bye if either player name is "bye".

skip to my lou
11-07-2004, 08:53 PM
0.0.5: many more search functions

skip to my lou
11-07-2004, 09:04 PM
Attached:

skip to my lou
11-07-2004, 09:19 PM
Benchmark:

732 Games. Average script execution time: 0.000001 seconds. :eek:

Rincewind
11-07-2004, 09:21 PM
Benchmark:

732 Games. Average script execution time: 0.000001 seconds. :eek:

Not bad. If you want 2.8M games, let me know. ;)

skip to my lou
11-07-2004, 09:35 PM
That would be helpful, please send to karths@gmail.com

I just copied the 700, and added it several times more, so there are a few thousand games now.

It processes 3000 games in 1.5 seconds, but 700 in 0.000001, hmmm something is wrong.

I will try again next weekend.

Garvinator
11-07-2004, 10:33 PM
Not bad. If you want 2.8M games, let me know. ;)
can you send me these games plz.

Rincewind
11-07-2004, 10:39 PM
It's 575Mb zipped (nearly 2Gb uncompressed). Any other ideas? :)

Garvinator
11-07-2004, 11:08 PM
It's 575Mb zipped (nearly 2Gb uncompressed). Any other ideas? :)
i dont think my download limit etc of 1gb would like that too much. My net access deal is that after the 1gb is used, my net is slowed to dial up speeds. Now i guess i could download the 575mb on the 31st of this month right near midnight ;)

So send along if you can using zipped format to garvingray@optusnet.com.au if you can. Would be much appreciated :D

skip to my lou
12-07-2004, 07:17 AM
Barry, do you have a CD-writer?

Alan Shore
12-07-2004, 07:47 AM
i dont think my download limit etc of 1gb would like that too much. My net access deal is that after the 1gb is used, my net is slowed to dial up speeds. Now i guess i could download the 575mb on the 31st of this month right near midnight ;)

So send along if you can using zipped format to garvingray@optusnet.com.au if you can. Would be much appreciated :D

Dude, you can't send 575MB through email, it's just not cricket! Plus it'd take Baz forever to upload... and lag him for days.

Rincewind
12-07-2004, 08:37 AM
Barry, do you have a CD-writer?

Yes, you want to email me an address and I'll post you a CD?

(Who said you can mail a 0.5Gb attachment?) ;)

Garvinator
12-07-2004, 09:05 AM
and lag him for days.and this is my concern how
;) :lol: :lol: :lol: :uhoh: umm ok maybe if that cd idea is done, that could work :confused: i dont know much about this stuff, would say four emails or something work? :confused:

Rincewind
12-07-2004, 09:43 AM
and this is my concern how
;) :lol: :lol: :lol: :uhoh: umm ok maybe if that cd idea is done, that could work :confused: i dont know much about this stuff, would say four emails or something work? :confused:

Youd want to do a lot more than 4 emails. Probably around 50. It could work but it would be tedious. Still there are binary news group utilities which would automate much of the work.

The other option is if Jeo had a ftp server with a good connection I could upload it from a network connection with a good upload transfer rate.

Garvinator
12-07-2004, 09:58 AM
The other option is if Jeo had a ftp server with a good connection I could upload it from a network connection with a good upload transfer rate.
seems like a lot of work :uhoh:

Alan Shore
12-07-2004, 10:38 AM
i dont know much about this stuff

Aren't you a computer programmer?? Or do some kind of computer related work?

Garvinator
12-07-2004, 10:41 AM
Aren't you a computer programmer?? Or do some kind of computer related work?
nope, i dont have any IT experience. I have done stuff with microsoft word, excel and all that stuff, but nothing to do with making web pages or programming. If i did have web page experience, i would make a new web page for the 2004 gp :whistle:

Alan Shore
12-07-2004, 10:43 AM
nope, i dont have any IT experience. I have done stuff with microsoft word, excel and all that stuff, but nothing to do with making web pages or programming. If i did have web page experience, i would make a new web page for the 2004 gp :whistle:

OK, what is it you do then? (Apart from full-time BB poster) :P

Garvinator
12-07-2004, 10:47 AM
OK, what is it you do then? (Apart from full-time BB poster) :P
is there anything apart from bb posting :uhoh:

skip to my lou
12-07-2004, 06:54 PM
Youd want to do a lot more than 4 emails. Probably around 50. It could work but it would be tedious. Still there are binary news group utilities which would automate much of the work.

The other option is if Jeo had a ftp server with a good connection I could upload it from a network connection with a good upload transfer rate.
Do you have a good upload connection? If you do, I would rather set up an FTP server for you.

Rincewind
12-07-2004, 09:40 PM
Do you have a good upload connection? If you do, I would rather set up an FTP server for you.

I can easily copy the file onto my laptop and ftp from a network with a good upload connection. If you want to proceed set up an account and email me the details.

skip to my lou
13-07-2004, 11:55 AM
Ahhh, I found out why its taking so long to search... I forgot to unload the fen module, so it was generating fen for each and every position in every game :classic:

At the moment the fen module just lags down the server like anything, so I will probably have to write the fen strings into the DB when the user uploads the pgn. Not yet sure about this yet either, because it would require massive amounts of space.

skip to my lou
13-07-2004, 12:18 PM
I have a couple of more ideas that I will implement soon:

1) Process the pgn and write the headers into different columns so that the engine doesn't have to loop through every games header. This will shave off a few ms for sure.

2) Only request the games from the mysql server that need to be displayed on that particular page. This will probably needed, especially when there are millions of games. (Done! All I had to do is use the LIMIT function together with the paging module. Script much faster now.)

skip to my lou
13-07-2004, 01:54 PM
Barry has uploaded the 2.8M games which is 562MB in total.

Im downloading at 500kb/s, so it may be slow for a little while.

http://www.chesskit.com/barry/

Garvinator
13-07-2004, 02:40 PM
Barry has uploaded the 2.8M games which is 562MB in total.

Im downloading at 500kb/s, so it may be slow for a little while.

http://www.chesskit.com/barry/
thanks :D i will download as end of month cause of download limits :(

skip to my lou
13-07-2004, 03:29 PM
The pgn file is 1.9gig once unzipped.

How would I go about inserting this into the mysql db?

PHP just returns a 404 when trying to read it, but when I reduce the file size to a PGN of a few megs, it works fine.

:(

skip to my lou
13-07-2004, 03:56 PM
How would I use chessbase 8 to automatically split this file into 1000 or more pgn files?

Rincewind
13-07-2004, 04:03 PM
How would I use chessbase 8 to automatically split this file into 1000 or more pgn files?

I guess you could but I wouldn't. Why not just use a text editor? Or write a quick script in perl or something to do the trick?

arosar
13-07-2004, 04:12 PM
Hey listen youse blokes....I got a bit of a question.

How do I delete games with a plycount value of nil from the CB interface? Or do I have to go into the PGN and delete them manually?

AR

skip to my lou
13-07-2004, 09:59 PM
I guess you could but I wouldn't. Why not just use a text editor? Or write a quick script in perl or something to do the trick?

Barry, I would have done it in Perl or PHP, but a 1.9GIG file is too big for it to handle, anyway, I found this pgn splitter thing, will try it and see how it works. Probably the computer im using can't handle it (3x256 DDR) maybe I should get on my other comp.

skip to my lou
13-07-2004, 10:12 PM
It stops after 55,638 games... :(

ok, how would I do it in chessbase?

skip to my lou
13-07-2004, 10:24 PM
Barry there is an error after 55,638 games.

Probably be better off doing it in chessbase.

Rincewind
13-07-2004, 11:13 PM
Barry there is an error after 55,638 games.

Probably be better off doing it in chessbase.

What sort of error?

skip to my lou
13-07-2004, 11:37 PM
Parse error.

So it just says done because it thinks its EOF.

skip to my lou
14-07-2004, 02:19 AM
I emptied the table and then added 49,500 games exactly to the MySQL DB.

It takes 0.03 seconds to search on my AMD 2100. I wonder how long it will take on the Dual Xeon :eek:

Anyway, Barry, I have come across a new problem. Some moves have space between the move number and notation, some do not, and it seems to be very random.

I tried using reg.ex in the SQL, but it takes 6 seconds to process the the 49,500 games with reg.ex.

skip to my lou
14-07-2004, 02:22 AM
After rebuilding the index, search time is 0.003 seconds. Although I am unable to regex to alow space between the dot and the move. Do you know of any custom wildcard in mysql? I know the _ char which finds any char, so it could even be a piece, so that doesn't work out.

skip to my lou
14-07-2004, 02:39 AM
Furthermore, _ or % do not attempt to find spaces.

skip to my lou
14-07-2004, 09:30 AM
I guess I can format the input on write and search..


"..." -> "$~."
". " -> "."
"$~" -> ".."

skip to my lou
15-07-2004, 08:28 AM
New pattern searching module.

E.g, you can search for patterns like- Knight Manoeuvre: Nb1-d2, Nd2-f1, Nf1-g3, Ng3-f5

Rincewind
15-07-2004, 10:11 AM
Parse error.

So it just says done because it thinks its EOF.

Perhaps it is just a problem with one game. Have you tried removing the offending game and start again?

I generated the file but exporting the games from a SCID database into PGN format. This should have resulted in a reasonably uniform format but quality control on a databse of that size would take more time than I have.

skip to my lou
18-07-2004, 02:01 AM
Perhaps it is just a problem with one game. Have you tried removing the offending game and start again?

I will try tommorow with the DBMS you suggested.

skip to my lou
19-07-2004, 01:27 AM
Tried ChessBase. Very annoying. The slightest changes take ages to make, and its easily crashing my computer.

Will try SCID soon.

paulb
19-07-2004, 08:47 PM
Benchmark:

732 Games. Average script execution time: 0.000001 seconds. :eek:

I think this is a *fantastic* project, but I *think* you're making a practical mistake. What I mean is that it *looks* as if you can only download/view one game at a time, and I think most people are going to want to download a whole file of games (eg all games that used a certain opening) or at least to very quickly view them on line, via a tree.

It would be useful to compare Jude Acers' ChessLib, which is *almost* useful - it's too slow, and it's too cumbersome to use, and it's too hard to *get the games*

Don't mean to be negative - as I say, this is a very good project - but I think you need to carefully consider what people will want.

Best of luck with it.

cheers - PaulB

skip to my lou
19-07-2004, 08:54 PM
I think this is a *fantastic* project, but I *think* you're making a practical mistake. What I mean is that it *looks* as if you can only download/view one game at a time, and I think most people are going to want to download a whole file of games (eg all games that used a certain opening) or at least to very quickly view them on line, via a tree.

This is no problem. This may not seem like it, but it is basically cosmetics. I will add it to my interface changes list and do it when I get a chance.

What I am concentrating on is to make the engine stable and perform basic functions very very fast. At the moment, it is searching 45,000 games in 0.002 to 0.003 seconds.


It would be useful to compare Jude Acers' ChessLib, which is *almost* useful - it's too slow, and it's too cumbersome to use, and it's too hard to *get the games*

Don't mean to be negative - as I say, this is a very good project - but I think you need to carefully consider what people will want.

Thanks. I will need more feedback like this so I can make it a good and useful program.

arosar
20-07-2004, 09:19 AM
Does this project have a scope of work? Let's see it so it's easier to add a wishlist.

AR

skip to my lou
20-07-2004, 11:23 AM
I will release 0.1.1 today along with alot more games

Trent Parker
20-07-2004, 12:42 PM
so what is chesslib 0.0.5? It's obvious that it contains a database of games.

skip to my lou
20-07-2004, 01:00 PM
An online chess database search engine that will contain millions of games. Useful for everyone, especially those who don't have access to a laptop. Eventually it will become much more than just a search engine.


I'm building a chess game search engine in PHP/MySQL. Currently, the search engine can search by headers (name, rating, tournament etc) or by moves (1. d4 d5 etc) or by fen position (search every position available in every game). Games are added to the database by a simple PGN uploader, similar to the one on this forum. There will be a base "library" and several different selections of games (like an iTunes playlist). So when I make it account based, you can manage your own favourites list, ratings etc. You can also add your own games (whether these games are made public or not is your choice).

skip to my lou
20-07-2004, 06:55 PM
DB now contains 1,200,000 games.

Not using the search engine, but simply browsing the table: Showing rows 1199970 - 1199999 (1200000 total, Query took 39.0526 sec)

For some reason the tables are locking up and CPU max during query is just 5% :confused: :(

Rincewind
20-07-2004, 07:01 PM
DB now contains 1,200,000 games.

Not using the search engine, but simply browsing the table: Showing rows 1199970 - 1199999 (1200000 total, Query took 39.0526 sec)

For some reason the tables are locking up and CPU max during query is just 5% :confused: :(

The queries are probably IO constrained not CPU constrained so you wont max out on CPU. Also the locking mechanism might be table level locking. Although I wouldn't expect a select to perform any locking. Still if you can configure the table or database to perform row-level locking this might help. Not sure how to do this in MySQL. But check out the documentation on row-level locking as this is the terminology used in other RDBMSs I've used.

Other idea is to spread the database across as many spindles as possible. 6x20Gb drives are better then 1x120Gb drive frmo a database performance point of view. You probably won;t be able to easily equip a single machine with 6 HDDs but the more physical disks the better and spread the database over them all.

skip to my lou
20-07-2004, 07:07 PM
Only InnoDB has row locking, MyISAM (the one I am using) only has table locking. But still it is locking on its own, for some odd reason.

Well If I got two or more RAID disks and striped them would it help?

I guess ram disks would help, if IO speed is the problem.

skip to my lou
20-07-2004, 11:12 PM
Ok, found out the problem. MySQL does not use index at all when there is a wildcard in the start of the search string.

This has improved search from move 1, but it does not affect search times for patterns or even browsing records towards the end of the table.

Hmm, this is confusing, there is absolutely no overhead on the table. :hmm:

Rincewind
20-07-2004, 11:53 PM
Only InnoDB has row locking, MyISAM (the one I am using) only has table locking. But still it is locking on its own, for some odd reason.

Well If I got two or more RAID disks and striped them would it help?

I guess ram disks would help, if IO speed is the problem.

Yes, or if you can configure the DB to use a memory cache it can help a lot.

Don;t know about RAID. Are devices added to the database like disk partitions? If so, perhaps you can just create partitions on every drive you have and then add these as devices in the database.

skip to my lou
21-07-2004, 12:02 AM
I am not exactly sure about the theory. My understanding was that the more physical disks, the quicker the db reading since there will be more than one pointer (assuming that it is doing a full table scan).

I can configure RAID to mirror the disk. There may be options to add devices as disk paritions but I have never used such a function so I will do some research about this when I get a chance.

skip to my lou
23-07-2004, 07:44 PM
Position searching update:

I will be hardcoding the FEN values into the DB. This has its advantages (much faster searching by position) and disadvantages (DB size is blown up at roughly 10 times its original size).

Garvinator
23-07-2004, 08:10 PM
disadvantages (DB size is blown up at roughly 10 times its original size).
will this increase the amount of mb needed for downloading? are all these games available for download now, if not when?

skip to my lou
23-07-2004, 08:16 PM
downloading what? why would you want to download ALL the games? That just totally defeats the whole purpose of this program.

There was a master.pgn that was up on barrys chesskit FTP. Its now gone and replaced with a split version. Almost 1000 files.


disadvantages (DB size is blown up at roughly 10 times its original size).

Garvin, for the online search program ChessLib, which this thread is about, since its titled ChessLib, I will be writing Fen Strings for each and every position physically to the database. This means they are hardcoded and are not generated each time you search, thus saving CPU power and alot of other dynamic resources.

Think about it, I have 2.8 million games in the database. Lets take the FEN for some move:

For the moveString: 1. d4 d5 2. Nc3 Nf6, I will have to store:

rnbqkbnr/pppppppp/8/8/3P4/8/PPP1PPPP/RNBQKBNR b KQkq d3 0 1
rnbqkbnr/ppp1pppp/8/3p4/3P4/8/PPP1PPPP/RNBQKBNR w KQkq d6 0 2
rnbqkbnr/ppp1pppp/8/3p4/3P4/2N5/PPP1PPPP/R1BQKBNR b KQkq - 1 2
rnbqkb1r/ppp1pppp/5n2/3p4/3P4/2N5/PPP1PPPP/R1BQKBNR w KQkq - 2 3

in a seperate field. I hope you see what I mean about blowing up the database size.

skip to my lou
25-07-2004, 03:32 PM
Currently you can search by position moves event site date round white black result
whiteelo blackelo eco plycount

What else should be searchable?

Rincewind
25-07-2004, 05:15 PM
Currently you can search by position moves event site date round white black result
whiteelo blackelo eco plycount

What else should be searchable?

Sometime particular material set ups are useful to search on. For example K+R+N+P vs K+R+4P, that sort of thing.

skip to my lou
25-07-2004, 05:47 PM
ok is there any documentation on how to develop those strings? I mean what are the possibilities of input?

Rincewind
25-07-2004, 09:51 PM
ok is there any documentation on how to develop those strings? I mean what are the possibilities of input?

Here is an example of the material search facility withni SCID. Take a look at it and it might give you ideas.

skip to my lou
30-07-2004, 09:04 AM
Thanks for the suggestion Barry.

I will start working on this in a couple of weeks, after I get the Open 05 site up and running.

skip to my lou
29-12-2004, 08:44 PM
:( Can't find where I put the DB structure. Will look through all my DVD's again.