Time has long passed. I’ve expressed interest in a chess engine in Julia, then the interest died down, and then some other idea, and then lost motivation due to the core code I need to write before I could get to the search logic.
In the meantime, I got 3 Stockfish patches passed (2 gainers and 1 simplification). I guess ideas are more easily tested in Stockfish?
So, is it over? But I don’t think so. There’s still one thing I want to explore. It deals with the fundamental system of engine development, the Sequential Probability Ratio Test (SPRT). Top chess engines nowadays use the process where they would write a new patch, test the patch in isolation, and then wait for the result. The problem is that it is INCREDIBLYYYY slow, resource intensive, only tests one patch at once, and so on.
So, I came back to ask “Can we do better?” Maybe ideas are much more easily tested in Stockfish, but testing a large amount of ideas take a lot of time?
Of course, testing this on Stockfish or other top chess engines is not viable. This is a risky proposition and it might not yield any benefit. So, a new engine would have to be made if you want to test it. However, is there even a way?
So, I’m asking the Julia community, not the chess engine community, specifically the optimization community.
Here’s the characteristic of the optimization problem.
-
We’re optimizing for the “playing strength” or “elo” of the chess engine.
-
The elo might not be perfectly transitive, but in practice, it is transitive enough to assume so.
-
Elo cannot be independently measured. The only way to measure elo is relative to other players. (By playing games between two versions)
-
The measurement is very noisy and expensive (Again, by playing games between two versions.)
-
Some variables are discrete. Some are continuous. Some variables are dependent on other variables being active (for example, an idea might have 3 variants or 3 parameters.)
-
The ideas of the programmers is not a bottleneck. It is not too hard to have the developers generate thousands of ideas. The bottleneck is testing them. These human-coded idea should be better than random ideas, but they’re still, on average, not easily improving the engine. If the optimization relies on millions of dimensions, however, you could automatically generate ideas, so, the number of ideas is not the bottleneck.
Julia has its strength in its ability to dynamically compile code, which allows patches to be tested in various combinations. However, the problem is what algorithm I should use to optimize given the constraints of the problem.
I know I might be giving the optimization community a hard time, but is there any insight on the optimization problem that I could do?
Thank you in advance.
P.S. History has shown that people trying to invent better system than SPRT have failed, so it won’t be easy, but it’s not proven that it can’t be done either.
So, what’s your take?