Neoclassical chess: Technical article
Foundations of Neoclassical Chess
Gabriel F. Bobadilla
The prevalence of computer-aided opening preparation have compromised in practice the interest of chess as a human competitive activity. This problem of the opening was addressed under a logical/mathematical perspective.
We specify all desirable conditions that new games must fulfill to be as similar to classical chess as possible. We ask which of this is “closest” to classical chess while satisfying constraints on the probability and cumulative distribution functions of the collection of possible initial positions from where the new game starts, which specify “sufficient randomness” at the beginning of play in the new game. For this we define a notion of distance in the universe of games and set the problem as a constrained optimization.
We define the Neoclassical chess universe and show that it is a collection of solutions of this problem for different levels of constraints, and show why Neoclassical chess at depth 3 (i.e. after three full moves) is the solution for human play, given its empirical distribution of openings at several depths.
We also show and that furthermore, this solution is “essentially” unique (i.e. except for minor variations).
This is what we mean by saying that Neoclassical chess is essentially the only possible evolution of the game of chess, i.e. that solves the above-mentioned problem and, at the same time preserves the integrity and history of the game and fulfills all desired conditions.
2 THE SOLUTION: NEOCLASSICAL CHESS
2.1 PROBLEM FORMULATION
We assume that the game of classical chess is the absolute ideal but that as mentioned, the last decades of opening study and later the prevalence of computer-aided opening preparation have compromised in practice its interest as a human competitive activity.
Hence we pose the following problem, stated in the broadest terms:
What is the minimal modification of the game of chess that eliminates the problem described in the previous chapter, i.e. makes the value of opening preparation based not on understanding but on rote opening memorisation decrease by a sufficiently large factor, subject to the additional conditions of preserving both the history and traditions of the game and providing for its future evolution, so that it is acceptable to a large majority of the chess world?
Thus, we demand the following conditions from the new game to be a solution:
1. A deterministic game equal to classical chess except in the opening. Nc chess must be a deterministic game, i.e. from its initial position the game will proceed as in classical chess and will follow all its rules, and White will initiate play. However the initial position in Nc chess could be anyone drawn from a collection of legal classical chess positions. Such process of choice will be universal and independent of the specific players participating in the game .
Note: This ensures that the middle-game and endgame are left untouched, only the start of the game may change. Also, note that some randomness must be allowed, otherwise the problem simply cannot be solved.
2. A game of skill, not chance. Any initial Nc chess position must be balanced, i.e. its evaluation must be practically equal to that of the initial classical chess position (which is: a very slight advantage to White). This is necessary to ensure that Nc chess is not a game of chance, but of “pure” skill, exactly as classical chess. This in turn is essential to the identity and history of the game, see for example [History of chess, Eales], B. Franklin and many others.
3. It integrates (subsumes/ includes) the history of the classical game, (“history” in the sense of the collection of classical chess games played up to the present moment). A large majority of relevant classical games of the contemporary (recent) past must be possible as a realisation of a Nc chess game, i.e. Nc chess must be “backwards consistent” with classical chess. A “large majority” because we have to discard the beginning of any classical games that is not fully correct.
4. It fully preserves the history of the classical game. All the future instances of Nc chess must have been possible at any time during the past of the game, i.e. Nc chess must be “forward consistent” with classical chess. Here we must allow all games.
5. It reflects the contemporaneous and future opening preferences of the human classical game. Neoclassical chess must incorporate the collective opening preferences of players of a sufficiently high skill level, and provide for the future evolution of opening theory. This incorporates the requisite that the probability distribution of openings in the new game not be arbitrary or random, but reflect actual recent human opening preferences.
6. Universal, easy setup process and start of play (practical condition): The setup of Nc chess should not be substantially more complex than that of chess, and must be based on an independent, clearly defined, universal process.
Note that we propose a parsimonious approach, with the explicit desire of modifying chess as little as possible. The modification is minimal in order to make possible widespread acceptance of the new game, as different from other variants of chess previously invented.
In this regard, note that chess960 (“Fischer-Random chess” or “FR”) fulfils 1, 2 (except in a few initial positions of FR where apparently the first move advantage counts too much) and also 3 and 6 but fails on 4 and 5. The key new conditions, 4 and 5 are the ones that embody the objective of achieving a true evolution of classical chess, and not a mere variant.
these are all very reasonable, if somewhat demanding, conditions. But 1 and 2 are essential (they were already achieved in FR, so we should not renounce them) and there is abundant anecdotal evidence that the chess community has shown that it needs also 4 and 5.
But additionally, Neoclassical Chess fulfils the following condition (sufficient randomness of the set of beginning positions of the new game), which classical chess does not fulfil, as there is 100% certainty of starting play from the one initial classical chess position.
Randomness Performance Criteria (RPC): the probability of starting the new game from any particular initial position must be sufficiently small, in order to sufficiently diminish the probability of occurrence of memorised opening lines. This would be sufficient to achieve some degree of rebalancing (so that the study of the opening does not overwhelm that of the other phases of the game) and to keep long-term knowledge, but not rote memorisation, of the opening as a desirable part of the game.
Note that as the openings distribution is not uniform, and in fact highly skewed, there is not a single factor (as K in the description of FR above) that if sufficiently large expresses the decrease in probability, but there are several conditions.
The first one is that the probability of starting the new game from the most likely initial position must be sufficiently small (lower than Z%), and the number of most frequent initial positions that taken together appear at least 50%, 66.6%, 75%, 90%, 95% of the time must be respectively higher than N1, N2, N3, N4, or generally constraints in the cumulative distribution function once we rank-order the openings in decreasing order of appearance , so that a sufficiently high number of possible initial positions appear for any given degree of assurance.
Note that classical chess does not fulfil the RPC, as there is 100% certainty of starting play from the single initial classical chess position.
Note that the performance criteria is also key. In our conceptual framework, if we do not demand it what we obtain as a solution is simply classical chess, which of course is the closest game to itself, as it already satisfies the criteria above from 1 to 6. But this emphasizes our deliberate idea of “minimum perturbation”: we want the solution to be as close to classical chess as possible. Also, what we are implying is that classical chess of course fulfils all desirable conditions that characterize it, but fails on the performance criteria, which embodies our intent of escaping from the opening problem.
2.2 PROBLEM SOLUTION: NEOCLASSICAL CHESS
The solution is the following. We define the Neoclassical chess universe as the collection of the following for all depths D = 0,1,2,3… up to a reasonably high number of (full) moves, Dmax = 30 is more than enough [get frontier opening / middle-game].
Neoclassical chess at depth D:
A game of chess where the players start the game from the position, after D correct full moves (i.e. D from each of White and Black), of a game drawn from the empirical probability distribution of relevant contemporary human master play.
Or in more practical terms:
A game of chess where the players start the game from the position, after D correct full moves, which is taken from a game chosen at random from a database of all relevant contemporary human master games.
For even more practical view, see “ Implementation” below. Note that classical chess is precisely Neoclassical chess at depth D = 0.
Hence the game which is the solution to the problem, which we will simply call “Neoclassical chess” is Neoclassical chess at depth D*, such depth being the minimal one that satisfies the desired “performance criteria” above: sufficient decrease in the probability of occurrence of memorised opening lines, and which we will take as the standard depth.
In practice, we will explore in 4.4 what number D* should be and will then find that highly likely D*= 3 (or failing that D*= 4) is the minimal depth that achieves sufficient performance.
Note that in the definitions above above we mean:
Chosen at random / Drawn from probability distribution: importantly, as many games will repeat positions, those positions are not equal-weighted, but appear in the same proportions as those in the collection, reflecting the distribution of collective opening preferences of “masters”. Note that as we draw from the empirical distribution, for a sufficiently large number of games we also get the same expected distribution of openings in Nc play (we here avoid any finesse of statistical correction factors as the number of games is very large)
Recent: In a sliding time window, the last N years up to the current moment (not from the beginning of the history of chess), i.e. so that it captures an evolving distribution of openings. As the window is sliding, it also provides for the evolution of chess opening preferences. Example: we may take all relevant games from 2010 to 2014, both included.
Correct: the evaluation after each move changes negligibly, so that the attained position does not deviate significantly from that of the initial classical position, i.e. if the initial evaluation is 0.2 it must be between 0 and 0.4. i.e. a fully playable position. In practice this criteria (that can be checked after sampling) is redundant in master play if we eliminating outliers (weird openings that do not reach for instance 20 games in 200,000 or less than 0.01% ), as mistakes are eliminated.
Relevant human master play: master in a broad sense, i.e. players of sufficiently high skill level, and taking (relevant) rated play competitions. I propose taking rated games where both players are above a certain rating (a typical IM/GM level but not super-GM level as too high a number may make the resulting universe to be narrowly influenced by conservative approaches at top competitive level). Also, a reasonable criteria is to eliminate blitz and blindfold games which can bias the distribution due to the specificity of these forms of play. This could be further sophisticated using a percentile of top-rated players and the rating limit could evolve slightly over the years; again this complication adds nothing given the robustness of the results.
Note that given our choice of definition of Nc chess, conditions 4 and 5 are now satisfied.
Note also that the performance criteria must be verified by some depth, hence there is always a D*.
An essential, explicitly sought after, feature of the solution is its practical lack of arbitrariness, as the openings distribution is very robust to changes of these parameters within reasonable ranges, while the key depth parameter is minimised as long as the performance criteria appropriate for top-level competitive play is verified.
2.3 UNIQUENESS OF THE SOLUTION
Claim: Neoclassical chess is a solution to the problem stated in 4.1 and is “essentially” the unique solution (under a mathematical perspective) that satisfies all conditions 1-6 and the performance criteria at any given pre-specified level.
To show that we have a solution, verify that the definition of Neoclassical chess above satisfies all the conditions and the performance criteria. Their verification is straightforward. Uniqueness is very important, and a little more difficult, requiring also to be more precise about the meaning of “closeness” to classical chess. See the Appendix for a sketch of proof. In fact, the solution for different levels of performance is the collection of neoclassical chess at different depths (which is a “base” of the solution), but also the probability mixtures of all these “pure neoclassical games”. We show that these more complex solutions add nothing in terms of performance.
2.4 THE STANDARD DEPTH D* IN NC CHESS
We have not specified the RPC exactly, as there is a discrete choice of depth, we will choose by increasing depth.
The criteria (or game collection, or opening distribution) can be updated at the start of each year, but given the slow evolution, once every few years is probably enough in order to publish an updated Nc chess standard.
Only the standard (optimal) depth is the crucially important parameter.
See below a typical result ( Minimum rating is 2600, for last 10 years). Similar in all others cases-
Note there is a significant stable singularity of the openings distribution of human play at depth 3
Table of number of initial positions
Depth d (in full moves) Most likey opening probability 50% most frequent number of initial positions 66.6% most frequent number of initial positions 75% most frequent number of initial positions 90% most frequent number of initial positions
3,7 3 4 5 11
17 5 9 15 57
17 16 42 76 252
19 61 162 —- —-
Note that initial positions must be multiplied by 2 to take into account preparation as both black and white.
Note the significant jump downwards of probability of most frequent opening from d=2 to d=3, (15% to 6%) then stabilization
d=1 and d=2 are clearly not high enough. Even d=2 is insufficient for many top players with extensive years of preparation of broad repertoires, and may even induce even more weight of opening preparation.
D = 3 Is sufficient. We want even the more memory-gifted players to enter into memory overload, and achieve it here.
D=4 Too large, if effects a higher degree of style imposition (think of the Nimzo, the Sicilian), but it’s highly likely that no higher depth is needed in any case.
This strongly indicates that depth is D*=3 is the solution for human play.
A standard implementation: using a database of games as described in 2.2
After experimenting with different values of the time window and rating level which define “master games”, the distributions are quite similar: i.e. extraordinarily robust to choice of parameters, and evolve slowly, and crucially they show a similar phenomenum for depth 3.
Truncation: instead of only outliers, in an introduction phase of Nc all opening outcomes below a certain threshold of probability may be discarded, instead of 0.001% something larger as 0.06% (or some aggregate tail of the distribution), as introduction for an initial tournaments, given that players will not have had time to adapt and study lesser-known openings outside their repertoire.
Very important: robustness w.r.t. the choice of exact algorithm parameters (ELO, time window).
2.6 ALTERNATIVE IMPLEMENTATION AND VARIANTS
The most significant alternative way in which the procedure described could be enacted, leading to very similar results, is by postulating a distribution of opening sequences (with their relative frequencies) as representative of reference play at a certain depth D, which we call “reduced-form” distribution, in place of the game database or distribution itself, and by checking that it does not differ in some or all of the main filtering and truncating criteria expressed above from the game database. Hence this non-uniform probability distribution or frequency distribution of opening sequences could be then used to obtain directly the opening sequence by means of generating a random choice in a non-uniform manner, i.e. taking into account the frequency (or “weight” of each distinct opening sequence) of each.
We claim that the first and the alternative procedure, using a “reduced form” of the database, is essentially the same.
Also, establishing a non-integer depth (i.e. allowing depths as 0.5, 1.5, 2.5… etc) or equivalently, and odd number of “plies” (where a “ply” is a half-move, i.e. a move from only the player with either the White or Black pieces) will result in a variant of the new game in which positions are obtained such that the last move is a move by White. This modification of the new game is not without interest as White would retain the advantage of having moved first, but Black as partial compensation has the slight advantage of making the first choice after the position is obtained randomly. However it is more different from chess than the whole integer depths described above. It could also have interest as an alternative variant of chess and for chess training purposes.
A further modification would be to select initially at random the depth to be used (step 3 above), instead of using a different, fixed depth for different purposes. For example, a random draw would be done from an initial distribution that assigns a 10% probability to using depth 1, 15% to depth 2 and 75% to depth 3. Then once the depth is obtained, the rest of the steps would follow similarly.
Another modification would involve using the reduced form above and “branching out” several of the “opening sequences” at different depths, using the actual probabilities involved in the branching. For example, most of the opening sequences could be of depth 3 except one where the sequence has been replaced with 2 subsequences of depth 4 with 2 respective frequencies adding up to the frequency of the sequence they replace. This possible modification may make possible a variant with opening possibilities conforming to some conventional opening classifications, preexisting or otherwise, i.e. having different depths, while they result in a complex game specification, where the depth of the initial position of the new game could vary. This is another possible variant, that cannot be described as a process independent of the actual statistical distribution of games, whereas the main form of the new game that we have described can be determined by the simple choice of the first D moves of a random relevant master game, i.e. independent of the actual distribution of games. However, it is yet a further simple modification of the original invention.
Using other depths higher than the optimal standard D=3
D=4 For decreasing the value of opening preparation even more, to less than other phases of the game.
D=5 Neoclassical brand similar to chess960 but with chess flavour
2.7 PREDECENTS OF NEOCLASSICAL CHESS
The work herein described (together with the companion paper by the author “Neoclassical chess: a bright future for the game”) has been carried out fully independently by the author. An ex post internet search has revealed two precedents of our proposal, i.e. two similar previous inroads into the problem:
Eric Schiller. “Why Fischer-Random is not the future of chess”, web entry (2011)
The earliest precedent that I have found. Besides a cogent critique of ches960 as the future of chess, he recognizes the importance of making the initial beginning of play less predictable, among several other key issues. He makes an interesting proposal for training and as an alternate form of chess, for instance starting play from a collection of sharp tactical balanced situations. This approach is close in spirit to the present work but falls short of a full method of play or an evolution of chess.
Daniel Lakeland “Randomized Chess” in “Models of reality”, web entry (2013)
A computer and handicap oriented approach. It lacks the crucial parsimony of the optimizing, minimal deviation approach here taken, and the idea of using the empirical distribution of actual master play.
These two authors, in our opinion, lack the conceptual problem framework that I have described, in their proposal the “database” is unspecified and the depth remains arbitrary, as the goal is not articulated as in here. As depth is unspecified, they end up with suggestions of too large depth that compromise excessively, and unnecessarily, the style and character of the resulting positions. All these results in two interesting new concepts, which to my mind are already superior to chess960 for chess lovers, but that do not quite achieve the status of a full evolution of the classical game as I intend here.
Note how the essence of neoclassical chess is the assumption that classical chess is the ideal, so we must depart from it the least possible to achieve our objectives, no more.
3 APPENDIX: UNIQUENESS
[This is in draft form – to be updated.] Conceive a stochastic process of correct master play. This is an evolving probability distribution of games, with its finite-dimensional projected sub-processes at depth d
Map out all the solutions to the problem.
Define mixtures i.e. another solution could be to take a 30% chance of starting at depth 0 (classical game), 50% of starting at depth 1 and 20% of starting at depth 2. These mixtures are cumbersome and hence confusing and undesirable for practical reasons. In any case, they do not improve on pure neoclassical solutions, i.e. there always exists a pure neoclassical solution at least as good in performance and that is, worst-case, closer to the initial position.
Mixed depth solutions (not distribution-independent). They arise from the “reduced form of probability distributions” or from expanding (branching-out) from the probability tree.
To formalise closeness to classical chess of mixtures of neoclassical games, define a metric space on the finite probability vectors [p1, p2, p3,…] where p1 = prob of No, p2 = prob N1 etc, p3 = prob N2 etc and No is the classical game, N1 is the neoclassical game at depth 1, N2 is the neoclassical game at depth 2, etc
Then dist(P,Q) = ind [max (k)[ abs(pk-qk)>0], defined as 0 if all components are zero
Proof that it is a distance: trivial dist=0 iff P=Q and symmetry. Triangular inequality: by induction.
This is a very natural definition of closeness (worst-case, risk aversion): distance to the initial position of classical ches is measured as the largest number of moves from which the players may have to start the game. It can be generalised to non mixtures (weak solutions)
This distance ensures solution. We aim to find distributions closest to [1,0,0,..], i.e. classical chess. For any mixed random solution, a pure neoclassical game is at least as good, i.e. achieves at least the same level of “performance” vs. objectives and is at least as close to classical. Hence mixtures of neoclassical chess do not offer better properties, while they are clearly more cumbersome and impractical.
It is in the sense that we predicate “essential” uniqueness, i.e. mixtures of neoclassical are never better and are more cumbersome. Hence we can stay with pure neoclassical games of different depth.
4 ABOUT THE AUTHOR
Gabriel Fernández de Bobadilla Osorio
He received a Master’s degree in Economics and Finance, a two-year full-time graduate program, from CEMFI (Centro de Estudios Monetarios y Financieros, Foundation of the Bank of Spain, Class of 1998), with 1st of class honors (“Premio Extraordinario”). He did his master’s thesis on stochastic models of the term structure of interest rates (the yield curve).
He also holds a Dr.- Ing. degree in Applied Mathematics from Universidad Politécnica de Madrid (U.P.M.), with a doctoral thesis on matrix models of linear dynamical systems, and a Master of Science in Electrical Engineering from Caltech (California Institute of Technology, Pasadena, USA, 1990), where he had the support of a Fulbright scholarship, studying robust control. He first graduated as an Industrial Engineer (U.P.M., Madrid, 1989), with specialization in control theory, electronics and computer science, achieving 1st of class honors and being awarded the distinction to the top nationwide graduate in his studies (“Primer Premio Nacional de Terminación de Estudios”). Since 2001 he holds the CFA (Chartered Financial Analyst) designation from the CFA Institute.
Gabriel F. Bobadilla works as an investment management professional for a Spanish asset management company and family office. He manages financial assets, with a special focus on alternatives.
Got something to say?