The Sweet Spot

I need to reevaluate my sanity for starting a blog entry at 3:00 am. But I have been promising a blog on my data mining of CTS tactician stats. I've done a lot of calculations, but tonight I'll focus on my most provocative findings. In the following graph, rating is plotted against accuracy for two groups of tacticians: those who have done between 1000 and 10,000 problems (blue circles) and the those who have done over 10,000 problems (red circles).

The idea behind having two groups differing in number of problems attempted is that the red group is representative of tacticians who have done a lot of problems and the blue group is representative of tacticians who have done only a few. So, the way I interpret this plot is that solving at just above 80% accuracy gives one the greatest score improvement over time. Interestingly, this interpretation is consistent with observations made by wormwood on the Message Board several months ago. His observations led me to experiment with accuracy, though I have taken it somewhat to an extreme--working for 98% lately, but achieving just over 96% in actuality.

I should note a couple of caveats about this graph. First, the two red points on the extremes (97.5% bin and 52.5% bin) should be more-or-less ignored because too few tacticians fall into these bins to give reliable statistics. Second, the error bars do not necessarily represent uncertainty about the actual rating of the group. These "errors" arise from the natural score distribution of the groups and their magnitudes are related to the robustness of the statistics and not related to their uncertainty. I have yet to perform proper error analysis on these numbers, but my feeling is that the numbers are robust.

One conclusion from the above graph may be that solving problems at greater than 90% accuracy results in little progress, given the information in the 92.5% bin. Its difficult at this point for me to argue with that conclusion. And anyone who has taken a gander at the message board lately will know I am eating my words right now.

But, despite the obvious conclusions one might make from the above graph, my belief is still quite the contrary. I believe that tacticians solving at greater than 90% accuracy are getting much more out of their training than tacticians solving at less than 90% accuracy. Difficult (and perhaps embarrassing) for me, however, is that this benefit is not revealed by analysis of rating alone as I have done thus far. So I am still short of proof on my theory.