To display crossings visually it is necessary to convert several kinds of data to comparable ratio scale measures. There is considerable arbitrariness in how this is done. In particular, the metrics applied to compare rules as to "difficulty for Machine Learning Algorithms" are ordinal, which supports the application of the Mann-Whitney test, to decide whether two rules are significantly different in difficulty. A typical point statistic for that analysis is the median value of the cumulated error curve, at whatever point is chosen for comparison. For ML, this number, E*, is well defined even if the learning curve is not flattening, and shows no signs that learning occurs.
For human learning (HL) there are two "natural" ratio scale metrics: M*, the median value of the "aha move" for those subjects who did discover the rule and F*, the fraction of all subjects who did discover the rule. For visual comparison with the E* value these two metrics are combined into a single measure that increases with increasing difficulty. In the graphic shown here the scale value, V, is computed as M*/F*. This increases with M*, and increases with decreasing F*
This shows four classes of rules. Within each of the first three classes comparisons are made between the underlying rule, and variants in which the rule is "underspecified" (also called "ambiguous". For the "shape match" (SM: each shape to one bucket) example, the code 1F means that one of the shapes can go to any bucket, while 2O means that every piece can go in either of two buckets. For the rule that pieces must be placed into buckets in clockwise order, AF means every alternate move is actually free, while 2F means the second and third moves after a constrained move are unconstrained. On the right in Figure 7 the top three lines show that for both AI methods, the rules with increasing under-specification are monotonically easier to learn. The top one is a strict shape match. The second allows one shape into any bucket. The third allows every shape into two buckets. For humans, at the left of diagram, the situation is more complex, with the fully specified rule falling between the ones with one and two free moves. This non-monotonicity although in the data not rising to statistical significance (Holm-Bonferroni corrected for multiple comparisons). It is a glimpse of the many ways in which human discovery of logical rules may surprise us. While a less constraining rule might be easier to discover for humans, as it is for machines, it seems that freedom represents an additional dimension of rule complexity. These, and more puzzling kinds of non-monotonicity will be explored in proposed research
Cite as: Kantor, Paul; Pulick, Eric. "Visualization of Difficulty Crossings." URL: https://wwwtest.rulegame.wisc.edu/ExampleFindings/DiffX.html
Roles. Analysis and visualization designed by PK, and executed by EP. All data shown here were collected in experiments conducted by EP, both for human leaarning and for machine learning.
