In radiological research, subjective image quality is frequently ranked using Likert-like scales and both parametric and non-parametric tests are performed on the results ¹. However, assigning numerical scores to individual items is inherently flawed if judges have no context and the frame of reference may change after judging several items. Scores should be independent of the order in which items are judged. Also, when differences are less obvious, assigning scores becomes more challenging. Calculating ranks based on pairwise comparisons can provide an alternative that is easy to understand, highly accurate and arguably more versatile ^1,2. Ranking and pairwise comparisons have been used in Condorcet elections for centuries. In the early 20^th century, mathematical models emerged in psychometric theory and for ranking incomplete chess tournaments or estimating the probability of winning a tournament which have found a wide range of applications ^3–5. Examples of the internet era include recommender systems and ranking students or exhibits ^6–8. Newer methods of active sampling maximize the information gain of each comparison ^9,10. Most pertinent to radiology are perhaps side-by-side comparisons used in visual perception and biomedical imaging ^1,2,11,12.