Tuesday, May 28, 2013

The Upset, Part II: All over the place

After discussing why we rank, it's time to say just how good each scheme is.  For the purposes of this study, I've looked at the rankings as of 1 Jan 2013.

or, how often did the team that won end up ranked lower?

This is the bit that causes the most confusion.  "But we beat them, why aren't we higher?"  Well, ranking algorithms generally aren't actually written to minimize this.  They're written to be more concerned about other things, and hope this comes along for the ride.  A scheme has been written, called Min-V, which is primarily concerned about minimizing retrodiction error.  More on its algorithm later.

How do Derby Chart, the European Roller Derby Rankings, and Min-V stack up in this category, as well as predictive ability?

Past upsets70559272
Past upset % 25.7% 20.2% 3.3% 272
Predict upsets1615 1745
Predict upset % 35.6% 33.3% 37.8% 45

Turns out, both rankings do a poor job of retrodicting bouts.  Only the Min-V system, with the sole purpose of minimizing retrodiction errors, has a low upset percentage.  For prediction, all three give at best a 2-in-3 chance of being correct.


Don't trust the rankings too much, unless they're Min-V.  And Min-V looks like this:

1London Rollergirls20.911
2Gent GO-GO Roller Girls [B]20.713
3London Rollergirls [B]20.711
4Paris Roller Girls20.703
5Bear City Roller Derby20.697
6Rainy City Roller Girls20.672
7Stockholm Roller Derby20.668
8Hellfire Harlots20.612
9Middlesbrough Milk Rollers20.602
10Glasgow Roller Derby [B]20.6
11Brighton Rockers20.592
12Glasgow Roller Derby20.587
13London Rollergirls [C]20.58
14Helsinki Roller Derby20.572
15Gent GO-GO Roller Girls20.568
16Leeds Roller Dolls20.558
16Tiger Bay Brawlers20.558
18Auld Reekie Roller Girls20.548
18Crime City Rollers20.548
20Central City Rollergirls20.537
21Dublin Roller Girls20.502
21Hot Wheel Roller Derby20.502
23Newcastle Roller Girls [B]20.5
23Leeds Roller Dolls [B]20.5
23Manchester Roller Derby20.5
26Bear City Roller Derby [B]20.498
27Copenhagen Roller Derby20.432
28Kallio Rolling Rainbow20.427
29Sheffield Steel Roller Girls [B]20.4
30Royal Windsor Rollergirls20.232
31Southern Discomfort20.2
32Lincolnshire Rolling Thunder20.1
32Quad Guards20.1
32Ruhrpott Roller Girls20.1
35Tyne & Fear20
37Roller Derby Bordeaux Club10.6
38Roller Girls of the Apocalypse10.5
38Herault Derby Girlz10.5
40Cork City Firebirds10.488
41Bristol Roller Derby10.442
42Birmingham Blitz Dames10.437
43London Rockin Rollers [B]10.421
44Stuttgart Valley Rollergirlz [B]10.402
45Paris Roller Girls [B]10.4
45Roller Derby Rennes10.4
47One Love Roller Dolls10.398
48Lutèce Destroyeuses - Paris10.3
49Brussels Derby Pixies10.2
49Roller Derby Metz Club10.2
51MRD: New Wheeled Order10.1
52Newcastle Roller Girls1
53Crime City Rollers [B]0.988
54Liverpool Roller Birds0.9
54Seaside Sirens Roller Girls0.9
56Sheffield Steel Roller Girls0.888
56London Rockin Rollers0.888
56Dolly Rockit Rollers0.888
59Stuttgart Valley Rollergirlz0.878
59Romsey Town Rollerbillies0.878
61Croydon Roller Derby0.8
61Rainy City Roller Girls [B]0.8
63Big Bucks High Rollers0.798
64Lincolnshire Bombers0.788
65Tiger Bay Brawlers [B]0.7
66Vienna Roller Girls0.6
67Rockcity Rollers0.5
67Milton Keynes Concrete Cows0.5
67Barockcity Rollerderby0.5
67Amsterdam Derby Dames0.5
71Bristol Roller Derby [B]0.4
71Rebellion Roller Derby0.4
71Dublin Roller Girls [B]0.4
71Harbor Girls0.4
71Portsmouth Roller Wenches0.4
71Liverpool Roller Birds [B]0.4
71Rotterdam Death Row Honeys0.4
78Munich Rolling Rebels0.3
78Blackland Rockin'K-Rollers0.3
78Bembel Town Roller Girls0.3
78Oxford Roller Derby0.3
78Royal Windsor Rollergirls [B]0.3
78Birmingham Blitz Dames [B]0.3
78Bad Bunny Rollers0.3
85Inhuman League0.2
85South West Angels of Terror0.2
85Manchester Roller Derby [B]0.2
85Bedfordshire Roller Girls0.2
85Mean Valley Roller Girls0.2
85Copenhagen Roller Derby [B]0.2
85Belfast Roller Derby0.2
85Hereford Roller Girls0.2
85Roller Derby Karlsruhe0.2
85Roller Derby Lyon0.2
85Dirty River Roller Grrrls0.2
85One Love Roller Dolls [B]0.2
85Gothenburg Roller Derby0.2
85Namur Roller Girls0.2
99Swansea City Roller Derby0.1
99Roller Derby Belfort0.1
99Crash Test Brummies0.1
99Evolution Rollergirls0.1
99Kent Roller Girls0.1
99Frankfurt Roller Derby0.1
99Dundee Roller Girls0.1
99Plymouth City Roller Girls0.1
99Dorset Roller Girls0.1
99Roller Derby Toulouse [B]0.1
99Norfolk Brawds0.1
99Helsinki Roller Derby [B]0.1
99Bruising Banditas0.1
99Kallio Rolling Rainbow [B]0.1
99Nantes Derby Girls0.1
99Roller Derby Metz Club [B]0.1
99Lincolnshire Bombers [B]0.1
99Dom City Dolls0.1
99Fierce Valley Roller Girls0.1
99Tampere Roller Derby0.1
99Central City Rollergirls [C]0.1
99Roller Derby Calaisis0.1
99Furness Firecrackers0.1
99Roller Derby Grenoble0.1
99Hell's Belles0.1
99Nidaros Roller Derby0.1
99Porto Roller Derby0.1
126Seaside Sirens Roller Girls [B]0
126Severn Roller Torrent0
126Shoetown Slayers0
126Barcelona Roller Derby0
126Les Quads de Paris0
126Imposters Roller Girls0
126Roller Derby Arras0
126Hell's Ass Derbygirls0
126Fair City Rollers0
126Brighton Rockers [B]0
126Velvet Sluts0
126Wolverhampton Honour Rollers0
126Nottingham Roller Girls0
126Kernow Rollers0
126Wirral Whipiteres0
126Wakey Wheeled Cats0
126Wiltshire Roller Derby0
126Vendetta Vixens0
126Roller Derby Angoulême0
126Eastside RocknRollers0
126Tenerife Roller Derby0
126Aarhus Derby Dames0
126Dolly Rockit Rollers [B]0
126Cardiff Roller Collective0
126Cornwall Roller Derby0
126Cherry Blood0
126Marseille Roller Derby Club0
126Spiders Black Widows0
126Oslo Roller Derby0
126Central City Rollergirls [B]0
126Roller Girls of the Apocalypse [B]0
126Amsterdam Derby Dames [B]0
126Auld Reekie Roller Girls [B]0
126Jakey Bites0
126Lahti Roller Derby0
126Royal Swedish Roller Derby0
126Luleå Roller Derby0
126Preston Roller Girls0
126Graveyard Queens Cologne0
126Lahti Roller Derby [B]0
126Roller Derby Lorient0
126Roller Derby Lille0
126Dresden Pioneers0
126Bairn City Rollers0
126Voodoo Vixens Besançon0
126Prague City Roller Derby0
126Plymouth City Roller Girls [B]0
126Grin n Barum0
126Kouvola Rock n Rollers0
126Southern Discomfort [B]0
126The Switchblade RollerGrrrls0
126Stockholm Roller Derby [B]0
126Dock City Rollers0
126Nantes Derby Girls [B]0
126Valencia Roller Derby0
126South Wales Silures0
126Panam Squad0
126Red Lion Roller Derby0
126Zurich City Rollergirls0
126Montpellier Derby Club0
126Limerick Roller Derby0
126Hulls Angels Roller Dames0
126Middlesbrough Milk Rollers [B]0
126B.M.O Roller Derby Girls0
126Roller Derby Toulouse0
126Big Bucks High Rollers [B]0
126Tyne & Fear [B]0
126Surrey Roller Girls0
126Milton Keynes Quads of War0
126Roller Derby Avingon0
126Bourne Bombshells0
126Kent Roller Girls [B]0
126Granite City Roller Girls0


  1. So by Min-V rankings, NWO beating TnF was a massive upset?

    1. Indeed. And, indeed, the ONLY men's upset in the latter half of 2012.

  2. At least 6 teams with B team ranked higher than A team?