Over the last three club seasons, the USAU has used the wrong regionals format in 46 out of 48 tournaments.
(First: A pdf of this is available at THIS LINK. I include the link here b/c I'm still not that good at formatting these posts and am posting this as a quick-hit while on Amtrak in response to this reddit conversation. If you can't see some of the numbers in the table below, plz follow the link.)
To be clear, this has had exactly zero deleterious effect on the Nationals field in any division. The formats have been meticulously designed and chosen to get this correct, and they have done so given that we accept the standards which the USAU Formats Manual lays out.* In every occasion, the best was done to determine the teams which continue to the next tier of play, in this case, Nationals.
To be clear, this has had exactly zero deleterious effect on the Nationals field in any division. The formats have been meticulously designed and chosen to get this correct, and they have done so given that we accept the standards which the USAU Formats Manual lays out.* In every occasion, the best was done to determine the teams which continue to the next tier of play, in this case, Nationals.
The issue arises when tiers are determined not just for the current season, but for the entirety of the following season. This is an adjustment that the USAU made starting with the 2014 season, and continued through 2015. Within the framework of relegation and creating tiers of play, this is a reasonable goal. The problem is in the application of the correct format.
From the formats manual:
“The fact is that most UPA Series tournaments do more than pick a single winner. This makes scheduling infinitely more difficult than non-Series tournaments. Ordinarily, it doesn't matter if you come in third or fourth at a tournament, but if the top three teams qualify to the next level, the schedule must be scrupulously fair to teams that have a shot at the third place spot. “
While I take issue with the use of the word “infinitely”**, the rest of this is spot-on. The pivot comes with “qualify to the next level”. When this document was written there were no tiers of play. There were no levels other than “Non-Series, Sectionals, Regionals, Nationals”. Now there are Pro, Elite, Top Select, Select, and Classic flights. On top of that foundation, there is a structure of tournaments to which teams are invited based in part or in whole on their flights. Which were determined the previous season.
See the problem yet?
If a team does not make it to the “Game-to-Go” or better, their ranking is specifically inaccurate w/r/t the other teams. The formats do not test it and are not designed to. If we run a thought experiment on this using a 16-team double-elimination bracket with two teams advancing (Use 16.2.1 in the USAU Formats Manual to refresh your memory). Consider TEAM A which loses its first two games 15-13 to what turn out to be the top two teams. Now, consider TEAM B which loses its first two games 15-0 to what turn out to be the 14th and 15th best teams.
In the 16.2.1 format, both teams are eliminated and can only finish as high as 13th. Any informed spectator would believe that TEAM A is better than TEAM B. Even our USAU rankings would agree on this. But the specific tournament format used makes no distinction because it is not designed to. The format is designed to pick the top two teams, and can be used to select the top three teams, as the loser of the game-to-go can reasonably be considered the third-place finisher.
However, the spots from four through sixteen are NOT accurately determined.
This is an incredibly depressing error considering the stated purpose of the the Formats Manual and all of the work that was put into its development. That is to say, the Formats Manual is a guide to finding the correct format given X [Total teams] and Y [Teams which “qualify to the next level”]. The problem is, like any tool, it can easily be misused as the hammer can’t swing itself.
The USAU has chosen to use “finish at regionals” to determine which teams “qualify to the next level” for the entire following season. However, the USAU has chosen the incorrect tool 95.83% of the time since taking this tack. The 4.17% they got correct? Pure luck.
It just so happens that some formats are correct for more than one number of teams advancing:
- At 2015 NE Mixed Regionals there were 16 teams and four bids to Nationals. The cutoff for the 2016 Select Flight was eighth place. The format used (16.4.2) is correct for either four or eight teams of sixteen teams advancing.
- At 2015 NW Women’s Regionals there were eight teams and three bids to Nationals. The cutoff for the 2016 Select Flight was seventh place. The format used (8.3) is correct for three, seven, or eight teams.
And then there is this:
- At 2015 NC Women’s Regionals there were five teams and one bid to Nationals. The cutoff for the 2016 Select Flight was second place. The difference between the format used (5.1) and the technically correct format (5.2) is largely academic (Bracket 4.1 vs Bracket 4.3) but the path to second place is more more complete in 5.2
Below is the full data collected from 48 regional tournaments in two years across three divisions:
Year
|
Division
|
Region
|
Teams
|
Bids
|
Select Cutoff
|
Format Used
|
Correct Format
|
# Acc Ranked
|
Off By
|
# Teams Affected
|
2014
|
Mixed
|
NE
|
16
|
2
|
6
|
16.2.1
|
16.6
|
3
|
3
|
13
|
2014
|
Mixed
|
SW
|
16
|
5
|
9
|
16.5
|
16.4.2
|
6
|
3
|
10
|
2014
|
Mixed
|
SE
|
16
|
1
|
5
|
16.1
|
16.5
|
2
|
3
|
14
|
2014
|
Mixed
|
MA
|
16
|
1
|
5
|
16.1
|
16.5
|
2
|
3
|
14
|
2015
|
Mixed
|
NE
|
16
|
4
|
8
|
16.4.2
|
16.4.2
|
8
|
0
|
0
|
2015
|
Mixed
|
SW
|
16
|
2
|
6
|
16.2.2
|
16.6
|
3
|
3
|
13
|
2015
|
Mixed
|
NC
|
16
|
3
|
7
|
16.3.2
|
16.7
|
4
|
3
|
12
|
2015
|
Mixed
|
MA
|
16
|
2
|
6
|
16.2.2
|
16.6
|
3
|
3
|
13
|
2015
|
Mixed
|
SC
|
16
|
1
|
5
|
16.1
|
16.5
|
2
|
3
|
14
|
2015
|
Mixed
|
SE
|
16
|
1
|
5
|
16.1
|
16.5
|
2
|
3
|
14
|
2014
|
Men's
|
NE
|
16
|
3
|
7
|
16.3.1
|
16.7
|
4
|
3
|
12
|
2014
|
Men's
|
MA
|
16
|
2
|
6
|
16.2.1
|
16.6
|
3
|
3
|
13
|
2014
|
Men's
|
SE
|
16
|
2
|
6
|
16.2.1
|
16.6
|
3
|
3
|
13
|
2014
|
Men's
|
GL
|
16
|
1
|
5
|
16.1
|
16.5
|
2
|
3
|
14
|
2015
|
Men's
|
GL
|
16
|
2
|
6
|
16.2.1
|
16.6
|
3
|
3
|
13
|
2015
|
Men's
|
MA
|
16
|
2
|
6
|
16.2.1
|
16.6
|
3
|
3
|
13
|
2015
|
Men's
|
NE
|
16
|
2
|
6
|
16.2.1
|
16.6
|
3
|
3
|
13
|
2015
|
Men's
|
SE
|
15
|
2
|
6
|
15.2
|
15.6
|
3
|
3
|
12
|
2014
|
Women's
|
NE
|
14
|
2
|
6
|
14.2
|
14.6
|
3
|
3
|
11
|
2015
|
Women's
|
NE
|
12
|
4
|
8
|
12.4
|
12.8
|
5
|
3
|
7
|
2014
|
Mixed
|
NC
|
12
|
2
|
6
|
12.2
|
12.6
|
3
|
3
|
9
|
2014
|
Mixed
|
SC
|
12
|
2
|
6
|
12.2
|
12.6
|
3
|
3
|
9
|
2015
|
Mixed
|
GL
|
12
|
1
|
5
|
12.1
|
12.5
|
2
|
3
|
10
|
2014
|
Men's
|
SC
|
12
|
2
|
5
|
12.2
|
12.5
|
3
|
2
|
9
|
2015
|
Men's
|
NW
|
12
|
2
|
6
|
12.2
|
12.6
|
3
|
3
|
9
|
2015
|
Men's
|
SC
|
12
|
2
|
6
|
12.2
|
12.6
|
3
|
3
|
9
|
2014
|
Mixed
|
NW
|
10
|
3
|
7
|
10.2
|
10.6
|
4
|
3
|
6
|
2014
|
Mixed
|
GL
|
10
|
1
|
5
|
10.1
|
10.4
|
2
|
3
|
8
|
2015
|
Mixed
|
NW
|
10
|
2
|
6
|
10.2
|
10.5
|
3
|
3
|
7
|
2014
|
Men's
|
NW
|
10
|
2
|
6
|
10.2
|
10.5
|
3
|
3
|
7
|
2014
|
Men's
|
NC
|
10
|
2
|
5
|
10.2
|
10.4
|
3
|
2
|
7
|
2015
|
Men's
|
NC
|
10
|
2
|
6
|
10.2
|
10.5
|
3
|
3
|
7
|
2015
|
Men's
|
SW
|
10
|
1
|
4
|
10.1
|
10.3
|
2
|
2
|
8
|
2014
|
Women's
|
NW
|
8
|
4
|
8
|
8.4
|
8.3
|
5
|
3
|
3
|
2015
|
Women's
|
MA
|
8
|
1
|
5
|
8.1.2
|
8.5
|
2
|
3
|
6
|
2015
|
Women's
|
NW
|
8
|
3
|
7
|
8.3
|
8.3
|
7
|
0
|
0
|
2015
|
Women's
|
SW
|
8
|
2
|
6
|
8.2
|
8.6
|
3
|
3
|
5
|
2014
|
Men's
|
SW
|
8
|
1
|
5
|
8.1
|
8.5
|
2
|
3
|
6
|
2014
|
Women's
|
SW
|
7
|
2
|
6
|
7.2
|
7.6
|
3
|
3
|
4
|
2014
|
Women's
|
MA
|
7
|
2
|
6
|
7.2
|
7.6
|
3
|
3
|
4
|
2015
|
Women's
|
SE
|
7
|
2
|
6
|
7.2
|
7.6
|
3
|
3
|
4
|
2015
|
Women's
|
GL
|
7
|
1
|
5
|
7.1
|
7.5
|
2
|
3
|
5
|
2014
|
Women's
|
SE
|
6
|
2
|
6
|
6.2
|
6.6
|
3
|
3
|
3
|
2014
|
Women's
|
SC
|
6
|
2
|
6
|
6.2
|
6.6
|
3
|
3
|
3
|
2014
|
Women's
|
NC
|
6
|
1
|
5
|
6.1
|
6.5
|
2
|
3
|
4
|
2015
|
Women's
|
SC
|
6
|
2
|
6
|
6.2
|
6.6
|
3
|
3
|
3
|
2014
|
Women's
|
GL
|
5
|
1
|
5
|
5.1
|
5.5
|
2
|
3
|
3
|
2015
|
Women's
|
NC
|
5
|
1
|
2
|
5.1
|
5.2
|
2
|
0
|
0
|
The first eight columns are factual and form the core of the problem. The final three are derived from the number of teams a given format is designed to accurately rank including the loser of the game-to-go (“Teams Accurately Ranked”), how many additional teams needed to be accurately ranked (“Off By”), and how many teams the format inaccurately ranked (“Teams Adversely Affected”).
The actions of the USAU on this are either ignorant or willfully damaging. This is exacerbated by the usefulness of the omnipresent USAU Formats Manual that players and teams have been rightly trained to trust. The problem is not with the manual, but with the application of the manual.
Beyond that, the problem is with those of us who like to think we fill the role of The Fourth Estate within the ultimate community who either didn’t notice or said nothing over two years of this. Certainly many of us are focused on playing and coaching (Which are both full-time jobs), but we play and coach in contexts of the USAU’s design. It would behoove us to investigate and understand all of the options available, choices made, intents, results, manuals, rules and all else if we are interested in making the best version of ultimate possible as it is clear that if we leave it to the USAU, errors will go unnoticed and thus uncorrected.
This is quite reasonable in a relatively young and diverse community which is rapidly expanding. We can all forget our collective power and responsibility to be not just members in a community, but citizens.
*- There is legitimate criticism to be written about the Formats Manual. This is not that.
**- If it were infinitely more difficult it would be impossible. The word which accurately describes the situation is “finitely”.
No comments:
Post a Comment