Want to know what's going on across the derby world? Check out our new calendar.
login | register
Enter your Derby News Network username.
Enter the password that accompanies your username.

you forgot it?!?

Bonnie Thunders and Teenie Meanie Talk Rankings

  • Will the same faces be gracing this year's Championships?
  • Will the calculator upset the old order?
  • Or are things already set in stone?

WFTDA’s rankings and divisioning white papers raised almost as many questions as they answered, so we’ve set out to clear a few things up by putting some of your concerns to the people behind the project and asking some questions of our own.

The ranking calculator project was masterminded by--among others--Bonnie Thunders, WFTDA’s director of gameplay, and managed by Teenie Meanie, chair of the WFTDA ranking committee.

Teenie skated with Sacred City before taking a leave of absence from skating in 2010, but has remained active with the league as an NSO and will be bench coaching Sacred’s Sacrificers in the 2013 season. She’s been involved with WFTDA since 2008 and has chaired the ranking committee for several years, overseeing the old vote-based system.

As someone who had been involved with the voting system for several years, Teenie wasn’t a fan: “I was ready for that system to die,” she said. “But, we had to have something to take it’s place to allow it to.” In 2011 she got the chance to help with that goal, being given oversight of the ranking calculator project.

A calculator was mooted back in 2009, at which point the project fell under the auspices of Bonnie Thunders, then a member of the WFTDA tournaments committee. She’s now gameplay manager at the WFTDA, with oversight over tournaments, rules, rankings and other gameplay-related matters.

What does Thunders think of the system as it’s come to be after three years of work? “In general I think it’s important for anyone who is looking at this new system to know that this isn’t the end; we can and will always makes things better,” Thunders said. Is she worried about how the system might turn out? “You need to start somewhere,” she said, “and I feel like we could have sat here for three more years and continued to try to make the most perfect system. I think the risk is not so much that we’re going to fail or what we might have not done right, but the risk is that we might not be able to be as nimble in the future to adjust and change things as we go.”

If winning isn't everything, why do they keep score?

Winning isn’t everything in this system, and some people have a problem with this. “The original version of the rankings calculator gave all points to the winner, regardless of point spread,” Meanie explained. “The membership rejected that.”

A later version saw one point awarded to the loser of a close game and that too was rejected by the membership. In the end, WFTDA members settled on a system in which points are shared by teams purely on the basis of point spread.

What about the bonus points awarded to teams in playoffs? Under the current system, teams get between 15 and 50 percent more points for post-season games than regular-season games, with that bonus dictated by which tournament the game is in and at what stage of the tournament it’s being played.

“This piece was fairly controversial within the membership, but was ultimately approved by a majority,” Meanie said. She also revealed that under the tournament committee’s initial proposals, games at championships would have had a 300 percent bonus attached to them. “Nearly all of the WFTDA membership felt that this was too high,” she explained, “and I am comfortable with what we ended up with.“

>>Read more on page 2

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Needed more details

I found the answers to my concerns disappointing, as they didn't get into the details. The answers "That's how FIFA does it" do not tell me how much they explored and discussed other potential options.

Rankings

I will say this much, using a math formula for determining rankings is much better than the previous subjective way of doing it.

In 2007, I designed the Arizona Power Rankings as a method of evaluating the different home teams that were playing in Arizona under several different rulesets. I used a simple math formula that rewarded teams for winning by giving them a prorate of their "points for" for the number of percentage of games they won and penalized for losses by prorating their points against for the number of games that they lost within a 365 day rolling period. Each non-WFTDA ruleset also contained a "factor" that scores were multiplied by in order to make the average scores similar to the WFTDA baseline. We would "true-up" and adjust those factors each season based on last season's results.

Just recently I have added a new "upset" function in the Power Rankings. If a team that is a 6 or more ranking underdog beats the higher ranked team, the factor is increased by as many as 4 times thus giving a underdog team a huge incentive for beating a top team (and penalizing that top team for losing) but if it goes the other way (the top team beating the underdog), there is no additional factoring in the rankings. Doing this method does not necessarily promote blowouts but does give an incentive for "David" to beat "Goliath".

One thing I do not agree with on the new WFTDA rankings is putting additional weight on tournament games. To me, a bout, is a bout is a bout. I would rather see a sliding scale that rewards underdog teams for taking out much higher ranked teams, regardless of the venue. Using WFTDA's methodology, the top teams will always be on top since they are the only ones who will be playing tournaments.

For more info on the derbydata.com Arizona Power Rankings, see:
http://derbydata.com/rec_assets/azpr-1210.pdf

=m

Benefit of the Doubt

While I don't fully understand the new rankings system, I am willing to give WFTDA the benefit of the doubt for its good intentions in seeking to promote more competition and opportunities for tournament play. We'll have to wait and see how new teams fare in this system. If using FIFA as a model, one wonders if there will be a similar system of relegation where the lowest teams in division one and the top ranked teams in division two switch places for the next season.

And it is good to see that WFTDA continues to be serious about giving teams in Europe (and Australia?) a more realistic shot at moving up in the rankings and participating in tournament play. They are working hard and improving quickly. It is also crucial for developing the sport's profile internationally. Let's hope it doesn't take too long to figure out how.

The multiplier

This may have been answered elsewhere... I skimmed the other comments but couldn't find it, so I'm going to ask it here:

A lot has been speculated about WINNING a game not being as important as winning by a large margin. So in that respect, what would happen if the winning team were given a larger multiplier than the losing team?

You've got the Point Total x Strength Rating x 1 (for regular season play) x 100 for BOTH TEAMS.

What if instead it was

Total Points for WINNING team = Points x Weight x 1.0 x 125 = xxx
Total Points for LOSING team = Points x Weight x 1.0 x 75 = yyy

That way, your total multiplier is still 200 total points, but the winning team gets an edge, even if it's a CLOSE game.

Does that make sense?

Good Idea?

Under your plan a team should never play another team it can't beat.

Shouldn't a WIN be worth more?

So make it 105 for the winner & 95 for the loser.

I just think that a WIN should count for more.

Why does winning require extra credit?

Winning is it's own reward, it doesn't need any help.

The problem is just winning isn't very informative, but who you played against and what the score was can give you idea about the relative strength of the two teams that played.

In 2011 London lost every sanctioned game it played, yet it when on to place 5th in Eastern regionals that year. London played some of the top teams in the East earlier that year and gave a good account of itself. In fact so good account of itself that it looked stronger than teams with wins. London was ranked 10th going into Easterns and it can be argued that it was under-ranked.

And...

because London would have had an opponent rating of 0.5 because they were a new team during Anarchy in the UK, the rankings of the teams that beat them would have been destroyed by their game with London because the close scores were more important than winning.

Actually, London would have gotten more points for those games than their opponents even though they were the losing team in every game.

3 full points should go to the winning team no matter what the score is.

What proves more?

Beating Gotham 151-150

or

Beating Brewcity 200-100

I think this argues for something different

I think this is a better argument for why it should be based off of rating and not ranking.

Sure. That could certainly

Sure. That could certainly improve it. I haven't dug into that but that would probably be better. However, no matter how you do it, beating Gotham by a point still only gets a team 300 points. Even if it was based on rating and not ranking, there would be plenty of teams that you could be beat by a certain number of points and do better than a win over Gotham.

I just don't see how beating Gotham doesn't prove more than beating anyone else, I don't care what the score is. You should get the full 600 points for beating the best team in the world.

Not only that but you diminish what it means to win. Sports are 99% about wins and losses. The score doesn't matter. If the score should matter this much, why do we have the playoff system that we do in WFTDA? Why don't we just add up the scores from each round and whoever did the best based on the final score and the ranking of their opponent moves on even if they lost?

Because...

...the playoffs are all about winning. By a point, by 100 -- it doesn't matter. If you keep winning, you get the Hydra.

Rankings is about trying to make sure the right teams get there.

The right teams

If by the "right teams" you mean best teams and/or strongest teams, you might be wrong. Nowhere in either of the two recently released WFTDA white papers does WFTDA mention that the aim of the ranking system is get the best teams to the playoffs or have them seated by strength.

This saddens me, I'd like to see this as a goal.

The right teams won't be at

The right teams won't be at playoffs. The teams that schedule the best will be at playoffs.

Beating Gotham isn't as good as beating Brewcity. Nobody sees a problem with that?

But this is an impossible problem

The problem is that there is no mathematical method can work in 100% of the situations. This is why WFTDA originally went with a voting method, and why they still hang on to a voting method in case something weird happens. There is no guarantee that you can have a well ordered list, and it's why in the end we have a tournament to decide the winner.

Sure

But, you can get close and you can actually have it make sense (at least in theory). For example, if a win is a win, then beating Gotham is better than beating Oly which is better than beating Denver, etc. That actually makes sense. Beating Brewcity by 99 more points than beating Gotham is not better no matter how you want to spin it.

And, really, the ranking isn't even the biggest issue with making wins meaningless. The biggest issue is that you make the end of games meaningless. "Oh, my favorite team won by 1 point in an awesome final jam? Cool. They improved their rating by .000000000001, that'll sure have huge consequences in the seeding for playoffs. Yawn."

Two things

1) What would your win-based system do you when you have something like Montreal beats Charm beats London beats Montreal? What if it gets into five teams that don't have a well-ordering series of bouts?

2) 1 point will only be 0.000000000001 if the W/L-factor*OpponentStrength*GameWeight is about the same as their current average. This could just as easily happen if it was a 100 point bout. Also, a 1-point win could be a huge upset and raise someone's average by a lot.

Edit: Nevermind point 2. I misunderstood what you meant.

Define: Winning

thebigchuckbowski wrote:

"Oh, my favorite team won by 1 point in an awesome final jam? Cool. They improved their rating by .000000000001, that'll sure have huge consequences in the seeding for playoffs. Yawn."

Firstly, this is better than the previous system where a win could be rendered meaningless by other results.

Secondly, a team winning yet getting a tiny increase in their rating implies that they performed exactly the expectation. If they wanted to move up in the world, they'll know what their target score is. Very much like in soccer towards the end of any small league-based competition, or even the second leg of a two-legged match--you know what you need to do when you step on the pitch. If you don't achieve that goal (which may be to not lose by X, or it might be to make damn sure you win by Y), then you've lost. Yeah, winning a one-point game is awesome; but if you're Gotham and that one point win is over Dutchland, are you really suggesting it's a cause for celebration or a rankings boost?

To me, if a team performs to expectation and beats the team one place (or ten, or whatever) below them in the rankings by 1 point, then it would be rather silly for that to be seen as a 'win' as far as rankings go. Teams will know what they have to do when they get on the track, and so should fans.

Finally, as is made pretty clear -- a winner takes all system was the first one WFTDA leadership proposed. The membership -- that's representatives from every single league that's part of the WFTDA -- rejected it outright. Like it or not, the sport is skater driven at its core, and the skaters said 'no' to that one very loudly and clearly.

Correcting the first portion

So, I also misunderstood what he was saying, but then I realized it when I was about to criticize the same thing. The issue is not about performing to expectation, it's that if it's the end of a close game and it comes down to the last jam, the system has no incentive to try to win other than increase your ratings by a few tenths of a point (all those decimals in his example were a gross exaggeration). In other words, the points gained from a 149-150 loss are nearly the same as a 151-150 win, and the difference is the same as a if you increased a blowout from 249-50 to 251-50. Right now, the only incentive to push that extra bit at the end is for bragging rights, and that appears to be a slight flaw in the system.

Winner take all is clearly a bad idea though.

Some prespective

All sports championships are about bragging rights. Are we going to look at the sport scientifically or emotionally?

Don't get me wrong

I definitely don't think WFTDA should implement a winner take all system and no matter how much I argue against it, this system is definitely better than the old way.

I like how the score works for the losing team. A multiplier of 0-1.5, depending on the score, makes a lot of sense and I wouldn't change anything about that.

However, there's a ton of problems with doing the same thing for the winning team. 1. It makes winning a close game completely meaningless. There's essentially no difference between 150-149 and 149-150 but there really IS a big difference between those two things because one is a win and one is a loss. This should be reflected in the rankings. 2. It promotes blowouts. Teams can't play their bench players at any point in the season because they always need to improve their score even in games that are already blowouts. 3. There are some pretty MAJOR scheduling issues with it.

I would think that most of us would agree that we want a regular season where teams play as many games as they can afford against teams that are of a similar competitive level. That would mean lots of really good games. That's what we all want, right?

Well, this system promotes the exact opposite of that. It's basically forcing teams to play less games and the games they do play should be against teams that they think they can blowout. That's the reality and that's because of two things 1. measuring wins by score instead of just giving 3 points. 2. The extra boost given to playoff games.

The playoff games thing has already been discussed so I won't get into that.

It really makes no sense to schedule a team that's ranked close to you. It would only make sense if you think you can blow them out. If it's probably going to be a close game, it's probably going to hurt your ranking even if you win. If your ranking is going to go down even with a win, why in the hell would you schedule that game? You wouldn't. What you want to schedule are teams that are ranked 20-40 spots under you that you are reasonably sure that you'll blow out. That's where you'll get the most points. Well, the scheduling issue with that is that both teams can't be ranked 20-40 spots above the other. One is going to be ranked lower. Why would the lower ranked team schedule a blowout which will damage their ranking? They wouldn't. So, you won't schedule opponents ranked near you, you won't schedule opponents ranked above you, and you can't schedule opponents ranked below you. How exactly is anyone going to schedule any games?

I think the idea that using the score for both teams somehow creates a perfect system where every team is ranked 100% accurately is completely absurd. No offense. There's no such thing as a perfect ranking system unless teams are playing a perfectly balanced schedule. That's impossible in WFTDA which means it's impossible to have a perfect system. If this system was what DNN or FTS or Derbytron was using to rank teams then it would be fine. But, it's not in a vacuum like those are. WFTDA's rankings actually decide something and teams will make scheduling decisions based on what will improve their ranking. That means that teams that exploit the system the best will be ranked higher and teams that don't exploit the system as well will be ranked lower. So, the reality is that this system will not produce perfect results, the better teams won't necessarily be ranked higher. There's always going to be little things teams can do to help their ranking, no matter the system, but there are such obvious exploits with this system, it's just mind blowing. And, the reason for these exploits almost completely comes down to giving teams points for wins based on the score.

If that's the case, then let's get past this idea that a 50 point win is better than a 5 point win.

I mean, what proves more? Beating 3 teams that are ranked 20-40 spots under you by 100 points or beating 3 teams that are ranked 1-5 spots below you by 1 point. Well, the WFTDA's system says that blowing out lesser teams proves more. I would argue the opposite all day long.

If WFTDA were to say all wins are the same, what would happen? Well, teams would actually schedule opponents ranked near them because they'd potentially get the maximum number of points they can expect. Teams would really schedule any games they have a chance of winning even if a team is ranked above them because a win would help their ranking so much. Scheduling blowouts wouldn't be nearly as worth it. That's what we want, right? Would the rankings suddenly be completely inaccurate? I don't think so. If a team is winning a bunch of games against closely ranked competition, then they should move up. If teams are losing a bunch of games against closely ranked competition, then they should move down. And, if teams go .500 against teams ranked near them then they shouldn't move too much.

when winning is losing.

thebigchuckbowski wrote:

The right teams won't be at playoffs. The teams that schedule the best will be at playoffs.

Beating Gotham isn't as good as beating Brewcity. Nobody sees a problem with that?

this.

wins HAVE to count for something. This weekend in Texas saw the Dallas Derby Devils and Assassination City play each other for only the 4th time in 4 years, despite being 30 miles from each other. These are two teams currently on the outside looking in - in the most recent South Central Rankings, DDD is 15th and ACRD is 26th.

DDD beat their crosstown rivals, 168-117. Yet they lost 4.3 points in the rankings, while ACRD gained those 4.3 points.

it shouldn't be that way - point differential shouldn't be the end all be all, unless you really want to see teams run up the score - something that often leads to cheap hits, and subsequently, injuries.

For a system to work, when one team beats another, it needs to move up the rankings, even if it's by mere percentage points. it's not rocket science, but when you start coming up with algorithms that make it rocket science, winning teams can get the short end of the stick.

a winning team ALWAYS needs to see an advantage in winning.

Every ranking system

Every ranking system that has had a factor for winning has been a miserable failure. It's why even FIFA removed their old factor for a win, in an effort the make their rankings something more than a joke.

And yes it is rocket science, or rather calculus. Calculus has created the modern world we live in, the very technology you're using to post, not simple arithmetic and intuition.

?

Huh? W-L record and 3-1-0 point systems both use wins and losses as pretty much the only factor and every professional team sports organization that I know of uses one of those.

Even if you're saying more intricate math based systems have been failures, there's like a million of those so it seems far-fetched to say they're all failures if they use winning as a factor. Plus, FIFA does use winning as a factor (not score) so I really have no clue what you're saying.

Perhaps you could explain your meaning a little more?

A little more

Professional sport organizations generally play a rather comprehensive season in which teams play all their divisional members, in which case W-L record is acceptable. WFTDA teams can only play a tiny sample of their divisional members, thus a more sophisticated math model has to be used to estimate team strength. I dislike the FIFA/WFTDA math model because it does not use the best tools available to estimate team strength, instead it uses an antique system.

Years ago FIFA gave a certain number of additional points for just winning the game, in 2006 that factor was removed as well as other modifications in an effort to “fix” the rankings as it was obvious to FIFA and fans that the rankings were ridiculous. The current system still produces some outstandingly weird results.

There are flaws in the win/loss, opponent strength and game weight factors and then it is averaged over a year of games. I'm sorry, but garbage in means garbage out. WFTDA should scrap this outdated model, better methods exists.

I think I agree

After thinking about it for a day, I think I agree with this. It seems like finishing a game 149-150 or 151-150 is pretty significant, but the calculator won't see them as being very different (like 0.66 difference if you're playing against Gotham regular season).

I'm not sure where the best value would be (125-75 seems too high, but it might actually be okay for all we know), but I think the best method to determine it would be to actually run data with a range of variables and see which ones make more sense and which make less sense. (More and less sense can be quantitatively determined by seeing how well predictions line up with performances).

Yay Winning!

Expected to see everyone agreeing with Megatron's point about winning...must admit I'm a little surprised by the amount of debate.

Here's a +1 vote for some sort of tweak that makes winning matter. My original thought was to just change how you divide the 3 points. Give 1 point for the win and then divide the other 2 points based on points scored. Or maybe 0.5 points for the win and then divide the other 2.5. Whatever...but some little incentive to ensure that no one's sitting there at the end of the game thinking about rankings rather than about trying to win the game.

Another related thought would be to cap the percentage of the three points that a team can get. The cap could even be really high -- like 85%. But if you say one team can only have 85% of the score-based points, then scores like 400-50 and 300-50 are equivalent (meaning the better team can justify playing its bench rather than continuing to run up the score).

The problem is the scale of the WFTDA

The reason the 3-1-0 point scale wouldn't work for the WFTDA is pretty simple: you can't get accurate rankings from them with the number of teams involved. Teams don't play even schedules, so you could get wholly unqualified teams going to playoffs, and wholly qualified teams missing them, because one team played a bunch of simps and rack-up three points a dozen times over, and the other played a schedule worthy of their level of play, but gets passed over because other teams used the rankings equivalent of the passive offense.

An even point-spread for win/tie/loss requires an even schedule, so you have an even comparison versus everyone else. Without it, Toaster City could beat up on Insane Clown Posse Roller Derby week after week and come up with more points than Gotham and take top seed at regionals.

As I see it, you have three options:
First, adopt the system WFTDA has now, or something similar.

Second, bring back the voting process.

Third, actually create conferences or confederations ala NCAA football or FIFA, where a group of teams play each other every year, and the top of the table moves on to playoffs (add some formalized promotion and relegation between divisions for good measure, too). But this requires an ability for everyone in the WFTDA to schedule games in the same period of time, and an increased expenditure in traveling (or playing the entire "conference schedule" over a weekend tournament, leaving the rest of the "season" no more the exhibition games).

I think, if money was no option, the third option would be great, but money is an issue, and it's a long way from being feasible.

But...

"Third, actually create conferences or confederations ala NCAA football or FIFA, where a group of teams play each other every year, and the top of the table moves on to playoffs (add some formalized promotion and relegation between divisions for good measure, too)."

This doesn't happen with FIFA. Yes, you get world cup qualfication groups and the like, but you also get friendlies (which could towards rankings). The whole point of the FIFA rankings (off which these are based) is that it is based on teams of a wide range of abilities, playing a wide range of schedules against a wide range of opponents. This way you can rate Brazil and the Solomon Islands in the same system.

Its accuracy is woeful, the rankings are comical, and it doesn't work -- but that doesn't change its aims, at least.

That's my point

FIFA rankings have no point in the system (system being World Cup for national teams, World Club Cup for individual teams) - it's there for marketing buzz. That the WFTDA's ranking system is based on FIFA's is fine - because the scale of the number of wide variety of skill levels is the closest match to anything else in regular use; but the qualification systems that FIFA and its member confederations use still involves teams playing each other in tournaments (even if the "tournament" takes two years to complete).

The problem with comparing FIFA and WFTDA rankings

Lex Talionis wrote:

Yes, you get world cup qualfication groups and the like, but you also get friendlies (which could towards rankings). The whole point of the FIFA rankings (off which these are based) is that it is based on teams of a wide range of abilities, playing a wide range of schedules against a wide range of opponents. This way you can rate Brazil and the Solomon Islands in the same system.

Wrong comparison to make here, I think. Brazil and the Solomon Islands are in the same "system," in that they both need to qualify for their respective regional/conference tournaments or the World Cup by winning, not by calculating their way up to a certain rank. That is, even if Brazil is ranked #18 and Solomon Islands #218, they both know that if they win all of their World Cup qualifying games over everyone else, they will have earned the right to be in the World Cup, regardless of their ranking.

The FIFA rankings only exist to seed/divide teams that have qualified for tournaments through their play, and for no other reason. What the WFTDA has done with its similar ranking system is make it the sole basis for qualification. This would be like treating the top 40 in the "comical-and-doesn't-work" FIFA world rankings as dogma and advancing those teams directly into the World Cup, with no exceptions.

Considering that World Cup qualifiers and WC Finals games are weighted more heavily (x2.5 and x4.0 over friendlies, respectively), as are WFTDA D1/D2 playoff games (x1.25~x1.5 over regular season games) those on the outside looking in would suddenly find themselves with no realistic chance of math-ing their way in, even if they swept the table in a quality schedule. Next year's fringe D1 teams without the weight of the previous year's tourney games may find themselves in a similar situation.

In the future, I think it would be better if the WFTDA adopted some kind of regional-national hybrid system,. similar to NCAA Basketball. Yeah, math-based rankings (RPI) can help determine who qualifies for the big tournament. But even a lowly-ranked team can still make it in if they win against everyone else in their local region. Sure, they'll probably get blown out in the early rounds, but isn't that going to happen anyway with using math-based seeds? Besides, wouldn't the divisional WFTDA tournaments benefit by having more local teams that won their way in over similarly-ranked tough competition, instead of one from outside the region that was mediocre but ranked high enough to be in no matter what, even if they may have wound up with the same tournament seed?