Rating Formula

Your suggestions.

Rating Formula

Joined: November 23rd, 2012, 5:55 am

August 17th, 2013, 3:50 pm #1

I've looked at the ranking formula in detail, and i would like to discuss some possible twitches with somebody.

It is fairly complicated so I don't think it can be dscussed here on forum, but if you have your opinion on the subject (either because you are a mathematician and looked into it, or just from experience) I'll be glad to hear you.
Quote
Like
Share

Joined: December 4th, 2012, 5:38 pm

February 8th, 2015, 2:40 pm #2

I was included in the discussion about the rating system... The math was very complicated and way over my head.
I do not like the rating system. Here is a reason why. Having a good rating is like a status symbol. Some players
will only play you if you have a certain rating. I have found that losing to a lower ranked player can cost a lot of points...
therefore taking away from my status. Gaining those points back is a slow and painful process and very unenjoyable process.
Today I was sent an invitation by a lower ranked player who I have played in the past. I lost a game and it cost me many
points... as I was having a bad game day. Therefore I declined to play them. Unfortunately I will continue to decline them
in-order to maintain my status. To me the ranking system is discouraging me to play instead of encouraging me. I do not know
the right answer here... as the math was so complicated....
I do not think the current rating system gives an accurate indication of a players abilities.
Options:
Give players more control over their points or status by letting them choose how many points they are willing to risk.

I think we should have a system in place which encourages play... not the opposite.
Quote
Like
Share

Joined: November 23rd, 2012, 11:59 am

February 9th, 2015, 8:32 am #3

Agree, dolphin, but my bigger problem is that you can win games against lesser opponents, and still lose points! There are several I would play, but cannot for this reason.
I only learned very recently, that there is a direct correlation between the points gained and lost by the winner/loser. This to me is at the heart of the problem; it needn't be so rigidly applied. I think this is an overreaction against the old site, were people would sometimes try to tactically take points off the higher ranked players.
If a lower player does beat a much higher ranked player, then by all means give them a heap of points, they have earned it; but that does not necessarily mean that the higher ranked player who lost should lose the same amount of points accordingly.
That to me is daft … and is discouraging me from playing.
Quote
Like
Share

infamouswhiteknight
Site Admin
Joined: October 23rd, 2012, 7:21 pm

February 11th, 2015, 12:09 pm #4

An update about the original post by Rubik87: we did start a discussion group about the rating system. We mostly did two unrelated things: one, spent time discussing what the goals of a rating system are, and two, spent time talking about some math related to the current rating system (and ELO-like systems in general). The discussion fizzled out, and we never concluded anything. This is as much (or more) my fault as anyone else's.

That said, I'm still happy to see some open discussion about rating systems from the community, and am not opposed to making changes based on ideas/decisions that come out of such a discussion. However, my current belief is that the vast majority of the players on this site are at the very least satisfied with the current system, and don't desire to see changes, and so I'm quite hesitant to actually make any changes at the moment. If a discussion group actually gets started, I would encourage it, and be willing to participate but I don't think that I should spearhead such a group.

In the interests of giving such a group a chance to get off the ground (should enough people be interested), I won't say too much about the specifics already mentioned in this thread. They may be things that other people want to pick up on in the future, or not. If anyone is interested in my personal opinions about these, or other questions related to the rating system (whether they be questions about mechanics, ideas or opinions with either the current rating system, or a prospective new idea), please feel free to approach me in chat, or send me an email to start such a discussion.

What I will do is give a brief summary of the main "goals" (for an "ideal" rating system) that we came up with during our discussion (because we basically completed that part of the discussion), and maybe mention briefly how they relate to the above concerns.

We concluded that the main 3 goals for an "ideal" rating system should be:
1. A rating system should encourage more games, and more interest in the game
2. A rating system should evaluate the relative skill level of (at least most) players
3. A rating system should be relatively simple

It might be a good idea to try to view complaints about the current rating system with these goals in mind, for example:
-Gaining points (back) shouldn't be a "quick and easy" process. Making it so would probably violate the 2nd goal, even if it seems to satisfy the 1st.
-Having points won = points lost in most games prevents point inflation, meaning that the system can still do a good job of evaluating players who haven't played in a while. Not doing so would likely violate the 2nd goal.
-The idea about "point wagering" that dolphinjcn mentions is very interesting. It seems hard to me to implement it without violating the 3rd goal in a major way, and if a future discussion group wants to proceed with that idea, keeping it simple should probably be one of the main lines of discussion.
Quote
Like
Share

Joined: November 28th, 2012, 10:23 pm

February 11th, 2015, 7:03 pm #5

I think that the ranking system, in general, is OK. However, I do think that it is too sensitive with regard to each game played, and that as a result, it is not ideal when considering points 1 and 2 above.

With regard to point 1, I personally find it very discouraging that one game can affect your points to the extent it does, and as a result typically enjoy playing unrated games more. Although I understand that the wins and losses eventually even out with time, I agree with dolphin in that I think it can be depressing to watch your points rapidly decline after having one bad day.

With regard to point 2, since each game can significantly affect your points, I find that there are some people (e.g., I am thinking of one or two super nice people that love playing everyone) that bounce around within the rankings a lot. It seems that this system cannot provide an accurate and/or precise evaluation of skill if a person can be at the purple star level one week and at the red dot level the next week. In fact, I think that the largest number of points I saw awarded/reduced for a single game was 48 points, which notably, is almost half of the points allotted to a C2 ranking level.

One game does not determine what level of player you are - I think it would be nice if the ranking system reflected this.

One approach in addressing these concerns would be, before altering the points, to multiply the points to be added/subtracted by some predetermined factor (i.e., percentage). This way the methodology would be the same, and the points won=points lost, but the total points awarded/lost each game would not be so large (i.e., relative to the ranking scale). It would take you longer to climb up the rankings, but you also would not fall as quickly.

With regard to wiganlatics post, I agree that you should not lose points after winning a game. In fact, I think this not only discourages mid- to high-ranked players from playing lower-ranked players, but if they do play, encourages them to win with the largest margin possible, thus also discouraging the lower-ranked players.

With regard to the point wagering, I agree that it would be challenging to implement and also might be challenging to use/explain to opponents, and thus may not be ideal. However, as IWK stated, it is very interesting. I can imagine that the first “high-stakes” tournament would draw quite the crowd!!
Quote
Like
Share

toby
Site Admin
toby
Site Admin
Joined: October 23rd, 2012, 7:15 am

February 12th, 2015, 2:00 am #6

Allow me to answer some of the points moodles has made:
It seems that this system cannot provide an accurate and/or precise evaluation of skill if a person can be at the purple star level one week and at the red dot level the next week.
There is no way to measure "skill" per se ... so we do the next best thing: we measure "results of play".
To the extent that everyone plays all their games to their highest level of ability, these correlate.
If someone plays a rated game and "fools around", or is not feeling well, or for some other reason plays
below their "level of skill", it will not.
In fact, I think that the largest number of points I saw awarded/reduced for a single game was 48 points, which notably, is almost half of the points allotted to a C2 ranking level.
Your memory or your observation were inaccurate. I ran the algorithm over a wide range of possible inputs
for player points and game results, and the largest swing was 16 points. Such large swings only occur
in games with wide margins (at least 20 points, and usually it requires more).
... if a person can be at the purple star level one week and at the red dot level the next week ...
To drop two color levels (200+ points) would require at least 12 "bad games" in a row,
or perhaps mixed in with some wins that did not gain as much.
And if a purple star level player is losing lots of games in succession by 20 points or more,
then he probably got lucky to reach that level in the first place.
It is my experience that players at that level only rarely lose games by big margins,
never mind 12 or more in a row.
And if these really are "abnormally bad games" ... maybe he should take a break
before piling up 12 of them.
One game does not determine what level of player you are - I think it would be nice if the ranking system reflected this.

One approach in addressing these concerns would be, before altering the points, to multiply the points to be added/subtracted by some predetermined factor (i.e., percentage). This way the methodology would be the same, and the points won=points lost, but the total points awarded/lost each game would not be so large (i.e., relative to the ranking scale). It would take you longer to climb up the rankings, but you also would not fall as quickly.
I agree with this point ... and in fact there already is such a "damping factor" built into the algorithm.
Your proposal is to reduce it - which is a possibility, but must be given some consideration.
I recall that during the development of the algorithm we experimented with different values before
settling on what we used.
I agree that you should not lose points after winning a game. In fact, I think this not only discourages mid- to high-ranked players from playing lower-ranked players, but if they do play, encourages them to win with the largest margin possible, thus also discouraging the lower-ranked players.
It is hard to disagree with your points about encouragement/discouragement,
but to some extent it is necessary to take into account the winning margin.

Unfortunately (in this context), it is the game itself which encourages this behavior,
since unlike chess, there is a score at the end of the game, not just a win and a loss.
The object of the game is "play as many pieces as you can", rather than
"play more than your opponent and then stop".

A system which merely took into account who won and who lost would not differentiate players as well.

The point adjustments are essentially trying to correct any errors in the relative rankings of
the two players, given the additional input of the just-played game. Based on the rankings
of the players and who plays first, there is an "expected result".
(E.g. a 500 level player vs a 300 level player "should win by at least X points".)
If the expectation is not met, the players' points are adjusted to bring them closer together
(or possibly even switch places if they were close to begin with).
Quote
Like
Share

toby
Site Admin
toby
Site Admin
Joined: October 23rd, 2012, 7:15 am

February 12th, 2015, 2:51 am #7

I would also like to address the issue of the overall accuracy of the ranking system.

I think it works extremely well.

I will give the following as an illustration.

I chose a player more or less at random (not anyone who has posted on this thread).

Using his statistics page, and the "36 most frequent opponents",
I noted the record shown (for rated games only) vs each of them,
and where they stand relative to him on the ranking scale.
I removed all the names.

Here are the results, which can be interpreted as follows:

L = losing record against that opponent (more losses than wins)
W = winning record against that opponent (more wins than losses)
= means same number of wins and losses.

The list is ordered by descending number of ranking points of the opponents.
So the first line means that opponent was rated 256 points higher than
our subject.

All the "W" lines indicating winning records occur below his points level.
All the records above his point level are "L" lines, i.e., losing records,
except for one "=", a player ranked just 15 points higher.

There are just a few "outliers", three losing records below
(and two of those are just one short of being even).

In short, relative to his opponents, this player is ranked "just right".

Yes, he wins games against players above him - sometimes,
and loses games against players below him - sometimes,
but in general, his games go according to form ... or the system
is assigning points "correctly".

I believe that is the case for almost all players on the list as well.
Which says to me the system as a whole is working very well,
even if it seems that sometimes the results don't "feel right" or are "unfair".
The overall situation says that is not the case.

L 256
L 235
L 218
L 167
L 166
L 108
L 104
L 101
L 87
L 65
L 58
L 51
L 42
L 33
L 33
= 15
L 5
THE PLAYER HIMSELF FALLS HERE ON THE LIST
W -12
L -18
W -27
L -32
= -40
L -41
W -50
W -54
= -58
W -85
W -88
W -93
W -100
W -119
W -139
W -153
W -205
W -266
W -268
Quote
Like
Share

Joined: November 28th, 2012, 10:23 pm

February 12th, 2015, 7:51 am #8

Sorry Toby, I was indeed mistaken in my calculation of total points lost for one game :(

I think you are also correct in stating that losing points (i.e., even when you win a game) is required in order to take into account the winning margin and/or correct any errors in the relative rankings of the two players. However, it is a bit of catch-22 in that if higher rank players won't play lower ranked player because they know they will likely lose a lot of points (e.g., even if they win), then the lower ranked players will find it more challenging to find games that help them move up in the rankings.
Quote
Like
Share

toby
Site Admin
toby
Site Admin
Joined: October 23rd, 2012, 7:15 am

February 12th, 2015, 11:21 am #9

it is a bit of catch-22 in that if higher rank players won't play lower ranked player because they know they will likely lose a lot of points (e.g., even if they win), then the lower ranked players will find it more challenging to find games that help them move up in the rankings.
It is a bit of a stretch to go from "you might lose points, even if you win"
to "likely to lose a lot of points".

The whole idea is that if you have a higher rating than another player -
and if it is deservedly so (that's the important part) - then it should NOT be "likely"
that you will lose (either the game or the points) and surely not "a lot of points".
If you have a bad game and lose by a lot, then, yes you can lose about a dozen points.

The points you can lose while still winning the game are rather more limited in scope.

For the morbidly curious, I have uploaded an extensive sampler of points changes
to http://pentolla.com/ranksamples.txt
Quote
Like
Share

Joined: November 23rd, 2012, 11:59 am

February 12th, 2015, 3:36 pm #10

Much of that information is interesting, and informative at many levels.

However, highlighting iwk's number one goal of a rating system, that it should encourage more games and interest, and I can only speak from my personal perspective, the current arrangement does not encourage me to play, I am incredibly selective in who/when I play now, and my interest has declined considerably, now that I have some understanding of how the rating system works. So, in response to toby's post, it isn't working for me.

Asking around recently, I could not find anyone who had any understanding of how the system works, and that includes many of the top players.
Quote
Like
Share

Joined: November 23rd, 2012, 11:59 am

February 13th, 2015, 7:11 am #11

But to put this into a bigger perspective, I am not saying that I believe that the current system is 'totally wrong', I just believe that it can be improved upon.
I remain deeply thankful, to the admin team, who have made on-line B**k*s possible; and appreciate all the time and no-little expense they have contributed to enable this.
Quote
Like
Share

toby
Site Admin
toby
Site Admin
Joined: October 23rd, 2012, 7:15 am

February 13th, 2015, 10:44 am #12

Asking around recently, I could not find anyone who had any understanding of how the system works, and that includes many of the top players.
The Ranking Guide Page explains the system in quite some detail.
It was viewed, during all of 2014, a total of 553 times.
Even the Code of Conduct page (which new players must state they have read, but they don't)
had more than 3 times as many views.
By contrast, the login-to-play page was loaded 107,100 times during the year.

While the advanced mathematics of the system may be hard to grasp,
the principles of the system should be fairly understandable.
Quote
Like
Share

Joined: November 1st, 2012, 10:02 am

February 16th, 2015, 12:21 pm #13

I agree with some of the points that Wiganlatics raised about the points system.

Since returning to the site due to work commitments, I have been trying to raise my duo rating to purple star. Not many people like playing duo so I know it is not easy.

As you get higher up the rankings it does appear that you have more chance of losing points rather then gaining. For example, I played a game as purple and won by 12 and my score stayed the same, I then lost the next match by 1 as orange, but lost 3 points. I think to get within 1 point in duo as orange was quite good.

No points system is perfect. This system is much better than the one on the old B**k*s site!

I know that levels and ranks don't mean much, I'm happy to stay where I am, but I do think the ranking system needs to encourage players to play. At the moment I feel happier just to play in unranked games.

I don't want to be ungrateful. I love the game and love the site but want to encourage people to play more.

Thanks for listening.

Red Dragon
Quote
Like
Share

infamouswhiteknight
Site Admin
Joined: October 23rd, 2012, 7:21 pm

February 18th, 2015, 10:25 am #14

reddragon: Your complaints are about the 2c (duo) rating system, not the c2 system.

You're correct that there are problems with the rating system for the 2c system. The system favours "big wins" too highly, so it's better to beat a player much below you by a lot than it is to beat a player close to your rating by a little. (Note: this is not the case with the c2 system).

This is not intentional. It's a problem that came up because when I designed the original formula I did a lot of testing with c2 to find the right settings for the rating system, but I didn't do as much 2c testing, and so the settings are a little bit "wrong".

I have known about this problem for a while, but I haven't spent the time needed to fix it. It's my fault. I'm sorry.

I will get to work on trying to correct these issues as soon as possible.
Quote
Like
Share

Joined: November 1st, 2012, 10:02 am

February 18th, 2015, 11:59 am #15

Hi IWK

Thanks for the reply. No need to apologise.

Please don't think I was criticising you. There is no perfect rating system.
I was just trying to make an observation about duo.

much appreciate your response
Quote
Like
Share

infamouswhiteknight
Site Admin
Joined: October 23rd, 2012, 7:21 pm

June 1st, 2015, 6:24 am #16

An update about the 2C rating formula: I have made some adjustments to the parameters, and I am hopeful that this will address some of the known problems with the current algorithm (specifically the ones mentioned here). I will create another post with the details of the changes shortly. It will appear once the new system is actually in place (very soon).

As a quick aside: I'm sorry this took so long. I have actually been spending a lot of time thinking about it since my last post, but I was very hesitant to make changes too quickly, find out they had unintended consequences, and then need to make more changes. One of the problems identified with the 2C rating system (that I already mentioned) was that the system favoured "big wins" too heavily, meaning that it was better to play players rated below you and win by a lot than players rated close to you and win by a little. The ability to "game the system" this way was unintentional, but changing it so that the system could be gamed in other ways wouldn't be better.

Also, I had a bit of a dilemma with what was "fair". On the one hand, we now have lots of statistics on file, and it would have been possible to tune the system to line up with statistics. But I'm not sure that those statistics are easy to understand. For example, the statistics say the following: If I am 100 points above my opponent, then when I'm orange, statistically I should win by 4 points.
If I tried to respect this, then I probably wouldn't have changed the old system at all - this falls closely in line with what was already expected there. But in talking to people, most people believe that this is an unreasonable expectation - even against someone rated 100 points below you, a win by 4 points with orange is better than one should expect. This is similar to the point reddragon made earlier in his post. He thought that even when he lost by only 1 point with orange, that was pretty good. Most people I've spoken to agree.

Normally, I'm of the opinion that if there is a difference between what people think intuitively, and what statistics say, we should trust the statistics. In this case, however, there are a number of reasons why I believe the statistics might be flawed. I won't go into them here, but suffice it to say that I have spent a lot of time thinking about this, and that is part of what took so long in bringing about the upcoming changes. If you want to speak to me about some of my thoughts on this matter, I'd be happy to - send me a private message, an email, or just come talk to me on the site.

Anyway, sorry again for the delay. I hope the upcoming changes make the system work better than it used to.
Quote
Like
Share

toby
Site Admin
toby
Site Admin
Joined: October 23rd, 2012, 7:15 am

June 18th, 2015, 9:25 am #17

The charts displayed below illustrate, to me, that the ranking system works pretty well.

I chose four different C2 players, at different levels, and plotted what
their winning percentage (x10) was against other players of different levels.

We see that, for all four of them, some highly ranked, some lower,
the percentage (shown in dark blue) generally goes down
as the opponent level (shown in red) goes up -
as you would expect.

The correlation is not perfect, and also the number of games against
the various opponents varies quite a bit, the overall pattern is clear.
wins-vs-ratings-charts.png
Quote
Like
Share