Improvements to the ranking system

Improvements to the ranking system

Postby Alayan » 28 Oct 2019, 17:34

After a few months since 1.0 came out, it's easier to see some of the issues with the ranking system. Some were already known earlier, but I'm going to try and list everything:
- A high rated player will often get more points from a new player than from a mid-level player with a stabilized ranking. This creates a "noob farming" element at the top which is out of place.
- For players that only rarely disconnect, the points lost from a single accidental disconnect are typically too high. This is especially felt for players on the top.
- Players that disconnect very often get a deflated ranking that make them take points from the top and give them to the bottom. Romero2019 is extremely annoying because of this, he's a 2.5K rated player that can drive at 6K level but keeps quitting races. Beating him gives you close to nothing but losing to him loses you points. Besides completely blocking abusive quitters from some servers, the ranking system should be more robust.
- Many players only play ranked races against a limited number of other players, or have a huge proportion of their races against one or a few players. For any ranking system to work well, you need players to play a variety of opponents so that the "links" between players are strong enough. If A and B never played each other, their relative ranking will be much more accurate if you can link them through other players they played against. If you have pools of players that play together inside the pool but rarely will play with someone outside the pool, the ratings get distorted.
- A related issue is the possibility to cheat by farming points against a fake account that just disconnects (or race badly on purpose).
- A player can refuse to play any ranked race on tracks he dislikes, or against players he doesn't want to face.
- A player's ranking never "fades away" after long inactivity. On one hand, the latest available ranking is the best estimate we have of the player's strength, but on the other hand the more time passes without playing, and the more uncertain the estimate becomes. Thus, a "ranking uncertainty" estimate that goes up when inactive and down when playing races against similarly-skilled drivers (if the rating difference is big, the result of a race doesn't tell a lot about how big the difference should be) is needed.

Are there other issues ?
Image
Alayan
STK Moderator
 
Posts: 236
Joined: 25 Mar 2012, 22:11

Re: Improvements to the ranking system

Postby kimden » 28 Oct 2019, 18:04

A player can refuse to play any ranked race on tracks he dislikes


I would add that while some official tracks are not present in Android version, ranked servers are forced to accept players with at least 70% of official tracks. So any player can actually remove 6 least favourite official tracks and still race on a ranked server. That doesn't only abuse the ranking system, imagine four such players with almost different absent sets, then they could block any ranked server.
One day, I'll put enough checklines on all your tracks.
User avatar
kimden
 
Posts: 52
Joined: 23 Jun 2019, 18:42

Re: Improvements to the ranking system

Postby Andet » 28 Oct 2019, 18:15

The only other thing I can think of is that the top scores don't seem to converge anywhere, so getting a really good rank requires that you play ~1,000 races and newcomers might be discouraged.
User avatar
Andet
 
Posts: 54
Joined: 08 Dec 2018, 01:40

Re: Improvements to the ranking system

Postby kimden » 28 Oct 2019, 21:47

I thought a bit and had some ideas.

1. For each player A whose points are recalculated, we can check if there is a 'suspicious' subset of his opponents in this game (e.g. if there is a player against whom A played at least B% of his ranked races, if there are two players against whom A played a least C% of ranked races and so on). If yes, the multiplier for the ranking change can be lowered. Though then we need to store all games count for each pair of players. Also I don't know if this actually works for a subset of rivals with ~16...30 players (or maybe this is ok?)

2. Similarly, we can check for a player if he/she played too much on a certain track, and if so, lower the change multiplier. Though I don't really know how to insert ranked addons in this system (and some rare picked tracks like Zen Garden or Abyss).

Of course, this shouldn't be done with players who has less than a certain amount of races. Also I guess the lowering value for multiplier shouldn't be constant (for example, 75% of Volcan Island vs 16% of six different tracks are different situations, but both 'suspicious').

About the frequently disconnecting players, maybe we can calculate their (and their rivals') changes based on their points without rage quits (which are greater than normal points, but also with other negative changes), but add changes to normal rating (for example, a player who has disconnected 16/16 times and obtained X < 2000.00 points will have rating changes later as if he had 2000.00 after 16 races, but instead of Y points (as without rage quits) he will get Y + (X - 2000.00)). Maybe this will help with reducing rage quits...

These ideas should be developed more (or thrown away) and are not final in any way.
One day, I'll put enough checklines on all your tracks.
User avatar
kimden
 
Posts: 52
Joined: 23 Jun 2019, 18:42

Re: Improvements to the ranking system

Postby eltomito » 29 Oct 2019, 22:28

What happens to me is that every once in a while, I have to quit in the middle of a race, because a cat jumps on my laptops and shuts the lid which screws me up, or something comes up which I have to deal with immediately, or there's a problem with my wifi that makes playing impossible.

I know these are all preventable issues (shut the cats in a box while playing, find a different girlfriend, use ethernet instead of wifi) but still, it would be nice if I was allowed to drop out of a race without losing any points, lets say, three times a week. Preferably, the system would be automatic, so that if the server detected I didn't finish a race in a reasonable time, it would pretend I never entered the race in the first place and it turn a blind eye towards my disastrous performance but no more than 3 times a week/a month or whatever you consider appropriate.

----

Regarding the fragmentation issue, I love racing the Brazilians but they're so far away that it's a bit hard to find a server where neither of us lags. Also, their time is very different, so we seldom meet online. I think this is something you can't really help. I'd love to race on the HK servers but Honk Kong is so far that I can never get decent ping there. So, the ranking system should take into consideration that a fairly strong degree of geographic fragmentation is inevitable.
User avatar
eltomito
 
Posts: 196
Joined: 15 Mar 2013, 09:25

Re: Improvements to the ranking system

Postby Alayan » 30 Oct 2019, 01:39

eltomito {l Wrote}:What happens to me is that every once in a while, I have to quit in the middle of a race, because a cat jumps on my laptops and shuts the lid which screws me up, or something comes up which I have to deal with immediately, or there's a problem with my wifi that makes playing impossible.

I know these are all preventable issues (shut the cats in a box while playing, find a different girlfriend, use ethernet instead of wifi) but still, it would be nice if I was allowed to drop out of a race without losing any points, lets say, three times a week. Preferably, the system would be automatic, so that if the server detected I didn't finish a race in a reasonable time, it would pretend I never entered the race in the first place and it turn a blind eye towards my disastrous performance but no more than 3 times a week/a month or whatever you consider appropriate.


Giving "wlidcards" that allow a player to completely avoid losing points is not something I consider a great idea, as it incentivize quitting the race when you're doing particularly bad (a top player unusually losing to average players after some big mistakes, etc.). The incentive should rather be to keep on trying.

I know that sometimes it's something completely outside the fault of the player (some connection issue with the server during the race), but quits still need to give a loss imho.

However, the idea with a count of quitted races (and so a quit frequency) would be to make the punishment in rating lower when the player only rarely quit races. It would also be to tune the system so that frequent quitters have not much influence on the rating of other players, at least when they quit.

Regarding the fragmentation issue, I love racing the Brazilians but they're so far away that it's a bit hard to find a server where neither of us lags. Also, their time is very different, so we seldom meet online. I think this is something you can't really help. I'd love to race on the HK servers but Honk Kong is so far that I can never get decent ping there. So, the ranking system should take into consideration that a fairly strong degree of geographic fragmentation is inevitable.


I think this is not something that can be "fixed" either when it comes to people living far away. The fragmentation talk was mostly related to the fact that there aren't that many people on STK ranked servers so it's very possible to play many ranked races against mostly the same players. If I were to do 300 races in a row against Andet, our rankings would evolve to better show our relative strength to each other, but if our average rating was overevaluated or underevaluated, then the same will be true after those 300 races.
Image
Alayan
STK Moderator
 
Posts: 236
Joined: 25 Mar 2012, 22:11

Re: Improvements to the ranking system

Postby QwertyChouskie » 30 Oct 2019, 17:15

Maybe maintain two ranking numbers internally: The user visible one, affected by quitting, and an internal one, not affected by quitting (or only slightly affected). The latter number would be used when calculating rating transfers, so that excellent players that regularly quit races have reasonable rating transfers when playing against others. (I remember Alayan recently mentioning a play that has a low rank but has the skills of a high-ranked player, which was causing rating transfers to be not accurate. This would theoretically solve that.)
Huge fan of/occasional contributor to STK!
User avatar
QwertyChouskie
 
Posts: 474
Joined: 29 Jun 2016, 14:57

Re: Improvements to the ranking system

Postby Alayan » 30 Oct 2019, 20:59

An illustration of the noob farming issue

6K+ players extracting 12 points from a new player that had quit the race.

Image
Image
Alayan
STK Moderator
 
Posts: 236
Joined: 25 Mar 2012, 22:11

Re: Improvements to the ranking system

Postby benau » 31 Oct 2019, 05:01

There is a high ping workaround in git servers now, so as long as the internet is stable (no packet loss) you can just use the default value of max-ping and jitter tolerance.

Brazilian servers fix this!
Image
benau
STK Moderator
 
Posts: 345
Joined: 08 Dec 2015, 17:32

Re: Improvements to the ranking system

Postby relay » 02 Nov 2019, 15:46

Hi all,

first of all: thanks alot for working on and improving this beautiful game since years, we enjoy it alot with our kids and family.

I'm new in online playing it, and I feel discouraged to race against higher rank players. Not because lack of fun, but because of loosing points against top players.

Imho there is an easy to implement way to fix this:

- Solution part 1: If a player wins against a higher rank player, the points should be adapted stronger.
- Solution part 2: If a player wins against a lower rank player, the points should be adapted minimally only, if the lower rank player is maximum 25 ranks below him. (Winning against more than 25 ranks below oneself should not change ranking.)

- Why: If e.g. rank 50 player wins agains rank 100 player, he should never become a higher rank than 50, as he has not proved to race better than the ones in the list on top of him, and rank100 player should not go lower, as he was not proven to drive worse than the 101, 102, 103, ...

Thanks alot for the beautiful free game. :-)

Kind regards, and thanks for reading/reflecting
Sincerely, Relay
relay
 
Posts: 1
Joined: 02 Nov 2019, 15:22

Re: Improvements to the ranking system

Postby kimden » 03 Nov 2019, 17:20

I don't think cutting the rating change based on the rank difference (not points) makes ranking better. I played with many players and being 7th currently, it can be sometimes difficult for me to beat players in top 50 (not necessarily because my rank is too high for my skill, a player can ride some tracks exceptionally well). Probably for ranks 50-100 it's even more unusual, why shouldn't they get points for winning against players in top 200 whose ranks are only 800 less?
Also it would mean that to change rating, top players need always to play together which is unlikely. This can probably cause decrease of races for them.
One day, I'll put enough checklines on all your tracks.
User avatar
kimden
 
Posts: 52
Joined: 23 Jun 2019, 18:42

Re: Improvements to the ranking system

Postby kimden » 03 Nov 2019, 17:47

Also I think the current system doesn't encourage players to be faster in the way that 3 seconds advantage is equal to 20 seconds advantage in terms of points change (for a standard 2-minute race). So the leader cannot increase his points other way than slowing down his opponents (there are few ways like putting the gums everywhere or using switches). Don't know if this is intended though, but probably at least a small bonus should be given.

(Yes, sounds like 'give more to the richest', but the speed should be rewarded somehow.)
One day, I'll put enough checklines on all your tracks.
User avatar
kimden
 
Posts: 52
Joined: 23 Jun 2019, 18:42

Re: Improvements to the ranking system

Postby kimden » 14 Jan 2020, 05:55

This won't probably be an issue in 2.0 since there will be for sure default number of laps on all ranked servers (however, I don't like that, a bit of diversity in races length depending on some players' taste is pretty good IMO).

Consider a long long race like this (20 laps Black Forest, happens if some very funny players decide to play it):

Screenshot from 2020-01-14 07-24-52.png


Do people who finished the race deserve some points? Definitely, especially looking at that list of disconnected players (who actually mostly chose this). Second place player got +1.67 only and the winner got +0.07 for lapping him (Note: both finished players have >5000 points). Is that good? Probably not. Any sufficiently large group of players can now choose a very long race 10.5 seconds before the countdown ends and one needs really much effort to not lose points (remember that one cannot even report them on european ranked servers!). Also it's just frustrating. 1 lap on Oliver's Math Class without a win probably gives more points.

One may say that eliminated players have lower rank. It is true; but it isn't the reason of very low points change, if the times were 10 times less, finished players would get +5.1 and +21.5. One may also say that longer races are much less random so more skilled players should outperform others with more power; but why the 2nd player got so little then? He outperformed all rivals with lower rank and put much effort. (Of course I'm not saying that changes for 20 lap race should be 10 times bigger, having +210 is even more ridiculous.)

This may also sound like I'm complaining about my points (or only about top players' points). It's partly true (I always do that), but this topic is right about that and such races happened not only with me.

In short: the ranking for long races is really strange. Even not mentioning the fact that basket balls usually make it impossible to win enough points.
One day, I'll put enough checklines on all your tracks.
User avatar
kimden
 
Posts: 52
Joined: 23 Jun 2019, 18:42

Re: Improvements to the ranking system

Postby Nickerson Romero » 16 Jan 2020, 13:17

how weird?
User avatar
Nickerson Romero
 
Posts: 35
Joined: 06 Oct 2019, 19:41

Re: Improvements to the ranking system

Postby RandomPerson » 26 Feb 2020, 07:27

Are there any known changes to the ranking system that you will make? When will these come into effect? And will there be a ranking reset? I also thought a little bit and developed my own ranking system. If someone is interested in it, I can send it (to you or in this topic?). It would fix many of the known bugs in the current ranking system.
RandomPerson
 
Posts: 15
Joined: 10 Feb 2020, 09:34

Re: Improvements to the ranking system

Postby kimden » 26 Feb 2020, 15:31

I used to think there will be a major change + ranking reset for 2.0 but probably it would be changed next release.

RandomPerson {l Wrote}: I also thought a little bit and developed my own ranking system.


I am interested in the discussion. Probably it would be easier for you to post the suggestions here rather than send them separately...
One day, I'll put enough checklines on all your tracks.
User avatar
kimden
 
Posts: 52
Joined: 23 Jun 2019, 18:42

Re: Improvements to the ranking system

Postby RandomPerson » 27 Feb 2020, 06:56

So really a ranking reset .... wow, that's a shame for all the current top players :-D (but good for everyone who has not yet made it into the high range of the ranking - like me :-D ;-) ) Where did you get this information from? You're not a developer, are you? Did the developers confirm it to you? And can you say more precisely what the "major changes" will be? Anyway, thanks for your information.

In the next few days I will send the basic idea for my own ranking system here. Or should I open a new topic? Unfortunately I have no idea about coding and programming (or whatever is necessary to implement a ranking system in a game :-D), but I wrote down my idea and also made an Excel spreadsheet as an example.

Greetings :-)
RandomPerson
 
Posts: 15
Joined: 10 Feb 2020, 09:34

Re: Improvements to the ranking system

Postby kimden » 27 Feb 2020, 08:23

No, I'm not a developer. I cannot say precisely, my info is based on what was said on STK irc/discord... (some info is pretty old) so better wait until devs say something officially.

RandomPerson {l Wrote}:In the next few days I will send the basic idea for my own ranking system here.


I'm looking forward! Ideas are better than their absence, and though problems are clearly visible right now I see no easy way to change that.
One day, I'll put enough checklines on all your tracks.
User avatar
kimden
 
Posts: 52
Joined: 23 Jun 2019, 18:42

Re: Improvements to the ranking system

Postby RandomPerson » 27 Feb 2020, 08:38

My idea for a ranking system is completely new. The old system is no longer preserved. To the developers, can you answer here? Or can someone who reads this ask the developers again? I myself don't like Discord or irc.
RandomPerson
 
Posts: 15
Joined: 10 Feb 2020, 09:34

Re: Improvements to the ranking system

Postby benau » 27 Feb 2020, 09:23

ok
Image
benau
STK Moderator
 
Posts: 345
Joined: 08 Dec 2015, 17:32

Re: Improvements to the ranking system

Postby RandomPerson » 27 Feb 2020, 09:34

What do you mean by "ok", Benau? I wanted to know which changes are already known for the ranking system.
(You're a developer, aren't you?)
RandomPerson
 
Posts: 15
Joined: 10 Feb 2020, 09:34

Re: Improvements to the ranking system

Postby benau » 27 Feb 2020, 11:11

There will probably be scores decreasing if long time no play and https://github.com/supertuxkart/stk-code/issues/3774 fixed
Image
benau
STK Moderator
 
Posts: 345
Joined: 08 Dec 2015, 17:32

Re: Improvements to the ranking system

Postby RandomPerson » 27 Feb 2020, 12:13

Oh, both are very good! How many points will you lose after how long you have been inactive?
RandomPerson
 
Posts: 15
Joined: 10 Feb 2020, 09:34

Re: Improvements to the ranking system

Postby benau » 27 Feb 2020, 12:19

Undecided
Image
benau
STK Moderator
 
Posts: 345
Joined: 08 Dec 2015, 17:32

Re: Improvements to the ranking system

Postby RandomPerson » 27 Feb 2020, 12:37

Thanks for giving information!
RandomPerson
 
Posts: 15
Joined: 10 Feb 2020, 09:34

Who is online

Users browsing this forum: eltomito and 1 guest