# Thread: General computer cheating discussion

1. Originally Posted by Bulldozer It can be interesting, especially for Kevin. http://centralsquares.com/2015/06/11...mihaela-sandu/
Bayesian inference makes sense. What do you think? Yes, we don't have enough statistical data about cheaters yet but the method is interesting.
Getting everything right in such a calculation is very difficult. The following are among the issues I have with it:

1. The analysis assumes equal probability for a win with each colour. In reality this is not the case. A win against a stronger player is significantly more likely with white and significantly less likely with black. This makes Sandu's streak somewhat less probable (not sure how much).

2. The analysis assumes that when the transmission is off, Sandu reverts to the scoring probability given by her rating. But that assumes that if Sandu is cheating now then she has never cheated before. It may be that she has cheated before and that she is actually weaker than her rating.

3. The same point you made: if Sandu was only cheating for some moves then her win probability if cheating is probably lower. Probably not as low as .8 though. However because the cheats who are caught are the stupid cheats (the ones who cheat more or less every move, who would have win probabilities higher than .9) we have no idea how prevalent careful cheating is. I hope it's not as common as 1/500 or even 1/1000 but it might be. Getting the prior for a low-probability event right is one of the very difficult things in this sort of analysis.

The analysis also assumes the non-transmission of one game was accidental. It was apparently deliberate on the part of the organisers. On that basis one could factor in the possibility that this adversely affected Sandu during that game giving her an increased chance to lose it (that again increases the probability of innocence.)

I think that the same thing happens whatever changes are made to the assumptions though: the probability of cheating may come out as well over 50% but it won't end up as 99%. This vindicates the FIDE approach that no matter how suspicious the results or games, you don't convict someone on game evidence alone - there has to be further evidence such as suspicious behaviour (even if that evidence is not by itself conclusive.)  Reply With Quote

2. Originally Posted by Vlad It is not interesting, it is plain wrong. I have already tried to explain why it is wrong on the crestbook forum. If Dr Ricy has time, he might give his 2 cents too. "4) Finally, the derivation of 0.3 probability makes no sense to me. Your earlier calculations were produced under different hypotheses. One can’t just combine all these numbers and plug them all into one formula."

Actually this is exactly what Bayesian analysis does. It computes the probability of an event occurring (Sandu cheating) given Sandu's results based on:

* the assumed a priori probability that any given player would cheat in this event
* the probability of Sandu obtaining her results assuming that she was cheating
* the probability of Sandu obtaining her results assuming that she wasn't cheating

It's a method that can work extremely well for assessing probabilities of a wide range of things - especially things like false positives in medical tests - but it is very sensitive to the assumptions made and very prone to "garbage-in-garbage-out."

In simple terms this is also about false positives. If a given crime is rare and someone can look guilty of it without being so, then a very high level of evidence is required to show they are actually guilty.

From an earlier post:

Strong players 2400+ (just cut this pesky Michael Baron) would put a very high weight on that Sandu was getting some help, while under 2000 prefer to believe in a fair tale.
It seems a certain retired very strong Australian player believes in fairy tales too.   Reply With Quote

3. No, that formula is applied incorrectly.

А - Sandu is a cheater, nА - Sandu is not a cheater.
B - sequence (4 wins, a loss, a win) happened, nB - did not happen.

Now we write the Bayesian formula

p(nА/B)=p(B/nA)*p(nA)/p(B).

She calculated that p(B/nA)=1/69,000 (I am happy to use this number). In the worst case scenario when nobody cheats p(nA)=1. Assuming p(nA)<1 will make my estimate better.

The probability of sequence B occuring is 1/3 in power 6; that is, p(B)=(1/3)^6=1/729. I am assuming that a win, a draw and a loss are equally likely. If you assume that draw is less likely than either a win or a loss it will make my estimate better. Combine all the values to derive:

p(nA/B)=0,01. So the correct result is 1% not 30% she claims.  Reply With Quote

4. Regarding Kevin's points above I don;t think they are major objections for the following reasons

1. The blogger did analyse the effect of higher draw percentages overall. There appeared to be a roughly linear dependence on the posterior probability so unlikely to be make a decisive difference.

2. I think this is tacitly built into the hypothesis. As mentioned in the article Sandu's rating was stable over a number of years in the range 2300-2400 and it is assumed that this is her base level of ability. However I understand she has been less active over the last couple of years and so we have less evidence if that is the correct base level (it could be higher or lower) but as that is the best we have then we use that.

3. Some more parameter analysis could determine the sensitivity of the posterior based on the expected result for cheating. I would probably assume a certain base-rating and cheat-rating or perhaps cheating-rating-boost. Rather than arbitrarily just pull probabilities out of the air. The reason being that we have better intuition about what a rating means.

The benefit of the Baysian approach is that the reasoning is sound and it focuses the discussion on the hypotheses being tested and the prior probabilities being posited. This this case the base rate of cheating seems to be the main driver. How likely is cheating is a difficult question to answer. Perhaps it is quite prevalent for low boosts but quite rare for large boosts. Whether rare means 1 in 1,000 or 1 in 10,000 is a very open question.

However even in large boost cheating is only 1 in 1,000 rare then there still is insufficient evidence to convict Sandu on the evidence of results alone. People can still have a good run of form and I don't believe there is any conviction possible without some physical evidence. There is cause for suspicion but that suspicion should only entail an investigation where the presumption of innocence is not infringed.  Reply With Quote

5. Originally Posted by Vlad No, that formula is applied incorrectly.

А - Sandu is a cheater, nА - Sandu is not a cheater.
B - sequence (4 wins, a loss, a win) happened, nB - did not happen.

Now we write the Bayesian formula

p(nА/B)=p(B/nA)*p(nA)/p(B).

She calculated that p(B/nA)=1/69,000 (I am happy to use this number). In the worst case scenario when nobody cheats p(nA)=1. Assuming p(nA)<1 will make my estimate better.

The probability of sequence B occuring is 1/3 in power 6; that is, p(B)=(1/3)^6=1/729. I am assuming that a win, a draw and a loss are equally likely. If you assume that draw is less likely than either a win or a loss it will make my estimate better. Combine all the values to derive:

p(nA/B)=0,01. So the correct result is 1% not 30% she claims.

Her assumption is either Sandu cheated or she didn't cheat

P(B/A) = 1/29000, P(B/nA) = 1/69000 and thus

P(B) = P(B/A) + P(B/nA) = 4.89755E-05

In your estimate of P(B) you assume results happen entirely an random and come up with P(B) = 1.37E-2 but B entails you observing the results games between players of specified ratings.  Reply With Quote

6. Originally Posted by Vlad The probability of sequence B occuring is 1/3 in power 6; that is, p(B)=(1/3)^6=1/729. I am assuming that a win, a draw and a loss are equally likely. If you assume that draw is less likely than either a win or a loss it will make my estimate better. Combine all the values to derive:
This part is completely wrong. P(B) is the prior probability of Sandu achieving that sequence in the absence of any assumption about whether she is cheating or not. It is not the prior probability of two completely random players playing each other. You cannot assume a win is 1/3 likely since if Sandu is not cheating, which on the prior assumption is overwhelmingly likely, then a win will be much less than 1/3. How much you increase the win probability above the rating expectation depends on how strongly the possibility of cheating is weighted in the prior assumption - but to get the win probability up from say .12 if not cheating to .33 you need an a priori assumption that there is at least a 1/4 chance of cheating.  Reply With Quote

7. Originally Posted by Rincewind Her assumption is either Sandu cheated or she didn't cheat

P(B/A) = 1/29000, P(B/nA) = 1/69000 and thus

P(B) = P(B/A) + P(B/nA) = 4.89755E-05
Not exactly. 1/29000 is P(B/A)*P(A) = (0.90×0.90×0.90×0.90×0.58×0.90) * (1/10000) = 1/29000

She calculated P(B) in this way:
P(B) = P(B/A)*P(A) + P(B/nA)*P(nA) = (0.9^5*0.58) * (1/10000) + (0.12^5*0.58) * (9999/10000) = 0.000034248 + 0.000014431 = 0.000048679

And, finally:
P(nA/B) = P(B/nA) * P(nA) / P(B) = (0.12^5*0.58) * (9999/10000) / 0.000048679 = 0.296448423

Do you agree?  Reply With Quote

8. Originally Posted by Vlad I have already tried to explain why it is wrong on the crestbook forum. If Dr Ricy has time, he might give his 2 cents too. On Crestbook, nobody understands even a little bit of what we are talking about. So, I decided to come here.   Reply With Quote

9. Originally Posted by Bulldozer Do you agree?  Reply With Quote

10. Originally Posted by Rincewind Her assumption is either Sandu cheated or she didn't cheat

P(B/A) = 1/29000, P(B/nA) = 1/69000 and thus

P(B) = P(B/A) + P(B/nA) = 4.89755E-05

In your estimate of P(B) you assume results happen entirely an random and come up with P(B) = 1.37E-2 but B entails you observing the results games between players of specified ratings.
Then I think there is a problem with the approach. You are not allowing me to take an average p(B), but taking an average p(nA) yourselve. That is sort of unfair. I can say that given that the player is old, performed poorly in the previous tournament, etc, the probability p(nA) should not be just an average across all players but rather much higher. More like p(nA)=1/4, as Kevin suggested.   Reply With Quote

11. Originally Posted by Vlad Then I think there is a problem with the approach. You are not allowing me to take an average p(B), but taking an average p(nA) yourselve. That is sort of unfair. I can say that given that the player is old, performed poorly in the previous tournament, etc, the probability p(nA) should not be just an average across all players but rather much higher. More like p(nA)=1/4, as Kevin suggested. I think the problem is that you are not calculating the conditional probabilities. Either Sandu cheated or she didn't and so P(B) = P(B|A)*P(A) + P(B|nA)*P(nA) (using the improved notation of Bulldozer). If you agree with this then how do you account for the additional probability mass in your value of P(B)?

PS I don't really understand where Kevin gets the 1/4 number from. The prior P(A) and P(nA) are the prevelance of cheating in the general chess population. I cannot accept that it is anything like 1 in 4. If you need that sort of prior for a conviction then your case is doomed.  Reply With Quote

12. PPS Using P(B|nA) = 1/69000 and P(B|A) = 1/3 then to get P(B) = P(B|A)*P(A) + P(B|nA)*P(nA) = (1/3)^6 what you need is P(A) = 1/250.

Note the original blog post also says to get P(A|B) > 0.99 then you need P(A) of 1/240 which explains why your value of P(B) implies a posterior of around 99%.  Reply With Quote

13. I do not think you have followed my argument. I said you should not average out across all chessplayers to derive the probability of cheating p(A). You should consider only old, rusty, etc chess players who are capable to achieve B. The probability p(A) in such a case is certainly much higher because you have a selection bias. You are selecting cheaters more often than on average.

The second problem with the calculation, that I mentioned on crestbook, but have not yet here, is the assumption that with computer insistence Sandu can achieve B only in 1/3 cases. Really? In my opinion, it is more like 90-100%.

Give me an access to a computer and I will kick Magnus Carlsen's butt. He may get occasional draw, but the score out of 10 games will be at least something like 9:1. It clearly indicates to me that she is not a chessplayer and does not understand that humans and computers are complementary. Well, at least at the master level.  Reply With Quote

14. Originally Posted by Vlad I do not think you have followed my argument. I said you should not average out across all chessplayers to derive the probability of cheating p(A). You should consider only old, rusty, etc chess players who are capable to achieve B. The probability p(A) in such a case is certainly much higher because you have a selection bias. You are selecting cheaters more often than on average.
I'm willing to accept rusty players as a reference class if you have some objective basis for the premise that they are more likely to cheat. Thinking of well publicised cases of cheaters and those suspected of cheating the main to in recent times are Feller and Ivanov. Neither were particularly old or rusty. Originally Posted by Vlad The second problem with the calculation, that I mentioned on crestbook, but have not yet here, is the assumption that with computer insistence Sandu can achieve B only in 1/3 cases. Really? In my opinion, it is more like 90-100%.
The argument is premised on those cheating at key positions and not all the time. That is players who are cheating but careful to avoid cheating allegation from game analysis and so just spot-cheating. I think on this assumption the probabilities are reasonable. But you can argue about different probabilities if you like. I think you will find it is difficult to get P(A|B) up to .99 based on that parameter. Originally Posted by Vlad Give me an access to a computer and I will kick Magnus Carlsen's butt. He may get occasional draw, but the score out of 10 games will be at least something like 9:1. It clearly indicates to me that she is not a chessplayer and does not understand that humans and computers are complementary. Well, at least at the master level.
I think that objection is dealt with the premise of spot cheating to avoid detection via game analysis. What would be your score over Magnus if you had to take care not to be detected as using a computer by (a) suspicious behaviour and (b) game analysis?  Reply With Quote

15. Originally Posted by Rincewind I'm willing to accept rusty players as a reference class if you have some objective basis for the premise that they are more likely to cheat. Thinking of well publicised cases of cheaters and those suspected of cheating the main to in recent times are Feller and Ivanov. Neither were particularly old or rusty.
Again you are missing my point. I suggested to consider the population that is both rusty and performing much better than their ratings. Among this population there will be a significant percentage of cheaters. Originally Posted by Rincewind I think that objection is dealt with the premise of spot cheating to avoid detection via game analysis. What would be your score over Magnus if you had to take care not to be detected as using a computer by (a) suspicious behaviour and (b) game analysis?
Yes, that is exactly what I mean. Say I have a device on me that sends me a signal each move but I only use it in 10-20% cases. In 80-90 % cases I am strong enough to make a judgement myself. If I train for a couple months, I doubt anybody in Australia will survive. First 15 moves I can just play remembering theory. In the next 10-15 I may have to use 5-8 moves. Once we come to an endgame and I have some advantage, I do not require any assistance. So 5-8 moves in the entire 50 + game.  Reply With Quote