Firstly, paying is not at all the correct incentive for the desired outcome. When the incentive is payment, people will optimize for maximum payout not for the quality goals of the system.

Secondly, it doesn't fix stupidity. A participant who earnestly takes the quality goals of the system to heart instead of focusing on maximizing their take (thus, obviously stupid) will still make bad classifications due to that reason.

> Firstly, paying is not at all the correct incentive for the desired outcome. When the incentive is payment, people will optimize for maximum payout not for the quality goals of the system.

1. I would expect any paid arrangement to include a quality-control mechanism. With the possible exception of if it was designed from scratch by complete ignoramuses.

2. Do you have a proposal for a better incentive?

1. Goodhart's law suggests that you will end up with quality control mechanisms which work at ensuring that the measure is being measured, but not that it is measuring anything useful

2. Criticism of a method does not require that there is a viable alternative. Perhaps the better idea is just to not incentivize people to do tasks they are not qualified for

> Secondly, it doesn't fix stupidity.

Agreed, and would add that it doesn’t fix other things like lack of skill, focus, time, etc.

An example is the output of the Amazon Turk “Sheep Market” experiment:

https://docubase.mit.edu/project/the-sheep-market/

Some of those sheep were really ba-aaa-ad.

I don't think there is any correct incentive for "do unpaid labour for someone's proprietary model but please be diligent about it"

edit: ugh. it's even worse, lmarena itself is a proprietary system, so the users presumably don't even get the benefit of an open dataset out of all this

Why would an unpaid human want to do that?

Exactly — they wouldn't.

Therein lies the problem.