Beschreibung
Another type of broadly-used method is training the crowd workers before assigning them to a given MCS task [69, 78]. This type of methods overcome the problem of embedding gold instances in the whole annotation process of crowd workers because a crowd worker without enough domain knowledge can get trained and crowd workers not performing well after the training can be excluded earlier. Nevertheless, in most existing solutions in this category, a ?xed training set is used to train the crowd workers and the ?nal selection of the crowd workers is mainly based on the number of questions they correctly answered in the training phase. We, via the following illustrative example, argue that the number of correctly answered questions during the training process may not be a good indicator for the ?nal selection of crowd workers, and we can do much better if we consider each individual user’s learning pattern.
