Active Crowd Toolkit

Benchmarking tools for crowdsourcing research

Test crowd consensus methods with one line of code

Use a set of existing data aggregation models to combine crowd labels and learn information about the workers.

The ActiveCrowdToolkit .NET v0.1 includes the methods: Majority voting, Vote distribution, Dawid&Skene [1], Bayesian Classifier Combination (BCC) [2], Community-Based BCC [3]

Build new active learning strategies

By combining existing data aggregation models, task selection methods and worker selection methods.

Play with your data sets

Import data in comma-separated format using integrated data parsing methods.

Integration with Infer.NET: Building machine learning models in ease

Take advantage of the Infer.NET framework for implementing probabilistic graphical models and running approximate Bayesian inference in just a few lines of code.

See some examples here and here, and check out the code in Github.

Analyse real-time performance of your algorithms

Use the graphical interface to understand the relative performance of different strategies, and visualise internal models such as the belief over a task’s true label and confusion matrices of individual workers.

Command-line API and cluster deployment

Want to run repeated experiments? Use the toolkit's command-line API to run experiments in batches. This is ideal for cluster deployment.

Publications

References

  1. A. P. Dawid and A. M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, Vol. 28, No. 1, pp. 20-28., 1979

  2. Kim, Hyun-Chul, and Zoubin Ghahramani. Bayesian classifier combination In: International conference on artificial intelligence and statistics, pp. 619-627. 2012.

  3. Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, Milad Shokouhi. Community-Based Aggregation Models for Crowdsourcing. In: the 23rd International World Wide Web Conference (WWW). Best paper runner-up, pp. 155-164, 2014

Credits

    John Guiver - Microsoft Research, Cambridge

    Pushmeet Kohli - Microsoft Research, Cambridge

    Po Ting Tse - University of Southampton