The cropland capture game: Good annotators versus vote aggregation methods / Baklanov A., Fritz S., Khachay M., Nurmukhametov O., See L. // Advances in Intelligent Systems and Computing. - 2016. - V. 453, l. . - P. 167-180.

ISSN:
21945357
Type:
Conference Paper
Abstract:
The Cropland Capture game, which is a recently developed Geo-Wiki game, aims to map cultivated lands using around 17,000 satellite images from the Earth’s surface. Using a perceptual hash and blur detection algorithm, we improve the quality of the Cropland Capture game’s dataset. We then benchmark state-ofthe-art algorithms for an aggregation of votes using results of well-known machine learning algorithms as a baseline. We demonstrate that volunteer-image assignment is highly irregular and only good annotators are presented (there are no spammers and malicious voters).We conjecture that the last fact is the main reason for surprisingly similar accuracy levels across all examined algorithms. Finally, we increase the estimated consistency with expert opinion from 77 to 91% and up to 96% if we restrict our attention to images with more than 9 votes. © Springer International Publishing Switzerland 2016.
Author keywords:
Crowdsourcing; Image processing; Votes aggregation
Index keywords:
Artificial intelligence; Crowdsourcing; Image processing; Learning algorithms; Learning systems; Accuracy level; Blur detection; Cultivated lands; Expert opinion; Perceptual hash; Satellite images; Sp
DOI:
10.1007/978-3-319-38884-7_13
Смотреть в Scopus:
https://www.scopus.com/inward/record.uri?eid=2-s2.0-84966526008&doi=10.1007%2f978-3-319-38884-7_13&partnerID=40&md5=2279f89080bc7bffb9b8a6386a6e0ad0
Соавторы в МНС:
Другие поля
Поле Значение
Link https://www.scopus.com/inward/record.uri?eid=2-s2.0-84966526008&doi=10.1007%2f978-3-319-38884-7_13&partnerID=40&md5=2279f89080bc7bffb9b8a6386a6e0ad0
Affiliations International Institute for Applied Systems Analysis (IIASA), Laxenburg, Austria; Krasovsky Institute of Mathematics and Mechanics, Ekaterinburg, Russian Federation; Ural Federal University, Ekaterinburg, Russian Federation
Author Keywords Crowdsourcing; Image processing; Votes aggregation
Funding Details 14-11-00109, RSF, Russian Science Foundation; 617754, ERC, European Research Council
Funding Text This research was supported by Russian Science Foundation, grant no. 14-11-00109, and the EU-FP7 funded ERC CrowdLand project, grant no. 617754.
References Comber, A., Brunsdon, C., See, L., Fritz, S., McCallum, I., Comparing expert and non-expert conceptualisations of the land: An analysis of crowdsourced land cover data (2013) Spatial Information Theory, pp. 243-260. , Springer; Dawid, A.P., Skene, A.M., Maximum likelihood estimation of observer error-rates using the em algorithm (1979) Appl. Stat, pp. 20-28; Dempster, A.P., Maximum likelihood from incomplete data via the EM algorithm (1977) JRSS Ser. B, pp. 1-38; Harvey, N.J., Ladner, R.E., Lovász, L., Tamir, T., Semi-matchings for bipartite graphs and load balancing (2003) Algorithms and Data Structures, pp. 294-306. , Springer; Jagabathula, S., Reputation-based worker filtering in crowdsourcing (2014) Advances in Neural Information Processing Systems, pp. 2492-2500; Khattak, F.K., Salleb-Aouissi, A., Improving crowd labeling through expert evaluation (2012) 2012 AAAI Spring Symposium Series; Kim, H.C., Ghahramani, Z., Bayesian classifier combination (2012) International Conference on Artificial Intelligence and Statistics, pp. 619-627; Li, H., Yu, B., (2014) Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing, , arXiv:1411.4086; Moreno, P.G., Teh, Y.W., Perez-Cruz, F., Artés-Rodríguez, A., (2014) Bayesian Nonparametric Crowdsourcing, , arXiv:1407.5017; Pareek, H., Ravikumar, P., Human boosting (2013) Proceedings of the 30Th International Conference on Machine Learning (ICML-13), pp. 338-346; Raykar, V.C., Eliminating spammers and ranking annotators for crowdsourced labeling tasks (2012) JMLR, 13, pp. 491-518; Raykar, V.C., Learning from crowds (2010) J. Mach. Learn. Res, 11, pp. 1297-1322; Salk, C.F., Sturn, T., See, L., Fritz, S., Perger, C., Assessing quality of volunteer crowdsourcing contributions: Lessons from the cropland capture game (2015) Int. J. Digital Earth, pp. 1-17; See, L., Building a hybrid land cover map with crowdsourcing and geographically weighted regression. ISPRS (2015) J. Photogramm. Remote Sens, 103, pp. 48-56; Simpson, E., Dynamic Bayesian combination of multiple imperfect classifiers (2013) Decision Making and Imperfection, pp. 1-35. , Springer; Tong, H., Li, M., Zhang, H., Zhang, C., Blur detection for digital images using wavelet transform (2004) 2004 IEEE International Conference on Multimedia and Expo, 2004, 1, pp. 17-20. , ICME’04, IEEE; Zauner, C., (2010) Implementation and Benchmarking of Perceptual Image Hash Functions, , Ph.D. thesis; Zhu, X., (2011) Co-Training as a Human Collaboration Policy, , AAAI
Correspondence Address Baklanov, A.; International Institute for Applied Systems Analysis (IIASA)Austria; email: baklanov@iiasa.ac.at
Editors Nguyen T.B.Thi H.A.L.Nguyen N.T.Van Do T.
Publisher Springer Verlag
Conference name 4th International Conference on Computer Science, Applied Mathematics and Applications, ICCSAMA 2016
Conference date 2 May 2016 through 3 May 2016
Conference code 174319
ISBN 9783319388830
Language of Original Document English
Abbreviated Source Title Adv. Intell. Sys. Comput.
Source Scopus