Vote aggregation techniques in the Geo-Wiki crowdsourcing game: A case study / Baklanov A., Fritz S., Khachay M., Nurmukhametov O., Salk C., See L., Shchepashchenko D. // Communications in Computer and Information Science. - 2017. - V. 661, l. . - P. 41-50.

ISSN:
18650929
Type:
Conference Paper
Abstract:
The Cropland Capture game (CCG) aims to map cultivated lands using around 170000 satellite images. The contribution of the paper is threefold: (a) we improve the quality of the CCG’s dataset, (b) we benchmark state-of-the-art algorithms designed for an aggregation of votes in a crowdsourcing-like setting and compare the results with machine learning algorithms, (c) we propose an explanation for surprisingly similar accuracy of all examined algorithms. To accomplish (a), we detect image duplicates using the perceptual hash function pHash. In addition, using a blur detection algorithm, we filter out unidentifiable images. In part (c), we suggest that if all workers are accurate, the task assignment in the dataset is highly irregular, then state-of-the-art algorithms perform on a par with Majority Voting.We increase the estimated consistency with expert opinions from 77% to 91% and up to 96% if we restrict our attention to images with more than 9 votes. © Springer International Publishing AG 2017.
Author keywords:
Crowdsourcing; Image processing; Votes aggregation
Index keywords:
Benchmarking; Crowdsourcing; Hash functions; Image processing; Learning algorithms; Learning systems; Blur detection; Cultivated lands; Expert opinion; Perceptual hash; Satellite images; State-of-the-
DOI:
10.1007/978-3-319-52920-2_4
Смотреть в Scopus:
https://www.scopus.com/inward/record.uri?eid=2-s2.0-85014154873&doi=10.1007%2f978-3-319-52920-2_4&partnerID=40&md5=ad4b775fa73cf33e3e543d84be1b8651
Соавторы в МНС:
Другие поля
Поле Значение
Link https://www.scopus.com/inward/record.uri?eid=2-s2.0-85014154873&doi=10.1007%2f978-3-319-52920-2_4&partnerID=40&md5=ad4b775fa73cf33e3e543d84be1b8651
Affiliations International Institute for Applied Systems Analysis (IIASA), Laxenburg, Austria; Krasovsky Institute of Mathematics and Mechanics, Ekaterinburg, Russian Federation; Ural Federal University, Ekaterinburg, Russian Federation
Author Keywords Crowdsourcing; Image processing; Votes aggregation
Funding Details 14-11-00109, RSF, Russian Science Foundation; 617754, ERC, European Research Council; EU-FP7, ERC, European Research Council
Funding Text This research was supported by Russian Science Foundation, grant no. 14-11-00109, and the EU-FP7 funded ERC CrowdLand project, grant no. 617754.
References Chatterjee, S., Bhattacharyya, M., A biclustering approach for crowd judgment analysis (2015) Proceedings of the Second ACM IKDD Conference on Data Sciences, pp. 118-119; Comber, A., Brunsdon, C., See, L., Fritz, S., McCallum, I., Comparing expert and non-expert conceptualisations of the land: An analysis of crowdsourced land cover data (2013) COSIT 2013. LNCS, 8116, pp. 243-260. , Tenbrink, T., Stell, J., Galton, A., Wood, Z. (eds.), Springer, Heidelberg; Dawid, A.P., Skene, A.M., Maximum likelihood estimation of observer error-rates using the EM algorithm (1979) Appl. Stat, 28, pp. 20-28; Dempster, A.P., Maximum likelihood from incomplete data via the EM algorithm (1977) JRSS Ser. B, 39, pp. 1-38; Jagabathula, S., Reputation-based worker filtering in crowdsourcing (2014) Advances in Neural Information Processing Systems, pp. 2492-2500; Karger, D.R., Oh, S., Shah, D., Iterative learning for reliable crowdsourcing systems (2011) Advances in Neural Information Processing Systems, pp. 1953-1961; Khattak, F.K., Salleb-Aouissi, A., Improving crowd labeling through expert evaluation (2012) 2012 AAAI Spring Symposium Series; Kim, H.C., Ghahramani, Z., Bayesian classifier combination (2012) International Conference on Artificial Intelligence and Statistics, pp. 619-627; Liu, Q., Peng, J., Ihler, A.T., Variational inference for crowdsourcing (2012) Advances in Neural Information Processing Systems, pp. 692-700; Moreno, P.G., Teh, Y.W., Perez-Cruz, F., Artés-Rodríguez, A., (2014) Bayesian Nonparametric Crowdsourcing, , arXiv preprint arXiv:1407.5017; Pareek, H., Ravikumar, P., Human boosting (2013) Proceedings of the 30Th International Conference on Machine Learning (ICML2013), pp. 338-346; Raykar, V.C., Eliminating spammers and ranking annotators for crowdsourced labeling tasks (2012) JMLR, 13, pp. 491-518; Raykar, V.C., Learning from crowds (2010) J. Mach. Learn. Res, 11, pp. 1297-1322; Salk, C.F., Sturn, T., See, L., Fritz, S., Perger, C., Assessing quality of volunteer crowdsourcing contributions: Lessons from the cropland capture game (2015) Int. J. Digit. Earth, 9, pp. 410-426; See, L., Building a hybrid land cover map with crowdsourcing and geographically weighted regression. ISPRS (2015) J. Photogramm. Remote Sens, 103, pp. 48-56; Sheshadri, A., Lease, M., Square: A benchmark for research on computing crowd consensus (2013) First AAAI Conference on Human Computation and Crowdsourcing; Simpson, E., Roberts, S., Psorakis, I., Smith, A., Dynamic Bayesian combination of multiple imperfect classifiers (2013) Decision Making and Imperfection, pp. 1-35. , Guy, T.V., Karny, M.,Wolpert, D. (eds.), Springer, Heidelberg; Tong, H., Li, M., Zhang, H., Zhang, C., Blur detection for digital images using wavelet transform (2004) 2004 IEEE International Conference on Multimedia and Expo, ICME 2004, 1, pp. 17-20. , IEEE; Zauner, C., (2010) Implementation and Benchmarking of Perceptual Image Hash Functions, , Ph.D. thesis; Zhu, X., Co-training as a human collaboration policy (2011) AAAI
Correspondence Address Baklanov, A.; International Institute for Applied Systems Analysis (IIASA)Austria; email: baklanov@iiasa.ac.at
Editors Loukachevitch N.Panchenko A.Vorontsov K.Labunets V.G.Savchenko A.V.Ignatov D.I.Nikolenko S.I.Khachay M.Y.
Sponsors Exactpro;IT Centre;OK.Ru (Mail.Ru Group)
Publisher Springer Verlag
Conference name 5th International Conference on Analysis of Images, Social Networks and Texts, AIST 2016
Conference date 7 April 2016 through 9 April 2016
Conference code 189269
ISBN 9783319529196
Language of Original Document English
Abbreviated Source Title Commun. Comput. Info. Sci.
Source Scopus