A Deep Choice Model for Hiring Outcome Prediction in Online Labor Markets

Yixuan Ma, Zhenji Zhang, Alexander Ihler


A key challenge faced by online labor market researchers and practitioners is to understand how employers make hiring decisions from many job bidders with distinct attributes. This study investigates employer hiring behavior in one of the largest online labor markets by building a datadriven hiring decision prediction model. With the limitation of traditional discrete choice model (conditional logit model), we develop a novel deep choice model to simulate the hiring behavior from 722,339 job posts. The deep choice model extends the classical conditional logit model by learning a non-linear utility function identically for each bidder within of the job posts via a pointwise convolutional neural network. This non-linear mapping can be straightforwardly optimized using stochastic gradient approach. We test the model on 12 categories of job posts in the dataset. Results show that our deep choice model outperforms the linear-utility conditional logit model in predicting hiring preferences. By analyzing the model using dimensionality reduction and sensitivity analysis, we highlight the nonlinear combination of bidders’ features in impacting employers’ hiring decisions.


deep choice model, hiring decision, convolutional neural network, conditional logit model, online labor markets

Full Text:



Abhinav, K.; Dubey, A.; Jain, S.; Virdi, G.; Kass, A.; Mehta, M. (2017). Crowdadvisor: A framework for freelancer assessment in online marketplace, Proceedings of the 39th International Conference on Software Engineering: Software Engineering in Practice Track, Buenos Aires, Argentina, 93-102, 2017.

Agrawal, A.; Lacetera, N.; Lyons, E. (2016). Does standardized information in online markets disproportionately benefit job applicants from less developed countries? Journal of international Economics, 103, 1-12, 2016.

Akiva, M. E.; Lerman, S. R.; Lerman, S. R. (1985). Discrete choice analysis: theory and application to travel demand, MIT press, vol. 9, 1985.

Boskin, M. J. (1974). A conditional logit model of occupational choice. Journal of Political Economy, 82(2, Part 1), 389-398, 1974.

Brynjolfsson, E.; Hitt, L. M. (2003). Computing productivity: Firm-level evidence. Review of economics and statistics, 85(4), 793-808, 2003.

Chan, J.; Wang, J. (2017). Hiring preferences in online labor markets: Evidence of a female hiring bias, Management Science, 64(7), 2973-2994, 2017.

Chollet, F. (2015). Keras. https://keras.io/, 2015.

Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, USA, 1251-1258,. 2017.

Sousa, W.; Montevechi, J.; Miranda, R.; Rocha, F.; Vilela F. (2019). Economic Lot-size using machine learning, parallelism, metaheuristic and simulation, 18(2), 205-216, 2019.

Fernández-Delgado, M.; Cernadas, E.; Barro, S.; Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research, 15(1), 3133-3181, 2014.

Goodfellow, I.; Bengio, Y.; Courville, A.; Bengio, Y. (2016). Deep learning, MIT press Cambridge, Vol.1, 2016.

Hagenauer, J. and Helbich, M. (2017). A comparative study of machine learning classifiers for modeling travel mode choice. Expert Systems with Applications, 78, 273-282, 2017.

Hensher, D. A.; Ton, T. T. (2000). A comparison of the predictive potential of artificial neural networks and nested logit models for commuter mode choice. Transportation Research Part E: Logistics and Transportation Review, 36(3), 155-172, 2000.

Hoffman, S. D.; Duncan, G. J. (1988). Multinomial and conditional logit discrete-choice models in demography. Demography, 25(3), 415-427, 1988.

Horton, J. J.; Tambe, P. (2015). Labor economists get their microscope: big data and labor market analysis. Big data, 3(3), 130-137, 2015.

Huws, U.; Joyce, S. (2016). Crowd working survey: Size of the uk's 'gig economy' revealed for the first time. University of Hertfordshire, 2016.

Ji, S.; Xu, W.; Yang, M.; Yu, K. (2013). 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1), 221-231, 2013.

Klein, G. A.; Orasanu, J. E.; Calderwood, R. E.; Zsambok, C. E. (1993). Decision making in action: Models and methods, Ablex Publishing, 1993.

Kokkodis, M.; Papadimitriou, P.; Ipeirotis, P. G. (2015). Hiring behavior models for online labor markets. Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, Shanghai, China, 223-232, 2015.

Krizhevsky, A.; Sutskever, I.; Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, Nevada, USA, 1097- 1105, 2012.

Kuek, S. C.; Paradi-Guilford, C.; Fayomi, T.; Imaizumi, S.; Ipeirotis, P., Pina, P.; Singh, M. (2015). The global opportunity in online outsourcing, Technical report, World Bank, Washington, DC, 2015.

LeCun, Y.; Bengio, Y. (1995). Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10), 255-258, 1995.

LeCun, Y.; Bengio, Y.; Hinton, G. (2015). Deep learning, Nature, 521(7553), 436-444, 2015.

LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324, 1998.

Lazear, P.; Shaw, L.; Stanton, T. (2016). Who gets hired? the importance of finding an open slot, Technical report, National Bureau of Economic Research, 2016.

Lhéritier, A.; Bocamazo, M.; Delahaye, T.; Acuna-Agost, R. (2019). Airline itinerary choice modeling using machine learning. Journal of Choice Modelling, 31, 198-209, 2019.

Lin, M.; Chen, Q.; Yan, S. (2013). Network in network. arXiv preprint arXiv:1312.4400, 2013.

Lin, W.-Y.; Hu, Y.-H.; Tsai, C.-F. (2012). Machine learning in financial crisis prediction: a survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(4), 421-436, 2012.

Ma, Y.; Zhang, Z.; Ihler, A.; Pan, B. (2018). Estimating warehouse rental price using machine learning techniques. International Journal of Computers Communications & Control, 13(2), 235- 250, 2018.

Maaten, L. V. D.; Hinton, G. (2008). Visualizing data using t-sne. Journal of machine learning research, 9, 2579-2605, 2008.

Mali, P.; Kuzmanovic, B.; Nikolic, M.; Mitic, S.; Terek, E. (2019). Model of leadership and entrepreneurial intentions among employed persons. International Journal of Simulation Modelling, 18(3), 385-396, 2019.

McFadden, D. (1974). Conditional logit analysis of qualitative choice behavior. New York: Academic Press, Chapter 4, 1974.

McFadden, D. (2001). Economic choices. American economic review, 91(3), 351-378, 2001

Moreno, A.; Terwiesch, C. (2014). Doing business with strangers: Reputation in online service marketplaces. Information Systems Research, 25(4), 865-886, 2014.

Mottini, A.; Acuna-Agost, R. (2017). Deep choice model using pointer networks for airline itinerary prediction. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, 1575-1583, 2017.

Nam, D.; Kim, H.; Cho, J.; Jayakrishnan, R. (2017). A model based on deep learning for predicting travel mode choice. Proceedings of the Transportation Research Board 96th Annual Meeting Transportation Research Board, Washington, DC, USA, 8-12, 2017.

Olugboyega O. (2016). Examining the protocols and platforms adopted for building information modelling processes by Nigerian construction professionals Journal of System and Management Sciences, 6(4), 1-45, 2016.

Omrani, H. (2015). Predicting travel mode of individuals by machine learning. Transportation Research Procedia, 10, 840-849, 2015.

Otsuka, M.; Osogami, T. (2016). A deep choice model. AAAI, Phoenix, Arizona USA, 850-856, 2016.

Pan, B.; Hsu, K.; AghaKouchak, A.; Sorooshian, S. (2019). Improving precipitation estimation using convolutional neural network. Water Resources Research, 55(3), 2301-2321, 2019.

Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958, 2014.

Yoganarasimhan, H. (2013). The value of reputation in an online freelance marketplace. Marketing Science, 32(6), 860-891, 2013.

Freelancer.com (2019). Freelancer.com reveals the fastest growing online jobs in 2018. http://www.freelancer.com. Accessed April 30 2019.

Upwork (2019). Upwork to report first quarter 2019 financial results on may 8, 2019. https://investors.upwork.com/news-releases. Accessed April 30 2019.

DOI: https://doi.org/10.15837/ijccc.2020.2.3760

Copyright (c) 2020 Yixuan Ma, Zhenji Zhang, Alexander Ihler

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

IJCCC is an Open Access Journal : CC-BY-NC.

Articles published in IJCCC user license are protected by copyright.

Users can access, download, copy, translate the IJCCC articles for non-commercial purposes provided that users, but cannot redistribute, display or adapt:

  • Cite the article using an appropriate bibliographic citation: author(s), article title, journal, volume, issue, page numbers, year of publication, DOI, and the link to the definitive published version on IJCCC website;
  • Maintain the integrity of the IJCCC article;
  • Retain the copyright notices and links to these terms and conditions so it is clear to other users what can and what cannot be done with the  article;
  • Ensure that, for any content in the IJCCC article that is identified as belonging to a third party, any re-use complies with the copyright policies of that third party;
  • Any translations must prominently display the statement: "This is an unofficial translation of an article that appeared in IJCCC. Agora University  has not endorsed this translation."

This is a non commercial license where the use of published articles for commercial purposes is forbiden. 

Commercial purposes include: 

  • Copying or downloading IJCCC articles, or linking to such postings, for further redistribution, sale or licensing, for a fee;
  • Copying, downloading or posting by a site or service that incorporates advertising with such content;
  • The inclusion or incorporation of article content in other works or services (other than normal quotations with an appropriate citation) that is then available for sale or licensing, for a fee;
  • Use of IJCCC articles or article content (other than normal quotations with appropriate citation) by for-profit organizations for promotional purposes, whether for a fee or otherwise;
  • Use for the purposes of monetary reward by means of sale, resale, license, loan, transfer or other form of commercial exploitation;

    The licensor cannot revoke these freedoms as long as you follow the license terms.

[End of CC-BY-NC  License for Website User]

INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL (IJCCC), With Emphasis on the Integration of Three Technologies (C & C & C),  ISSN 1841-9836.

IJCCC was founded in 2006,  at Agora University, by  Ioan DZITAC (Editor-in-Chief),  Florin Gheorghe FILIP (Editor-in-Chief), and  Misu-Jan MANOLESCU (Managing Editor).

Ethics: This journal is a member of, and subscribes to the principles of, the Committee on Publication Ethics (COPE).

Ioan  DZITAC (Editor-in-Chief) at COPE European Seminar, Bruxelles, 2015:

IJCCC is covered/indexed/abstracted in Science Citation Index Expanded (since vol.1(S),  2006); JCR2018: IF=1.585..

IJCCC is indexed in Scopus from 2008 (CiteScore2018 = 1.56):

Nomination by Elsevier for Journal Excellence Award Romania 2015 (SNIP2014 = 1.029): Elsevier/ Scopus

IJCCC was nominated by Elsevier for Journal Excellence Award - "Scopus Awards Romania 2015" (SNIP2014 = 1.029).

IJCCC is in Top 3 of 157 Romanian journals indexed by Scopus (in all fields) and No.1 in Computer Science field by Elsevier/ Scopus.


 Impact Factor in JCR2018 (Clarivate Analytics/SCI Expanded/ISI Web of Science): IF=1.585 (Q3). Scopus: CiteScore2018=1.56 (Q2);

SCImago Journal & Country Rank

Editors-in-Chief: Ioan DZITAC & Florin Gheorghe FILIP.