Bridging the gap: Online job postings, survey data and the assessment of job vacancies in Canada

OJP data from firms like Vicinity Jobs have emerged as a complement to traditional sources of labour demand data, such as the Job Vacancy and Wages Survey (JVWS). Ibrahim Abuallail, PhD Candidate, University of Ottawa, presented research relating to bias in OJPs and a proposed approach to effectively adjust OJP data to complement existing official data (such as from the JVWS) and improve the measurement of labour demand.

1. Bridging the Gap: Online Job Postings, Survey Data and the Assessment of Job Vacancies Author & Presenter: Ibrahim Abuallail Special thanks to Brittany Feor & Sukriti Tehran Co-author: Anne-Lore Fraikin  

2. Introduction & Motivation

3. Motivation Importance of Vacancy Data in Economics Pissarides (1986) Macro-labour economists studying business cycles ( Yashiv (2007)). Conventional Approach: Survey Data (JVWS)

4. Limitations of JVWS Data Time lag in reporting. Impractical to inform real-time policy decisions The sample size is less than 10% of all employers in Canada. Data suppression for confidentiality. Exclusion of federal, territorial, and provincial employees. This also includes a considerable number of management occupations that are not captured in the data

5. Data & Methodology

6. An Alternative Approach: Online Job Postings Turell et. al (2019) use a weighting function to improve online job posting data representativeness using UK data. Evans et. al (2023) use a winsorization algorithm to improve online job postings forecast errors (with respect to survey) using Australian data. Several studies have already begun leveraging online job postings data as a reliable source (see for example Marinescu (2017), Deming and Kahn (2018), and Goldfarb et. al (2020)). Related Literature

7. Vicinity Jobs Data Web scraping tool that collects online job postings from multiple website sources Contains over 103 million job postings for the period between 2018 and October 2023 Around half of the data sources are from actual employer websites Other third-party non-employer job posting websites may also be included; Ex: indeed and monster.com Most third-party data sources included are cost based (employers have to pay to post), or include high administrative costs

8. Vicinity Jobs Data Figure 1: Top 10 website sources for VJ data. Employer websites and government job boards comprise the majority of the data sources

9. Vicinity Jobs Data → Graph for survey Figure 2: Distribution of Broad Occupational Categories in Online Job Postings Data. Breakdown of NOC categories in → NOC 2016 Table 7

10. Limitations of Online Job Postings Data Potential for duplication Authenticity Job posting not for the purpose of hiring Postings by fraudsters, with job scams growing in recent years Employers could attempt to recruit multiple people using a single job posting (LMIC 2020). Online job postings also suffer from a lack of historical data, with the Vicinity Jobs dataset only starting from 2018 Type of Data (stock vs flow)

11. Flow vs Stock Variable Convert VJ flow variable to stock variable Research from the Josh Bersin Company 2023 Where d represents a day, and q is specified as the quarter of the JVWS survey and is equal to the 44 days preceding the end of the quarter.

12. Online Job Postings vs Survey Data Figure 4: Total Stock of Vacancies Comparison of VJ and JVWS

13. Granger Causality - Wald Test The null hypothesis ( H ): VJ t does not Granger cause JVWS t : The alternative hypothesis ( H 1 ): VJ t Granger causes JVWS t :

14. Granger Causality Test Results Granger Causality Test Model 1: JVWS ∼ Lags(JVWS, 1:2) + Lags(VJ, 1:2) Model 2: JVWS ∼ Lags(JVWS, 1:2) Res. Df Df F Pr (>F) 1 6605 2 6607 -2 29.514 1.743e-13 Table 1: Granger test results

15. Machine Learning Forecasts Figure 5: Comparison of model predictions

16. Goals Improve the bias in online job postings towards certain sectors Improve forecast-ability of online job postings data for job vacancies from the survey

17. Algorithms & Results

18. Weighting Function

19. Vacancy Stocks – Weighting Function Figure 6: Total Stock of Vacancies

20. Weighting Function - Errors Figure 7: Average Errors by Quarter

21. Winsorization Algorithm The first step would be to split the dataset by job posting sources, and identify sources j = 1, …, J with a certain cut off, k , for postings per source. Then we compute the percentage change between q − 1 and q for the online job postings per source:

22. Winsorization Algorithm 1 2 3 1 2

23. Winsorization Algorithm Figure 8: Average Source Errors by Quarter

24. Robust Regression with Huber Weights

25. Robust Regression with Huber Weights Figure 9: Average Errors by Quarter – Winsorized and Robust

26. Combined Approach The goal is to solve the bias / representation issue, while also improving forecast ability. Combine weights function with winsorization . Combine weights function with robust regression.

27. Combined Approach Figure 10: Comparison of combined algorithm errors with weighting function errors, and original errors.

28. Combined Approach Figure 11: Performance of Machine Learning Models on combined algorithm data.

29. Conclusion

30. Conclusion Vacancy data has important applications in Economics The conventional source of data has documented flaws, mainly a reporting lag The alternative being used, online job postings, has its own limitations and drawbacks

31. Conclusion Develop a new enhanced methodology to improve the forecast-ability of online job postings. The robust regression method has not been used in this context before. Develop a combined approach (robust method that is also weighted by survey). Using machine learning models to forecast survey data. Applying this analysis on Canadian labour market data. Our main contribution

32. Questions?

33. Appendix

34. NOC Version 2016 Figure 5: NOC 2016 Version 1.3 Data source: NOC, Government of Canada Back to VJ Data

35. JVWS Broad NOC Distribution Figure 13: Distribution of Broad Occupational Categories in Survey Data. Breakdown of NOC categories in → NOC 2016 Table → Back to VJ Data