Job analysis is the scientific study of jobs to determine the knowledge, skills, and abilities (KSAs), among other things, one needs in order to successfully perform that job.
It is the foundation of most HR programs and is therefore critical to the success of high-performing human capital operations. Using KSAs, we can understand what to test and select for when predicting who will be successful in a given role.
The Problem
At Wonderlic, a major source of pain for our clients is the difficulty they have in trying to choose the correct testing profile for the job they are hiring for. Our clients are not experts in the assessment space and look to us for guidance. Therefore, if we could provide an automated ability to take in their job description and return a recommended testing profile based on the KSAs identified in it, that would provide tremendous value.
A single job description (JD) is typically too deficient to identify the KSAs needed in a job. However, thousands of JDs for a single job should contain enough signal to identify the totality of the KSAs required in that role. Therefore, we proposed collecting a large library of JDs, tagging each with its relevant KSAs, and then training supervised learning models to predict which KSAs are required for a job described by any given JD. The resulting tool would use the identified KSAs to return prioritized testing profiles for our customers.
Project Development
To get us started, we acquired a data set of over one million job descriptions from listings on Monster.com. After examining the data, it became clear that to avoid low-quality job descriptions, the best thing we could do was filter down to only include full-time job postings. Job postings that were not full-time were generally of lower quality, often spam not truly advertising a job. Filtering these out, we were left with about 250,000 job descriptions.
Next, we noticed a number of jobs were posted multiple times with only slight differences between them. In some cases, a certain job was posted a couple thousand times with very minor changes between each post (i.e. UPS warehouse worker). This created a critical problem. The number of nearly-duplicate JDs was so large that it was likely to make our predictions more difficult. The frequency of the features that would be extracted from these JDs is not representative of the entire real-world population of JDs that exist, specifically, within the Wonscore platform (i.e. Wonscore customers are not setting up testing profiles over and over again for the exact same job with very slight differences). Therefore, the fastest and easiest option for dealing with the duplicates was to remove them because we would still be left with a sufficiently large dataset of about 50,000 job postings.
Criterion Development
We decided to connect our AI-powered job analysis tool to a previously established and widely accepted KSA taxonomy. This was critical to the success of the project since so much of what job analyses are used for, such as talent assessment, performance management, and compensation setting, is subject to intense legal scrutiny. The set of KSAs we landed on came from O*NET, which has conducted rigorous job analyses of over 1,000 different roles, broadly determining which KSAs are required for which jobs.
While we wanted to tag all the job postings with the KSAs required based on O*NET, there was no easy way to automate this mapping, and doing it by hand was not scalable. Therefore, we used a pre-trained model to identify which job profiles from O*NET were the closest matches for each job description based on the associated job title. We used these matches and their connected KSA profiles to determine the final mapping of KSAs to each JD.
Feature Engineering & Model Building
After cleaning the text data and extracting several descriptive data points, we developed a TF-IDF (term frequency-inverse document frequency) model for use in analysis. TF-IDF is used to help the model learn patterns from the words that are most important in each JD. We also leveraged an unsupervised approach—topic modeling, using an implementation of LDA (latent Dirichlet allocation)—to add additional features for our model to learn from. Topic modeling is used to identify groups of words that together make up a coherent “topic”. For our purposes, the prevalence of these topics across the text body of each JD were used as additional features for training our classifiers.
At this point we were ready to begin testing the performance of different classification models, ultimately landing on the XGBClassifier from the XGBoost library. We used precision, recall, F1 score, and Matthews correlation coefficient (MCC) to measure the performance of our models. After evaluating all the models, we noticed that a number of KSAs were being predicted with high precision but very low recall. When diving into the predictions to troubleshoot this suboptimal performance, it seemed that the models were more likely to return the KSAs that were present in almost all jobs, such as “Communication”, and avoiding assigning KSAs that were more specialized, such as “Programming”.
At the root of this challenge was the fact that many KSAs either were in almost all jobs or were in only a small subset of jobs. Imbalanced data like this is a common challenge in classification problems. To solve for this, we used a technique called SMOTEENN (Synthetic Minority Over Sampling Technique with Edited Nearest Neighbors), which is an advanced method for creating synthetic data to balance the frequency of the occurrence of the target classes. This creates a more defined decision boundary, which makes it easier for the model to identify a classification pattern in the data. When we implemented SMOTEENN prior to training XGBClassifier models, the recall scores improved significantly without much of a reduction in the precision scores. Additionally, the MCC scores saw dramatic improvement.
At this point, we felt confident that we had a production-ready version of the AI-powered job analysis tool and were ready to start preparing it for deployment and a follow-up validation study with our I/O psychology team. Try it for yourself by checking out the AI-powered job analysis interactive demo.
