Increasingly, job seekers need to pass a series of tests in the form of artificial-intelligence games just to be seen by a hiring manager. In this third of a four-part miniseries on AI and hiring, we speak to someone who helped create these tests, and we ask who might get left behind in the process and why there isn’t more policy in place. We also try out some of these tools ourselves.
This miniseries on hiring was reported by Hilke Schellmann and produced by Jennifer Strong, Emma Cillekens, Anthony Green, and Karen Hao. We’re edited by Michael Reilly.
Jennifer: Often in life … you have to “play the metaphorical game”… to get the win you might be chasing.
(sounds from Matthew Kirkwold’s cover of XTC’s Complicated Game, “And it’s always been the same.. It’s just a complicated game.. Gh – ah.. Game..”
Jennifer: But what if that game… was literal?
And what if winning at it could mean the difference between landing a job you’ve been dreaming of… or not.
Increasingly job seekers need to pass a series of “tests” in the form of artificial-intelligence games… just to be seen by a hiring manager.
Anonymous job seeker: For me, being a military veteran being able to take tests and quizzes or being under pressure is nothing for me, but I don’t know why the cognitive tests gave me anxiety, but I think it’s because I knew that it had nothing to do with software engineering that’s what really got me.
Jennifer: We met this job seeker in the first episode of this series …
She asked us to call her Sally because she’s criticizing the hiring methods of potential employers and she’s concerned about publishing her real name.
She has a graduate degree in information from Rutgers University in New Jersey, with specialties in data science and interaction design.
And Sally fails to see how solving a timed puzzle… or playing video games like Tetris… have any real bearing on her potential to succeed in her field.
Anonymous job seeker: And I’m just like, what? I don’t understand. This is not relevant. So companies want to do diversity and inclusion, but you’re not doing diversity and inclusion when it comes to thinking, not everyone thinks the same. So how are you inputting that diversity and inclusion when you’re only selecting the people that can figure out a puzzle within 60 seconds.
Jennifer: She says she’s tried everything to succeed at games like the ones from Cognify she described… but without success.
She was rejected from multiple jobs she applied to that required these games.
Anonymous job seeker: I took their practice exams. I was practicing stuff on YouTube. I was using other peers and we were competing against each other. So I was like, all right, it’s not me because I studied for this. And I still did not quote unquote pass so…
Jennifer: I’m Jennifer Strong and in this third episode of our series on AI and hiring… we look at the role of games in the hiring process…
We meet some of the lead creators and distributors of these tools… and we share with them some feedback on their products from people like Sally.
Matthew Neale: The disconnect I think for this candidate was between what the assessment was getting the candidate to do and, and what was required or the perceptions about what was required on the job.
Jennifer: Matthew Neale helped create the Cognify tests she’s talking about.
Matthew Neale: I think the intention behind Cognify is to look at people’s ability to learn, to process information, to solve problems. You know, I would say, I suppose that these kinds of skills are relevant in, in software design, particularly in, in software design where you’re going to be presented with complex difficult or unusual problems. And that’s the connection that I would draw between the assessment and the role.
Jennifer: So we tested some of these tools ourselves…
And we ask who might get left behind in the process… Plus, we find out why there isn’t more policy in place … and speak with one of the leading US regulators.
Keith Sonderling: There has been no guidelines. There’s been nothing specific to the use of artificial intelligence, whether it is resume screening, whether it’s targeting job ads or facial recognition or voice recognition, there has been no new guidelines from the EEOC since the technology has been created.
Frida Polli: So I’m Frida Polli. I’m a former academic scientist. I spent 10 years at Harvard and MIT and I am the current CEO of a company called Pymetrics.
Jennifer: It’s an AI-games company that uses behavioral science and machine learning to help decide whether people are the right fit for a given job.
Frida Polli: I was a scientist who really loved the research I was doing. I was, at some point, frustrated by the fact that it wasn’t there wasn’t a lot of applications, real-world applications. So I went to business school looking for a problem, essentially, that our science could help solve.
Jennifer: When I spoke to her earlier this year she called this her fundamental “aha” moment in the path to creating her company.
Frida Polli: Essentially, people were trying to glean cognitive, social, and emotional aptitudes, or what we call soft skills, from a person’s résumé, which didn’t seem like the optimal thing to do. If you’re trying to understand somebody more holistically, you can use newer behavioral science tools to do that. So ultimately just had this light bulb go off, thinking, okay, we know how to measure soft skills. We know how to measure the things that recruiters and candidates are looking to understand about themselves in a much more scientific objective way. We don’t have to tea-leaf-read off a résumé .
Jennifer: The reason companies score job seekers is because they get way too many applications for open roles.
Frida Polli: You know if we could all wave our magic wand and not have to score people and magically distribute opportunity. I mean, my God, I’m all in all in. Right? And what we can do, I think, is make these systems as fair and predictive as possible, which was always kind of the goal.
Jennifer: She says Pymetrics does this using cognitive research… and they don’t measure hard skills…like whether someone can code… or use a spreadsheet.
Frida Polli: The fundamental premise is that we all sort of have certain predispositions and they’ll lead us to be more versus less successful. There’s been a lot of research showing that, you know, different cognitive, social and emotional, or personality attributes do make people particularly well suited for, you know, role A and less well suited for role B. I mean, that research has, you know, predates Pymetrics and all we’ve done is essentially make the measurement of those things less reliant on self-report questionnaires and more reliant on actually measuring your behavior.
Jennifer: These games measure nine specific soft skills including attention and risk preference… which she says are important in certain jobs.
Frida Polli: It’s not super deterministic. It can change over time. But it’s a broad brush stroke of like, Hey, you know, if you tend to be like, let’s take me for a second, I tend to be somewhat impulsive, right. That would make me well disposed for certain roles, but potentially not others. So I guess what I would say is that both hard and soft skills are important for success in any particular role and the particular mix… it really depends on the role at hand, right?
Jennifer: Basically it works like this. — Employees who’ve been successful in a particular job a company is hiring for… are asked to play these games. That data gets compared against people already in a Pymetrics database… The idea is to build a model that identifies and ranks the skills unique to this group of employees… and to remove bias.
Jennifer: All of that gets compared against incoming job applicants… And it’s used by large global companies including KraftHeinz and AstraZeneca.
Another big player in this field is a company called Arctic Shores. Their games are used by the financial industry… and by large companies mainly in Europe.
Safe Hammad: The way we recruit was and in many cases is broken.
Jennifer: Safe Hammad is a cofounder and CTO of Arctic Shores.
Safe Hammad: But companies are recognizing that actually they can do better. They can do better on the predictability front to improve the bottom line for the companies. And also they can do better on the bias front as well. It’s a win-win situation: by removing the bias, you get more suitable people, the right people, in your company. I mean, what’s not to like?
Jennifer: The same as Pymetrics, Arctic Shores teases out personality traits via AI-based “video games.”
Safe Hammad: So, the way we measure something like sociability isn’t “Oh, you’re in a room and you want to go and talk to someone,” or, you know, actually, you really wouldn’t realize, you know, there’s a few tasks where we ask you to choose left, choose right, and press you a little bit. And we come out with a measure of sociability. I mean, for me, it’s magic. I mean, I understand the science a little bit underneath. I certainly understand the mathematics, but it’s like magic. We actually don’t put you in a scenario. That’s anything to do with sociability. And yet, if you look at the stats, the measurements are great.
Jennifer: He says the company’s tools are better than traditional testing methods, because the games can pull out traits of job applicants that might otherwise be hard to figure out. Like whether they’re innovative thinkers… something most candidates would probably just answer with yes if they were asked in a job interview.
Safe Hammad: When you ask questions, they can be faked. When I ask a question about, you know, how would you react if you’re in this position? You’re not thinking, oh, how would I react? You’re thinking, oh, what does the person asking me want me to say, that’s going to give me the best chance of getting that job. So without asking questions, by not asking questions, it’s a lot harder to fake and it’s a lot less subjective.
Jennifer: And to him, more data equals more objective hiring decisions.
Safe Hammad: It’s about seeing more in people before you bring in some of this information which can lead to bias, so long as in the first stage of the process, you’re blind to their name, the school they went to, what degree they got, and you just look at the psychometrics, what is their potential for the role? That’s the first thing we need to answer.
Jennifer: He says games give everyone a fair chance to succeed regardless of their background. Both Pymetrics and Arctic Shores say their tools are built on well-established research and testing.
Safe Hammad: Now, these interactive tasks are very carefully crafted based on the scientific literature based on a lot of research and a lot of sweat that’s gone into making these, we actually can capture thousands of data points and, and a lot of those are very finely nuanced. And by using all that data, we’re able to really try and hone in on some of the behaviors that will match you to those roles.
Jennifer: And he says explainability of results is key in building trust in these new technologies.
Safe Hammad: So we do use AI, we do use machine learning to try and inform us to help us build that model. But the final model itself is more akin to what you would find a traditional psychometrics. It means that when it comes to our results, we can actually give you the results. We can tell you where you lie on the low to medium to high scale for creativity, for resilience, for learning agility. And we will stand by that.
Jennifer: And he says his company is also closely monitoring if the use of AI games — leads to better hiring decisions.
Safe Hammad: We’ll always be looking at the results, you know, has the outcome actually reflected what we said would happen. Are you getting better hires? Are they actually fulfilling your requirements? And, and better doesn’t necessarily mean, Hey, I’m more productive. Better can mean that they’re more likely to stay in the role for a year.
Jennifer: But not everyone is feeling so optimistic. Hilke Schellmann is a professor of journalism at NYU who covers the use of AI in hiring… She’s been playing a whole lot of these games… as well as talking to some critics. She’s here to help put it all into context.
Hilke: Yeah Jen.. AI-based video games are a recent phenomenon of the last decade or so. It’s a way for job applicants to get tested for a job in a more “fun” way…(well… that’s at least how vendors and employers pitch it), since they are playing “video games” instead of answering lots of questions in a personality test for example.
Jennifer: So, what types of games are you playing?
Hilke: So…. And I played a lot of these games. For one company, I had to essentially play a game of tetris and put different blocks in the right order. I also had to solve basic mathematical problems and test my language skills by finding grammar and spelling mistakes in an email.
Jennifer: But these aren’t like the ones you might play at home. There’s no sound in these games… and they look more like games from the 1980’s or early 90’s… and something that surprised me was all the biggest companies in this space seem to really be into games about putting air in balloons… What’s that about?
Hilke: Well.. The balloon game apparently measures your appetite for risk. When I played the game… I figured out pretty early on that yellow and red balloons pop after fewer pumps than the blue balloons, so I was able to push my luck with blue balloons and bank more money. But while I was playing I was also wondering if this game really measures my risk taking preferences in real life or if this only measures my appetite for risk in a video game.
Jennifer: Yeah, I could be a real daredevil when playing a game, but totally risk averse in real life.
Hilke: Exactly… And.. that’s one concern about AI-games. Another one is whether these games are actually relevant to a given job.
Jennifer: Ok, so help me understand then why companies are interested in using these games in the first place…
Hilke: So.. Jen From what I’ve learned these AI-games are most often used for entry level positions. So they are really popular with companies who hire recent college graduates. At that point, most job applicants don’t have a ton of work experience under their belts and personality traits start to play a larger role in finding the right people for the job. And oftentimes.. traits like agility or learning capabilities are becoming more important for employers.
Jennifer: Right… And companies are more likely to need to change up the way they do business now… so it means some skills wind up with a shorter shelf life.
Hilke: Yeah.. so in the past it may have been enough to hire a software developer with python skills, because that’s what a company needed for years to come. But these days, who knows how long a specific programming language is relevant in the workplace. Companies want to hire people who can re-train themselves and are not put off by change. And… these AI-games are supposed to help them find the right people.
Jennifer: So, that’s the sales pitch from the vendors. But Walmart.. (one of the biggest employers in the U-S)… they shared some of their findings with these technologies in a recent episode of an industry podcast called Science 4-Hire.
David Futrell: There’s no doubt that what we run is the biggest selection and assessment machine that’s ever existed on the planet. So, I mean, we test, everyday, between ten and fifteen thousand people and that’s just entry level hires for the stores.
Jennifer: That’s David Futrell. He is the senior director of organizational performance at Walmart.
David Futrell: When this machine learning idea first came out, I was very excited by it because, you know, it seemed to me like it would solve all of the problems that we had with prediction. And so we really got into it and did a lot of work with trying to build predictors using these machine based algorithms. And they work. But whenever it’s all said and done, they don’t really work any better than, you know, doing it the old fashioned way.
Jennifer: And he told the host of that podcast that Walmart acquired a company that was using a pure games based approach…
David Futrell: And uh we found that it just didn’t work well at all. I won’t mention the company, but it’s not the big company that it’s in this space. But they were purported to measure some underlying aspects of personality, like your willingness to take risks and so on.
Jennifer: And a concern with these and other AI hiring tools (that goes beyond whether they work better than what they’re replacing… is whether they work equally on different groups of people…
[sound of mouse clicking]
…including those with disabilities.
Henry Claypool: I’m just logging in now.
Jennifer: Henry Claypool is a disability policy analyst… and we asked him to play some of these games with us.
Henry Claypool: Okay, here we go…complete games.
Jennifer: He’s looking at one of the opening screens on a Pymetrics’ game. It asks players to select if they want to play a version that’s modified for color blindness, ADHD or dyslexia… or if they’d rather play a non-modified version.
Henry Claypool: Seems like that alone would be a legal challenge here.
Jennifer: He thinks it might even violate the Americans with Disabilities Act…. or A-D-A.
Henry Claypool: This is a pre-employment disclosure of disability, which could be used to discriminate against you. And so you’re putting the applicant on the horns of a dilemma, right — do I choose to disclose and seek an accommodation or do I just push through? The thing is you’re not allowed to ask an applicant about their disability before you make a job offer.
Jennifer: Claypool himself has a spinal cord injury from a skiing accident during his college years… It left him without the use of his legs and an arm.
But this hasn’t held back his career… He worked in the Obama administration and he helps companies with their disability policies.
Henry Claypool: The fear is that if I click one of these, I’ll disclose something that will disqualify me for the job, and if I don’t click on say dyslexia or whatever it is that I’ll be at a disadvantage to other people that read and process information more quickly. Therefore I’m going to fail either way or either way now my anxiety is heightened because I know I’m probably at a disadvantage.
Jennifer: In other words, he’s afraid if he discloses a disability this soon in the process… it might prevent him from getting an interview.
Henry Claypool: Ooops… am I… oh I’m pumping by the mouse.
Jennifer: Pymetrics’ suite of games starts with one where people get money to pump up balloons… and they have to bank it before a balloon pops.
Henry Claypool: Okay. Now carpal tunnel is setting in..
Jennifer: A few minutes into the game it’s getting harder for him.
Henry Claypool: I really hate that game. I just, I don’t see any logic in there at all. Knowing that I’m being tested by something that doesn’t want me to understand what it’s testing for makes me try to think through what it’s anticipating.
Jennifer: In other words, he has a dialogue going in his head trying to figure out what the system might want from him. And that distracts him from playing… such that he’s afraid he might not be doing so well.
And the idea that he and his peers have to play these games to get a job… doesn’t sit right with him.
He believes it’ll be harder for those with disabilities to get hired if that personal interaction early on in the process is taken away.
Henry Claypool: It’s really, it’s too bad that we’ve los
A second wave fuelled by the spread of the Delta variant is seeing Indonesia's COVID-19…
Newcastle United defender Florian Lejeune completes a move to La Liga side Alaves on permanent…
On Thursday afternoon a Newcastle United Official Announcement has confirmed that Florian Lejeune has now…
I don’t blame Joelinton. He is what he is. A very average player who generally…
Newcastle United are maintaining total silence on Martin Dubravka, as is the case with pretty…