The Variability of Resumes In The Hiring Process – Irrational Behavior

This spring I took an online course from Coursera, Irrational Behavior, as taught by Dr. Dan Ariely. It was a fascinating course, one that I enjoyed immensely.

Irrational Behavior with Dr. Dan Ariely on Coursera

Irrational Behavior with Dr. Dan Ariely on Coursera

See my comments on the course from my previous post on Irrational Behavior and Hoarding.

The other writing assignment in the course, was to “Design An Experiment”.  In this course, “experiment” refers to a psychological experiment.  Not a physics, chemistry, or computer science experiment.

From the guidelines – “Design An Experiment” – Please include the following in your paper:
A brief summary of previous (relevant) research. Be sure to use and cite at least one piece of research or theory that your experiment builds on.
Research question. What do you want to find out?
Your hypothesis. What do you think you will find?
A brief overview of the proposed design including identification of your independent and dependent variables.  For more details, check out this guide.
Implications. If your study turns out the way you expect it to, what does that mean?


I’ve often thought of the shortcomings of resumes. As so, in many ways, using resumes is rather irrational.  So, I designed an experiment to test the variability of resumes.

For those of you who are not familiar with statistical variability and standard deviation, please read what Wikipedia says on the subject.


With over 100K students signed up for the course, it’s too much effort for the professor, or teacher’s assistants to do the grading. Instead the grading is peer reviewed. All students are given the rubric, and you grade the papers of others, and get points for doing the peer reviews. I found it really interesting to see the essays of the others.

Writing has often been a weak point of mine. But just as in the previous assignment, I ended up getting perfect marks from every peer reviewer! As you can imagine, I was very pleased with this.


Notes On Plagiarizing:

The essay follows. For those of you who are taking online courses, you should not need to be told not to plagiarize this work. For one thing, it violates the Coursera honor code. For another reason, the big data capabilities of Coursera and other companies will most certainly discover your copying soon enough, and you will fail your project. That will be most counterproductive.




The Variability of Resumes In The Hiring Process

In the principles of Dr. Deming’s Statistical Quality Control (Deming, 1982), if a a tool or process  is to produce high quality output, must be very consistent and have very little variability in the output.

However, in the typical US hiring process, resumes (or curriculum vitaes) produce massive variability in the results. Even within the same work group, one resume can be accepted by one person, but rejected by others. At the same time, those in different roles such as recruiters, human resources, and the hiring manager will each have their own criteria for evaluation, which adds to the variability.

In the end, sometimes perfectly qualified people are rejected, not even getting an interview. Or, the wrong people get hired. Some will be later released in a few weeks or months, whereas others even might remain for years. These can be compared to Type 1 and Type 2 errors.

While there is not a lot of formal research to be found on resume variability, there is research on the shortcomings of the interview process. In the study by Dawes (1979), 50 applicants were rejected from medical school, based on their interview. Later however, when 50 new spots became available, they were admitted to medical school. The subsequent results showed that they did just as well as everyone else in all aspects of their training.

The studies by Dana, Dawesy, and Peterson (2013) note that “Unstructured interviews are a ubiquitous tool for making screening decisions despite a vast literature suggesting that they have little validity.”, and their “simple recommendation for those making screening decisions is not to use them”.

Step #2: Establish a clear research question

How much variability or consistency is found in the various screening tools used in hiring process: resumes, written test, or samples of one’s work?

Step #3: State the hypothesis to be tested

I hypothesize that in the hiring process, the use of resumes results in a lot of variability. At the same time, written tests, and samples of one’s work produce more accuracy, consistency, and less variability.

Step #4: Design your experiment

In this between-group experiment, at least 30 participants are randomly assigned to three different groups, ten or more per group. Each group is given the same job description, and the qualifications of ten or more applicants. The participants’ task is to then choose the three best candidates for the position, and rank them (1, 2, 3).

Each group reviews the qualifications for the same ten applicants, each group having a different screening format. Group 1 reviews self written resumes of the candidates. Group 2 reviews the results of short answer tests that the candidates typed into electronic forms (to eliminate screening based on handwriting). Group 3 reviews work samples of the candidates (their portfolio).

The results from each group is compared internally, and the variability is measured. In each group, was there consistency, an overlap of candidates selected by the ten different reviewers, or not?

The results and variability between the groups (resume, test, or portfolio) are also measured. Did the different methods result in the same candidates being selected, or not?

The results are also compared to pure random sampling.

Because the two conditions are independent samples that receive different treatments, the experiment is between-groups. There is no control group.

Independent variable: Format of candidates’ qualifications (resume, test results, portfolio).

Dependent variable: Variability of results within each method, and between methods.

Since the test is specifically designed to test variability, and variability is measured, I believe that the test has construct validity. Since the format of qualifications is manipulated, and the results are expected to vary according to the format, the test has internal validity. Since the tasks are already quite familiar tasks found in hiring or in school, I believe that the experiment has external validity.

Step #5: Implications

The implications of this study for business could be huge. If resumes produce highly variable and inaccurate results, then it is most irrational to bet your business on on them, and use resumes so much. Perhaps resumes should not be used at all in the hiring process. Completely new hiring processes would need to be developed.

As so many stretch the truth on their resume, by using other methods to determine suitability, such as written tests or work samples, where cheating is more difficult, then puffery can be reduced.

If better results can be attained using other methods, then the costs of firing employees that were regrettably hired would be reduced, both for the employer, and the employee. At the same time, the erroneous rejection of perfectly suitable candidates would also be reduced.



Deming, W. Edwards  (1982).  Quality, productivity, and competitive position. Cambridge, MA:  Massachusetts Institute of Technology, Center for Advanced Engineering Study.  ISBN 0911379002, PUB ID: 102-232-322


Heath D. & Heath C. (2009)
Why It May Be Wiser To Hire People Without Meeting Them. Retrieved from:

Refers to the study on interviews by Robyn Dawes, at the University of Texas Medical School, 1979,.


Dawes R (1979)
The robust beauty of improper linear models in decision making.
Source: American Psychologist, Vol 34(7), Jul 1979, 571-582. doi: 10.1037/0003-066X.34.7.571


Dana, Dawesy, Peterson (2013).  Belief in the unstructured interview: The persistence of an illusion.
Source: Judgment & Decision Making . Sep2013, Vol. 8 Issue 5, p512-520. 9p.
Retrieved from:


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: