Employee Engagement Blog

Compared to the Fastest Company in the Slow Lane

Comparing the results of your employee survey with normative data is often seen as being valuable. However, upon closer and more thoughtful examination, such comparisons at best do little more than make nice slide presentations. At worst, they lead to complacency and can be destructive to your organization’s ability to achieve its objectives. Essentially all larger companies today conduct employee surveys. Many conduct such surveys at regular intervals, such as annually or bi-annually. Others do so less frequently, or only when a specific event or environment precipitates the perceived need by management.

A significant percentage of these companies look to compare the results produced by their surveys with industry norms. Clearly, the thinking is that by creating such comparisons, they will be able to assess and demonstrate how their organization stacks up against others that compete within their respective industry.

The fact is, the comparison of your company’s survey data with the so-called normative data from your industry is at best a meaningless exercise, and at worst a destructive one. Not because you shouldn’t be comparing any of your company’s metrics to those of your competitors. Comparisons with objective data such as return on assets, market share, year-over-year growth, same store sales, and countless others can be invaluable to a company’s management.

Comparing normative survey response data should be avoided simply because normative data comparisons represent “junk science”.. First of all, industry norms are averages. And anytime you compare one number with an average, the comparison can be rife with problems.

Next, we must reconcile the fact that when comparing subjective data from two organizations that are in the same industry, may not even be appropriate. Consider two successful manufacturing companies. Company-A competes on product leadership and quality while Company-B competes on low cost. Company-A is an open culture that encourages complete candor. Company-B is a very closed culture where many employees feel quite fearful of speaking their mind. Employees in Company-A know that they can speak their mind while feeling completely confident that they will be rewarded for their candor. Employees in Company-B can also speak their mind but could easily limit their career. Now the big question- Guess which company has higher engagement scores? Yep, that's right, Company-B! Wait a minute! The company with the fear culture has higher engagement scores than a company that values open and honest communication? You know it's true! When people are afraid to speak their mind, they will tell you what they think you want to hear! So we are supposed to compare ourselves to the culture we are trying to avoid?

We can set that little problem aside for the moment. Next, let’s say we have the following percentages in our norm: 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%. The average or norm is 50%, and your company’s score being compared with the norm is 65%. That means that your company scored 15 points higher than the norm. Yea!

Of course, it also means your company scored 25 points behind the leader... maybe. The actual industry leader may not even be included in the sampling of companies whose results went into the normative data you’re using. In plain English... the survey company supplying you with your normative data may not have had the industry leader as a client. If in actuality, the true industry leader scored even higher, then your company’s score is even further behind the company with which all are trying to compete. But, instead of recognizing that your company needs to initiate efforts to improve its 60% score, everyone seeing the normative data comparison goes back to work happy to have beaten the industry norm by 15 points.

Now, let’s say that the 65% was related to “customer satisfaction.” Should the conclusion drawn after seeing how that score compared with the industry norm be happiness at being 15 points above the norm, or alarmed at the fact that 35% of your customers are less than satisfied? What if your company’s above the norm 65% actually represents a 10- point drop in customer satisfaction as compared with prior years? In that case, this hypothetical company whose executives become pleased with being better than the norm may actually be piloting a sinking ship and not realize it until too late.

Objectifying the Subjective

Another problematic issue with normative data comes from compiling the norm using survey data received over many years... that asks different questions... in different ways... using varying scales... at companies with widely differing cultures. Here’s why:

1. A given survey’s results can have enormous variations depending on when the survey was taken. Just imagine the results of a survey taken just prior to 9/11, and the results to one just after.
2. The questions asked or not asked can dramatically alter a survey’s results. Just imagine two surveys seeking to measure “customer satisfaction”. If one survey includes a question about satisfaction with the company’s pricing, but the other doesn’t, the two can’t later be accurately compiled as a “norm”.
3. How questions are asked also makes an enormous difference in survey outcomes. So, unless the questions in each survey used to compile the normative data were identical, the so-called “norm” can’t be representative of a true average response.
4. Just as the questions, and the way in which they are asked must be identical, the response scales must also be the same in each survey used to compile a representative “norm”. If one survey asked respondents to rank their response on a scale of 1–5, while another offered a different option, then averaging the two will not produce an accurate outcome.
5. Even if the surveys used to compile the norm were identical in terms of design, scale and time frame, which is rarely if ever the case!

The bottom-line is that all companies, and the survey instruments, are each individually unique, so the most meaningful comparisons of a company’s survey results are found when a company compares the results of this year’s survey with what employees said in previous years, or when a company establishes benchmarks against which progress toward a goal can be measured.

Managing to the Norms vs. Establishing Benchmarks

Obviously, and it might go without saying, when a company manages to what is purported to be the industry average or norm, at best it is measuring itself against the average, as opposed to the leaders. In this competitive age that in itself is a dangerous proposition. Average companies today often fail. Just ask the folks over at Mervyn's, Montgomery Ward, Circuit City, or innumerable others... if you can find out where they’re employed today, that is.

The use of well-crafted and properly executed employee surveys in today’s ultra-competitive, global marketplace can offer companies powerful and even invaluable insights that can mean the difference between success and failure. But, not by comparing their results to what someone is claiming to be the “industry norms”. All these comparisons can hope to produce are interesting Power Point presentations. At best, these comparisons are essentially meaningless, and at worst they end up detracting from a company’s ability to achieve its individual potential.

On the other hand, establishing measurable benchmarks against which a company’s relative performance and progress is assessed through the use of employee surveys is an excellent way to capitalize on what today’s leading employee survey technologies offer. Such benchmarks, of course, can be based on a company’s past performance, or they can simply be more arbitrarily set, such as basing them on management’s vision or long-term goals for the organization. Either way, however, the performance benchmarks established by management should always be set with an eye to realism or achievability. For example, establishing an “employee satisfaction” benchmark of 100% satisfaction can only remind one of the Queen of Hearts, in Lewis Carroll’s classic book, “Alice in Wonderland,” when she said: “I have often believed five or six impossible things before breakfast.”

Establishing benchmarks that can never be achieved doesn’t result in a more motivated workforce, to the contrary, it results in benchmarks that quickly become ignored out of hand.

The bottom-line is that all companies, and the survey instruments, are each individually unique, so the most meaningful comparisons of a company’s survey results are found when a company compares the results of this year’s survey with what employees said in previous years, or when a company establishes benchmarks against which progress toward a goal can be measured.

In Conclusion...

Today’s leading technologies can make the use of employee surveys more valuable than ever before. Clearly, the capability of today’s computers and systems represents an enormous leap forward when compared to the paper surveys and manually compiled results of years gone by. Today’s employee surveys can report results that are more specific, measurable, and actionable. Their use can provide a company’s management with insight and the ability to see important trends that would otherwise be unavailable.

In fact, even the use of normative data as one of the data points that go into creating realistic benchmarks can be valuable. It is not the “normative data” that is the problem. It is the over reliance on normative data, the lack of awareness about where the data came from and when it was collected. The best employee survey platforms improve management decision-making, increase the level of employee engagement, uncover new and invaluable knowledge, reduce the potential for employee litigation, facilitate cultural change, and much more. But, none of this occurs as the result of comparing survey response data to what someone claims to be the “industry average” or “norm.”

So, whenever you hear someone quote a statistic akin to: “The industry average is 66%.” Just remember that: “77% of statistics are made up on the spot.”

We welcome your views on the use of normative data.