Validating Group Results to increase data quality and understanding

Now one of the things that many associations are going to run into is errors or differences in results when compiling benchmarking results. The larger a reference group is the higher the chances of there being anomalies in the results for each group. Often times it is directly related to the questionnaires which weren’t filled out responsibly. So, associations will need to go back and validate that change in data.

Validating data is time-consuming but very important. If data isn’t validated correctly, it can render all that effort put into benchmarking pointless. So, every step you take in this regard has to be done correctly.

Tracing and Dealing with Invalid Values

Generally speaking, if you often see strange values associated with certain questions, it is imperative to go back and check individual data for mistakes. Some times the syntaxes of the calculations may be incorrect and have to be fixed. Take for instance a very common mistake for price per number of man-hours. If the value is ‘0’ that’s obviously incorrect. So, you’ll need to trace back that erroneous data to the individual participant and fix it. That may require you to call the participant to get the right information. While this should be part of your validation step, yet many times these types of errors may escape pass people who are checking the questionnaires.

Tip: When drafting questionnaires building a fail safe into them can help save a lot of time. A fail safe prevents the entry of invalid values. For instance, some questions can be listed as ‘yes’ or ‘no’.

Now you also need to apply that same approach to weighted averages. You can only use weighted averages with KIPs that have a denominator and nominator. To get a weighted average, you take the sum of both the nominator and denominator values and then divide them. Though the figures will be calculated for the same group of participants. That means you should have the same number of participants for both the denominator and nominator. Then check the results and fix it.

Creating Panels

Over the years we’ve seen that one of the solutions of dealing with differing participants for each reference group is to create panels for the participants for each of the reference groups which covers all periods. Now that may sound a little complex but let us unpack this idea. As we go along, I’m sure you’ll begin to understand the importance of panels.

Panels can be created for two or more periods for analysis. However, you will need to create a different panel for each timeframe.

Tip: Panels are easily created in a spreadsheet program like excel. You can create a template and then reuse it multiple times for each period. Though always double check the values and ensure that the formulas are correctly used throughout the spreadsheet.

When you are using a panel for the complete group, there is the possibility that a participant or more may have shifted to another group. It could also have become either bigger or smaller or perhaps moved to another region etc. So, you have three options the first being to accept the difference in the sample, the second being to create panels for each reference group or you can define the latest or the first reference group which is assigned to every period.

Some times a participant may have answered a question in the first period but not in the second one. So, even though the total number of participants who have answered a questionnaire for a certain period may be the same, the number of members who responded to each question could differ.

If the sample number of participants keep fluctuating, then the method could end up resulting in too few participants entered in the panel. So, steps will have to be taken in order to even this out to make the panels more useful.

Tables with Relevant Reporting Values

Once all the group data has been checked, the next step in the process is to create what we like to call a book of tables. This so-called book includes all the relevant reporting values drawn from all the relevant reference groups of those variables. The values which are needed in that book of tables may differ based on a number of variables. Take for instance that in the case of a qualitative variable you will require the number of responses and the associated structure. For KPIs, you will need quartiles, and mean, as well as the number of responses.

A book of tables will be required for the analysis step of our Compare Method. Period development analysis will require the results of the panels from varying periods. That way things are a lot easier and more reliable when you get down to analyzing overall group development.

Tip: You will always want to make sure that the values in the book of tables are correct. After all, no association wants to make assertions based on incorrect values. You will need to double check all formulas then use a benchmarking tool to create that book of tables.

Conclusion

The entire process of validation and conversion is part of creating an overview of each segment of the industry participating in the benchmarking. Once you have run through these steps and created a book of tables with all the correct values the next step is analysis. The analysis step is a crucial part of running and making sense of benchmarking data which is something we will discuss in our next post.

Learn more

Other articles you might like

How Benchmarking Can Help Business Associations Make Money and Attract Member

It is interesting to note that many of the reasons why businesses join an association correlate with reasons to benchmark...

The Major Don’ts of Benchmarking by Associations

Knowing how to use benchmarking for performance improvement does not guarantee that businesses will use it...

The Importance of Training and Business Advise in Benchmarking

While benchmarking is an essential part of growing businesses within an industry by measuring their progress the key to successfully...

We supply the tools

Our intelligente intelligent online do-it-yourself Benchmarking Survey Tool has been created to make you successful in benchmarking. We support you with usefull gadgets in all parts of the process managing your benchmark.

  • Online inquiry with realtime calculations
  • Analysis and validation tools embedded into the same system
  • Online reporting through dashboards
  • Offline reporting through generated pdf reports
 
×