Validating Group Results to increase data quality and understanding

Now one of the things that many associations are going to run into is errors or differences in results when compiling benchmarking results. The larger a reference group is the higher the chances of there being anomalies in the results for each group. Often times it is directly related to the questionnaires which weren’t filled out responsibly. So, associations will need to go back and validate that differences nge in data.

Validating data is time-consuming but very important. If data isn’t validated correctly, it can render all that effort put into benchmarking pointless. So, every step you take in this regard has to be done correctly.

Tracing and Dealing with Invalid Values

Generally speaking, if you often see strange values associated with certain questions, it is imperative to go back and check individual data for mistakes. Sometimes the syntaxes of the calculations may be incorrect and have to be fixed. Take for instance a very common mistake for price per number of man-hours. If the value is ‘0’ that’s obviously incorrect. So, you’ll need to trace back that erroneous data to the individual participant and fix it. That may require you to call the participant to get the right information. While this should be part of your validation step, yet many times these types of errors may escape pass people who are checking the questionnaires.

Tip: When drafting questionnaires building a failsafe into them can help save a lot of time. A failsafe prevents the entry of invalid values. For instance, some questions should have a answer between certain minimum and maximum values.

Now you also need to apply that same approach to weighted averages. You can only use weighted averages with KIPs that have a denominator and nominator. To get a weighted average, you take the sum of both the nominator and denominator values and then divide them. Though the figures will be calculated for the same group of participants. That means you should have the same number of participants for both the denominator and nominator. Then check the results and fix it.

Creating Panels

Over the years we’ve seen that one of the solutions of dealing with differing participants for each reference group is to create panels for the participants for each of the reference groups which covers all periods. Now that may sound a little complex but let us unpack this idea. As we go along, I’m sure you’ll begin to understand the importance of panels.

Panels can be created for two or more periods for analysis. However, you will need to create a different panel for each timeframe.

Tip: Panels are easily created in a sophisticated Benchmark Tool. The Tool can easily calculate which participants have participated in every period the panel covers. This ensures that you compare like for like

When you are using a panel for the complete group, there is the possibility that a participant or more may have shifted to another group. It could also have become either bigger or smaller or perhaps moved to another region etc. So, you have three options the first being to accept the difference in the sample, the second being to create panels for each reference group or you can define the latest or the first reference group which is assigned to every period.

Sometimes a participant may have answered a question in the first period but not in the second one. So, even though the total number of participants who have answered a questionnaire for a certain period may be the same, the number of members who responded to each question could differ.

If the sample number of participants keeps fluctuating, then the method could end up resulting in too few participants included in the panel covering a longer time. So, steps will have to be taken in order to even this out to make the panels more useful.

Tables with Relevant Reporting Values

Once all the group data has been checked, the next step in the process is to create what we like to call a book of tables. This so-called book includes all the relevant reporting values drawn from all the relevant reference groups of those variables. The values which are needed in that book of tables may differ based on a number of variables. Take for instance that in the case of a qualitative variable you will require the number of responses and the associated structure. For KPIs, you will need quartiles, and mean, as well as the number of responses.

A book of tables will be required for an effective analysis of results. Period development analysis will require the results of the panels from varying periods. That way things are a lot easier and more reliable when you get down to analysing overall group development.

Tip: You will always want to make sure that the values in the book of tables are correct. After all, no association wants to make assertions based on incorrect values. You will need to double check all formulas and then use a benchmarking tool to create that book of tables.

Conclusion

The entire process of validation and conversion is part of creating an overview of each segment of the industry participating in the benchmarking. Once you have run through these steps and created a book of tables with all the correct values the next step is analysis. The analysis step is a crucial part of running and making sense of benchmarking data which is something we will discuss in our next post.