Section 5: The Reliability and Validity of Survey Information (2024)

Note: This guide is intended to ensure that surveys conducted in the OAG meet reasonable requirements and expectations of survey professionals as well as the VFM audit standards of the Office of the Auditor General. The use of the terms "must" and "should" in this guidance document do not necessarily have the status of OAG standards and policies. However, they reflect methodological requirements and expectations in the conduct of surveys.

Introduction

The concepts of reliability and validity are core issues in determining the quality of survey information. Determining the reliability and validity of survey data is a highly technical undertaking that is key to assessing the adequacy (see VFM Manual, section on Sufficient Evidence) of the information for supporting audit observations.

Reliability, validity and significant error

In order for a survey to provide sufficiently sound, consistent, and relevant evidence, the information it provides must be both reliable and valid. The concept of reliability has a specific meaning within survey research that is narrower than the general use of the term in audit. To be reliable, measurement must be consistent from individual to individual surveyed, across settings and at different times. Consistency of information is essential for making general statements.

Validity is the extent to which the survey information is relevant to the conclusion being drawn and is sufficiently accurate and complete to support the conclusion. Validity is commonly thought of as the answer to the question "Are you actually measuring what you want to measure?" For example, employees may be asked whether promotion processes are "fair". Their answers may reflect not their view of the processes themselves, but with the results, i.e. whether or not they received a promotion.

Reliability requires the use of standardized information collection instruments and survey procedures [10, 11, 12 ] that are designed to enhance consistency. Relevant information requires careful planning to ensure that the information is clearly related to audit objectives, and is collected from the individuals best suited to providing the information. Obtaining information that is complete and accurate requires well planned information collection instruments and survey administration procedures.

Error in survey data can result in poor reliability and validity. Error, i.e. inaccuracies or missing information, can arise at various points in the collection, maintenance, processing, and reporting of data. At data collection, information can be recorded incorrectly. For example, in a structured interviewing situation, interviewers may incorrectly record information provided by a respondent. Inadequate procedures for maintaining data, such as on computer information systems, can result in the loss or alteration of data. For instance, data from client survey forms can be incorrectly entered into computer information systems. Processing and reporting of data can also add error; for example, through the incorrect transfer of data from computerized databases into analysable or reportable forms.

In addition to error being introduced through errors in survey data, error is also added in sampling. Summary data from a sample is only an approximation of what would be found by examining the entire population. The discrepancy between sampled values and population values is known as sampling error. This type of error can be larger or smaller depending upon sample size and sampling procedures. Sampling error can be statistically estimated when using probability based samples.

Error as a source of bias

No survey data are completely free from error. However, in order for survey information to be reliable and valid, the information needs to be free from significant error. Error is significant when it is of such a nature or magnitude that it would affect audit conclusions. Bias especially creates a risk of significant error. Bias is error that is caused by a systematic source. Because it is systematic in nature, bias it is likely to lead to incorrect conclusions. For example, in surveying small businesses receiving federal contributions or grants, those businesses that have failed might be less reachable than those that are successful. Because the sample would contain a lower proportion of failed businesses than is actually the case, the success of the contributions and businesses satisfaction with them could be overestimated.

In surveys, an important source of potential bias occurs when information cannot be obtained from some of the population members sampled. For this reason, the survey must implement procedures to reduce and compensate for non-response and estimate its impact.

Another important source of potential bias can occur when non-probability sampling procedures are used or when probability sampling is not properly administered. Sampling error of this type often cannot be estimated statistically.

Controlling error

Although error can never be completely eliminated, it can be reduced to acceptable levels through the careful design of standardized data collection instruments, the implementation of appropriate sample designs and sampling procedures, the implementation of adequate survey administration procedures, and the implementation of data verification and correction procedures, especially procedures for corroborating answers received in response to questionnaires or survey interviews. In particular, data collection instruments are required to be pretested in order to minimize flaws in their design.

Error is more difficult and expensive to control in some data than others. In planning the audit, the auditor must determine the potential for significant error and whether the value to be obtained from the survey data merits the cost and time required to reduce survey error to acceptable levels (see VFM Manual, section on Sufficient Evidence).