Statistical Significance
Paid feature
Log in to see if this feature is included in your plan.
You can view if there's a statistically significant difference between how certain response groups answered the questions in your survey. To use the statistical significance feature in SurveyMonkey:
- Turn on statistical significance while adding a Compare Rule to a question in your survey. Choose the groups you want to compare to break down your survey results by group in a side-by-side comparison.
- Examine the data tables for the questions in your survey to see if there are statistically significant differences in how different groups answered the survey.
Viewing Statistical Significance
The following steps will guide you through creating a survey that can display statistical significance.
Step 1: Add Closed-Ended Questions to a Survey
To display statistical significance when analyzing results, you'll need to apply a Compare Rule to a question in your survey.
You must use one of the following question types in your survey design to be able to apply a Compare Rule and calculate statistical significance:
- Multiple Choice
- Dropdown
- Matrix/Rating Scale
- Ranking
Make sure your answer options break down into meaningful groups. The answer options you choose to compare when creating the Compare Rule will be used to cross-tabulate your data across the rest of the survey.
Step 2: Collect Responses
After you finish designing your survey, create a collector to distribute it. There are several ways to send your survey.
You need at least 30 responses per answer option you plan to use in your Compare Rule in order to turn on and view statistical significance.
TIP! You can buy survey responses using SurveyMonkey Audience. Target specific attributes, like gender, age, income, employment status, and more to get answers from the people whose opinions you need.
To turn on statistical significance, you need to add a Compare Rule to a question in your survey to divide respondents into groups.
To apply a Compare Rule:
- Go to the Analyze Results section of your survey.
- Click +COMPARE in the Current View section of the left sidebar.
- Click Compare by Question and Answer, unless you used an A/B test in your survey.
- Use the drop-down menu to find the desired question.
- Select the answer options you'd like for your groups. These groups will be used to cross-tabulate your data across the rest of the survey.
- Click the toggle next to Show statistical significance to turn them on. At least 2 of the answer options need to have 30 responses or more in each group order to turn on statistical significance.
- Click Apply.
To edit your groups after you create the rule, click the down arrow to the right of the rule in the left sidebar and click Edit rule.
TIP! You can create multiple Compare Rules, but you can only apply or view one Compare Rule at a time.
Step 4: Examine Your Data Tables
After you apply the Compare Rule, you'll see how each group answered each question in a side-by-side view so you can easily spot similarities and differences.
We highlight answer options that are statistically significant in the data tables below question charts.
- Statistical significance is calculated for the following question types: Multiple Choice, Dropdown, Matrix/Rating Scale, and Ranking.
- Each row in the data table represents a response group. Each response group is also assigned a letter, which you can reference in the data table.
- Each column in the data table represents an answer choice.
Hover over any answer option to see additional information. You'll see one of the following messages:
Message Description Significantly lower than The group is significantly less likely to select the answer option compared to other highlighted groups. Significantly higher than The group is significantly more likely to select the answer option compared to other highlighted groups. More responses needed You need at least 30 response to calculate statistical significance for the group. Combined answer choices At this time, we can't calculate statistical significance for combined answer choices. Please uncombine them. Hidden answer choices At this time, we can't calculate statistical significance for hidden answer choices. Please unhide them. No significant difference The group selected this answer choice about as often as other groups. Scroll down the page to the section on "What is a statistically significant difference?" for more information.
Step 5: Share Your Results
Shared Data Pages
If you share your survey results online, people who view your shared data page can view statistical significance.
To create a shared data page that includes statistical significance:
- Under Current View in the left sidebar, click the Compare Rule you created with statistical significance.
- Under Saved Views in the left sidebar, click +Save as...
- Enter a name for the view and click Save.
- Click + Share All in the upper-right corner of the page.
- Finish setting up the shared data page.
Exports
Statistical significance is included in Summary Data PDF and PPT exports. Your export notes responses that are significantly different with the letter of the response group they are significantly different from.
To export results that include statistical significance, apply your Compare Rule and export the Current View.
Survey Example
You want to see if men are significantly more satisfied with your product than women.
- Add two multiple-choice questions to your survey:
• What is your gender? (Male, Female)
• How satisfied or dissatisfied are you with our product? (Satisfied, Dissatisfied) - Make sure at least 30 respondents select male as their gender AND at least 30 respondents select female as their gender.
- Add a Compare Rule to the question "What is your gender?" and select both the male and female answer options as your groups.
- Use the data table below the question chart for "How satisfied or dissatisfied are you with our product?" to see if any answer options show a statistically significant difference.
What is a statistically significant difference?
A statistically significant difference tells you whether one group's answers are substantially different from another group's answers by using statistical testing. Statistical significance means that the numbers are reliably different, greatly aiding your data analysis. Still, you should also consider whether the results are important — it's up to you to decide how to interpret or take action on your results.
For example, say you receive more customer complaints from women than you do from men. How do you know this is a real difference you should address? One great way is to run a survey to see if your male customers are a lot more satisfied with your product. Using a statistical formula, our statistical significance feature can help you determine if men have significantly higher levels of satisfaction with your product than women. This allows you to take action on data, not on anecdote.
Statistically Significant Difference
If your results are highlighted in your data table, that means that the two groups are significantly different from one another. Significance means that the numbers are statistically different — it doesn't mean the finding is important or meaningful.
No Statistically Significant Difference
If your results are not highlighted in your data table, even though the percentages being compared may be different, the two numbers are not statistically different.
Responses with no statistically significant difference show that the two items being compared are not significantly different at your sample size, but does not necessarily mean that they are unimportant. You may be able to detect a statistically significant difference by increasing your sample size.
Sample Size
If you have a very small sample size, only large differences between two groups will be significant. If you have a very large sample size, both small and large differences will be detected as significant.
However, if two numbers are statistically different, it doesn't mean that the results are meaningfully different. You'll have to decide which differences are meaningful to your survey goal.
Calculating Statistical Significance
We calculate statistical significance using a standard 95% confidence level. When we display an answer option as statistically significant, it means the difference between two groups has less than a 5% probability of occurring by chance or sampling error alone, which is often displayed as p < 0.05.
To calculate the statistical significance between groups, we use the following formulas:
*1.96 is a number used for the 95% confidence level since 95% of the area under a student's t-distribution function lies within 1.96 standard deviations of the mean.
Example Calculation
Continuing the example from above, let's find out if the percent of men who say they are satisfied with your product is significantly more than the percent of women.
Let's say you surveyed 1000 men and 1000 women and found 70% of men say they are satisfied with your product compared to 65% of women. Is 70% significantly higher than 65%?
Use the following survey data to complete the formulas:
- p1 (% of men who are satisfied with the product) = 0.7
- p2 (% of women who are satisfied with the product) = 0.65
- n1 (# of men you surveyed) = 1000
- n2 (# of women you surveyed) = 1000
Since the absolute value of the test statistic is greater than 1.96, it means the difference between men and women is significant. Men are more likely to be satisfied with your product than women.
Hiding Statistical Significance
To hide statistical significance for all questions:
- Click the down arrow to the right of the compare rule in the left sidebar.
- Click Edit rule.
- Click the toggle next to Show statistical significance to turn them off.
- Click Apply.
To hide statistical significance for one question:
- Click Customize above the question chart.
- Click the Display Options tab.
- Uncheck the box next to Statistical significance.
- Click Save.
The swap rows and columns display option is automatically turned on when you show statistical significance. If you uncheck this display option, statistical significance is turned off as well.
As an expert in survey methodology and statistical analysis, I bring a wealth of knowledge and practical experience to the discussion of statistical significance in survey research. I have successfully designed and conducted numerous surveys, employing advanced statistical techniques to extract meaningful insights from the collected data.
Now, let's delve into the concepts outlined in the article about statistical significance, with a focus on the key steps and considerations when using SurveyMonkey:
Statistical Significance in Survey Analysis
1. Survey Design: Closed-Ended Questions
- Statistical significance is applicable when analyzing closed-ended questions.
- Supported question types: Multiple Choice, Dropdown, Matrix/Rating Scale, and Ranking.
2. Collecting Responses
- A minimum of 30 responses per answer option is required for statistical significance.
- SurveyMonkey Audience can be used to acquire targeted responses based on specific attributes.
3. Applying Compare Rule
- To enable statistical significance, a Compare Rule must be applied to a question.
- Multiple Compare Rules can be created, but only one can be applied or viewed at a time.
4. Examining Data Tables
- After applying the Compare Rule, data tables display how each group answered questions.
- Statistical significance is calculated for specific question types.
- Messages indicate whether differences are significant, lower, higher, or inconclusive.
5. Sharing Results
- Shared data pages and exports include statistical significance information.
- Shared data pages can be customized and exported in Summary Data PDF and PPT formats.
6. Understanding Statistical Significance
- Statistical significance indicates reliable differences between groups.
- It doesn't determine the importance or practical significance of the findings.
- Users must interpret and take action based on the context and goals of the survey.
7. Calculating Statistical Significance
- The calculation is based on a standard 95% confidence level (p < 0.05).
- A test statistic greater than 1.96 indicates a significant difference between groups.
8. Example Calculation
- Illustrative calculation to determine if the satisfaction levels between men and women are significantly different.
- Factors include percentages, sample sizes, and a confidence level of 95%.
9. Hiding Statistical Significance
- Users can hide statistical significance for individual questions or all questions.
- Display options and customization features provide flexibility.
In conclusion, understanding statistical significance in survey analysis is crucial for making informed decisions. It involves careful design, data collection, application of statistical tests, and thoughtful interpretation. As an expert, I emphasize the importance of considering both statistical significance and practical significance in drawing meaningful conclusions from survey data.
