When should panel data be used?
Panel Data vs. Property Database
Published on: September 24, 2024 by Kirk Wakefield
When assessing the brand impact of sports sponsorships, the choice of data sources matters. We’ve recently explored this by collecting over 2,000 responses from NFL fans across four teams, matching panel data to key indicators like passion and attendance. The findings were then compared to over 1 million responses from team database samples, representing 25 NFL teams.
đWhat did we find?
Panel members completed the same surveys nearly 50% fasterâa median time of 5.63 minutes, compared to 8.37 minutes for team database respondents. This isn’t just about speed; it’s about quality of insight.
The chart illustrates how quickly different groups completed the survey across 10 deciles. The key takeaway: over 40% of panel members finish in just 3-5 minutes, whereas hardly any database fans do. The variance in response times signals a deeper issue.
đWhy does this matter?
The impact is twofold:
1. Lower brand lift: In two major NFL sponsorships, panel members reported brand lift half that of team database respondents. Faster completions may lead to less thoughtful responses.
2. Lower variance: The statistical variance is critical in understanding respondent behavior. For example, in brand equity scales (0-100) the standard deviation (SD) in responses is 26.22 for panel data, compared to 35.18 for team databases. A narrower range of responses suggests that panelists donât deviate much in their answers, leading to less differentiation between brands.
In layman’s terms, less time spent on surveys equals less variance in responses, which impacts the richness of insights we can gather for sponsors.
đSo, are panel members different?
Panel members might be quicker readersâor perhaps they’re just less engaged. Attention checks show that 10-20% can’t accurately input their favorite team’s name. While we match panelists for factors like attendance and passion, on average they differ in several key areas:
-Younger (by 10 years)
-Lower income (37.5% earn less than $75k vs. 23.4% of NFL fans)
-More female (+12%)
-More single (+11%)
-More renters (+13%)
-Watch less linear TV (-10%)
When it comes to evaluating the true brand lift of sponsorships, representative samples from team databases provide a deeper, more accurate view than panels.
When Should You Use Panel Data? (Hint: Not Often)
Panel respondents typically earn less than $1 for completing a survey. This means:
Panel data is useful for:
1. Quick insights: Ideal for short questions like “Who are you voting for?” or “Do you plan to buy a car in the next 12 months?”
Panel data is not ideal for:
2. Longer surveys (over 2 minutes): Respondents are motivated to finish as quickly as possible, leading to rushed answers.
Why the rush? If survey takers earn $1 for 5 minutes, that equates to $12 per hour. Many are professional survey-takers (think Uber drivers), maximizing earnings during downtime. Time spent on careful answers equals less income per hour.
Panel participants also share tips and shortcuts on forums (e.g., MTurk Forum), discussing which surveys offer the best return for time spent.
Bigger issues
–Bots: Automated programs completing surveys for profit. (More on this here: https://lnkd.in/gbr2gbYv).
–Fraud: Non-US participants posing as US respondents for financial gain.
While there are ways to mitigate these risks, doing so increases costs. See:
https://lnkd.in/ghy2kxQ9
Bottom line: Be cautious with panel data. Do your homework to ensure it’s reliable. Even large research firms face these issues. Respondents are less invested than fans of a brand or team, who engage more meaningfully in surveys.
When should you use non-fans as a comparison? (Hint: Never)
The idea of using non-fans as a control group makes sense. They presumedly have been exposed to the rest of the brand’s marketing but not the sponsorship.
The problem is that individuals who are not fans (at all) of a team in their market have unusual characteristics.
In a national study (N =1,166), non-fans (27% of Americans) have the lowest income (most have <$50,000 household income), the least education (most have high school or some college), are the oldest (M = 56), and are most likely to be White/Caucasian (81.7%), compared to those who have favorite pro teams to follow.
Moreover, non-fans will always have low brand preference scores compared to sports fans who follow one or more favorite teams across the major leagues and NCAA.
Statistically speaking, comparing non-fans to avid fans is looking at the opposite tails of a normal distribution and somehow being surprised they are different. Firms may use this approach because it will always show significant differences (even if no sponsorship exists).
