Published on: November 26, 2024 by Kirk Wakefield
When we ask fans, “Which of these brands do you prefer to use in the future?” responses are captured on 0–100 slider scales. These scales are anchored by four descriptive labels (adapted from the BAV Group):
Never → If no other choice → One of several → The one I prefer
From 2,206,516 fan responses, we’ve analyzed brand preference scores for 2,019 sponsors and their competitors. The data shown below illustrates how good survey questions do three things:
- Promote variance in responses:
Poorly designed questions often result in skewed distributions, with most responses clustered at one end of the scale. In contrast, well-designed questions generate more balanced distributions, revealing a richer picture of fan preferences. Leading questions consistently produce distorted, skewed results. - Measure attitudes in degrees:
Preferences, like most attitudes and opinions, exist on a spectrum—not as simple yes/no choices. Effective survey questions capture these degrees, allowing for more nuanced insights. Whether it’s preference, willingness to recommend, or likelihood to use, all are better understood on a continuum rather than a binary scale. - Enable precision in ranking:
The average scores demonstrate clear distinctions:- Sponsors: 47.36
- Competitor #1: 42.24
- Competitor #2: 38.57
Furthermore, aided recall boosts sponsor scores significantly (M = 59.92), offering an exact quantification of the advantage (+12.56). Such precision provides actionable insights into brand positioning.
Reliable questions yield consistent results over time, unlike formats such as unaided recall, rank-order, or select-one. For example, asking fans “What wireless brands come to mind?” or to “Name your top three movies” can produce varying answers from month to month—you might forget one or discover a new favorite.
In contrast, rating specific options on a consistent scale delivers stability. If people were asked to rate Gladiator, Gladiator II, and Wicked 1, on a 1-10 scale we would find on average:
- Gladiator (IMDB = 8.5), Wicked (IMDB = 8.2) and Gladiator II (IMDB = 7.0).
This precise, comparative data underscores Gladiator as an all-time great, while Gladiator II aligns more closely with other good but not extraordinary films. In comparison, Wicked (I, a sequel is coming) looks like a big winner as well.
By asking the right questions, we gain deeper, more reliable insights into consumer preferences, helping sponsors and competitors alike make informed decisions.
Data collected using scales (like rating something from 1 to 10) can be analyzed with advanced methods to find patterns and relationships. In contrast, data from simple questions like picking one option (nominal) or ranking items in order (ordinal) can’t be analyzed as thoroughly or flexibly.
Bad questions
Leading and loaded questions find few if any responses in disagreement because doing so makes no sense. Some examples:
- I am more inclined to consider products and services offered by official partners of [your favorite team].
- Does being the “Official Health Care Partner of the [team]” make you have a more or less favorable opinion of the [partner]?
- I am more likely to try a product from a sponsor of the [team] versus a product from another company.
- Sponsors support my favorite teams and leagues and I support them in return.
Leading questions contain the answer in the question itself. Any question starting with “I am more likely to” is a leading question.
Loaded questions contain bias, assumptions or information that can influence responses. The question design subtly pushes toward the desired response rather than an impartial, honest answer.
Double-barreled questions ask about two or more issues. Respondents might be positive about one but negative about the other. For example, post-game surveys often ask attenders to rate ease of entrance and exit from parking areas on one scale.
Each of these lead to skewed data, reduced credibility, as well as ethical concerns for knowingly fielding faulty questions.