“Surveys show that surveys never lie,” is a saying that you may have seen before. It can be interpreted multiple ways, and one is that surveys, even if they are properly designed to extract pertinent information, are not perfect because they inherently have bias in some way, shape, or form. If you have ever relied on research to make decisions, you already know this about surveys. That’s not to say that survey’s don’t have their benefits, such as explaining the “why” behind the way respondents think, establishing segmentation, etc.. However, when the scope of a survey expands beyond that purpose, such as to quantifying claimed behavior, we begin to see the limitations of survey research become glaringly obvious.
One of the ways to help alleviate this bias behind surveys is to replace or supplement with actual behavioral data. This is where Verto Analytics comes in – we provide cross-device behavioral data so you do not have to rely solely on survey data’s claimed data.
In fact, we at Verto Analytics have fielded survey data meant to mimic behavioral data in order to show where similarities and differences occur. In this example, we will look at claimed vs. actual behavioral data behind the usage of Spotify, the audio streaming service.
To compare claimed vs. behavioral data, we asked panelists to best describe their Spotify usage habits in a survey by choosing one of the following:
- Never tried
- Tried, but no longer using very often
- Using monthly
- Using weekly
- Using daily
Using Verto’s cross-device behavioral panel, we’re able to see how those survey respondents actually engaged with Spotify across PC, Smartphones & Tablets. With the metrics listed below, we compared respondents’ stated usage to their actual usage.
- Average Spotify Sessions (Past Six Months)
- Average Number of Active Days (Past Six Months)
- Average Spotify Sessions Per Active Day (Past Six Months)
How did the data line up? We aligned claimed daily, monthly and weekly usage on Spotify at a responder level versus actual usage of the service across the last 6 months since they took the survey. What’s interesting is that 28% of responders who claimed monthly usage did not use the app at all on PC, Smartphone or Tablet.
Among those claiming to use Spotify daily, just about all actually had some observed cross-device Spotify usage (94%) – even if it was not as often as they claimed as later charts will show. However, as the claimed usage incidence decreased, so did the accuracy behind the claims: only 83% of those who claimed weekly usage had actually used Spotify, compared to 72% of those who claimed monthly usage.
When we take a closer look at how claimed usage aligns with actual usage, we see quite a bit of overlap among those claiming to use Spotify daily, weekly, and even monthly.
In the graph below, we compared respondents’ behavioral usage of Spotify. If people were accurate in their surveys, then we’d expect to see the monthly users cluster around 6 days of use, the weekly users cluster around 24 and the daily users cluster around 180. But instead, we see a large overlap – there are claimed daily users who are less engaged in Spotify than those who claimed they were monthly users, and monthly users more engaged than claimed weekly users.
Even while those who claimed they were daily users of Spotify are more likely to actually have used Spotify, people struggle to recall their actual frequency of usage. We saw this across those who claimed to have used Spotify monthly, weekly and even daily.
If we break down the claimed users based on summary stats, we see that the claimed daily users of Spotify do have higher intensity given the 8.4 average sessions on days the app is used. However, the average number of active days among responders who claim daily usage is only 85 out of a possible 183 days. This shows that people are more likely to recall usage intensity or bursts of activity as opposed to consistent usage across days.
So what does this mean? Surveys are still very useful to understand why someone takes an action. But, the more granular you get in asking people to recall how often they do certain things, the more error you’ll get in your reporting. If you want a true understanding of how people engage on online, using cross-platform behavioral data provides the clearest picture of actual consumer behavior.
Interested in more insights about the digital behaviors and demographics of Music Lovers? Download the full report here.