New study reveals a "smartphone penalty" that distorts survey results - marketplace-tech Recap

Podcast: marketplace-tech

Published: 2026-02-09

Duration: 10 minutes

Guests: Carly Urban

Summary

The transition to smartphones for survey responses is likely skewing data, as users on these devices tend to provide less accurate answers due to lower engagement and increased fatigue.

What Happened

A recent study suggests that the use of smartphones for responding to surveys might be distorting the data, particularly evident in the FINRA Foundation's National Financial Capability Study, which showed a 15% decrease in financial knowledge from 2009 to 2021. Carly Urban, an economics professor at Montana State University, conducted research showing that smartphone users are more prone to incorrect or 'don't know' responses compared to those using desktops or laptops. Urban's curiosity was sparked by the puzzling decline in financial literacy despite increased educational efforts, leading her to explore the impact of device choice on survey responses. The study employed a randomized experiment through the Understanding America survey, forcing participants to respond on specific devices, revealing that smartphones led to worse performance on financial knowledge questions.

Urban noted that the quality of smartphones has significantly improved since 2009, but their use in surveys only became prevalent by 2021. The research demonstrated that smartphone users often spend less than three seconds on a question, indicating a lack of effort rather than mere attentiveness issues. Furthermore, when financial literacy questions were placed later in the survey, the performance gap widened, suggesting fatigue played a significant role.

The study also considered the 'fat finger' hypothesis, but found that attentiveness was consistent across devices. General knowledge questions yielded similar results, reinforcing the notion that the issue was not limited to financial topics. Urban hypothesizes that survey fatigue is more pronounced on smartphones, causing users to disengage faster than they would on other devices.

The implications of this research extend beyond financial surveys, suggesting that surveyors need to reconsider how they collect data, especially in terms of device usage. Urban pointed out that younger, less educated, and lower-income individuals are more likely to respond on smartphones, which might introduce new biases into survey data. She emphasizes the importance of disclosing device usage in survey methodologies to provide clarity on the data's reliability.

Urban also highlighted potential future challenges with AI and chatbots, which could further alter how survey responses are gathered and interpreted. She stressed the necessity of adapting research methods to account for these technological changes to ensure accurate data collection. The conversation underscores the complexities of measuring knowledge and the potential pitfalls of relying on devices that encourage minimal engagement.

Overall, the episode sheds light on how technological advancements can inadvertently impact data integrity, calling for a reevaluation of survey techniques in the face of evolving digital landscapes.

Key Insights