When managing the economy, governments make choices that not only impact overall economic growth but also how it is distributed. How do voters judge incumbents for this? I argue that they look to the economic performance of their social in-groups, a phenomenon I call group-based retrospection. By punishing and rewarding the incumbent for their groups’ performance relative to the national economy, voters can incentivize policy-making that aligns with their interests. I first examine this argument using panel survey data to estimate the relationship between in-group performance and incumbent voting over time. I further test the causal mechanism with three experiments fielded in Denmark and the United States that randomize information about the economic performance of groups based on geography, age, education, ethnicity and class. My findings suggest that group-based retrospection is a substantial driver of voters’ reactions to economic change, and this has important implications for electoral accountability and party competition.