Here, we explored the impact of finger counting habits on arithmetic in both adults and children. Two groups of participants were examined, those that begin counting with their left hand (left-starters) and those that begin counting with their right hand (right-starters). For the adults, performance on an addition task in which participants added 2 two-digit numbers was compared. The results revealed that left-starters were slower than right-starters when adding and they had lower forward and backward digit-span scores. The children (aged 5–12) showed similar results on a single-digit timed addition task—right-starters outperformed left-starters. However, the children did not reveal differences in working memory or verbal and non-verbal intelligence as a function of finger counting habit. We argue that the motor act of finger counting influences how number is represented and suggest that left-starters may have a more bilateral representation that accounts for the slower processing.