This study explores the effects of memory retrieval in task switching. To this end, item-specific stimulus-to-task mappings were manipulated in two “alphabet-arithmetic” experiments. Letter-stimuli were presented and the responses were verbal letter names. The task was either to name the next letter in the alphabet, (e.g., C → “D,” task “plus”), or to name the preceding letter (e.g., C → “B,” task “minus”). The mapping of individual stimuli to the two tasks (and thus to responses) was either consistent (CM) or varied (VM). In Experiment 1, performance was worse for VM items relative to CM items, indicating item-specific task-mapping effects. These task-mapping effects also contributed to mixing costs (i.e., worse performance in mixed-task blocks than in pure-task blocks) but not to switch costs (worse performance in task-switch trials than in repeat trials within mixed blocks). Experiment 2 manipulated pure and mixed tasks between-participants, and the data again showed differential effects of the task-mapping manipulation on mixing costs and switch costs. This suggests that, in these memory-dependent, alphabet-arithmetic tasks, interference due to involuntary task (and/or response) retrieval primarily increases general multi-task effects, such as maintaining activation of the current task.