Ssshhhh......Don't tell the card guys
I once removed a pencil notation on a 1964 Willie Mays card. I used a white retractable Clic eraser. Worked well. If you have a light hand you can take off stray marks pretty well without affecting the surface. It will remove sharpie from a glossy photo too. Avoid the gum or kneaded erasers as those have a small amount of abrasives embedded into them and they will, at best, leave small marks behind.