First off...this is my 5000th post!
That out of the way, the residue left by a light erasure is too small for me to be concerned that the card is altered. technically speaking, if you hold a card between your fingers the natural you're probably adding an infinitesimal layer of oil to it, thus altering it. But no one would actually consider that an alteration. Likewise, a pencil erasure is a virtually undetactable change to the surface.
And if nobody sees it, including TPG, then I don't really care.