When you wade into the deep end of the pool, the risks increase exponentially.
I don't have a deep end of the pool budget, so I keep myself satisfied with my Mantles and Koufax's and a Ty Cobb check and a Maris plus my Mariano Rivera collection. I do my homework and network with knowledgeable collectors and dealers. I have a high level of confidence my stuff is good.
Collect what you like and what you are comfortable with and you'll be just fine.