Collectors should generally be comfortable with the value in their collection, whatever the circumstances might be. If you feel you have too much tied up in it it is easy to right-size it. It is way more fun when it isn't financially burdening, not that any of us would ever do that.
To reiterate others, I don't think baseball card collecting will die off in any of our lifetimes. That said there are fewer and fewer 50s-70s Topps set collectors, just starting sets, and there is an overabundance of those cards in low to mid grade. If I had millions of those I might be a little concerned.
.