I lost my SSN card many years ago, and have not needed it to do typical things like vote, work, travel in USA and out of USA, start businesses, and file lawsuits. It's irrelevant.
The fact that you don’t use yours doesn’t make the thing you lost not an ID. If you lose your house key and always go in through the garage, the key you lost is still a key.