Because silence is a common good, like clean air. It's everyone's. When people fill it with their noise they effectively privatize it for the duration. When they shout on speakerphone or play their music or blare sound from their apps it's especially selfish.
Imagine trying to live your life where other people’s desires by default overrode you own.