If I had to place a bet - my money would be on your theory. I've noticed the exact same thing, which makes it even worse as I then have to pull up the app drawer and go back to my player app. At least it never gets totally kicked out in that the state is preserved/not reloading like a fresh open.

I guess my question from there would be why does the device suddenly feel the need to release resources when the music/podcast pauses? I'm experiencing it on an iPhone 13 Pro which should have no shortage.

If it was a once in a blue moon thing I would brush it off as just unlucky timing, but it happens very regularly.