Practically speaking, I've been using a yubikey plugged permanently into my laptop for many years, and it works fine. I use it to authenticate to my work VPN (with a password as second factor).
The only downsides are: one fewer USB port, and the green light on the yubikey which is permanently lit.
You've made a great list of the downsides, perhaps you could at some point list the upsides? A yubikey you don't move around, and that you don't take extra steps to ensure the security of, seems no more secure or useful than a cert file on your hard drive.
As others have said, you have to touch the yubikey to get it to generate a one-time password (it acts like a USB keyboard).
The yubikey is one factor, the VPN also requires a second factor (memorized password). These are concatenated so you type the password without pressing the enter key, then tap the yubikey (which "types" the OTP + enter key). This process works in web forms, shells, etc. Could hardly be simpler.
If the laptop is lost/stolen, I can deactivate the token.
Unlike a cert file, a hacker can't steal the keys in a Yubikey if the machine is compromised. Same reason why many enterprises prefer using TPMs for storing machine certs.
The advantage of a Yubikey over a TPM in this case is that the Yubikey requires a physical tap before it'll sign a request, which prevents certain MITM attacks.
If I understand correctly, you must touch the Yubikey to reply to a second factor request. Simply being present in the USB port is not sufficient to utilize its credentials.
I have one that's permanently installed too, and the green LED isn't ever lit unless an app is asking for touch. Perhaps it depends on what mode you have it it? (I'm using mine in U2F + PGP Card mode.)
The only downsides are: one fewer USB port, and the green light on the yubikey which is permanently lit.