I'd rather suspect the Windows API behind for this behavior, rather than the file manager itself. Apparently there might be leaked Windows 3.1 source code (but not all of it): https://www.betaarchive.com/forum/viewtopic.php?t=41781 (only comments here, didn't check further to see if/where this can be downloaded), but not sure if the relevant source is visible there. Windows 3.1 doesn't seem to use the same conversion as for Alt+XXX to read file names (which are stored in the OEM charset on the disk, e.g. code page 437), as typing Alt+255 enters a normal space (not event a non-breaking one).
You are right, it's likely to be the conversion from OEM to ANSI. In that case, NT wouldn't have the same bug, since it uses Unicode internally (and the tables translate OEM character 255 to U+00A0).
Quickly had a look in the KERNEL31 source. The LDOPEN.ASM file contains calls to a routine called MyOemToAnsi, which does the character set conversion. In turn, that function either calls the actual conversion routine in the keyboard driver, or does nothing if there's no keyboard driver yet (boot phase). Didn't find the keyboard driver part, probably not part of the actual kernel so it makes sense that it's not there. Anyone knows where to find one?
Great question, what would happen if NT sees a FAT filesystem (therefore with 8.3 names stored in OEM charset) with a filename containing a Alt+255 (0xFF) character? Anybody with a ready VM around to test this?