If women who were married to people who worked with Feynman had consensual sex with Feynman, what exactly is the problem? It might create a tense work environment, but you should be able separate work from private life.