I suspect you're referring to the widely debunked claims of Ralph Neder here?
Plutonium is, in itself, not all that toxic - it is significantly less toxic than many other heavy metals, and can be handled safely. I wouldn't suggest going out of your way to inhale plutonium dust - but that holds true for such large swathes of the periodic table such that it seems somewhat silly to base a power generation strategy on that alone.
It is of course radioactive, but that then falls into the problems I described above (the tradeoff between not-that-dangerous and not-that-long-lived). However the main reason people complaining about plutonium dust sound rather silly to me is this: plutonium is extremely dense - almost double the density of lead - and thus encouraging it to float around the atmosphere isn't all that easy. Unless you have someone machining doorknobs out of Pu-238 in your bedroom (in which case I would strongly urge you to find different housemates) then inhaling plutonium dust seems rather low on the list of things to worry about.
There is sufficient evidence in humans that inhalation of plutonium-239 aerosols causes lung cancer, liver cancer and bone sarcoma. Exposure to plutonium-239 also entails exposure to plutonium-240 and other isotopes. /Plutonium-239/
