YUI was more extensible, followed (mostly) good JavaScript patterns (making heavy use of the module pattern), had relatively good cross-platform support and an easy to understand interface.
Serious? That's what drove me away. It might have been top notch academically but for real life use I found it painful to get answers and simple code problems fixed.
You're looking at the current version of the documentation. jQuery didn't have nearly as good documentation five years ago, and YUI's was much better than it is today (I think YUI3 is a step back for Yahoo).
1. Yahoo hosted scripts, including roll-ups
2. UI components and styles
3. The best documented library, both outside and inside thde code, hands down
4. More feature complete (ex. one I can recall off the top of my head, form handling of upload inputs didn't exist in jquery, and extending jquery internals was/is a nightmare in comparison)
5. The best documentation, hands down
6. Yahoo hosted scripts, including roll-ups
7. The best documentation
jquery has come a long way with documentation, but then YUI has evolved even further in becoming a modular library.
Edit: Had my dates wrong.