Look at the list of contributors at the end of the release notes of the upcoming 0.9.0 release of the bitcoin reference client[1]. Or look at the activity of other projects, e.g. the bitcoinj google group[2]. There's a lot more than 5 people working on bitcoin.
the number of people contributing is extremely small compared to the people who know about it/make money of it/are enthusiastic about it/could contribute. There is not a deep bench of developers. Many open issues which don't get solved because the 3-4 main devs (laanjw, sipa, gavin) are to busy. look at coinbase: they get rich of it, take 1% fees and add nothing back whatsoever.
[1] https://bitcoin.org/bin/0.9.0/test/README.txt
[2] https://groups.google.com/forum/m/#!forum/bitcoinj