I feel like k-means in in Lab color space would lend itself well here too. Either by picking a large k that you then narrow down by picking the highest contrast colors, or instead by running k-means multiple times with different k's until you get an error threshold that is tolerable and results that have a high enough contrast. K-means is also a pretty easy algorithm to understand/implement.
Google uses k-means clustering in Chrome on their new tab page (though they use RGB). The most visited section shows a color strip below each site, with a color chosen from the site's favicon.
I am going to straight up lift that idea into a web app I've built. Providing users a hint more context about their content makes interfaces less bland and anonymous. Thanks for raising it!
K-means clustering gives you a set of centroids which are the calculated means of each cluster. If you only want actually occurring colors, just look up the color closest to each centroid.
