Pretty interesting decision to include a lengthy "technical digression" in their SEO apology:
<< Technical Digression: How to scrape 178k URLs in Ruby in less than 15 minutes - Ok, so you have 178k URLs in a postgres database table. You need to scrape and analyze all of them and write the analysis back to the database. Then, once everything’s done, generate a CSV from the scraped data. >>
As I see it, the "technical digression" serves two purposes:
- first, they use it with Google to show how sincere they were about "cleaning up"
- second, and perhaps more important, it gives Google a public excuse to justify why they restored RG so quickly in rankings. As in, "Oh they did a very thorough job of complying by our standards that we had no option but to restore their status".
Unfortunately in this case Google's response speed comes across as a bad thing. Because it applies to only a very select group of sites that are either very rich (aka large advertisers) or very well connected.
My favorite is that they seemed to have spent a few hours (more than 3!) improving their code so that it would run in 15 minutes instead of three hours.
<< Technical Digression: How to scrape 178k URLs in Ruby in less than 15 minutes - Ok, so you have 178k URLs in a postgres database table. You need to scrape and analyze all of them and write the analysis back to the database. Then, once everything’s done, generate a CSV from the scraped data. >>