they’re pretty good at getting info that is very up to date by using tools to access the web
Yeah that's a charitable way to phrase "perform distributed denial of service attacks". Browsing github as a human with their draconian rate limits that came about as a result of AI bots is fucking great.
I don't run personal sites worth millions of dollars. I do, however, use sites like Sourcehut, DigiKey, Github, Mouser, Farnell, etc, etc, etc. that have opted to put everything behind bullshit captchas because of the DDoS (nee AI) bots.