There are people behind the web crawler. If they’re so well funded they can exert a little effort to not so badly inconvenience people as they steal their training data.
I've downvoted you for being incredibly aggressive in your responses. I'm not sure why you're ad homineming the parent commenter, but it's not helping the discussion.
I don’t even really get what they are saying. I am also saying that they are hostile, and with all of their money they can afford to not be hostile. So I feel like we agree?