This is a meaningless simplification. In this framework "robots.txt" has no role, because your server "can choose" not to respond. Heck, even DDOS is fine, because "protocol"