The key distinctions are: we can hold people accountable, and the amount of shit produced by one person is limited. Neither of those are true for LLMs.
The recent book Unaccountability Machine (Dan Davies) made the rounds a while back on this site a few years back. A few years before that, Ted Chiang's essay in the New Yorker entitled "Will AI Become the New McKinsey?" did likewise.
There's a bright red through line here. I get the sense that the intellectual ferment is starting to develop an awareness of the risk (and if you're cynical, potential) of LLM deployments in business as a systematic strategy for absorbing accountability for decisions.
For the time being, it usually seems like there's someone who is accountable, or at least can be scapegoated. But how long will that last? As Davies points out in his book, we didn't need LLMs to create bureaucracies where the buck fails to stop anywhere and instead irretrievably slides between the tracks. As Chiang points out, the "efficiency maximization" of McKinsey served as a way for organizations to outsource accountability for major decisions to an entity very, very good at working backwards from desired outcomes while acting like they were just led to a fait accompli by the numbers.
