Many IT projects are failing because their budgets and deadlines are deliberately set too optimistically.
The reason for this comes from too sides: on the one hand, there is the client (either internal or external) who wants to have as many features for as little money as possible and who is many times not very good at judging if their demands are realistic.
On the other hand there is the IT organization which has an incentive to comply with unrealistic targets.
If it is an external client, this unrealistic promise is the advantage over the competition that might get them the project.
But I also see this happen regularly with internal projects where there is no competition between implementors. In that case the motivation is usually that IT management wants to have the consent of higher management to get this project started; they know that once a project has been going for a year or so, it won't be terminated if it's over budget or past deadlines. At the same time, if they would have tried to give realistic estimates too higher management the project might never have been approved.
Construction projects are the same. It seems like every week I read about another construction project that’s far beyond its initial estimate. The difference is that a half-finished construction project is a gaping wound in the landscape, but a half-finished software project is just some code hidden in a corporate server. The incentive to finish construction projects is greater.
> The difference is that a half-finished construction project is a gaping wound in the landscape, but a half-finished software project is just some code hidden in a corporate server.
The difference is that missing deadlines in construction projects come with bigger monetary penalties (citation needed,
based on the low penal damages I've personally seen in IT project contracts).
From my own professional experience, I'd say it's got more to do with the artificial delivery constraints that come with the mandatory annual and quarterly financial reporting of publicly traded companies. They announce their yearly results to the market and next years IT budget is broadly defined as a portion of the previous year's profits.
Everything is planned & tracked as a portion of an arbitrary 12-month time frame & the internal politics means each department are competing for funds on a "use-it-or-lose-it" basis. If you've never sat in on department/geography/company-wide governance forums where they agree next year's "change management" budget, the basic format is you present a business case & cost forecast based on FTEs over the expected duration of the project, all the mandatory regulatory compliance projects get approved automatically and the remainder of the budget is set based on how well each middle-manager can defend the interest of their department. They have a final number for total approved spend & approve as many projects as they who's forecast cost fits into that, Tetris-style, with a flat 20% contingency spread across everything to account for the fact that historically, trying to plan in this way is so inaccurate that it's basically a series of pseudo-random guesses. It's worth noting that even stuff badged as "Agile" is funded & budgeted for in this way.
Every programme I've been involved with has had some variation on "we can't start work on X in that month, because it won't finish until February and will cannibalise part of my budget for next year".
Unfortunately, given the economic & political status quo this problem is impossible to solve (but for the libertarians reading this, an interesting example of the 'higher order' unintended consequences of sincerely designed industry regulation.
IT projects are often something that the organization has never done before. Given that fact, it seems ridiculous that anyone my presuppose how long and how much a thing will cost without actually having ever done the thing.
The reason for this comes from too sides: on the one hand, there is the client (either internal or external) who wants to have as many features for as little money as possible and who is many times not very good at judging if their demands are realistic. On the other hand there is the IT organization which has an incentive to comply with unrealistic targets.
If it is an external client, this unrealistic promise is the advantage over the competition that might get them the project. But I also see this happen regularly with internal projects where there is no competition between implementors. In that case the motivation is usually that IT management wants to have the consent of higher management to get this project started; they know that once a project has been going for a year or so, it won't be terminated if it's over budget or past deadlines. At the same time, if they would have tried to give realistic estimates too higher management the project might never have been approved.