I'll be the devil's advocate and say that a lot of software development is not particularly novel. A lot of software development is essentially doing minor changes to existing systems, in the same way and format as everything existing, with nothing novel or particularly challenging.
If you're asked to add a couple of fields on a CRM form that you've worked on for years, being able to estimate how long changing the storage, data, business and presentation logic should be expected and easy, with high accuracy and very little to no risk. Or maybe you're making yet another minor derivative of a data import task that's slightly different from the hundreds you did before. This is often why businesses demand very specific skillsets, because while I would have no idea how long a Wordpress integration module would take, I would bet that someone who works with it a lot would.
There are a lot of software developers who, when given such a task, will try to make it more challenging by rebuilding everything in the process, then throwing up their hands about the complexity involved. It's understandable why business partners get upset about this.
There is absolutely inventive software development, but we don't make a credible case when we lump it all in together.
You're right, but in my experience, even non-novel software features can be hard to accurately estimate, especially due to yak shaving.
Last week I was asked to add a button to a form that calls a service endpoint, same as the other five buttons that I already implemented, and which took me around 30m, so this should be easy, right?
Except during testing the service throws this weird error that makes no sense, and so I have to start debugging, by tracing the code and possibly looking at the network traffic. I find that the library I'm using is sending a weird request, which with a bit of Googling tells me it's a known bug, and which has a patch, but that doesn't apply to our deployed version of the library.
So now I'm two hours into this half-hour task, and I'm trying to decide whether I should spend the time to backport this fix and re-test all the code using the library, or implement some kludge that solves this problem but increases the effort of anyone in the future that touches this code.
And a lot of these seemingly trivial, familiar tasks require far different time than estimated (both less or more) precisely because no one went back and rebuilt things, thereby leaving an inconsistent, incoherent spaghetti mess behind them.
Editing code is more akin to rewriting essays than re-engineering a machine in that way: It's going to take a lot more effort to make a 3rd grader's essay print-worthy than one of Hemingway's.
thereby leaving an inconsistent, incoherent spaghetti mess behind them
Generally my experience has been that the inconsistent, incoherent mess develops when people tasked with doing trivial changes try to "fix technical debt" in the process, leaving you partly on the way to a "better" model, at least until the next guy comes along. And then the next guy. Until you have a layer cake of different approaches and technologies and philosophies, degrading to a worse and worse state.
Look, I've been doing this for 20 years. Almost everything I work on tends to be novel code. I'm terrible at estimating, and now refuse to do it at all. But there is a lot of self-serving, well, nonsense that we ply in this field, justifying our failures (and the reality that most developers don't want to learn other people's code, so they just short circuit the whole thing and pretend it's higher ground), and the whole "technical debt" thing is one of the most egregious. Most developers, when asked to do something to existing code, will throw their hands up and declare it the worst code they've ever worked with, and they can't possibly make some trivial change without completely rewriting all of it, etc (usually somehow pulling in whatever the pet technology of the month is...sorry, I can't change the web form without turning it into a single-page app built on io.js and React and...). We all tell each other this on boards that we dominate, not realizing how completely ridiculous and transparent it really is. It's always more fun to build something novel than to just modify things that exist.
Which is why I really think maintenance programmers are a unique and valuable breed for businesses to have. I personally have a weakness that I always need to rebuild things. I'm terrible as a maintenance programmer, so I simply don't do it. But there are some people who are perfectly happy adapting to whatever they are working on, learning its idioms and unique traits, and then competently and quickly making necessary changes without drama and theatrics about how dire of a situation they've been put in.
To abuse the analogies that have been plied, it's being asked to drive somewhere, and first re-building the car into an electric car, then a self-driving car, and then deciding that you want it to be a hyperloop, and then talking about relativity, all while the business just wanted to get from A to B.
Ideally you need something in the middle of "Rewrite this whole damn thing from scratch" and "don't touch a thing more than you need to, never change any architecture, just make it work and the tests pass."
You need to understand the existing architecture and idioms and traits. And you need to understand the business domain. Then you need to decide when it makes sense to refactor a sub-system first to make the change easier (and support future such changes, which requires you to make a _guess_ as to how likely future such changes are), or to resolve existing lingering problems while making your change easier. And when it doesn't. And you hardly ever, ever, need to rewrite the whole thing (but not never, just hardly ever).
Personally, I think this is what the art of crafting good code _is_, and something that needs to be the goal of everyone writing software, not something you can say you Just Don't Do because It's Not Your Style.
Exactly this. The problem is by getting to B, the customer thinks they need to visit C, D and E and they don't tell you until later that they also need to visit G but only if you haven't visited H first.
The book The Nature of Software development covers this concept much better than I could..
Even minor changes to existing systems can be hard to estimate if the system has accumulated technical debt over time.
I agree that most software development isn't very novel, but I also lost count of the number of developers scared to change parts of a codebase in fear of what it could break, even if that codebase is less than a year old.
If you're asked to add a couple of fields on a CRM form that you've worked on for years, being able to estimate how long changing the storage, data, business and presentation logic should be expected and easy, with high accuracy and very little to no risk. Or maybe you're making yet another minor derivative of a data import task that's slightly different from the hundreds you did before. This is often why businesses demand very specific skillsets, because while I would have no idea how long a Wordpress integration module would take, I would bet that someone who works with it a lot would.
There are a lot of software developers who, when given such a task, will try to make it more challenging by rebuilding everything in the process, then throwing up their hands about the complexity involved. It's understandable why business partners get upset about this.
There is absolutely inventive software development, but we don't make a credible case when we lump it all in together.