Having said that, the alternative of "estimating is a waste of time" is also ridiculous.
"Developer, will I be able to catch the next train if I walk, or do I need to get a taxi?"
"Estimating is a complete waste of time"
"But I just want to know if I need to spend money on a taxi. It's two miles away and I don't have any bags, and the train leaves in one hour"
"Start thinking post-industrially"
"FFS, I just want to know how long it'll take me to get to the station!"
So what's new here? That estimating an unknown task is silly? Well of course it is! But saying estimates are useless is also daft since there are plenty of times we know a fair amount about the problem, and knowing how long things are likely to take is very helpful in deciding what to do next.
If someone asks me how long it'll take to add a bit more information onto a page that we already have in the database, I can give them a fairly good answer since it's a problem that we've done plenty of times and therefore have a good idea about how long it's likely to take. If they want me to find new data sources and integrate them then I can tell them we've never spent less than X weeks doing that so we have to assume it'll take at least that long. If X weeks doesn't seem like a worthwhile investment of time, then even this level of estimation has been useful.
This analogy is flawed, and I think represents the exact problem developers experience with non-developers. There's an underlying assumption that we know what walking or a taxi are. We don't. In this analogy, developers would need to invent both before giving an accurate estimate. The methods of transport are the mechanism for delivering the result of arriving at the desired destination. Software development projects are also about creating the mechanism for arriving at a result. The software is bespoke almost by definition, or you could go buy it off the shelf. Bespoke software is unknown. It's invention. We may be able to apply past experience inventing similar things to our estimations, but there is always some amount of uncertainty. Sometimes there is a lot because we've never built anything like the thing requested. We often may not even understand what is being asked for. If I don't know what a car is, asking me how long it will take to invent one so that you could use it to get to the train is inviting trouble.
> There's an underlying assumption that we know what walking or a taxi are. We don't.
Of course we do! The idea that every task involves some groundbreaking research into an entirely novel field is ridiculous. We often know a great deal about what it is we're about to do. Yesterday I needed to fix some chef recipes to get some monitoring working on a different OS, I could give a fairly decent estimate about how long that would take and it had some uncertainty in it. I built a simple site with stripe integration and had a good idea about how long it would take. I know more about doing that next time, too and have a better idea about how long it will take.
> We may be able to apply past experience inventing similar things to our estimations, but there is always some amount of uncertainty.
Having uncertainty in an estimate does not make the estimate pointless. Even having a lower bound can be enough. If I know a feature will not be done in under a week (because similar ones take two weeks, and this has a few extra complexities so my base assumption should be "longer than two weeks") then that might be enough for someone to not choose to implement it. I've had that before on projects, where I've given a minimum time and the feature has been dropped.
> If I don't know what a car is, asking me how long it will take to invent one so that you could use it to get to the train is inviting trouble.
But if someone said "Invent a car for me, I need to get to the station tomorrow" you could easily say that it's not going to happen within a day, and that information would still be useful.
That's exactly the point why "software engineering" still isn't.
It's tinkering, and as much as we software developers would like to keep it that way, at some point mankind will just have to solve that problem once and for all.
And today's software development will be called "software art" or whatever.
What we likely will do, is develop higher level abstractions for more and more sets of problems so the subset of problems developers works on will change.
Just like an exceedingly small percentage of developers today needs to care about hitting the hardware directly.
How long will it take to invent a flying car? That's engineering right? I want a precise estimate. Also it needs to be made out of a material that is both transparent and can withstand crashing into a concrete wall at 85 mph. And the entire thing needs to be powered by a fuel that can be harvested from inorganic compounds. It must seat 6 people as wel as be able to interact automatically with a aerial car traffic control system that is being built by another team using standards that haven't yet been developed.
How is that any different than software? Software engineering is a thing. There are constraints however each implementation has the potential to be novel. Electrons or steel, it's all engineering. Even engineering has trial and error. Would you say the Wright Brothers didn't engineer the airplane?
What's the definition of engineering? Let's refer to Wikipedia:
Engineering (from Latin ingenium, meaning "cleverness" and ingeniare, meaning "to contrive, devise") is the application of scientific, economic, social, and practical knowledge in order to invent, design, build, maintain, research, and improve structures, machines, devices, systems, materials and processes.
Even "tinkering" is engineering. In fact every invention comes from tinkering. There's no definition of engineering in existence that would preclude software development from being considered authentic engineering.
In terms of constraints and fixed boundaries, to suggest software doesn't have those is just nonsense. Just try leaving out the curly braces next time you write something in C. Unlike a field such as writing, software does have rules. I can write a misspelled sentence and forgo the rules of grammar and my sentence will still be "compiled" and probably understandable. A book doesn't stop working because of a bug. Software certainly does. Thus, there are plenty of fixed boundaries and constraints on our work. The characteristics of steel are analogous to the charactistics or a specific software class. Except in our business, many times we have to invent the steel ourselves.
It's not. People think that because inventions tend to result from tinkering that the two are causally related. They're not. Tinkering is merely one of the ways to provide a mental space in which invention can occur; invention itself is not engineering.
I'll be the devil's advocate and say that a lot of software development is not particularly novel. A lot of software development is essentially doing minor changes to existing systems, in the same way and format as everything existing, with nothing novel or particularly challenging.
If you're asked to add a couple of fields on a CRM form that you've worked on for years, being able to estimate how long changing the storage, data, business and presentation logic should be expected and easy, with high accuracy and very little to no risk. Or maybe you're making yet another minor derivative of a data import task that's slightly different from the hundreds you did before. This is often why businesses demand very specific skillsets, because while I would have no idea how long a Wordpress integration module would take, I would bet that someone who works with it a lot would.
There are a lot of software developers who, when given such a task, will try to make it more challenging by rebuilding everything in the process, then throwing up their hands about the complexity involved. It's understandable why business partners get upset about this.
There is absolutely inventive software development, but we don't make a credible case when we lump it all in together.
You're right, but in my experience, even non-novel software features can be hard to accurately estimate, especially due to yak shaving.
Last week I was asked to add a button to a form that calls a service endpoint, same as the other five buttons that I already implemented, and which took me around 30m, so this should be easy, right?
Except during testing the service throws this weird error that makes no sense, and so I have to start debugging, by tracing the code and possibly looking at the network traffic. I find that the library I'm using is sending a weird request, which with a bit of Googling tells me it's a known bug, and which has a patch, but that doesn't apply to our deployed version of the library.
So now I'm two hours into this half-hour task, and I'm trying to decide whether I should spend the time to backport this fix and re-test all the code using the library, or implement some kludge that solves this problem but increases the effort of anyone in the future that touches this code.
And a lot of these seemingly trivial, familiar tasks require far different time than estimated (both less or more) precisely because no one went back and rebuilt things, thereby leaving an inconsistent, incoherent spaghetti mess behind them.
Editing code is more akin to rewriting essays than re-engineering a machine in that way: It's going to take a lot more effort to make a 3rd grader's essay print-worthy than one of Hemingway's.
thereby leaving an inconsistent, incoherent spaghetti mess behind them
Generally my experience has been that the inconsistent, incoherent mess develops when people tasked with doing trivial changes try to "fix technical debt" in the process, leaving you partly on the way to a "better" model, at least until the next guy comes along. And then the next guy. Until you have a layer cake of different approaches and technologies and philosophies, degrading to a worse and worse state.
Look, I've been doing this for 20 years. Almost everything I work on tends to be novel code. I'm terrible at estimating, and now refuse to do it at all. But there is a lot of self-serving, well, nonsense that we ply in this field, justifying our failures (and the reality that most developers don't want to learn other people's code, so they just short circuit the whole thing and pretend it's higher ground), and the whole "technical debt" thing is one of the most egregious. Most developers, when asked to do something to existing code, will throw their hands up and declare it the worst code they've ever worked with, and they can't possibly make some trivial change without completely rewriting all of it, etc (usually somehow pulling in whatever the pet technology of the month is...sorry, I can't change the web form without turning it into a single-page app built on io.js and React and...). We all tell each other this on boards that we dominate, not realizing how completely ridiculous and transparent it really is. It's always more fun to build something novel than to just modify things that exist.
Which is why I really think maintenance programmers are a unique and valuable breed for businesses to have. I personally have a weakness that I always need to rebuild things. I'm terrible as a maintenance programmer, so I simply don't do it. But there are some people who are perfectly happy adapting to whatever they are working on, learning its idioms and unique traits, and then competently and quickly making necessary changes without drama and theatrics about how dire of a situation they've been put in.
To abuse the analogies that have been plied, it's being asked to drive somewhere, and first re-building the car into an electric car, then a self-driving car, and then deciding that you want it to be a hyperloop, and then talking about relativity, all while the business just wanted to get from A to B.
Ideally you need something in the middle of "Rewrite this whole damn thing from scratch" and "don't touch a thing more than you need to, never change any architecture, just make it work and the tests pass."
You need to understand the existing architecture and idioms and traits. And you need to understand the business domain. Then you need to decide when it makes sense to refactor a sub-system first to make the change easier (and support future such changes, which requires you to make a _guess_ as to how likely future such changes are), or to resolve existing lingering problems while making your change easier. And when it doesn't. And you hardly ever, ever, need to rewrite the whole thing (but not never, just hardly ever).
Personally, I think this is what the art of crafting good code _is_, and something that needs to be the goal of everyone writing software, not something you can say you Just Don't Do because It's Not Your Style.
Exactly this. The problem is by getting to B, the customer thinks they need to visit C, D and E and they don't tell you until later that they also need to visit G but only if you haven't visited H first.
The book The Nature of Software development covers this concept much better than I could..
Even minor changes to existing systems can be hard to estimate if the system has accumulated technical debt over time.
I agree that most software development isn't very novel, but I also lost count of the number of developers scared to change parts of a codebase in fear of what it could break, even if that codebase is less than a year old.
> So what's new here? That estimating an unknown task is silly? Well of course it is! But saying estimates are useless is also daft since there are plenty of times we know a fair amount about the problem, and knowing how long things are likely to take is very helpful in deciding what to do next.
What's more, if we don't know enough about the problem to come up with a reasonably-confident estimate, that indicates that we need to define or scope out the problem better.
"Developer, will I be able to catch the next train if I walk, or do I need to get a taxi?"
"Estimating is a complete waste of time"
"But I just want to know if I need to spend money on a taxi. It's two miles away and I don't have any bags, and the train leaves in one hour"
"Start thinking post-industrially"
"FFS, I just want to know how long it'll take me to get to the station!"
So what's new here? That estimating an unknown task is silly? Well of course it is! But saying estimates are useless is also daft since there are plenty of times we know a fair amount about the problem, and knowing how long things are likely to take is very helpful in deciding what to do next.
If someone asks me how long it'll take to add a bit more information onto a page that we already have in the database, I can give them a fairly good answer since it's a problem that we've done plenty of times and therefore have a good idea about how long it's likely to take. If they want me to find new data sources and integrate them then I can tell them we've never spent less than X weeks doing that so we have to assume it'll take at least that long. If X weeks doesn't seem like a worthwhile investment of time, then even this level of estimation has been useful.