> However, the fact that Linus Torvalds, who is in charge of a critical piece of our infrastructure made of 15 million lines of code (the Linux kernel), does not use a debugger tells us something about debuggers
It _may_ tell us something about debuggers. Probably that they are not as useful when it comes to OS kernels.
Just as well, it _may_ tell us something about Linus Torvalds. Probably what tools he is more proficient with, or likes more.
> Debuggers were conceived in an era where we worked on moderately small projects, with simple processors (no thread, no out-of-order execution), simple compilers, relatively small problems and no formal testing.
Unix was conceived in another era too, but modern Unices are totally different and have been adapted to modern times. So have debuggers. When I debug a Java program in IntelliJ I can stop and run threads as I like; OOO execution is transparent; so is both high-level compilation (javac) and low-level JIT compilation by the JVM.
> in order to catch bugs, we often need to be able to run with sufficiently large and representative data sets. When we’re at this point, the debugger is usually a crude tool to use
You'd typically use a debugger to understand why one of the tests with your large dataset fails, after you have already noticed that there is a bug. You don't use a debugger to notice that a bug exists at all.
This article really has a point, but it's buried under crude logic, and the elephant in the room is that "debuggers vs. tests" is a false dichotomy.
This conjures up the memory of articles talking about why syntax highlighting is bad. [1]
It's sort of funny that the author even admits "Ok, well, I do use a debugger but only when it makes sense". Like, are there devs using debuggers when they don't make sense? I don't often drop into a debugger just for fun.
Debuggers (and, particularly the java debugger) are amazing tools for quickly figuring out why things went wrong. I rarely step through code (though sometimes that's useful) but I frequently will drop a breakpoint on an exception and then jump around the call stack to see exactly how a particular exception developed. With a debugger, I get all the inputs for everything up and down the stack. I can pop the stack and replay the exception over and over again. I can evaluate code in the current scope to test and see what's going wrong.
Particularly with really complex software stacks, debuggers are essential.
Consider, for example, this error case I ran into.
Some code took a view on a map. Unfortunately for us, modifications to that view are both allowed AND mutated the underlying map. When the code that produced that view called other code, at about 30 layers down some code said "If there's not a value here, add the value 0". This caused the unusual state where with apparently the same input data when you first called a method, it would pass no problem. The second time, it would error out.
How on earth would you discover this behavior without a debugger? The root method where this all happened was already 20 layers deep in this fairly complicated piece of software (yay finance!). A simple unit test would show everything working as expected, it's not until that second run of the same code that things blew up.
Without actually catching the fact that the inputs were mutated from an earlier invocation there's really no way to know what went wrong. A debugger made that obvious.
> Like, are there devs using debuggers when they don't make sense? I don't often drop into a debugger just for fun.
Yes. Older developers tend to debug with console/echo/print etc. Somewhere along the line this was deemed “bad practice” and now there are devs who will exclusively use the debugger for everything because they were taught it’s the “correct” way.
People will do what they’re taught. I grew up with console/echo/print so that’s how I roll. I never use the debugger, to my detriment. Conversely, people who grew up exclusively using the debugger will use the debugger for everything even when it would be way faster to just dump some data.
I learned to program, even in assembly, before quality debugging tools were available (or I knew about their existence), so very much relied on "debug manually using log output". But once proper debuggers were a thing and I learned how to use them I never looked back.
In fact I often use debuggers to step through new (or new-to-me) code even if it does appear to be producing the right output as I've seen too many times there are bits of unexpected behaviour along the way that are virtually guaranteed to cause a problem at some point (including cases where errors were occurring but being swallowed/ignored).
These takes on debugger use always mystify me. They always seem to indicate that they think debuggers are used by single-stepping thousands of times as if they are using them to watch their program execute at 1,000,000,000x slowdown.
Even primitive debuggers like gdb are not intended to be used like that. You theorize a failure mode, identify the proximal causes, then set breakpoints at the earliest suspicious points. You then examine the actual state to determine if your mental model of the code matches. You can then single-step to verify that the suspicious piece of code does or does not do the correct thing.
If you are feeling really adventurous and are willing to try technology that is only 20 years old you can use time travel debugging. Then you do not even need to think about how you debug. You just hit the bug then run backwards to the cause of the error. Then you examine the entire state of the universe at the time trivializing basically every bug.
You can use even newer stuff like high grade visualization of the entire program execution which trivializes bug detection, performance enhancement, and system understanding.
These articles always seem stuck 50 years in the past and even then seem to always be arguing that hammers suck because they keep trying to hit the nail with the handle.
> You can use even newer stuff like high grade visualization of the entire program execution which trivializes bug detection, performance enhancement, and system understanding.
Very interesting... what is the gdb-reverse-debugging equivalent for high grade visualizations of, for ex. cpp or python software? Something that's free, open source, generalizable
I am very much in the opposite camp, I am 10x more productive in tech stacks where I can use an interactive debugger.
It’s not that I enjoy stepping through code line by line, I rely on a debugger to verify if my assumptions about “starting conditions” of a certain piece of code are correct and, if so, at which point my understanding of how things should be no longer reflect reality.
This can all be done with printf, if you knew ahead of time what you want to print. But most of the time I don’t, the whole point is that I do interactive exploration of the program state (“evaluate expression” is the main feature of a good debugger). When I find a problem I dig deeper, which is a lot faster within a debugger session than any recompilation loop I’ve ever seen (even in Go).
In similar spirit I play with SQL queries when given a data set that I need to extract something from.
I do not use a multimeter. Because mostly I plug together things with connectors. Because I believe the things have (more or less) been tested to do what they are supposed to and it's seems that the connectors themselves usually work. Some people build the things I plug together, and they probably reach for the multimeter more often.
I find this take odd. To me, debugging with breakpoints is just a more powerful and flexible version of thinking about where to put printfs. You still have to think carefully about where to put the breakpoints, too.
If I sprinkle a couple of prints and my hypothesis about what state I need to see is wrong, then I have to do another build and run cycle to get more information. If I'm stopped at a breakpoint I can dynamically and interactively root around without having to go through the whole scenario again.
Maybe this is just a different mode of using the debugger than what's being argued against. I will say that stepping through is really only fruitful in my experience when there's an extremely narrow region of code that I've identified as causing the problem.
Hah, yep. And further, sometimes a logging statement can cause a few ms of delay that causes whatever you're debugging to just magically start working again (yay, race conditions). A breakpoint has a better chance of catching it in the act than a logging statement does. At least in my experience.
Interesting. This probably varies by tech stack, but in my experience debugging concurrent/async stuff is where I will actually lean away from the debugger for fear of that kind of timing skew.
The case they are describing is when you know exactly where the error appears, but do not know how it appears. So you put in one breakpoint that will be hit exactly once when it rears its head instead of continuing over breakpoints which causes the timing skew you are talking about.
In any event, standard console logging is an abomination and there are much better encoded logging techniques that are exactly as usable and just as easy to add to your code, while not introducing material timing skew.
Yeah, but debuggers have a hard time capturing state history and displaying it in a useful way. Often times, the bug is a lot easier to understand if you see the historical values printed to the terminal, but is not at all obvious just by inspecting the current value of variables. This is especially true in any system with feedback, which is nearly any non-trivial system. Bugs can become so much more obvious if you see how state is changing through time, which print debugging is better at than debuggers and "plot debugging" is best at.
That is because those debuggers use primitive technology from over 50 years ago. Use a time travel debugger in the last 20 years and you have perfect state history of all values over all time.
Print debugging only captures what you think to capture. Time travel debugging captures everything. And with a proper implementation the overhead is minor.
Right, but again, you can do exactly that with breakpoints (especially if they can print and then auto-continue), and you can change your output, adding or disabling them while still running.
With prints, I'm usually not targeting one spot to see if it looks right, I'm putting them broadly over everything that gets hit to find where the data diverges from my expectations. Maybe I'll do another round to narrow it down some more, but it's only after finding the right place like that that I'd use the debugger, so I don't have to waste time stepping through everything.
100% agree, i don't even see any difference with print statements, what was all this pontificating about? it's literally just a different way to do the same thing as print statements - find out if the program reached that step and some values.
The problem with debuggers is that some programmers find them absolutely necessary to their workflow and they don't get that some of us simply don't.
I use debuggers sparingly, I think they can help in some cases but most of the time I feel much better reading code and traces. Especially when debugging difficult async code.
Being condescending with coders who have different habits than yours is the issue.
There is no "one true way" of doing things, even in our beloved craft.
This is one of those funny things. Debuggers are amazingly useful as a print statement machine. You get prints for every line. You can stop and think as the program executes. But at the same time, most bugs can be found with, as said, prints and thought.
I generally reach for a debugger when code is complex enough that I don't know where to begin putting my prints. But often end up reducing to prints and other output when I get close to the issue.
That said. For buggy code, where the bug appears seldomly, conditional breakpoints are a lifesaver.
Another aspect of this is when you only know about the failure after it has happened (such as with segfaults that can't be reproduced with tools like valgrind). Being able to use something like `rr` to record the entire runtime and then rewind it to the point where the program explodes can be a life saver.
I'm not big on using a debugger but when all else fails, having access to one can be the difference between a swiftly identified and resolved bug and weeks or months of hair pulling and stress.
>However, the fact that Linus Torvalds, who is in charge of a critical piece of our infrastructure made of 15 million lines of code (the Linux kernel), does not use a debugger tells us something about debuggers
It tells us that its possible to program without a debugger It tells us absolutely nothing about the utility of doing so because anecdotal preference cannot possibly reveal this.
> Will your debugger scale to hundreds of processors and terabytes of data, with trillions of closely related instructions? I’d rather not take the risk.
Almost nothing almost anyone is working on needs to work on TB of data across hundreds of processors to reveal a logic error or in fact at all.
The article could profitably be shortened to "I feel like debuggers are a bad investment of my time because I would prefer to spend that time revising and writing tests because my mental model of the tech I work with is sufficiently good that I rarely need to poke the code manually and I have confused my own preference and circumstances with universal truth because that's what humans do"
One of the first things I do when starting work on a new codebase is to step through every single line of code (especially the framework bits). I'm usually the first one to discover silent misconfigurations that have been there for years.
That being said, once I fully understand the entire stack and how we're using it. I can abuse the hell out of it and make it do anything I want it to do; even beyond anything the original authors ever considered.
Most people only take the time to get surface-level knowledge of the entire stack, and either re-implement something they can get for free, or don't take full advantage of what the stack gives them.
That being said, getting that knowledge without a debugger is nearly impossible.
Since I write mostly in Rust and Python, I seldom need a debugger any more. I used one back in the C/C++ era. Sometimes, when some C/C++ level below Rust breaks, I use gdb. Not often.
Somewhat to my surprise, I was able to use gdb to find out why a multi-thread program Rust had a 100x performance degradation under Wine.
The trouble was down in Wine's DLLs which emulate Windows. The memory allocator is vulnerable to futex congestion. They layered spinlocks 3 deep, and when enough threads are allocating, all the CPUs spend most of their time in spinlocks. The Wine devs
acknowledge the problem but are still struggling with a fix.
I would have used a profiler, but "tracy" under Wine is broken at the moment.
I used a debugger last year when a C++ JPEG 2000 decoder called from Rust was crashing. Valgrind was more helpful, though. Once someone got the decoder to fail under valgrind, they were able to fix the problem.
The year before that, I used gdb to find a bug in the Rend3 graphics library. That's in Rust, but has unsafe code. The bug was in the unsafe code, of course.
There are times like this when the high-level language abstraction has broken down, and you need to delve into the machinery underneath. But that doesn't come up too often in modern languages.
To each their own but the notion that the debugger "will not scale to hundreds of processors and terabytes of data" just seems bizarre to me. If a debugger cannot scale to such quantities (which on its own can analyze a lot of data because it's running on the same/another system), then why could I manage to see what is going on? I guess what is meant here is that with sufficient knowledge about the codebase I'm working on, I'll be able to localize the issue much quicker than a debugger, simply because I have an understanding of what to actually look for. But I still don't know why a smart debugger couldn't help me localize the issue.
The one thing I agree about is that debuggers got stuck in a point in time and they cannot do much more than set a breakpoint and trace some code (with exceptions) - but this is not a limitation of the concept, it's rather that the tools we have today often fail to help us. We resort to using printf() instead because then we're looking at just one variable and when it's printed, rather than stopping the whole program or hitting the continue shortcut 50 times.
It's why I've been researching the topic quite a bit myself with some other ideas than "stop the whole thing and look at state". One thing I found is that if the debugger is not integrated into your workflow (it's not working by default after you compile/run), you'll be much less likely to actually use it. Another way to look at it is: would you still not utilize the information the debugger displays, even if it displayed the state at a line you highlighted immediately?
IMO, debuggers, logging, metrics, tracing and the other tools focus on different things and are good at different things.
Debuggers are extremely powerful in the correct situation. If you can reproduce an issue in a controlled fashion, debuggers are nuts. In many cases, you drop a breakpoint where the exception/panic/segfault occurs, trigger your reproduction, and the current context + variable view just throws the problem in your face. If that fails, you breakpoint earlier and that's usually it. I've very much changed my opinion there: If you have a reproducible problem, and the source code, use a debugger. It's better.
On the other hand, good logging and tracing are better in other cases. If we call a debugger spear fishing - targeting a concrete target specifically - good logging and tracing are more of a drag net. Like, in my last job, we had a bug that effectively nulled user accounts, which possibly voided thousands of euros of investment. Except, it occured about once or twice a week, across billions and trillions of interactions with the service. This turned out to be a very, very specific synchronization issue in a rarely used and rarely contested component, but very active players were more likely to be affected by it. Very messy.
We couldn't have caught that error without just dumping all errors and a lot more events into a central bucket and then analyzing from there.
But, in a similar way, this kind of event collection in turn allows analysis like: How critical is this thing? An error occuring 20 times a day on a production system for the last 30 days probably isn't that important, else we'd have had a ticket 20 days ago. Not something a debugger can tell.
Exactly. What are the available debugging tools, and what are their pros and cons? That's a much more fruitful question.
Today I had to investigate a bug in a C program. With a conditional breakpoint in gdb I could spot the problem and devise a solution in a matter of minutes.
When I'm debugging SQL (which happens frequently), that depends. If the query is light, I often use a "Prolog-like" approach: make the expression more general until I find the clauses responsible for the bug. On the other hand, if the query is too heavy, I consider spending some time reading the expression very carefully to see if I can find the issue without incurring in a costly execution.
It all boils down to what sorts of problems you have, and which of the available solutions will be more appropriate.
Hard disagree. Most of the time I have a specific failure case in mind when I open my debugger. It's very easy to see which if statements the failure case takes and to run again looking at what variables caused it to take those if statements. Usually the problem becomes obvious after that and it's not unusual for bad variables to be passed into the function; which a debugger can help with as well
I think the real source of friction is from setting a project up to be debuggable ...and keeping that configuration current.
It's a lot easier for single binary projects (like games) than multi- multi- target projects (web frontend, backend, etc) - plus debugging tools are less mature in those worlds.
If debuggers mostly Just Worked in that kind of environment, people would use them far more often.
I am personally also not using debuggers much -- I think within the two last years it was only the backtrace functionality of gdb twice for a crashing C program :)
I still wonder whether it would make sense to learn more about using them, though. My reasons for not using a debugger often are primarily:
* I work with different kinds of systems. Printf or some equivalent is always available, Debuggers always require a system- and programming-language dependent setup and incur an additional learning curve.
* The hardest bugs I have encountered so far were timing-dependent and could not be reproduced when running under a debugger.
Instead I prefer to work with the printfs and some tools which work non-interactively like strace and valgrind. To me, it makes a lot of sense to look at the code when there is a problem and consider the problem in my mind, although I would not go as far as saying that this is in any way fundamentally superior to stepping through the code interactively?
It's been years since I dealt with C/C++. But, from what I remember the debuggers and their integration with the IDEs wasn't great. In particular, it seemed like embedded dev with debuggers was really painful.
I've found that higher level languages (Java/javascript/python/etc) tend to have a much better integration and experience with debuggers than C/C++ did.
Before that I also encountered similar issues at the other end of the performance spectrum: When working with parallelized C++ code that required more than 64 GiB of RAM the overhead of a debugger can make the difference between being able to run the program or not.
I still believe that in almost all cases it is possible to use a debugger even there, but it is always a significant setup required.
And in the modern languages like Java and Python there are stack traces which already greatly help with the “debugging” such that even there I rarely missed the option to step through a program. I never worked with any really complicated Java or Python codebases, though.
I only use a debugger when it's well integrated with my IDE/editor, which is, uh, almost every case there days.
I think debugger vs print is more an ergonomic problem. Yes I can write print and if/else instead of using debugger and conditional breakpoint... but it's often much slower, and I need to be careful to not commit them into production code.
> Doing “something else” means (1) rethinking your code so that it is easier to maintain or less buggy (2) adding smarter tests so that, in the future, bugs are readily identified effortlessly.
I fail to see how debugger prevents me from doing this. To me it's mostly a different way to print/if. It would be very weird if using print prevents me from writing maintainable code.
A debugger does not preclude you from thinking. You just have less information about the process you're trying to understand. If using a debugger makes a portion of your brain shut off, that's a personal problem.
I clicked on the article knowing that the writer would open with something like this:
"I learned to program with BASIC back when I was twelve. I would write elaborate programs and run them. Invariably, they would surprise me by failing to do what I expect. I would struggle for a time, but I’d eventually give up and just accept that whatever “bugs” I had created were there to stay."
People that dont use debuggers for serious software development had their brains develop programming ability without one. They are stuck in their ways, and generally arent aware of it. You cant teach an old dog new tricks
It's what you call a local optimum. Their programming and debugging process is stuck in one. They've never learned to be proficient with a debugger as part of the development process and they won't take the hit to their current productively to learn.
Personally it's all about fighting for observability. If you aren't using all the tools at your disposal you're not as effective as you could be.
The author used a very specific definition of a debugger: interactive line-by-line execution. So this would not include dumping an array using a debugger, adding watchpoints in a debugger, generating stack traces in a debugger, among other features. So a better title would be, "I do not step my code in a debugger."
> I think that my blog post is clear on what I mean by “debugger” which is interactive line-by-line execution. I specifically say that I use tools that are called debuggers. But I do not use them as debuggers.
For my client-side stuff (native Swift/Xcode), I use the Xcode symbolic debugger. Works a treat.
For my server-side stuff, I usually program in PHP, and I've always used echo(), var_dump() and die(). I've just never found all the overhead, preparing the xdebug stuff, and using a symbolic debugger, to be worth it. I find and fix bugs fairly quickly. string dumps have always worked for me.
I sometimes use debuggers such as gdb (occasionally it is very useful), but mostly do not need them and can use printf debugging and valgrind and other methods.
(However, sometimes a debugger is more useful when I am trying to debug a program that I did not write and do not have the source code of.)
I usually roll my eyes when people declare that they don't use debuggers, but this laid out some very good reasons why stepping through your code to solve your bugs comes at an opportunity cost. Namely time, and the chance to fix the problems in your software in a more systemic manner.
I certainly see now why the first instinct to step through gnarly code might not be the best one!
Yeah, the majority of why I once used a debugger was to dinf memory aliasing errors or indexing errors in old c++. Now I mostly use rust, or ”modern c++” and I bjust realized I only end up using the debugger when trying to use bindings to older libraries. In the case of rust, with good bindings, that means I'm commenting out code to get it compile, then stepping through to understand some types that aren't visible, or stepping through a runtime error or panic to see what part of the contract is being violated. Occasionally bad bindings or libraries leave me looking for the source of UB or aliasing issues, but that has become rare. So in greenfield rust projects, yeah, t don't think I would need a debugger... That is an odd thought after using gdb directly for a couple decades.
To be fair, most people who don’t use debuggers have no reason other than habit because they didn’t learn to debug using the debugger.
I started writing JS/PHP over 20 years ago. Back then I debugged with console/echo/var_dump etc. This stuck with me and is still how I debug today. Every time I see someone use the JS debugger I’m impressed by its utility. Still, I always end up reverting to console logs. It works for me and I can’t find a reason to break the habit.
this is one of those weird self-deprivation exercises that some "advanced" people like to engage in
real men don't use debuggers
real men don't use ide's
can program on a piece of paper with both hands tied behind their back
etc...
The actual article is much more polite than this, but I have to deal with these people, who think of themselves very highly, at work every day. Crap i'd get at the code reviews from some bizarre cult-ish schools of thought like "remove all your comments because comments in code never age well", or "logging is distracting and unnecessary", "unit tests spoil programmers so they become sloppy", people actually enjoy imposing these bizarre ideas on others.
The more relevant point, actually, is that the OP seems to do a bunch of multithreaded code and for their use case debuggers may not be utilized to their full extent. But I'm sure some half-wits at work would just read the title and run with it.
Debuggers would be great if they had more speeds than 0 or infinity.
Also, I find debugging JS to be really difficult these days because the ability to set breakpoints and inspect state just doesn't work half the time in Chrome.
What I really want is a "debugger" statement I can turn on/off, at least then it'll break where I want it to.
Debuggers allow you to examine a call stack with their states with a single breakpoint and yes sometimes you don’t event think about where to place the breakpoint like anywhere within a call stack. Another thing is that modern debuggers allow custom code execution which also in a way powerful tool.
* The logs don't have enough info (if there's even logs); and
* I can't write a quick test to reproduce the issue, either because there's no tests or because they're so overengineered that it's not worth the effort.
I’m with you on that. It just doesn’t fit my problem solving process. Some people love them and that’s great. My only annoyance is when those people condescend to me, like I’m choosing life on hard mode.
You do not like using debuggers for stepping through code - your choice. I am of the opposite and do use it quite often. There is absolutely no need for us to change our ways. To each their own.
The problem here is that if the idea of using debuggers becomes an unfashionable niche then the next language won’t provide one at all.
Given how great Java/C#/JS debuggers are, one could hope that every new language will ship with a working debugger, that it would be table stakes.
But there’s a whole generation of working programmers that started their careers in Go and were told debuggers are lame all along, so it’s hard to blame them for not using one.
As for fashion and being told what is cool - I do not give a flying fuck about propaganda. I use what I think fits me and provides best ROI (I am my own business and pay all the expenses).
I've often used "strace" and friends for debugging, especially where there is a lack of source code and ability to build an appropriate binary.
strace is all about logging the state over time. Of course, it's limited to system calls, and not C library stuff or algorithmic computations, but if your application is system-call dependent, such as networking or file I/O, strace can be a great tool.
"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead."
It _may_ tell us something about debuggers. Probably that they are not as useful when it comes to OS kernels.
Just as well, it _may_ tell us something about Linus Torvalds. Probably what tools he is more proficient with, or likes more.
> Debuggers were conceived in an era where we worked on moderately small projects, with simple processors (no thread, no out-of-order execution), simple compilers, relatively small problems and no formal testing.
Unix was conceived in another era too, but modern Unices are totally different and have been adapted to modern times. So have debuggers. When I debug a Java program in IntelliJ I can stop and run threads as I like; OOO execution is transparent; so is both high-level compilation (javac) and low-level JIT compilation by the JVM.
> in order to catch bugs, we often need to be able to run with sufficiently large and representative data sets. When we’re at this point, the debugger is usually a crude tool to use
You'd typically use a debugger to understand why one of the tests with your large dataset fails, after you have already noticed that there is a bug. You don't use a debugger to notice that a bug exists at all.
This article really has a point, but it's buried under crude logic, and the elephant in the room is that "debuggers vs. tests" is a false dichotomy.