Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You want to trust that thing to review your code? It will often give you eloquent explanations, trying to convince you that code has a certain bug, while completely missing the actual bug. You'll be gaslit into chasing down dead ends, distracted from actual issues.

Example: "In JS, create a Promise that is resolved with some emitted value from an EventEmitter, or rejected if an 'error' event is emitted first. Write that concisely, using '.once' and only removing the other event respectively."

Fairly everyday thing in node. Every competent programmer would grasp the intent here pretty quick and write correct code, but somehow it's generally beyond ChatGPT.

Example ChatGPT output (comments mine):

    function eventToPromise(emitter, event) {
      return new Promise((resolve, reject) => {
        emitter.once('error', reject); // memory leak: not removing the other listener
        emitter.once(event, (data) => {
          emitter.removeListener('error', reject);
          resolve(data);
        });
      });
    }
Come on now... I just start the whole prompt over to let it try again:

   function promiseFromEvent(emitter, eventName) {
     return new Promise((resolve, reject) => {
       const resolveHandler = (data) => {
         emitter.removeListener(eventName, errorHandler); // memory leak: should be removeListener('error', errorHandler)
         resolve(data);
       };

       const errorHandler = (error) => {
         emitter.removeListener(eventName, resolveHandler);
         reject(error);
       };

       emitter.once(eventName, resolveHandler);
       emitter.once('error', errorHandler);
     });
   }
Now I prompt "You have a bug in there, fix it."

ChatGPT: "Apologies for the oversight. You are right; there is a bug in the previous code. The issue is that if the 'error' event is emitted first, both the 'error' and 'data' event listeners should be removed, and the Promise should be rejected."

Nope. The error event listener is actually the only listener working correctly... It proceeds to introduce a cleanup function that removes both listeners (despite keeping '.once'), fixing the bug by accident, not intent. Asking it to change it so that the code adheres to the original prompts starts a downward spiral.



Interesting. When I tried it (using GPT4), and told it "but won't that leak memory because we're not removing the other listener?", it gave me:

  const EventEmitter = require('events');
  let eventEmitter = new EventEmitter();

  function eventPromise(eventEmitter, eventName) {
   return new Promise((resolve, reject) => {
    const successHandler = (data) => {
     eventEmitter.off('error', errorHandler);
     resolve(data);
    };

    const errorHandler = (error) => {
     eventEmitter.off(eventName, successHandler);
     reject(error);
    };

    eventEmitter.once(eventName, successHandler);
    eventEmitter.once('error', errorHandler);
   });
  }

  // Use like so:
  eventPromise(eventEmitter, 'data').then(console.log).catch(console.error);
Which seems to be what you're asking for?

https://chat.openai.com/share/89cf1cc8-4ae1-40f6-aa34-0629eb...


Kind of: The point is making it find and correct the mistake itself, instead of spelling out the fix. We want it to replace a competent coder who can do code reviews by themselves, rather than just being a fancy input method that turns English into code.

That said, sometimes you can even get it to write correct code on the first try. It's completely random.


A tool is only as useful as the one wielding it, and it doesn't have to be perfect to be useful. I'm faster with LLM guided development than without in unfamiliar territory. Jack of all trades, LLM-grade proficiency in all of them.

See, the thing is I don't actually want it to take my job, I like having a job, for many reasons. As a translator that turns Jira tickets into English and then turns them into code, if it makes my job easier and I'm more productive, great! If it takes deep domain knowledge before it's useful and it can be tricked into missing subtle things, that sounds like job security.


90-95% of programmers I've worked with can't "grok" asynchronous or multi-threaded programming, and when forced to write async code it is littered with bugs.

ChatGPT doesn't need to find 100% of bugs in 100% of code out there.

Think of it as a "better linter". It's cheap, and finding just 50% of bugs would be an enormous step up for quality in most code bases.


> Think of it as a "better linter".

No way. A linter is deterministic. ChatGPT is all over the place and, for me, it’s wrong almost every time I ask it anything. I wouldn’t trust it to tell me the ingredients in a pepperoni pizza. I’m definitely not letting it give me programming advice.


Ironically, I had a lot of trouble with a particular recipe (pan-fried gyoza with a crispy bottom), and it was only GPT 4 that gave me a working recipe!

The lack of determinism can be considered a type of strength. Run it multiple times! It might find different bugs each time.

Humans are the same, by the way. If you show a random set of programmers random snippets of code, you'll get a non-deterministic result. They won't all find all of the bugs.


> The lack of determinism can be considered a type of strength. Run it multiple times! It might find different bugs each time.

I’ve only tried it with code a little bit, but what I find is that it gives me hallucinations that I need to spend time figuring out. I don’t know what it’s saying, because it’s gibberish, and then I have to spend my time figuring out it’s not accurate. I don’t want to run on that treadmill.

I’m guessing software will continue the trend of getting less reliable as more people are willing to generate it via an AI.


Did you try GPT 4? It hallucinates a lot less.


You said, "You'll be gaslit into chasing down dead ends, distracted from actual issues."

When the ideal situation is that your Code-writing bot is attempting to gaslight your Project-management bot, and they go around in tireless circles, 24/7 arguing with each other, while their human masters are sleeping or not paying attention :)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: