Developers turn to chatbots like ChatGPT from OpenAI to help them code. So far so good, but a recent study reveals there is a problem. 52% of programming responses generated by ChatGPT are incorrect. That's right, you read that right, 52%!
For the study, researchers analyzed 517 questions on Stack Overflow and analyzed ChatGPT's attempted response.
We found that 52% of ChatGPT responses have incorrect information, 77% of responses are more detailed than human responses, and 78% of responses have varying degrees of inconsistency compared to human responses
It should be noted that for the study the researchers chose GPT-3.5 instead of GPT-4, the most recent version of the language model at the time of the study. It should be noted that the researchers also carried out parallel tests with GPT-4 and concluded that, although the newer model performed “slightly better”, both had a high inaccuracy rate.
