Cookies Preferences
Close Cookie Popup
Cookie Consent
By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage and assist in our marketing efforts. More info
So I recently tried using ChatGPT to help me create Unit Test for my code. The idea was to have ChatGPT identify and create the test for me so I could just review them. This is my short proof of concept steps and conclusion.
The first thing to try out was just using the web https://chat.openai.com/ since that was a low effort approach.
The question was:
“Please write unit tests using jest and testing-library/react for this react code <and inserted the entire code for a button component>.”
Which gave me following build error:
So I needed to post the error in a new question and then it managed to give an answer that built. Although 2 tests failed.
I told it there were 2 errors and provided one of them in a new question. It fixed the first problem and actually identified the second. Although the solution to the second problem didn’t solve the problem.
Then there were a couple of back and forth where I got one bad “solution” after the other and sometimes repeated answers. After a while I gave up and fixed the problem on my own and gave the AI a hint to what would solve it. Some more chatting and after a while it almost got it, close enough.
So it gave me many “solutions” that didn’t work but the most interesting was that even if I provided the entire code it still seemed as if it just skimmed the code and gave a guess of a solution. When I pointed out some interpretation error it had made it gave me better answers. And when I pointed out that the answer it had given me had 2 errors it found the issues.
So it was more like a lazy human that took a quick look at the code and just gave an answer it thought would work. On the plus side it is very polite.
I have a friend that has access to ChatGPT-4 through Azures AI, so I asked him to post the same question I posted to ChatGPT. The answer had more instructions on how to actually set up and run the tests, and they ran without errors directly. So there seems to be an improvement between 3.5 and 4 (which I also have heard from other users that have tried both 3.5 and 4).
Since I didn’t have direct access to it myself, this was the only question I posted.
Another way to ask ChatGPT is to use the API instead of the Web Solution and for this Postmanis a great tool. Setting up Postman was really easy, I just followed this step by step. https://quickstarts.postman.com/guide/chatgpt/index.html?index=..%2F..index#0
Again, I posted the same question as before. I used https://www.freeformatter.com/json-escape.html#before-output to escape the code to be able to post the question in JSON.
This also gave me an answer that didn’t work, so I needed to continue the dialog. After a couple of questions, I hit the quota, so I couldn’t continue.
Just to try and get the answer to work, I moved the conversation to the Web service from my first try. Again there was a lot of back and forth with different errors. After a while, it varied its answer between 2 different “solutions” that gave 2 different errors, so I ended up in some infinite loop conversation and just gave up. Another reflection I made here was that it didn’t seem to remember its previous answers.
From this small evaluation, my answer is a Yes-ish. It can help you create the boilerplate code, and from the one test on ChatGPT-4, it seems to be running directly. Although I don’t think (but more testing needs to be done on ChatGPT-4) that you can put a person without technical or testing knowledge to just ask AI to write unit tests and add them to your code. We still need to review the tests if they actually check what we want (technical part) and if they check relevant things (tester part). But a good tester with some technical skills to understand the code could ask specific questions to get correct unit tests and just make sure it seems to add up without any deeper programming skills.
And as a huge plus I think we can get rid of the interview question “have you worked with this specific tool before” since if you can read code you don’t need to know the specific syntax for a specific tool in order to get code and understand it, AI can help you with that.
As a bonus here is the list of Unit Tests for my button component the different approaches gave me (note the difference from web vs API ChatGPT-3.5 which should be similar response).
