Top 10 Things I Have Learned from ChatGPT

Michael Hansen, St. Albans School

In addition to teaching AP Statistics for 25 years, I have taught high-school computer science for about 13 years. This summer, I used ChatGPT to help me write a Python application of about 3,000 lines. Here are my top 10 takeaways from that experience.

1. Artificial intelligence is a game-changer for software development.
   I cannot imagine writing code in the future without using a chatbot or similar AI tool. It would be like writing a long document without a spell-checker or navigating a car without GPS. Possible? Yes, but not what most modern people would do.
2. We need a complete re-think of how to teach computer science,
   or at least the coding aspect of computer science. Coding is no longer the challenge students thought it was. Indeed, ChatGPT writes beautiful code for all but one of the exercises in the Java textbook I use for my beginning students. Paste the text of the exercise into the ChatGPT query window, press enter, and voila! In most cases, you receive code an A+ student would envy. Not only that, but ChatGPT’s code includes coherent comments to explain what it is doing. (Try getting your beginning students to do that!)
3. There is no longer a barrier to coding for anyone.
   The reason students formerly thought coding was challenging is it seemed arcane. Now that coding is easy, it hardly qualifies as arcane. In the past, some people thought coding was key, meaning every high-school student should learn how to code. I never thought that, but I did not contradict people who did because they supported a cause near to my heart: computer science education for all.

   In light of the new AI chatbots that will revolutionize how code is produced, I would say computer science is key, and understanding how to formulate requirements is key, but coding, per se, is not. It is akin to when knowing what the standard deviation measures is key, but knowing how to compute s.d. by hand is not. I still make my students compute s.d. manually a few times in hopes they will see how larger errors contribute more to the statistic, but it is hard to make a case that s.d. computation is a key skill for statistics.

4. Some ‘old school’ assessments may remain valid, but definitely not as take-home exercises.
   One assessment type I use to gauge student fluency with terminology and notation—namely, fill-in-the-blank—is tailor-made for ChatGPT, which was built from the outset as a conditional probability engine for finding the most likely word or phrase to finish a thought. On a fill-in-the-blank assessment, ChatGPT easily outscores even my best students. Nevertheless, I believe the skill being measured is important for humans to learn, even if AI bots can run rings around them. Tellingly, ChatGPT is already ‘better’ than many students at scoring well on the measures they care most about, namely the College Board’s Advanced Placement exams … and the technology will keep improving each month.
5. If you are not happy with ChatGPT’s initial response, ask it to try again.
   On the second or third try, it almost always does better. However, be aware that ChatGPT always responds with a confident tone, even when it is wrong. On my summer project, I had to reformulate one set of function requirements more than a dozen times, but it was still worth it, since the code ChatGPT produced for me would have taken me hours to research, write, and debug on my own.
6. The key challenges for students remain the same:
   formulating the requirements of a problem in sufficient detail and clarity; testing/recognizing problems in the code (whether written by humans, AI, or a combination); understanding what the computer is actually doing with the code; using that knowledge to debug and refine software; and designing an entire system to begin with. AI helps with the debugging, but the other challenges remain uniquely human—at least for now.

   I am old enough to remember sales pitches 40 years ago for software products that would supposedly make developers obsolete. What happened instead is the field evolved to use different tools, but not fundamentally different skills. Something hard always remains; there is no silver bullet for software development. The classic book on the subject, The Mythical Man-Month by the late Frederick Brooks, is still in print almost half a century after its release.

7. Debugging is easier.
   For the most common student bugs (e.g., fencepost errors, lines inside a loop that should be outside the loop or vice versa, improperly copied-and-pasted code that should have been encapsulated, etc.), ChatGPT will usually find the bugs if you paste your code into the query window, even if you do not describe the symptoms. There is grave danger here, because I believe the most educational experience in computer science is the adventure (cough, cough) of spending hours tracking down logical errors in a program and learning—through the school of hard knocks—how to write better code. I fear students, if deprived of the education serious debugging entails, will never fully mature as computer scientists. Then again, I may be as misguided as people in the 1970s who thought students would lose something important by not extracting square roots with pencil and paper.
8. ChatGPT will reveal to everyone what computer scientists have known for decades:
   translating requirements from English into computer code was never the central problem. (Proof: Translating detailed requirements from English into code was a task historically given to junior programmers. Now, with the advent of ChatGPT, that task will become largely a matter of copy-and-paste.) The hardest task of all is requirements analysis. Once the requirements of a problem are well understood and formulated at a sufficient level of detail, the problem is mostly solved. For large systems, architecture and communication among the developers (as Brooks noted) become daunting challenges, as well.
9. Even though ChatGPT will write code, I need to persuade (and insist) my students learn how to write their own code.
   A certain set of core knowledge is essential in any field. Yes, you could skip learning “Happy Birthday,” but you would look rather pathetic if you had to look up the words and music on your phone every time your coworkers gathered around a cake in the company break room. In computer science, the best way to formulate detailed requirements is to write pseudocode, and it is hard to write good pseudocode without having written real code.
10. I need to write new grading rubrics that recognize that much or all of the code in student projects may have been written by ChatGPT,
    regardless of what I tell the students to do and regardless of any honor-related penalties I may threaten. My time-honored trick of writing project assignments with requirements too specific to permit plagiarism from the internet will no longer work, since ChatGPT has no trouble writing code to satisfy virtually all student-level project requirements.

A colleague proposed that for take-home projects, I could have students write detailed requirements and submit those to me without code. The grade would then be based on the quality of the code ChatGPT writes in response and the quality of the enhancements the students suggest (and justify). That idea holds promise because it recognizes the reality of AI in the workplace of the future and leverages that reality as part of an authentic assessment.

Michael Hansen studied mathematics and computer science at Bradley University (BA) and the University of Illinois at Urbana-Champaign (MS). After working 12 years in the computer field as a government contractor, he joined the faculty of St. Albans School, an independent school for boys in Washington, DC. He plays violin, viola, and piano semi-professionally.

Editor’s Note: This article originally appeared in the September 2023 issue of Amstat News. Hansen also discussed ChatGPT in a recent K–12 chat/webinar.