How Students Use - And Don't Use - ChatGPT

Generative artificial intelligence (AI) promises numerous benefits for accountants, but concerns exist about its possible negative impact on learning in the classroom. Accounting educators are grappling with the perceived threat that generative AI tools will erode learning and jeopardize the quality of future accounting professionals.

Is the threat real? We have gathered experiment and survey results from students in various accounting courses at Sam Houston State University to provide insight into whether student use of one generative AI tool (ChatGPT) helps or hinders learning, how students use it in accounting courses, and their perceptions of the tool.

The experiment: ChatGPT and learning outcomes

In fall 2023, business students in three courses — "Introduction to Financial Accounting," "Managerial Accounting," and "Accounting Information Systems (AIS)" — could choose to use ChatGPT to help complete selected assignments in their respective courses. Quantitative assignments in the "Introduction to Financial Accounting" and "Managerial Accounting" courses included problems on transaction analysis, cost behavior, and cost-volume-profit (CVP) analysis. The assignments in AIS involved students first completing enterprise resource planning (ERP) software tutorials and then answering internal control questions about the tutorials.

Students could use ChatGPT in answering the questions portion of these qualitative assignments in AIS, and they were then tested on material covered in those assignments.

In comparing the test grades of the 59 students (34.8%) who chose to use ChatGPT to the 111 students (65.2%) who chose not to use ChatGPT, we found no difference in test score averages between the two groups. The result indicates that ChatGPT did not affect academic performance. This lack of difference was consistent across majors, class modality (online vs. in-person), assignment type (qualitative vs. quantitative), and course levels (introductory vs. upper-level).

Responses to open-ended questions answered after completing the assignment about the use or nonuse of ChatGPT were then used to generate more specific survey questions. These questions were asked in a follow-up study the next semester in the same courses to understand how students use and perceive ChatGPT.

The survey: Student perceptions of ChatGPT

In spring 2024, a new group of students in the three courses were again encouraged to use ChatGPT for selected assignments and then asked to complete a survey. The survey featured a series of 7-point Likert scale statements in which students indicated their level of agreement, where 1 = "strongly disagree" and 7 = "strongly disagree."

There were 129 attempts (58.6%) to use ChatGPT to complete the assignments, with the remaining 91 (41.4%) not making use of the technology. The respective percentages of usage and nonusage nearly flipped from the previous semester, which indicates students' growing embrace of ChatGPT (a new version of ChatGPT, 4.0, was available that semester, but only one student chose to use it).

The themes that emerged from the survey responses were the use of ChatGPT as a supplement to class material, the helpfulness of ChatGPT, trust in ChatGPT output, and the question whether its use is perceived as cheating.     

Use of ChatGPT as a supplement to class material

Considering the promising benefits of generative AI, it is interesting how few students in the spring relied heavily or exclusively on the use of ChatGPT. Students reported a preference for traditional class resources, such as class notes and their textbooks, to complete the assignments.

Of the 129 attempts using ChatGPT in the spring, 43.4% used traditional class resources more than ChatGPT, 43.4% used both resources equally, and 10.1% used ChatGPT more than traditional class resources. Only 3.1% of the 129 attempts that made use of ChatGPT relied exclusively on it. (See the graphic, "Use of ChatGPT in Spring 2024 for Accounting Assignments").

The survey also highlighted preferences for ChatGPT use between majors and on the type of assignment completed. (See the chart, "Student Assessment of ChatGPT.")

Accounting majors relied on ChatGPT notably less than nonaccounting majors (question 1: 2.8 vs. 3.4, respectively). Additionally, students reported far less reliance on ChatGPT for qualitative assignments compared to quantitative ones (question 1: 2.5 vs. 3.5, respectively). And there was a stronger reliance on class resources for qualitative assignments than for quantitative assignments (question 2: 4.9 vs. 4.0, respectively).

Student feedback indicates ChatGPT was used more for solving basic accounting calculations than for assignments requiring deeper critical thinking skills.

Based on these results, it appears that the students perceive ChatGPT to be a better supplement than a substitute for traditional class resources. The following student comments convey this sentiment:     

"Whenever I used ChatGPT — I ended up cross-referencing the responses with notes and the textbook, and I found that the majority of the time there [were] minor differences."

"I am very interested in AI but prefer to use its responses as more of a supplement and aid to my work or to help me find a jumping-off point if I am at a loss."

Perceived helpfulness of ChatGPT

Students, in general, reported that ChatGPT was somewhat helpful. They found it more helpful for quantitative assignments than for qualitative ones. Further, nonaccounting majors found it more helpful than accounting majors. This is emphasized by students' willingness to use ChatGPT in nonaccounting courses (question 11: 4.6) versus other accounting courses (question 10: 3.6).

Students had mixed feelings regarding the helpfulness of ChatGPT with finding correct answers (question 6: 3.5) and specific formulas (question 8: 3.1). They were just above neutral (question 4: 4.2) regarding the perceived helpfulness of the technology with learning the material covered in the assignment. Instead, they reported that ChatGPT helped clarify the understanding of questions (question 7: 4.7), provided detailed descriptions (question 5: 5.1), and improved the quality of their responses to the assignments (question 3: 4.6).

Key student comments reveal the overall thoughts about the helpfulness of the technology:

"I mostly use ChatGPT to expand my thoughts and gather ideas."

"I have used ChatGPT as a resource when it comes to understanding concepts."

Is using ChatGPT cheating?

Interestingly, students, in general, were neutral (question 12: 4.0) about whether using ChatGPT was considered cheating. Also interesting, nonaccounting majors would be more willing than accounting majors to use it only if instructed by their professors (question 13: 5.4 vs. 4.9, respectively).

The following student comments provide thoughts about whether the use of AI is cheating:

"I don't see ChatGPT as a useful tool because it does the thinking for you, and so I see it as cheating."

"Where exactly is the line between you learning something and being able to apply it later on and AI just doing the work for you?"

Trust (or lack of) in ChatGPT

In another finding, students reported a mediocre level of trust in the output provided by ChatGPT (question 14: 3.7). However, this trust varied significantly between nonaccounting and accounting majors (question 14: 4.1 vs. 3.2, respectively) and on quantitative versus qualitative assignments (question 14: 4.2 vs. 3.0, respectively).

A student who used ChatGPT to complete both a quantitative and qualitative assignment expressed their skepticism, commenting:

"After using it a couple of times for this class and Dr. Henderson's class, I found that I trust myself more than I trust ChatGPT."

Despite this lack of trust, students, regardless of major and assignment type completed, reported a higher level of confidence in using ChatGPT for a future assignment (question 15: 4.6).

The table below shows how the 129 Sam Houston State University students who used ChatGPT for one of three accounting classes in spring 2024 assessed the generative AI tool. The results are broken down by assignment type.

Student assessment of ChatGPT

What does this mean for accounting and education?

Based on these findings, the fear of eroding knowledge learned in the accounting classroom due to generative AI may be overstated. With AI being increasingly embraced in the profession, this is another opportunity for professionals to inform and collaborate with educators, so they understand just how the technology is being used in practice and how to integrate it effectively in the classroom.

Accounting educators should aim to create efficiencies in learning, just as AI promises to streamline accounting tasks.

The introduction of AI is perhaps reminiscent of the introduction of spreadsheets, which transformed the accounting profession but did so by massively expanding the value and importance of accounting professionals. Accounting educators should seize the opportunity to prepare students for the evolving future of the profession, and this includes AI.

The following student comment emphasizes that ChatGPT should and will be a valuable tool among other tools that accounting professionals should embrace to continue providing valuable professional services:

"ChatGPT is a great resource but should never be the only resource."

— Jacqui Wukich, CPA, Ph.D., is an assistant professor of accounting at Baldwin Wallace University in Berea, Ohio. Cassy Henderson, Ph.D., is a professor of accounting and interim chair of the accounting department and Ronald J. Daigle, Ph.D., CPA, CFE, CGMA, is a distinguished professor of accounting at Sam Houston State University's College of Business Administration in Huntsville, Texas. To comment on this article or to suggest an idea for another article, contact Jeff Drew at Jeff.Drew@aicpa-cima.com.