Evaluating GPT's Programming Capability Through CodeWars' Katas
File version
Accepted Manuscript (AM)
Author(s)
Wen, Lian
Zhang, Shaoyang
Chen, David
Jiang, Yanfei
Griffith University Author(s)
Primary Supervisor
Other Supervisors
Editor(s)
Cao, C
Chen, H
Zhao, L
Arshad, J
Asyhari, T
Wang, Y
Date
Size
File type(s)
Location
Birmingham, United Kingdom
License
Abstract
Understanding the capabilities and limitations of programming-oriented AI models is crucial. This paper evaluates the programming proficiency of GPT-3.5 and GPT-4 using Codewars coding problems of varying difficulty. The experiments reveal a distinct boundary at the 3kyu level, beyond which these models struggle. This led to proposing a complexity measure that includes problem difficulty and solution time. The research emphasizes the need for validation and creative thinking in AI models to better emulate human problem-solving. Future work aims to refine the complexity measure, enhance AI capabilities, and develop an objective programming problem difficulty measure. These insights are valuable for advancing AI programming and problem-solving abilities.
Journal Title
Conference Title
Knowledge Science, Engineering and Management: 17th International Conference, KSEM 2024, Birmingham, UK, August 16–18, 2024, Proceedings, Part V
Book Title
Edition
Volume
14888
Issue
Thesis Type
Degree Program
School
Publisher link
Patent number
Funder(s)
Grant identifier(s)
Rights Statement
Rights Statement
This work is covered by copyright. You must assume that re-use is limited to personal use and that permission from the copyright owner must be obtained for all other uses. If the document is available under a specified licence, refer to the licence for details of permitted re-use. If you believe that this work infringes copyright please make a copyright takedown request using the form at https://www.griffith.edu.au/copyright-matters.
Item Access Status
Note
Access the data
Related item(s)
Subject
Information and computing sciences
Persistent link to this record
Citation
Zhang, Z; Wen, L; Zhang, S; Chen, D; Jiang, Y, Evaluating GPT's Programming Capability Through CodeWars' Katas, Knowledge Science, Engineering and Management: 17th International Conference, KSEM 2024, Birmingham, UK, August 16–18, 2024, Proceedings, Part V, 2024, 14888, pp. 17-26