This is a repository for my Text Mining course final project. All the code generated can be found under the eponyme directories. Most of the code used for the project can be found in TextMiningproject.ipynb, with some helper functions in utils.py. The images used in the report are in the images directory.
Large Language models (LLMs) have demonstrated remarkable advancements in natural language processing, but their performance on competitive programming tasks remains an open question as benchmarks usually focus on more traditional coding tasks. Competitive programming tasks require not only linguistic understanding but also algorithmic and mathematical reasoning, precision and efficiency. This paper evaluates the problem-solving capabilities of 5 small size LLMs on a set of competitive programming problems from the publicly available coding platform Open.Kattis. Each code generated is evaluated on compiling correctness, success on sample input/output combinations and finally on the online judging platform directly. The results reveal a significant gap between the ability of the LLMs to produce correct code, which is pretty good, and the ability of the LLMs to produce code that actually solves the competitive programming tasks, which is not very good. These finding offer a valuable insight to the current state of most LLMs, which is that although they appear to produce valuable outputs, they actually lack strong reasoning capabilities to produce qualitative outputs.
I am concious that using generative AI might go against the Kattis website TOS. This is for educational purposes only, I used this project to learn to fine tune a LLM and uses LLMs in a real cases scenario. All code generated was submitted on a seperated account from my main account.*
Further more, here are 2 extratxs from the Kattis term of services that are relevant for this project:
2. License for content
Permission is granted to download the materials (information or software) on Kattis's website for viewing. This is the grant of a license, not a transfer of title, and under this license you may not:
- copy the materials, unless a specific license allowing you to copy the content is shown;
- use the materials for commercial purposes, or for any public display (commercial or non-commercial) without prior agreement with Kattis;
- attempt to decompile, reverse engineer or circumvent any security features in Kattis;
- remove any copyright, license or other proprietary notations from the materials; or
- transfer the materials to another person, "mirror" or "proxy" the materials on any other server.
This license shall automatically terminate if you violate any of these restrictions and may be terminated by us at any time. Upon the termination of this license, you must destroy any downloaded materials in your possession whether in electronic or printed format.
6. Automated access
You may operate a tool to automatically access content from Kattis, such as a search engine, bot, scraper, data mining tool, bulk downloader, wget, submission client or similar. However, you are subject to the following rules when doing so:
- a. You must provide a descriptive user-agent when connecting to Kattis
- b. You must obey robots.txt, X-Robots-Tag and relevant tags
- c. You must rate limit your access not to cause adverse effects on our services
- d. You must make it clear how to contact you in the user agent string, by supplying an e-mail or URL with contact information
- e. We reserve the right to ban any form of automated access at our discretion