Can My Graphics Card Qualify for the American Invitational Mathematics Examination?

Over the past few years, I have been intrigued by how fast frontier Large Language Models (LLMs) have improved. Having trained students for math competitions for over a decade, I paid most attention to results related to math and math competitions. This year, results have been very impressive, with LLMs achieving perfect scores on the International Math Olympiad1 and solving famous mathematical conjectures that have been open for decades2.

However, I have also been a little wary of the corporate hype and intellectual property controversies involved with running online proprietary LLMs. I am not surprised when Google Search, powered by their Gemini AI, can solve a math problem, when the first link in the search result is a solution written in a forum or wiki. Further, I am jaded by ever-increasing costs for using online LLMs, or phrases like “with usage limits that are 4x higher” in subscription plans that do not say what baseline usage limits are.

Parallel to the rise in online LLMs has been the growth of publicly released, local LLMs. While these models are not without their own controversies,3 they are free to download and run on local hardware, even without an internet connection. I decided to take the plunge and see what I could accomplish with local LLMs on my mid-to-high end gaming desktop (luckily bought before RAM prices went through the roof).

Since a local LLM is constrained to local hardware and much less powerful than frontier LLMs, I thought a natural place to start would be a high school math competition, the AMC 10.4 I decided to have the models compete on last year’s 2025 AMC 10A. Spoiler: I was pleasantly surprised by how well various local LLMs performed!

Details and Qualification Results

The models listed below were run on a 4070 Ti Super with 16 GB of VRAM. With the exception of Qwen 3.8, I picked versions of the models (quantizations or ‘quants’ for short) that could run in the 16 GB of VRAM I had available. To me, this qualifies as a happy medium where I could get decent results that help us understand what is possible with local LLMs, while still using models whose performance could be replicated for a hobbyist on local hardware. (Qwen 3.8 was just released, so curiosity got the best of me and I wanted to test its performance, even though the model’s size meant that it ran 8-12 times slower than the others.)

List of models tested:

  • Gemma 4 12B (Q6_K quant)
  • Qwen 3.8 27B (Q4_K_M quant)
  • Qwen 3.5 9B (Q8 quant)
  • DeepSeek R1 14B (Q6_K quant)
  • Phi-4 (Q6_K quant)
  • Qwen 2.5 Math 7B (Q8 quant)

The 2025 AMC 10A had 25 multiple-choice questions.5 Each correct question is worth 6 points and a question that is skipped is given 1.5 points. For purposes of this contest, models were instructed to return the multiple choice answer as part of their response. If they did not (typically from being stuck in a loop), they were given the 1.5 points as if they skipped the answer.6 The results are shown below:

Students needed a score of at least 105 to qualify for the American Invitational Mathematics Examination (AIME) from the 2025 AMC 10A, so the top 4 models all would have qualified for AIME!

Deeper Dive and Further Questions

LLMs measure input and output in terms of tokens. Very roughly speaking, one token can be thought of as one word.7 Developer plans for online LLMs are charged based on token use. While this is not as relevant for local LLMs, lower token use means less time spent and less memory used. Token use for each model, broken down by whether the model gave a correct, incorrect, or blank (N/A) answer is shown below:

First, some cold water on Gemma 4’s performance. Recall that a lot of a model’s expertise depends on its training data and that general conclusions should not be drawn from specific data. Gemma’s performance is so much better (all questions correct, vastly smaller token use) that I suspect that math competitions were used more prominently in training the model. More data could be collected (performance on older competitions, more advanced competitions, etc.). One basic reminder that is especially key to local LLMs: experiment with different models and see which one works best for your use case.8

Second, of note is that the top values of ~4300 in the token usage chart are not random! When asking the models to generate responses, I set a 4096-token limit. This should be plenty for almost any AMC 10 problem solved in an efficient manner; note that all models had an average of less than 2500 tokens for their correct answers. To me, it actually shows the strength of the other top models (Qwen 3.8, Qwen 3.5, and DeepSeek) that their incorrect and blank answers used the maximum numbers of tokens. For most of these responses it is clear when reading the solution that some bizarre things are happening, and the model might be struggling to get the correct answer. Based on these results, we can be confident that a solution should not take that many tokens, so those responses could be flagged for further review. For the lower-performing models (Phi-4 and Qwen 2.5 Math) this was less true. They still have shorter incorrect or blank answers without hitting the token limit.9

For final reference, below is a chart showing token use for the 25 problems given to the models. The color represents the “difficulty” of the problem to the models. Bright green was answered by all 6 while darker orange was missed by most models. (Hover to see each model’s results per problem.)

This data supports the above discussion that easier questions can be answered in fewer tokens. Future exploration could examine specific solutions to notice patterns in how the models solve the problems or if there are specific problem types that models struggle with. Reach out if you are interested in helping me explore more!

Python Scripts and Raw Answers/Solutions

The Python scripts I used to generate the results and charts above are included in a GitHub Repo: click here to access the repository. In particular, the full output from each LLM is available, including their solutions and answers for each problem.

  1. Discussed here: https://www.digitalapplied.com/blog/imo-2026-perfect-scores-ai-benchmark-saturation ↩︎
  2. See here for an interesting article summarizing some recent results: https://www.quantamagazine.org/why-the-legendary-erdos-problems-are-falling-to-ai-20260803 ↩︎
  3. For example, how public and open are these models really? See a discussion here: https://opensource.org/ai/open-weights ↩︎
  4. Shameless plug to an old blog post I wrote: https://areteem.org/blog/amc-10-12-all-you-need-to-know ↩︎
  5. Each question was fed to the model as text, including LaTeX math formatting. Images were skipped, except for one question (15) where the image gave additional information not described in the text of the question. ↩︎
  6. The rationale: It is less damaging to be shown a wrong solution if it is clear it is wrong (since no final answer is given), than a wrong solution with a definitive (wrong) answer. ↩︎
  7. More specifically, one token corresponds to roughly 4 characters. See a more in-depth explanation here: https://blogs.nvidia.com/blog/ai-tokens-explained/ ↩︎
  8. Qwen is typically lauded as one of the best local LLMs for programming, so would probably perform better for programming tasks. Further, models themselves can be run or customized with different settings, parameters, etc. Optimizing each model was not the focus here. ↩︎
  9. In fact, both Phi-4 and Qwen 2.5 Math had examples of giving a numeric answer but not choosing the answer choice. Since the instructions given to the models were clear to choose the multiple-choice answer, this was not manually fixed for scoring purposes. Further, there were cases where the numerical answer given was correct and cases where it was incorrect, so I felt that keeping these scenarios counted as a blank answer was acceptable. ↩︎