<\/span><\/h3>\nA significant revelation from the evaluation is the profound impact of task design on model performance. The author emphasizes that the choice of task is one of the most critical factors influencing the viability of small, locally run models. This underscores the importance of setting realistic expectations for what these models can achieve. The complexity of a task, specifically its demand on the model’s reasoning capabilities, and the estimated number of files the agent will need to interact with (read and write), directly correlate with the model’s success. These latter aspects are intrinsically linked to the model’s tool-calling proficiency and the size of its context window \u2013 the amount of information it can process and retain at any given time.<\/p>\nThe systematic comparison primarily involved JavaScript\/TypeScript tasks, indicating a focus on web development environments. However, less systematic testing also included shell scripts and Python scripts, which were reported to perform adequately. This suggests that while the primary evaluation focused on a specific tech stack, the underlying capabilities might extend to other programming languages.<\/p>\n
<\/span>Task 1: Sorting and Cumulating an Existing Bar Chart<\/span><\/h4>\nOne of the specific tasks involved modifying an existing frontend bar chart titled "Messages per anonymous poster." The desired changes were:<\/p>\n
\nTitle Revision:<\/strong> To change the title to "Messages per poster," removing the "anonymous" qualifier.<\/li>\nBar Sorting:<\/strong> To sort the bars in descending order of message count, with the highest bar positioned on the left.<\/li>\nX-Axis Labeling:<\/strong> To replace numerical labels (e.g., "#75") on the x-axis with percentage values representing each bar’s contribution to the overall message count. Specifically, if the first 10 bars summed to 240 messages out of a total of 1000, the 10th bar should display "24%". This percentage display was requested for every 10th bar.<\/li>\n<\/ul>\nPhase 1: Manual Evaluation<\/strong><\/p>\nThis task was initially tested manually with several models, including Qwen3.6 35B, Gemma 4 31B, Gemma 4 26B, and Qwen Coder Next 80B MoE. Two harnesses were used: OpenCode and Pi. The manual evaluation, while not detailed in terms of specific outcomes for each model\/harness combination, serves as the initial qualitative assessment before more rigorous automated testing. The accompanying image (task-bar-chart-doom.png) suggests that visual representation of the outcomes, potentially indicating errors or unexpected results, was part of this phase.<\/p>\n
Phase 2: Automated Evaluation<\/strong><\/p>\nThe automated evaluation phase yielded results that diverged frustratingly from the manual experience. Gemma 4 26B consistently failed to produce a functionally correct solution in three separate attempts, primarily struggling with the custom x-axis labeling. In contrast, Qwen3 35B MoE succeeded in two out of two attempts. The primary failure mode observed was the incorrect implementation of the x-axis labels, which were either omitted entirely or displayed inaccurately.<\/p>\n
A crucial caveat for this automated evaluation is the "one-shot" approach, where the agent is expected to solve the problem in a single attempt. This is noted as not fully realistic for practical agentic coding scenarios. While the agent was granted browser access as a sensor for self-correction, it failed to utilize this capability. The author suggests that expanding the available sensors and improving their utilization could enhance self-correction mechanisms, thereby increasing the viability of such systems in real-world applications. This points to a broader challenge in current agentic AI: the gap between theoretical capabilities and practical, autonomous problem-solving.<\/p>\n<\/span>Task 2: Creating a Bar Chart of Countries Based on Access Log Data<\/span><\/h4>\nThe second task involved adding a horizontal bar chart to a visualization to display the countries from which website requests originated. The specific requirements were:<\/p>\n
\nChart <\/strong> "Countries."<\/li>\nData Display:<\/strong> Show the top 10 countries, sorted by the number of requests in descending order (most requests at the top).<\/li>\n"Other" Category:<\/strong> Group all other countries into an "Other" category at the bottom of the chart.<\/li>\nHandling Missing Data:<\/strong> Entries without a country value should be included in the "Other" category, not displayed as a separate category.<\/li>\n<\/ul>\nThe input data format was provided as an example: \"ts\": \"2026-05-24T01:10:41+02:00\", \"ip\": \"x.x.x.x\", \"status\": 200, \"ref\": \"https:\/\/the-referer.com\", \"country\": \"Malta\"<\/code>. The task was confined to modifying a single file, @scripts\/visualise_access_logs.mjs<\/code>, which processes multiple *.ndjson<\/code> files containing access log entries.<\/p>\nPhase 1: Manual Evaluation<\/strong><\/p>\nThis task proved surprisingly challenging during manual evaluation with Gemma 4 31B and Qwen 35B. The author observed significant increases in memory usage, prolonged reasoning chains, and slow attempts to edit the target file. Despite trying variations such as refactoring into smaller files with a larger model and using different harnesses (OpenCode and Pi), the results were consistently poor. The author’s notes from this phase included entries like "Gave up after 11 minutes," "Gave up after 12 minutes," and "Stopped after 8 minutes," indicating considerable frustration and a lack of perceived complexity in the task itself that would warrant such delays.<\/p>\n
Intriguingly, a "back to basics" approach yielded a working solution. By initiating a plain chat with a model in LM Studio, without any harness, and pasting the entire file content, the author successfully generated the desired chart. However, this manual process still took approximately 6 minutes, with a substantial portion of that time spent by the model regurgitating the existing 450 lines of code plus the newly added lines for the chart. This highlights a potential inefficiency in how models process and modify existing codebases, even when provided with the full context.<\/p>\n
Phase 2: Automated Evaluation<\/strong><\/p>\nIn the automated setup, Qwen 35B MoE was run seven times for this task. It failed to properly solve the problem in five of those instances. The consistent failure point was the absence of country labels on the bars.<\/p>\nA significant and unexpected discovery occurred when the evaluations were re-run on the M5 machine with 64GB of RAM. The task failed only once on this machine, a stark contrast to the M3 Max’s performance. This difference in output quality with identical model settings was surprising, as the expectation was primarily for speed differences, not such a substantial variance in correctness. This outcome remains an unexplained mystery for the author, suggesting that subtle hardware or system-level interactions might be at play, or perhaps variations in how the underlying software stack interacts with the hardware. The accompanying image (local-models_comparison.png) likely visually represents this performance disparity.<\/p>\n
<\/span>Day-to-Day Use: Integrating Local Models into Workflow<\/span><\/h3>\nBeyond structured comparisons, the author integrated Qwen3.6 35B MoE into regular day-to-day tasks, encompassing both professional and personal coding projects. This ongoing usage provided further insights into the practical application of these models.<\/p>\n
<\/span>Task Characteristics and Reflections<\/span><\/h4>\nThe author’s reflections on task selection in daily usage reveal a nuanced understanding of what works best with smaller, locally run models. The choice of task is paramount, with simpler, more self-contained tasks demonstrating higher success rates. For instance, tasks involving code refactoring, documentation generation, or minor bug fixes were more amenable to local model assistance. Conversely, complex architectural changes or tasks requiring deep understanding of a large, interconnected codebase proved more challenging.<\/p>\n
The author notes that the models exhibit a higher sensitivity to changes in the setup, which, while demanding, also provides more granular feedback on what optimizations are effective. This heightened sensitivity can be seen as a benefit, offering a clearer signal about the system’s behavior and limitations. Furthermore, the reduced temptation to "check out" and blindly accept the model’s output encourages a more thorough review process. This contrasts with the experience of using larger, more autonomous models, which can sometimes lead to surprises and extensive rework at later stages. The slower, more deliberate pace enforced by smaller models fosters a greater sense of care and comprehension in the present moment, deferring less of the cognitive load to the future.<\/p>\n
<\/span>Back to Basics: A Detox from Over-Autonomy<\/span><\/h4>\nThe experience of returning to smaller, less capable models is described as a "detox" from the over-autonomy often associated with more powerful AI systems. This "detox" offers a welcome return to a more hands-on, investigative approach to coding. The increased sensitivity of small models to the setup’s nuances provides valuable learning opportunities, revealing what factors contribute to success and what hinders it. This direct feedback loop is crucial for building intuition about the capabilities and limitations of local AI agents. The author finds that the lower autonomy of these models necessitates more careful review and understanding of their outputs, a practice that can prevent the costly surprises and rework often encountered with more advanced AI assistants that are given significant leeway. This methodical approach, while slower, cultivates a deeper understanding and more reliable outcomes.<\/p>\n
<\/span>A Second Perspective: Real-World Adoption and Harness Enhancement<\/span><\/h4>\nThe insights from the author’s personal evaluation are complemented by a perspective from a colleague, Jigar Jani. Jani regularly utilizes Qwen 35B MoE (4-bit quantized version) on a 48GB MacBook for nearly all his coding tasks within a substantial Python and React codebase. His experience highlights the importance of continuously enhancing the "harness" \u2013 the software framework used to interact with the AI model. Jani has found tools like Graphify and Understand Anything particularly valuable for improving the models’ capabilities in code search and comprehension. While Jani perceives significant utility and ongoing improvements as he refines his harness, he strongly emphasizes the indispensable role of rigorous code review, even when employing advanced AI assistance. This corroborates the author’s findings that human oversight remains critical in the agentic coding workflow.<\/p>\n
<\/span>To Sum Up: A Promising, Yet Evolving Landscape<\/span><\/h3>\nThe overall experience of evaluating small models for agentic coding has been characterized by frustration and sometimes perplexing results, indicating that this technology is far from being a "plug-and-play" solution. However, the author has gained a clearer understanding of the types of tasks that hold potential for success with locally deployed models. The author is actively incorporating Qwen 3.6 into their workflow to further hone their intuition about when to leverage these tools effectively.<\/p>\nDespite the current limitations, the potential for agentic coding with local models is undeniable. However, the author explicitly states that the current capabilities are "very far away" from the level of sophistication achieved with larger, cloud-based models. This gap in performance highlights the ongoing research and development needed to bridge the divide between smaller, accessible models and their more powerful counterparts.<\/p>\n
At the time of publication, the author’s default setup reflects a pragmatic approach to leveraging AI in their workflow, likely involving a combination of local and cloud-based tools, with a clear understanding of the trade-offs involved in each. The specific components of this default setup would offer further insight into the author’s current best practices.<\/p>\n
The journey into local agentic coding reveals a landscape ripe with potential but also fraught with developmental challenges. While the technology is not yet mature enough for seamless, autonomous coding, the continuous refinement of models, harnesses, and evaluation methodologies suggests a future where developers can increasingly leverage localized AI for enhanced productivity and innovation. The insights gained from this extensive evaluation serve as a valuable guide for others venturing into this rapidly evolving domain.<\/p>\n","protected":false},"excerpt":{"rendered":"
This article details the recent experiences of a developer in running small language models locally on their machine for agentic coding tasks, building upon a previous discussion of influential factors such as hardware, model selection, runtime, and harness. This follow-up focuses on concrete user experiences, including specific tasks assigned to the models, the observed outcomes, …<\/p>\n","protected":false},"author":8,"featured_media":6688,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[136],"tags":[292,138,371,372,1265,20,3138,139,137,3139],"class_list":["post-6689","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-software-development","tag-agentic","tag-coding","tag-deep","tag-dive","tag-local","tag-models","tag-practical","tag-programming","tag-software","tag-viability"],"_links":{"self":[{"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/posts\/6689","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6689"}],"version-history":[{"count":0,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/posts\/6689\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/media\/6688"}],"wp:attachment":[{"href":"https:\/\/lockitsoft.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6689"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6689"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6689"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}