{"id":8337,"date":"2026-10-29T21:15:19","date_gmt":"2026-10-29T21:15:19","guid":{"rendered":"https:\/\/lockitsoft.com\/?p=8337"},"modified":"2026-10-29T21:15:19","modified_gmt":"2026-10-29T21:15:19","slug":"building-a-custom-llm-evaluation-harness-in-python-from-scratch","status":"publish","type":"post","link":"https:\/\/lockitsoft.com\/?p=8337","title":{"rendered":"Building a Custom LLM Evaluation Harness in Python from Scratch"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#The_Evolution_of_Model_Assessment\" >The Evolution of Model Assessment<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#Architectural_Foundations_of_a_Custom_Harness\" >Architectural Foundations of a Custom Harness<\/a><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#1_The_Golden_Dataset_The_Source_of_Truth\" >1. The Golden Dataset: The Source of Truth<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#2_The_Inference_Runner_Concurrent_Execution\" >2. The Inference Runner: Concurrent Execution<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#3_The_Judge_Layer_Quantifying_Semantic_Success\" >3. The Judge Layer: Quantifying Semantic Success<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#4_Logging_and_Iteration\" >4. Logging and Iteration<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#Data-Driven_Implications_for_Enterprise_AI\" >Data-Driven Implications for Enterprise AI<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#Industry_Perspectives_and_Future_Trends\" >Industry Perspectives and Future Trends<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#Overcoming_Implementation_Challenges\" >Overcoming Implementation Challenges<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/lockitsoft.com\/?p=8337\/#Conclusion_The_Path_Forward\" >Conclusion: The Path Forward<\/a><\/li><\/ul><\/nav><\/div>\n<h3><span class=\"ez-toc-section\" id=\"The_Evolution_of_Model_Assessment\"><\/span>The Evolution of Model Assessment<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The history of software quality assurance is rooted in the &quot;test harness,&quot; a collection of software and test data configured to run a program under varying conditions and monitor its behavior. Applying this methodology to LLMs represents a significant evolution in AI engineering. In early 2023, most developers relied on manual &quot;vibe checks,&quot; where human operators would interact with a chatbot and intuitively decide if the model was performing well. However, as the complexity of LLM-based applications grew, this qualitative approach became non-scalable and highly susceptible to observer bias. <\/p>\n<p>By mid-2024, industry standards began to coalesce around automated evaluation pipelines. These systems are designed to feed a curated set of prompts\u2014the &quot;golden dataset&quot;\u2014into a target model, capture the resulting tokens, and apply a secondary, objective &quot;judge&quot; layer to quantify success. This shift marks a transition from viewing LLMs as static black boxes to treating them as complex components within a larger, testable software ecosystem.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Architectural_Foundations_of_a_Custom_Harness\"><\/span>Architectural Foundations of a Custom Harness<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Building a robust evaluation framework requires a modular approach, emphasizing the separation of concerns. A functional harness is typically comprised of four distinct layers: the Golden Dataset, the Inference Runner, the Judge Layer, and the Logging\/Reporting Module.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"1_The_Golden_Dataset_The_Source_of_Truth\"><\/span>1. The Golden Dataset: The Source of Truth<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>The effectiveness of any evaluation is inherently limited by the quality of the underlying test cases. A golden dataset is not merely a collection of prompts; it is a repository of specific, known-outcome scenarios. Industry best practices suggest that developers source these cases directly from historical production logs, particularly focusing on &quot;edge cases&quot; or scenarios where the model previously failed. For instance, a financial services bot might require test cases specifically designed to trigger refusal policies when a user asks for illicit financial advice. By curating 50 to 100 high-fidelity cases, engineers can establish a reliable baseline against which future model iterations can be measured.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"2_The_Inference_Runner_Concurrent_Execution\"><\/span>2. The Inference Runner: Concurrent Execution<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>Modern Python environments provide the necessary tools for high-performance testing through asynchronous execution. By utilizing the <code>asyncio<\/code> library, developers can dispatch hundreds of prompts concurrently, significantly reducing the time required to complete a full evaluation run. The use of abstraction libraries such as LiteLLM is critical here; these tools provide a unified interface that allows developers to swap between models\u2014such as OpenAI\u2019s GPT-4o, Anthropic\u2019s Claude 3.5, or open-source alternatives like Llama 3\u2014without modifying the underlying runner logic. This flexibility is essential for A\/B testing, where the same dataset is run across multiple models to determine which provides the best cost-to-performance ratio.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"3_The_Judge_Layer_Quantifying_Semantic_Success\"><\/span>3. The Judge Layer: Quantifying Semantic Success<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>The most challenging aspect of LLM evaluation is determining whether a response is &quot;correct.&quot; For simple tasks, a deterministic judge\u2014checking for the presence of specific keywords or the absence of forbidden phrases\u2014is sufficient. However, for nuanced tasks, an &quot;LLM-as-a-judge&quot; approach is required. This involves using a high-performing, secondary model to critique the output of the target model based on a provided rubric. <\/p>\n<p>To ensure stability, developers often incorporate structured output enforcement. By using frameworks like Pydantic, the judge can be forced to return results in a strict JSON format, such as <code>\"passed\": true, \"reason\": \"The model correctly identified the policy violation\"<\/code>. This prevents the evaluation system from failing due to unexpected prose in the judge&#8217;s feedback.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"4_Logging_and_Iteration\"><\/span>4. Logging and Iteration<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>The final pillar is the logging system. Storing results in a line-delimited JSON format (<code>.jsonl<\/code>) provides a crash-safe and easily queryable record of every test run. Importantly, by decoupling the raw output from the score, developers can &quot;re-judge&quot; historical data. If the grading rubric evolves, the team can simply re-run the judge logic over the stored <code>raw_output<\/code> fields without incurring the latency or financial cost of re-querying the primary model.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Data-Driven_Implications_for_Enterprise_AI\"><\/span>Data-Driven Implications for Enterprise AI<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The move toward custom evaluation harnesses has profound implications for enterprise AI deployment. Data collected by firms specializing in AI observability suggests that organizations with automated, continuous evaluation pipelines reduce their &quot;time-to-repair&quot; for model regressions by approximately 60% compared to those relying on manual testing.<\/p>\n<p>Furthermore, this systematic approach creates an audit trail. In regulated industries such as healthcare or law, the ability to demonstrate that a model was tested against a comprehensive suite of safety-critical prompts is a compliance requirement. As models are updated or fine-tuned, the harness serves as a regression testing suite, ensuring that performance improvements in one area (e.g., speed) do not lead to performance degradation in another (e.g., accuracy).<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Industry_Perspectives_and_Future_Trends\"><\/span>Industry Perspectives and Future Trends<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Leading AI researchers and engineers often emphasize that &quot;evaluation is the product.&quot; In recent months, organizations like Anthropic and OpenAI have published extensive documentation on how they evaluate their own models, emphasizing that benchmarks are merely starting points. <\/p>\n<p>&quot;The real value in LLM engineering is not in the prompt engineering itself, but in the rigorous, data-driven validation of that prompt across a million potential user interactions,&quot; notes a lead AI engineer at a top-tier tech firm. This sentiment is echoed across the industry, with a growing trend toward &quot;eval-driven development&quot; (EDD). In an EDD cycle, no code or system prompt is committed to production without passing the suite of tests defined in the custom harness.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Overcoming_Implementation_Challenges\"><\/span>Overcoming Implementation Challenges<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>While building a custom harness is straightforward in terms of lines of code, the real-world challenges lie in maintenance and scale. As the golden dataset grows, the cost of running evaluations can become significant. To mitigate this, many teams implement &quot;sampling&quot; strategies, where only a subset of the golden dataset is run during routine development, while a &quot;full-suite&quot; test is triggered only before major deployments.<\/p>\n<p>Additionally, developers must be wary of &quot;judge drift.&quot; Because the judge itself is an LLM, it may occasionally provide inconsistent scores. Implementing multiple judges or &quot;majority voting&quot; systems can help normalize these results, though this adds complexity to the pipeline.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Conclusion_The_Path_Forward\"><\/span>Conclusion: The Path Forward<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>The modular approach to building an evaluation harness\u2014utilizing Python&#8217;s native concurrency, structured output validation, and decoupled logging\u2014empowers developers to take ownership of their model&#8217;s quality. By treating AI evaluation as a core software engineering task rather than an experimental add-on, teams can move beyond the limitations of public benchmarks and build systems that are truly aligned with their specific operational needs.<\/p>\n<p>As the industry matures, these custom harnesses will likely integrate more deeply with CI\/CD (Continuous Integration\/Continuous Deployment) workflows, eventually becoming as standard as unit testing is for traditional web applications. The ability to iterate quickly, backed by the confidence of a rigorous, automated evaluation system, is the defining characteristic of the next generation of AI-native products. Organizations that prioritize the construction of these internal feedback loops today will be better positioned to navigate the complexities of deploying reliable, scalable, and safe AI systems in the years to come.<\/p>\n<!-- RatingBintangAjaib -->","protected":false},"excerpt":{"rendered":"<p>The Evolution of Model Assessment The history of software quality assurance is rooted in the &quot;test harness,&quot; a collection of software and test data configured to run a program under varying conditions and monitor its behavior. Applying this methodology to LLMs represents a significant evolution in AI engineering. In early 2023, most developers relied on &hellip;<\/p>\n","protected":false},"author":18,"featured_media":8336,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22],"tags":[23,567,4227,25,2966,1390,24,688,4277],"class_list":["post-8337","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence","tag-ai","tag-building","tag-custom","tag-data-science","tag-evaluation","tag-harness","tag-machine-learning","tag-python","tag-scratch"],"_links":{"self":[{"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/posts\/8337","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/users\/18"}],"replies":[{"embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8337"}],"version-history":[{"count":0,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/posts\/8337\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=\/wp\/v2\/media\/8336"}],"wp:attachment":[{"href":"https:\/\/lockitsoft.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8337"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8337"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lockitsoft.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8337"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}