Thoughtworks Future of Software Development Retreat Explores Harness Engineering, Self-Hosting, and the Evolving Role of AI in Software Creation
The recent Thoughtworks Future of Software Development Retreat, held earlier this year, has become a focal point for industry discussions surrounding the rapidly evolving landscape of software engineering. Building on the foundational discussions from their inaugural retreat in Utah, this latest gathering saw a significant shift in emphasis, with "Harness Engineering" emerging as a prominent topic of conversation, moving from an obscure concept to a dedicated session. This evolution reflects the growing recognition of the need for sophisticated frameworks to effectively leverage the power of large language models (LLMs) in practical software development.
The Rise of Harness Engineering: Guiding AI’s Potential
Harness Engineering, as explored during the retreat, centers on developing methodologies and tools to manage the interaction between human developers and AI agents. A key area of focus within this domain is context management. While the capacity of LLMs to process increasingly larger contexts has expanded, the fundamental challenge of ensuring the AI focuses on the most pertinent information remains. Attendees shared strategies for optimizing context windows, with one participant noted for maintaining a remarkably concise context, limiting their agents.md file to under 200 lines to ensure sharp focus. This highlights a pragmatic approach to harnessing AI, prioritizing directed attention over sheer volume of information.
On the "sensor" side of harness engineering, which refers to the methods for gathering and processing data for AI, a notable trend is the increasing adoption of computational sensors. This involves a deliberate shift towards programming languages that offer greater control and predictability, such as Rust, moving away from more dynamic languages like Python for critical components. Furthermore, there’s a growing emphasis on "leveling up" validation approaches. This includes the wider adoption of property-based testing and techniques borrowed from formal methods. The sentiment expressed by one attendee encapsulates this shift: while not possessing the expertise to write formal specifications themselves, they recognize the value of understanding and verifying these specifications within their specific domain, demonstrating a collaborative approach to leveraging advanced validation techniques.
The longevity of this focus on harness engineering remains a subject of debate. The question of whether LLMs will become so advanced that specialized harnessing frameworks become obsolete is a pertinent one. Those with a deep understanding of LLMs, often described as possessing "mechanical sympathy," generally believe that such frameworks will continue to be essential. They argue that even highly sophisticated models benefit from structured guidance to ensure optimal performance and cost-efficiency. The retreat participants noted that a well-managed harness can significantly reduce token usage, thereby lowering operational costs. Crucially, it also democratizes AI by enabling weaker, more accessible models, including those that can be hosted locally, to become viable tools for a wider range of applications.
The Self-Hosting Imperative: Cost, Control, and Sovereignty
The increasing token costs associated with proprietary LLM services have made the prospect of self-hosting open-weight models increasingly attractive. This trend is further accelerated by the rapidly diminishing time it takes for open-weight models to close the gap in performance with leading frontier models. However, cost is not the sole driver. A significant motivation for self-hosting stems from a desire for independence from the major frontier model providers. Recent governmental interventions that have restricted access to certain AI models have amplified concerns about model sovereignty and the potential for external control over critical AI capabilities.
Information security also plays a pivotal role. Many organizations are hesitant to share sensitive or proprietary data with third-party AI providers. Self-hosting ensures that such data remains within the organization’s control. Furthermore, when models are hosted externally, it is the host’s model that learns and improves, not the user’s. While the interest in self-hosting has surged recently, several participants shared that their companies have been engaged in this practice for up to two years, indicating a growing maturity in this area.
The discussions also touched upon the potential pitfalls of self-hosting, drawing parallels to the "half-arsed private clouds" of the past that led to significant, often inefficient, expenditures. The crucial differentiator for self-hosted AI is whether the operational complexity of hosting a model proves to be simpler than that of hosting a full cloud infrastructure, potentially due to more streamlined interaction protocols. The primary hurdle identified is the specialized talent required for efficient GPU utilization. Managing an inference data center is not yet a widely possessed skill. The operational costs, encompassing capital expenditure on GPUs and ongoing electricity consumption, are significant. The physical design of data centers can also impact optimal usage, presenting an opportunity for professional services firms to offer expertise in managing these complex environments.
Cost control also extends to educating users on selecting the appropriate model for specific tasks. The potential for AI itself to act as a broker, intelligently routing jobs to the most suitable model, was explored as a promising avenue. This could empower engineers and other users to leverage less powerful, and thus more cost-effective, models when full computational power is not required.
The rise of self-hosting is also expected to drive increased adoption of fine-tuning. Currently a niche activity, fine-tuning models for specific domains could lead to reduced reasoning overhead, lower token consumption, and consequently, more economical operation. The development of models specifically trained to support programming tasks is a testament to this trend. Ultimately, in a field characterized by rapid and unpredictable change, the key strategic advantage lies not in predicting the future, but in building resilient strategies that can adapt to inevitable shifts.
Unifying Themes: Agentic Control and the Nature of Expertise
A compelling overarching narrative emerged from the retreat, articulated by Kief Morris, which synthesized the diverse discussions. Morris posits that the core of nearly every session revolved around a fundamental question: "How much do we let an agent decide, and how do we stay confident in what it does?" This central theme underpins discussions ranging from code review to production incident response, highlighting the constant negotiation between AI autonomy and human oversight.
The retreat explored how traditional software development rigor, such as code review, might manifest in the context of AI agents. Disagreements surfaced regarding the level of trust teams place in agents to identify and rectify production incidents. The extent of autonomy granted to AI agents appears to be highly context-dependent, influenced by the specific operational environment and the perceived risks involved. Morris’s observation that teams are making consistent choices about the "unit of work" to be delegated to an agent – its size, scope, preparation, and verification – underscores the critical importance of defining these boundaries.
"Bring Me a Rock": Evolving Management Paradigms in the Age of AI
Sam Ruby’s session, provocatively titled "Bring Me a Rock," delved into a management style historically criticized for its inefficiency and lack of clear direction. The phrase evokes a scenario where a manager repeatedly rejects proposed solutions without articulating specific criteria, forcing subordinates to guess the desired outcome. Historically, this approach was seen as a sign of a manager’s "unfinished thinking."
However, Ruby argued that with LLMs, this approach transforms from a management dysfunction into a potentially defensible working method. The tireless and patient nature of AI, capable of generating numerous iterations rapidly, makes a process of serial refinement viable. This "brainstorming register" approach, where the AI acts as an inexhaustible generator of options, allows for exploration through elimination.
The ensuing discussion revealed a nuanced concern: who should be empowered to steer these AI agents? With product managers and people managers increasingly engaging directly with LLMs, and seasoned engineers achieving demonstrably superior results, questions arose about the appropriateness of non-engineers directing AI. Ruby reframed this, suggesting that when a manager utilizes an LLM instead of routing work to their team, they are, in essence, making a "hire." This perspective shifts the focus from seeking permission to manage to understanding how to manage effectively when the worker (the AI) possesses more specialized knowledge than the manager. The challenge then becomes mastering "management by objective," a principle articulated by Peter Drucker, rather than "management by method," a critical distinction in guiding AI effectively.
The core of Ruby’s argument is that managing an LLM involves setting clear objectives rather than dictating granular tasks. This raises the persistent question of confidence: how can we ensure the AI’s actions align with the desired outcomes? While many tasks can be outsourced, the acceptance criteria remain a human prerogative. The risk lies in unstated objectives, those implicitly assumed but never articulated. For instance, an objective to "create a daily to-do list from emails" carries numerous unstated assumptions about data privacy, security, and the exclusion of unwanted functionalities. While recent advancements suggest LLMs are improving at identifying and mitigating security vulnerabilities, the challenge of defining all necessary "conformance tests" to prevent undesirable outcomes remains significant.
The process of software development itself is inherently exploratory, involving the evolution of workflows as AI becomes integrated. Human guidance is crucial in this evolutionary process. While LLMs can actively participate in model construction, the human role in teaching and understanding these models remains indispensable, as they help us conceptualize and communicate the objectives for the AI.
Local Models and Developer Education: Emerging Trends and Challenges
Complementing the discussions on harness engineering and self-hosting, the retreat also highlighted practical developments in working with local AI models. Birgitta Bäckeler’s recent memos exploring the viability of local models for programming and her subsequent evaluation experiences provided valuable insights. Alongside these, Sebastian Raschka’s detailed guide to his local model environment offered a practical blueprint for developers. Both researchers identified the Qwen 3.6 model as a current frontrunner for local agentic programming tasks.
In a related development, Simon Willison shared a cost-saving tip for utilizing Anthropic’s Fable model: instructing it to delegate smaller tasks to other, less expensive models, thereby optimizing resource allocation and reducing overall expenditure.
The economic impact of AI on developer education was also a significant talking point. Josh Comeau, a prominent figure in front-end web development education, reported a substantial decline in online course sales. He attributes this trend to several factors: the uncertainty surrounding the future job market due to AI, and the ability of AI to provide personalized, often free, tutoring. This raises concerns about the sustainability of high-quality, free educational content when AI can synthesize and disseminate existing knowledge without direct compensation to creators. The sentiment expressed is one of apprehension regarding the potential devaluation of original work and the economic viability of content creation in an AI-saturated landscape.
The Future of Cross-Platform UIs and the Nature of Expertise
A point of contention arose regarding the persistence of cross-platform front-end development frameworks, specifically highlighted by John Gruber’s critique of Claude’s macOS desktop application using Electron. Gruber’s strong criticism of Electron’s tendency to create a suboptimal user experience across all platforms, and his pointed remarks about the developers involved, underscore a broader debate. The increasing capability of AI coding agents to generate native applications for multiple platforms efficiently suggests that the era of "least-denominator" cross-platform UIs might be nearing its end.
Finally, Dan Davies introduced a distinction between "interactional" and "contributory" expertise. Contributory expertise lies with those actively advancing a field, while interactional expertise is gained through engagement with contributory experts. Davies posed a critical question: can machines achieve a form of expertise through the industrial consumption of vast amounts of data that is equivalent to genuine contributory expertise capable of addressing novel situations?
The author reflects on their own position, aspiring to be a contributor rather than merely an interactor, while acknowledging a career built on synthesizing and explaining the ideas of others. This self-awareness highlights the nuanced boundaries of AI capabilities. While machines can excel at pattern recognition and information synthesis, the capacity for true innovation and tackling entirely novel problems remains a frontier. The exploration of LLM limitations and the potential for human expertise to adapt and thrive alongside AI continues to be a central theme in the evolving discourse on the future of software development.
