The intersection of artificial intelligence, creativity, and user curiosity on mobile devices was the central theme of a recent presentation by Bhavuk Jain, a product leader who has been instrumental in developing innovative AI-powered features. Jain detailed his personal experience working on two distinct AI products: one focused on generating visually stunning images and another on making visual search intuitive and seamless. His insights shed light on the intricate process of transforming foundational AI models into user-facing products, emphasizing the critical steps of post-training, fine-tuning, retrieval, grounding, and the implementation of robust guardrails.
The presentation, delivered at a prominent tech conference, aimed to demystify how cutting-edge AI capabilities translate into tangible applications that address everyday user pain points. Jain began by posing relatable scenarios to the audience: the common frustration of needing to screenshot an item seen on social media, switch apps, and upload an image to initiate a search for product information, or the time-consuming effort of sifting through countless images to find the perfect wallpaper. These seemingly minor inconveniences, he explained, served as the direct inspiration for the two features he would later discuss.
From Foundational Models to User-Facing Products: A Four-Step Journey
Jain articulated a clear roadmap for how modern AI systems evolve into deployable products. He stressed that these products rarely start from scratch, but rather leverage massive foundational models. However, these raw models, often unpredictable and unconstrained, require a rigorous refinement process through four essential architectural steps:
-
Post-Training: This crucial phase aligns the foundational model with human preferences, stylistic nuances, and essential safety rules. Jain likened this to teaching a brilliant but unrefined student manners, personality, and safety protocols. Techniques like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are employed here, with DPO often offering a simpler, more scalable path. The emphasis is on rigorous evaluation to ensure desired behavior, often requiring the engineering of specific metrics to quantify subjective qualities like summary quality. This step necessitates an early architectural decision regarding flexibility versus a simpler, scalable approach.
-
Fine-Tuning: Once aligned, the model is still a generalist. Fine-tuning specializes it for specific use cases, tasks, or domains. This can involve task and instruction tuning, where the model learns to map input-output pairs to desired results, or Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA and QLoRA, which allow for the training of smaller, specialized modules attached to the larger frozen model. Jain highlighted that sometimes, fine-tuning isn’t necessary; if the issue is a lack of fresh or factual knowledge, grounding the model with external data is the more effective solution. This step involves a trade-off between greater specialization and increased complexity, maintenance costs, and financial outlay.
-
Retrieval and Grounding: This step has gained prominence due to the challenge of AI hallucinations. Instead of relying solely on the model’s internal, frozen knowledge, this layer connects the AI to external sources of truth, such as structured knowledge bases, real-time web results, or on-device context. This significantly reduces factual inaccuracies and enables the model to provide answers based on real knowledge rather than speculation. This is where modern AI systems integrate search capabilities, embeddings, Retrieval-Augmented Generation (RAG), or tool usage.
-
Inference and Guardrails: The final stage involves deploying the refined model within a robust infrastructure. Crucially, this includes implementing essential guardrails to ensure the AI operates reliably, safely, and at scale. Guardrails are multi-layered safety mechanisms designed to constrain what the AI can see, say, and do, even after fine-tuning. These include data guardrails (filtering input data), model guardrails (shaping model behavior through prompts and safety classifiers), and runtime guardrails (real-time inspection of inputs and outputs for issues like PII, toxicity, or jailbreak attempts). Governance and user experience (UX) elements, such as logging, audits, human review flows, and user-facing controls, form the human safety net. This entire infrastructure bridges the gap between a trained model and a product used by millions.
Case Study 1: AI Wallpapers – Engineering Creativity
The first case study explored the development of "AI Wallpapers," a feature designed to empower users to create unique, high-quality wallpapers on their mobile devices by simply describing their ideas. Jain described this as an example of the "generative phase" of AI engineering, where the primary optimization goal was creativity.
The motivation behind AI Wallpapers stemmed from several key factors. Firstly, it offered clear, immediate user benefit through personalization, aligning perfectly with Android’s Material You design philosophy, which dynamically adapts the phone’s interface to match the wallpaper. Secondly, there was a significant, documented user need; changing wallpapers is the most popular way users customize their phones, with many spending considerable time searching for the ideal background. Finally, generative AI technology had advanced to a point where it could provide an "infinite range of high-quality content," moving users beyond static galleries to become creators themselves.
However, the development of AI Wallpapers presented four major challenges:
- The UX Challenge of Prompt Engineering: Complex prompts requiring technical jargon like "cinematic lighting" or "8K" were deemed impractical for average users. The challenge was to translate this complexity into an intuitive, fun, and fast mobile UI.
- The Quality Challenge: Generative models can produce technically correct but artistically awkward or "weird" images. The goal was to achieve consistent artistic control that users would find appealing for their wallpapers.
- The Safety Challenge: For a feature deployed on millions of devices, preventing the generation of inappropriate, harmful, or copyrighted content was a non-negotiable, foundational requirement.
- The Emerging Viability Challenge: Initial server-based models were powerful but prohibitively expensive to run, posing a long-term viability concern for a popular feature.
To address these challenges, the team made a "non-obvious bet" on simplicity and restriction over maximum freedom. Instead of a blank text box, they opted for a guided creation experience, positioning the AI as a "creative partner." This approach allowed them to engineer specific, measurable quality criteria related to format, conciseness, and accuracy. This decision was validated through extensive user experience research.
"Our role was to be their creative partner," Jain stated. "Instead of just giving our users a powerful but complex tool, we decided our role was to be their creative partner."
This restricted input model, coupled with an obsessive focus on "delight" – through subtle animations and playful error states – created a magical experience even within constraints. The team invested heavily in micro-interactions, ensuring that even moments of failure were handled with creativity, such as displaying a T-Rex attempting to paint instead of a standard error message. This deliberate choice to favor restriction and simplicity directly reduced the likelihood of catastrophic safety failures, such as generating inappropriate content like weapons in a creative context, and made risk assessment more manageable.
The high-level architecture involved a frontend with a simplified user experience, which fed into a prompt engineering service that translated user selections into detailed, structured prompts. Backend services implemented the first layer of server-side guardrails using classifiers and blocklists. The inference platform, powered by a fine-tuned model, included a second safety layer of visual classifiers before final image upscaling.
A key insight into achieving quality was recognizing that AI models are not monolithic; they excel at specific styles. The team focused on identifying and building around the model’s natural strengths, constraining user choices to domains where it consistently produced outstanding results. This involved extensive large-scale testing with human review, where designers and researchers evaluated images not just for technical correctness but for aesthetic appeal. This process led to the refinement of prompt keywords and the design of suggestion chips that reliably produced beautiful images. Over a million images were generated and reviewed to select the final UI elements. This rigorous approach, combined with leveraging internal research teams for model improvements and fine-tuning, ensured a continuously improving user experience.
The philosophy for scaling AI Wallpapers was built into the process from day one. Relentless UX exploration informed evaluation criteria and metrics. A rigorous evaluation pipeline with human raters ensured consistently high-quality results. Critically, the feature was engineered to use a scalable, internally reusable endpoint across multiple Google products, significantly reducing costs and improving reliability. The successful launch, characterized by user praise and delivery at Google scale across various devices and OEM partners, validated their strategy.
Case Study 2: Circle to Search – Engineering Understanding
The second case study shifted focus to "Circle to Search," a feature designed for intuitive visual search without leaving the current application. Jain described it as a complex, real-time, multimodal retrieval task that translates a visual input and a gesture on a screen into a high-fidelity search query.
The motivation for Circle to Search was to eliminate the friction inherent in the traditional visual search workflow. Users previously had to take screenshots, switch apps, upload the image, and then search. Each step was a potential point of user abandonment. The "North Star" for this project was a seamless experience with zero context switching, moving from curiosity to answer instantaneously. This vision was made achievable by building upon Google Lens’s powerful visual understanding capabilities and integrating them with deeper, system-level OS integration.
Circle to Search evolved from Google Lens by becoming "screen-facing" rather than "world-facing." By replacing multiple steps with a single gesture, it removed friction and enabled users to act on curiosity instantly. This seemingly minor shift had profound implications for the underlying technology, transforming the task into a real-time retrieval problem demanding extremely low latency and high relevance.
The high-level architecture begins on-device, where a user gesture triggers the capture of on-screen content. This "multimodal prompt" includes pixels and structural layout information. This rich prompt is then processed by an AI pipeline. The core AI reasoning and heavy lifting occur here. Finally, the answer is sent back to the device and presented as an overlay, seamlessly integrated into the user’s current context.
The AI pipeline itself starts with a core multimodal model, specifically fine-tuned to understand the relationship between images, user queries, and screen context. This model initiates parallel reasoning and retrieval across multiple sources, querying the knowledge graph for structured facts and web/image search for visually similar items. For example, circling a handbag would trigger queries to identify the brand and model via the knowledge graph, while simultaneously searching for shopping links, reviews, and alternative color options. The final stage involves a generative AI overview synthesis model that crafts a natural language answer. Before delivery, this answer must pass through a non-negotiable checkpoint of safety and responsibility filters.
Key to this seamless experience are Google’s Multitask Unified Model (MUM), which moves beyond keyword matching to intent understanding across modalities, and Gemini integration, powering richer AI overviews and contextual help by handling complex reasoning.
Crucially, deep system-level integration was a game-changer. Unlike Lens, which required users to hunt for an app, Circle to Search was built directly into the Android OS, enabling a universal gesture that works everywhere. This system-level integration provided the AI with deeper context, allowing it to precisely identify the selected item amidst the screen’s semantic layout. This hybrid advantage meant the AI could focus on the exact pixels belonging to the circled object, leading to highly accurate search results.
Safety and responsibility were paramount. The visual AI was grounded in factual data by cross-referencing recognized entities with a structured knowledge base like the Knowledge Graph, serving as a first line of defense against misinformation. A dedicated safety layer acted as a core architectural component, ensuring all synthesized results, especially those from generative models, passed through a responsible AI filter designed to catch harmful, hateful, or inappropriate content. Furthermore, the system was designed to handle ambiguity gracefully, offering broad suggestions in low-confidence scenarios rather than providing potentially incorrect answers, thereby building user trust.
The hypothesis that removing friction through deep system-level integration would unlock new engagement proved correct. Circle to Search launched on over 300 million devices, driving a significant 70% year-over-year increase in visual searches. It quickly became a fast-growing query type, particularly among younger users, with shopping and translation functionalities proving highly effective.
Key Learnings and Actionable Takeaways
Reflecting on both AI Wallpapers and Circle to Search, Jain distilled several key learnings applicable to AI engineering:
- Generative Products (AI Wallpapers): The primary challenge was managing the "blank canvas problem." Success hinged on clear principles, tight product definitions, a non-obvious bet on guided creation, and an obsessive focus on the evaluation pipeline for artistic quality. Over-investing in moments of "delight" was crucial for making a restrictive experience magical.
- Understanding Products (Circle to Search): While model improvements were vital, the transformative element was deep OS integration, making the feature seamless to invoke. This frictionless experience, coupled with accurate search results, was highly appreciated. The integration also unlocked deeper intelligence by providing semantic layout context. Building trust required grounding visual AI in factual data, cross-referencing model output with structured knowledge bases for reliability.
For AI engineers, the actionable takeaway is to think about "productizing" AI, not just deploying it. This involves building a complete system around the model, including evaluation pipelines, cost and latency optimizations, and safety guardrails, to create a reliable and trustworthy experience. Tech leaders are urged to champion "principle-led development," using principles like quality and safety as a North Star, especially when making difficult, counterintuitive decisions. Product managers and designers should obsess over the user’s workflow, identifying and eliminating friction, as demonstrated by the success of Circle to Search.
The journey from raw AI models to user-centric products is complex, demanding a meticulous blend of technological innovation, product strategy, and a deep understanding of user needs. As Jain’s presentation illustrated, success lies not just in the power of the AI itself, but in the thoughtful engineering and principled design that brings its capabilities to life in a safe, intuitive, and delightful manner.
