## Overview This case study comes from a podcast discussion featuring HubSpot's CMO Kip Bodner and VP of Marketing Emmy Jonathan, where they discuss real-world AI experiments being conducted within HubSpot's marketing organization. The primary use case examined is the transformation of their "first conversion nurturing" email flow—a high-volume automated email sequence sent to leads who download educational content—from traditional cohort-based personalization to true one-to-one personalization powered by large language models. HubSpot, as a CRM and marketing automation platform, has significant resources and existing infrastructure that gave them advantages in implementing this solution, including a large library of educational content (courses, guides, templates) and robust data collection on user behavior. The case study provides valuable insights into both the organizational approach to prioritizing AI initiatives and the technical architecture of a production LLM system. ## Organizational Framework for AI Prioritization Before diving into the technical implementation, it's worth noting HubSpot's approach to prioritizing AI use cases, as this represents a practical framework for LLMOps project selection. The team received over 100 AI project ideas from across the marketing organization and needed a systematic way to evaluate them. They used a 2x2 matrix framework with two axes: - **Y-axis**: Impact on top-of-funnel metrics (demand generation and brand awareness) - **X-axis**: Breadth of use across the marketing team (internal productivity gains) This framework helped them balance revenue-impacting use cases against internal efficiency improvements. The email personalization use case scored highly on the demand impact axis due to the massive volume of leads flowing through the first conversion nurturing system—representing their largest cohort of prospects (those with educational intent, which they estimate is at least 10x larger than the cohort actively evaluating software). The prioritization process was kept deliberately lightweight and agile, using Slack messages and Google Forms for idea collection. They maintained bi-weekly review meetings to allow for rapid reprioritization as new technologies or market conditions emerged. This "perfect is the enemy of good" philosophy extended to their implementation approach as well. ## The Problem: Limitations of Cohort-Based Personalization HubSpot's existing first conversion nurturing workflow used traditional segmentation-based personalization. When a lead downloaded an educational content offer (ebook, template, guide), they would be placed into a segment based on: - Demographic characteristics - Inferred interests based on the content they downloaded The system would then send emails with content tailored to that segment—for example, marketing-related content for leads who downloaded marketing resources. However, after years of A/B testing and optimization, they had reached a plateau with only incremental gains possible. This is a common pattern in conversion optimization where initial tests yield significant improvements but returns diminish over time. The fundamental limitation was that they were doing "group guessing"—placing people into cohorts and making assumptions about what the group might want, rather than understanding individual needs. ## Technical Architecture The AI-powered solution uses GPT-4 from OpenAI combined with a vector database to achieve one-to-one personalization at scale. The architecture follows a RAG (Retrieval-Augmented Generation) pattern with several distinct processing stages: ### Data Collection and Context Building When a lead enters the system, the following data is collected: - **Business URL**: Provided by the user on the form, which is then scraped to understand what the company does - **Form data**: First name, email address, company size, etc. - **Conversion context**: What content offer they downloaded - **Behavioral signals**: Other actions taken on HubSpot's website after the initial conversion This multi-source data collection provides rich context for personalization that goes far beyond simple demographic segmentation. ### Job-to-be-Done Inference The LLM's first task is to synthesize all available information and generate a summary of what the person is likely trying to accomplish—their "job to be done." This is a critical insight from the case study: the key to success was accurately inferring intent, not just personalizing surface-level copy. An example provided in the discussion shows how the system analyzed a small online coffee company whose user downloaded influencer marketing content and subsequently showed interest in content calendars. The LLM generated a summary interpreting this as preparation for seasonal promotions and a strategic approach to brand growth, connecting the dots between different behavioral signals. ### Ideal Content Generation Rather than immediately searching the existing content library, the LLM first imagines what a "perfect course" would look like to help this specific person accomplish their inferred goal—regardless of whether such content exists. This is an interesting approach that allows the system to reason about ideal outcomes before constraining to available resources. ### Vector Database Retrieval The hypothetical ideal course description is then sent to a vector database containing embeddings of all HubSpot's actual courses and their relationships. The database returns the top 10 most similar real courses based on semantic similarity to the ideal course. ### Final Recommendation Selection The LLM reviews the candidate courses in the context of everything it knows about the user and selects the single best option. This multi-stage filtering approach (generate ideal → retrieve candidates → select best match) appears more sophisticated than a simple single-pass retrieval. ### Personalized Copy Generation Finally, the system generates personalized email copy that: - Uses the person's name - References their company specifically - Connects the recommended course to their specific business context - Creates compelling, personalized subject lines and body text The example shown generated copy like "Turn every sip into a story that captivates and converts" for the coffee company prospect—demonstrating genuine personalization rather than simple mail-merge style token replacement. ## Results and Iteration Process The system achieved impressive results: - **Conversion rate**: 82% improvement - **Open rate**: Approximately 30% improvement - **Click-through rate**: Over 50% improvement The team emphasized that these results took approximately two months of iteration to achieve. A critical learning was that their initial hypothesis was incorrect. They first assumed the personalized email copy would drive conversion improvements, but discovered that the real value came from accurately inferring the job-to-be-done and recommending truly relevant content. The personalized copy was "icing on the cake" but not the primary driver. This finding has important implications for LLMOps practitioners: it suggests that investing in better retrieval and recommendation logic may yield higher returns than investing in more sophisticated text generation. ## Key LLMOps Learnings Several practical LLMOps lessons emerge from this case study: **Ship early and iterate**: The team repeatedly emphasized that AI models cannot be perfected in isolation—they need real user feedback to improve. Waiting to launch until the system is "perfect" is counterproductive because perfection is impossible without real-world data. **Combine domain expertise with AI expertise**: The project paired Josh Bliss (AI/technical expertise) with Jordan Douglas (email automation and persona domain expertise). This pairing of subject matter experts with AI practitioners was highlighted as essential to success. **Start with the right problem**: By focusing on a high-volume, high-impact use case (first conversion nurturing), the team ensured that even modest percentage improvements would translate to significant absolute gains. **Infrastructure matters**: HubSpot's existing library of educational content gave them a significant advantage. The more content available for matching, the more likely the system can find something truly relevant for each user. Organizations considering similar implementations should assess their content assets. **Measurement and validation are critical**: The team took "double takes and triple takes" to validate their results, recognizing that 82% conversion improvements sound almost too good to be true. ## Caveats and Considerations While the results are impressive, several contextual factors should be considered when evaluating this case study: The discussion is from a podcast where HubSpot is promoting their own AI capabilities, so there may be selection bias toward highlighting successful experiments. The specific definitions of "conversion" and baseline performance levels are not provided, which makes it difficult to fully contextualize the improvements. Additionally, HubSpot has significant advantages that may not be available to smaller organizations: a large content library, substantial first-party behavioral data, dedicated AI resources, and existing marketing automation infrastructure. The transferability of these results to organizations without similar assets is unclear. The compute costs, latency considerations, and operational complexity of running this system at scale are not discussed. For organizations considering similar implementations, these operational factors would be important to evaluate. ## Team and Resources The implementation was led by a centralized team within HubSpot's Marketing Technology group, headed by Mark. Dave G. was brought on to lead initial AI efforts, and the team grew over time. The core implementation appeared to involve: - One person spending the majority of their time for several months - Building and maintaining the vector database - Training and fine-tuning the LLM - Ensuring proper data annotation - An OpenAI subscription (GPT-4) This relatively lean resourcing suggests that similar implementations may be achievable for mid-sized organizations with the right technical talent.