Morgan Stanley's quantitative research team developed AlphaLab, an agentic harness system designed to automate quantitative research for financial modeling and prediction tasks. The system takes time series data and natural language descriptions as input, then autonomously conducts research, builds evaluations, and performs mass experimentation to generate trained machine learning models. Alpha Lab 1.0, released in April 2026, demonstrated success on academic benchmarks and achieved top 12% performance on a Kaggle competition, while also producing meaningful improvements to internal production models. The team is now developing Alpha Lab 2.0, which focuses on self-improving capabilities through carefully constructed evaluation environments and meta-optimization, with the goal of encoding enterprise knowledge and human expertise through high-quality evaluation frameworks rather than hardcoded workflows.
Morgan Stanley’s quantitative research group, consisting of approximately 30 PhD AI researchers, developed AlphaLab as an automated research agent specifically designed to automate quantitative financial research tasks. The team operates with a hybrid model, balancing academic research activities including publishing papers and open-sourcing code with applied internal work. Their problems are often well-posed, resembling Kaggle-style challenges with input time series data requiring prediction of future values, often with constraints around model calibration.
The motivation for building AlphaLab stemmed from several organizational needs. First, many algorithms in production were believed to have room for improvement if more computational cycles could be dedicated to better hyperparameter tuning or exploring ensemble methods. Second, there was a strong belief that models built for one desk, such as credit bonds, could transfer effectively to related areas like municipal bonds through an automatable translation process. Third, many trading desks had low-hanging fruit opportunities where minimal machine learning or AI automation was being used, and even reasonably trained models could deliver significant impact.
The team noted that while the possibility of long-horizon task automation had existed for about a year and a half, it wasn’t until December 2025 that building such a system felt truly feasible. This turning point was attributed to the release of Opus 4.5 and the emergence of harnesses like Claude Code and Codex, which demonstrated that models had reached a maturity level where they could handle long-horizon tasks effectively.
AlphaLab was built as a custom agentic harness rather than using off-the-shelf agent frameworks. This decision was made to maximize flexibility and enable easy customization of any component. The team embraced the philosophy that in the era of advanced code-generation models, building from scratch provides maximum freedom. All tool calls are implemented using functional tool calling, which facilitates provider-agnostic operation across OpenAI, Anthropic, or open-source model providers.
The system was designed with several key requirements in mind. It needed to integrate seamlessly with existing data infrastructure and pre-built scaffolding around backtesting and evaluations. The harness was intended to run across a full spectrum of use cases, from completely new datasets requiring natural language descriptions to well-established models with existing evaluation frameworks that simply needed additional optimization cycles. Model agnosticism was a priority, ensuring the system could leverage any frontier provider or open-source model. Importantly, the architecture was designed to rise with the tide of model improvements rather than becoming obsolete as models evolved. Finally, careful consideration was given to encoding Morgan Stanley’s enterprise knowledge and the team’s human expertise as quantitative researchers.
AlphaLab provides three main tool categories to its LLM agents. First, full shell access enables writing bash commands for code creation, Python environment setup, code editing, and execution. Second, web search capability allows the system to access arXiv papers, technical blogs, and other public domain resources to understand state-of-the-art methods. Third, a Slurm abstraction layer manages GPU cluster resources, allowing the model to request specific hardware configurations such as multiple H100 GPUs and CPUs without handling low-level orchestration details.
The first phase involves comprehensive research and context building, functioning as an enhanced initialization process. The system is prompted to create a to-do list encompassing all tasks necessary to begin meaningful experimentation. This scaffolding approach allows continuous re-prompting if the system attempts to exit prematurely. Tasks typically include setting up Python environments, data loading, and various statistical tests. Each to-do list item requires the agent to take notes in a markdown file, which future agents can query to manage their context dynamically. Web search is heavily utilized during this phase, with the system reading academic papers and technical content to build understanding of public domain state-of-the-art approaches. This research phase typically requires approximately 3 to 4 hours to complete.
The second phase focuses on constructing evaluation frameworks, which the team acknowledges as the most critical component. While LLMs are not malicious, they can make serious mistakes, and optimizing against a flawed evaluation renders the entire system ineffective. To address this, Alpha Lab 1.0 employed a multi-agent framework for robustness. One agent builds the evaluation code, which is then reviewed by two critic agents. One critic focuses on high-level conceptual errors and potential forward leakage of information, while the other takes a programmatic approach, writing unit tests and integration tests. Any identified issues are documented and returned to the builder agent for correction. This iterative loop continues until all agents agree the evaluation is satisfactory.
The third and central phase implements mass experimentation using a Kanban board or Jira-style formulation. A strategist agent analyzes and queries context from previous phases to continuously propose new experiments, submitting them to an implementation column. Worker agents pull these experiment cards and handle the complete implementation workflow: writing code, configuring Slurm specifications for hardware requirements, submitting jobs to the GPU cluster, monitoring job completion, analyzing training curves for underfitting or overfitting patterns, examining evaluation results, and producing postmortem analyses.
These postmortem reports feed back to the strategist agent, enabling self-evolution where the strategist learns from results. For example, if multiple transformer variants underperform while XGBoost shows promise, the strategist can pivot to exploring more tree-based methods. Human researchers can steer this process by canceling cards, adding custom experiment cards, or using a chat interface to guide the strategist toward more creative approaches or specific methods. An ever-growing leaderboard tracks model performance against both the defined evaluation and a held-out private validation set, with full traceability from initial idea through code implementation.
The initial version of AlphaLab was released in early April 2026, accompanied by a comprehensive 40-page technical report and fully open-sourced code on GitHub. The team evaluated the system on several benchmarks to validate its capabilities. Academic datasets included CUDA kernel optimization and traffic time series prediction. Following the LM speed-running approach popularized by Andrej Karpathy, they tested the system’s ability to find optimal training configurations for language models, where AlphaLab outperformed a simpler single-agent loop approach.
The system was also deployed on a Kaggle competition hosted by Nvidia focused on fine-tuning their NeMo Tron model for reasoning capabilities. Despite joining late and having only 10 iterations to work with, AlphaLab achieved top 12% performance among submissions. The team noted that AlphaLab’s performance improves with more iterations, suggesting better results would have been possible with additional time.
Internally at Morgan Stanley, several models underwent optimization through AlphaLab, particularly cases where decent baseline models existed but additional refinement was desired. The system identified meaningful improvements in multiple instances, with these enhanced models progressing through risk management processes toward production deployment.
Despite these successes, the team encountered significant challenges and hard questions that drove the evolution toward Alpha Lab 2.0. First, while some results were promising, there were cases of clear failure, raising questions about the system’s true capabilities and reliability. The difficulty in measuring real-world effectiveness became apparent, particularly when the system failed on the hardest problems.
Second, the architectural choices in Alpha Lab 1.0, while reasonable, felt somewhat arbitrary. The research phase structure, the strategist-worker arrangement, and other design decisions lacked strong theoretical motivation. This raised the fundamental question of whether humans should be making these design choices at all, or whether this meta-optimization should itself be an LLM-driven verifiable loop.
Third, the original formulation where the LLM simply receives data and goals and operates autonomously did not effectively encode Morgan Stanley’s enterprise knowledge or the team’s expertise as quantitative researchers. This expertise was largely missing from the picture.
The team concluded that building custom evaluations and environments provides the answer to all three challenges. This represents a lesson repeatedly learned: starting with good evaluation is fundamental and non-negotiable. While there was initial enthusiasm to treat the system more like a human researcher, clear measurement mechanisms are essential.
Alpha Lab 2.0 adopts a highly opinionated stance on evaluations, explicitly modeling the system after Kaggle competitions. The input consists of data and natural language descriptions, the harness operates in the middle, and its sole responsibility is submitting containerized models. The system receives public leaderboard scores as feedback while users access private leaderboard results on held-out validation data. This strict format enables clear measurement for any given task.
Critically, the team views evaluations and environments as functionally equivalent, with the distinction being that environments are used for training. They have constructed between 10 and 20 carefully designed environments that serve as reinforcement learning signals. This enables what they describe as AlphaLab training AlphaLab—a meta-optimization loop.
With robust measurement in place, human tuning of the harness becomes possible through manual hill climbing, experimenting with configurations like multiple strategists engaged in debate. More significantly, the team is implementing meta-harness optimization where LLMs analyze execution traces and results to improve the harness itself. A parallel effort involves collecting high-quality traces from open-source models and applying techniques like GRPO (Group Relative Policy Optimization) or other on-policy distillation methods. The optimal configuration is envisioned as an orchestration of open-source and closed-source models, with the entire system optimized holistically.
Regarding the challenge of encoding human expertise, the team firmly believes this is achieved through environment construction. Building high-quality environments represents genuinely difficult work requiring deep expertise. While proprietary data serves as one component, designing verifiable metrics is only the beginning. The team also develops qualitative rubrics for evaluating agent behavior, assessing traces based on characteristics of good research practices and thought processes. Each rollout receives grades on how well it follows established research processes. These carefully crafted rubrics become the learning signal for models, representing the primary mechanism for embedding organizational expertise into the system.
The Alpha Lab 2.0 architecture centers on strict environment and evaluation frameworks, with the implementation details of the middle layer becoming increasingly abstracted. While initialized to Alpha Lab 1.0 configurations, the system is designed as fundamentally self-improving. Currently, manual tuning against environments continues, but the long-term vision sees auto-research researching itself in a completely autonomous self-improvement process.
The team’s strategic perspective is that general auto-research capabilities will become commoditized, citing evidence like GLM 5.2 as indicators of this trend. They argue that all enterprise value derives from building high-quality environments rather than the research automation itself. Their willingness to continue releasing code publicly reflects this philosophy—they believe environments encode the true value, while the harness mechanics will inevitably become widely available.
From an LLMOps perspective, this case study demonstrates several critical production considerations. The system handles long-running workflows spanning hours, requires orchestration of multiple model calls with different roles, manages GPU cluster resources through job submission systems, implements containerization for model deployment, and maintains comprehensive traceability from experiment conception through code implementation to results.
The multi-agent framework with specialized roles (builders, critics, strategists, workers) represents a sophisticated production architecture requiring careful orchestration. The integration with existing data infrastructure and backtesting frameworks highlights the importance of LLM systems fitting into broader enterprise workflows rather than operating in isolation. The emphasis on model-agnostic design anticipates the rapidly evolving LLM landscape and positions the system to leverage improvements across providers.
The evolution from Alpha Lab 1.0 to 2.0 illustrates a maturation in thinking about production LLM systems. The shift from handcrafted agent workflows to environment-driven self-improvement represents recognition that hardcoded patterns may not scale as model capabilities advance. The focus on evaluation quality as the foundation for all system capabilities reflects hard-won lessons about LLM behavior in production settings.
While the presentation showcases impressive technical achievements and thoughtful architectural evolution, several aspects warrant balanced consideration. The claim that environments and evaluations encode all organizational value may be somewhat optimistic given that the harness design, prompt engineering, and orchestration logic clearly contribute substantial value. The distinction between what should be open-sourced versus proprietary may prove more nuanced in practice.
The relatively small number of internal production successes mentioned, while positive, represents limited validation compared to the broader vision. The acknowledgment of failure cases without detailed analysis leaves questions about failure modes and mitigation strategies. The self-improving meta-optimization vision, while compelling, remains largely aspirational as of the presentation, with manual tuning still necessary.
The comparison to Kaggle competitions, while useful for evaluation design, may not fully capture the complexities of production financial modeling where distributional shift, model monitoring, and risk management introduce challenges beyond static leaderboard optimization. The presentation focuses heavily on model development and experimentation while giving limited attention to deployment, monitoring, model governance, and other operational concerns central to production LLMOps.
Nevertheless, this case study represents a sophisticated application of agentic LLM systems to a complex production domain, with thoughtful consideration of how to evolve the approach as model capabilities advance. The emphasis on evaluation quality, the willingness to move from handcrafted to learned system designs, and the recognition that human expertise manifests through environment construction rather than workflow engineering all represent valuable insights for the LLMOps community.
Thinking Machines, a new AI company founded by former OpenAI researcher John Schulman, has developed Tinker, a low-level fine-tuning API designed to enable sophisticated post-training of language models without requiring teams to manage GPU infrastructure or distributed systems complexity. The product aims to abstract away infrastructure concerns while providing low-level primitives for expressing nearly all post-training algorithms, allowing researchers and companies to build custom models without developing their own training infrastructure. The company plans to release their own models and expand Tinker's capabilities to include multimodal functionality and larger-scale training jobs, while making the platform more accessible to non-experts through higher-level tooling.
Poolside AI, a foundation model company focused on code generation, developed a comprehensive "Model Factory" system that enables them to train and deploy models from scratch to production in 5-8 weeks with a team of fewer than 70 researchers. Their approach treats model building as 90% engineering, emphasizing automation, reproducibility, and rapid experimentation (10,000-20,000 experiments per month). The result is the Laguna S model (118B parameters, 8B active), which demonstrates that smaller models with better behaviors—persistence, verification, and backtracking—can compete with models 10x their size, suggesting a path toward commoditized, open-weight foundation models.
Shopify's CTO discusses how the company has achieved near-universal AI adoption internally, with nearly 100% of employees using AI tools daily as of December 2025. The company has developed sophisticated internal platforms including Tangle (an ML experimentation framework), Tangent (an auto-research loop for automatic optimization), and SimGym (a customer simulation platform using historical data). These systems have enabled dramatic productivity improvements including 30% month-over-month PR merge growth, significant code quality improvements through critique loops, and the ability to run hundreds of automated experiments. The company provides unlimited token budgets to employees and emphasizes quality token usage over quantity, focusing on efficient agent architectures with critique loops rather than many parallel agents. They've also implemented Liquid AI models for low-latency applications, achieving 30-millisecond response times for search queries.
