This case study details Shopify's journey in implementing and scaling Vision Language Models (VLMs) for product understanding and classification across their e-commerce platform. The evolution of their system represents a significant advancement in applying LLMs to practical business problems at scale, with particular attention to production deployment and optimization. ### System Evolution and Architecture Shopify's approach to product classification has evolved significantly since 2018. They began with basic machine learning methods using logistic regression and TF-IDF classifiers, then moved to multimodal approaches in 2020, and finally to their current sophisticated VLM-based system. The current implementation is built on two key foundations: * A comprehensive product taxonomy covering 26 business verticals, 10,000+ product categories, and 1,000+ associated attributes * Advanced Vision Language Models that provide true multimodal understanding, zero-shot learning capabilities, and natural language reasoning The technical implementation shows careful consideration of production requirements and scalability. The system uses a two-stage prediction process: * First stage: Category prediction with simplified description generation * Second stage: Attribute prediction using the category context from the first stage ### Model Selection and Optimization The team has shown a thoughtful approach to model selection and optimization, documenting their progression through different model architectures: * Started with LLaVA 1.5 7B * Moved to LLaMA 3.2 11B * Currently using Qwen2VL 7B Their optimization strategies for production deployment are particularly noteworthy: * Implementation of FP8 quantization to reduce GPU memory footprint while maintaining prediction accuracy * Sophisticated in-flight batching system using Nvidia Dynamo for dynamic request handling and optimal resource utilization * KV cache optimization for improved inference speed, particularly beneficial for their two-stage prediction process ### Production Infrastructure The production system runs on a Kubernetes cluster with NVIDIA GPUs, using Dynamo for model serving. The pipeline architecture demonstrates careful attention to reliability and consistency: * Dynamic request batching based on real-time arrival patterns * Transaction-like handling of predictions with automatic retry mechanisms * Comprehensive monitoring and alerting for prediction quality * Validation against taxonomy rules and structured output processing ### Training Data and Quality Assurance The team implemented a sophisticated multi-stage annotation system to ensure high-quality training data: * Multiple LLMs independently evaluate each product * Structured prompting maintains annotation quality * Dedicated arbitration system resolves conflicts between different model annotations * Human validation layer for complex edge cases and novel product types * Continuous feedback loop for ongoing improvement ### Production Performance and Scale The system demonstrates impressive production metrics: * Processes over 30 million predictions daily * Achieves 85% merchant acceptance rate for predicted categories * Doubled hierarchical precision and recall compared to previous neural network approaches * Successfully handles the complexity of spanning all product categories with structured attributes ### Monitoring and Quality Control The implementation includes robust monitoring and quality control mechanisms: * Continuous validation against taxonomy rules * Automatic retry mechanisms for partial failures * Monitoring and alerting for prediction quality * Regular quality audits of annotation standards ### Challenges and Solutions The case study highlights several challenges in deploying LLMs at scale and their solutions: * Resource optimization through quantization and batching * Handling complex product relationships through taxonomy structure * Maintaining consistency across millions of predictions * Balancing accuracy with computational efficiency ### Future Developments The team has identified several areas for future improvement: * Planning to incorporate newer Vision LM architectures * Migration from tree-based taxonomy to a Directed Acyclic Graph (DAG) structure * Enhancement of metadata extraction capabilities * Expansion of attribute prediction to more specialized categories * Improvements in handling multi-lingual product descriptions This case study is particularly valuable as it demonstrates a practical application of LLMs in a business-critical context, with careful attention to production deployment, optimization, and scaling. The team's approach to balancing model sophistication with practical constraints, and their focus on measurable business outcomes, provides useful insights for others implementing LLMs in production environments.