## Overview of the Swedish Tax Authority's AI Journey The Swedish Tax Authority (Skatteverket) represents a compelling case study in how public sector organizations can systematically adopt and operationalize large language models and AI systems over an extended period. The organization features an AI strategist role (held by Gita Gintautas) that bridges technical development and management, working closely with product owners and development teams. This role emerged from years of data science work within the organization, reflecting a maturation of AI capabilities from experimental projects to strategic, production-scale implementations. The Authority's digitalization journey spans multiple decades, with digital tax returns beginning as early as 1998 according to the transcript discussion. However, the meaningful integration of AI technologies, particularly for text processing, began around 2018-2019. This timing coincided with organizational transformation including the adoption of Agile SAFe frameworks for software development, creating a modern foundation for iterative AI development and deployment. ## The Text Processing Foundation A critical insight from this case study is the recognition that text represents the dominant modality of interaction between citizens and the tax authority. According to their annual reports, the largest interaction pools include website visits, personal page logins, calls, and email contacts—all primarily text-based communication channels. This reality drove the initial AI strategy to focus heavily on NLP applications as the foundation for automation and digitalization efforts. The organization developed a portfolio of text processing applications including text categorization systems, transcription services, and OCR pipelines for digitizing paper tax declarations. These foundational capabilities serve as reusable components across different business domains within the organization. The approach reflects a platform mindset where core AI capabilities are developed once and then adapted for multiple use cases, creating efficiency and consistency across the organization. ## Production RAG Systems and Model Evaluation One of the most concrete LLMOps examples discussed involves a RAG (Retrieval-Augmented Generation) solution for question answering and support, tested in 2023-2024. The team conducted systematic benchmarking of multiple models including Llama 3.1, Mixtral 7B (specifically mentioned as Mixtral 78 in transcript, likely Mixtral 7B), Cohere for AI, and GPT-3.5 as a commercial baseline. The evaluation methodology incorporated LangChain for the RAG implementation and tested across different levels of question complexity. The results revealed nuanced performance characteristics that challenge simplistic narratives about commercial versus open-source model superiority. For simpler questions (typically one or two sentences), the open-source models performed very well and in some cases slightly outperformed GPT-3.5. However, for more complex queries, GPT-3.5 demonstrated better performance. This finding has significant implications for production deployment decisions—suggesting that organizations can potentially use smaller, open-source models for a substantial portion of use cases while reserving more expensive commercial models for complex scenarios, creating a tiered approach to model deployment. The discussion also touched on the broader challenge of benchmark saturation and the need for more sophisticated evaluation frameworks. The team referenced a framework called "360" from an AI University (likely referring to comprehensive evaluation approaches) that attempts to provide broader spectrum testing including aspects like truthfulness, bias, and safety considerations beyond simple accuracy metrics. ## Open Source Strategy and On-Premise Requirements A distinctive aspect of this case study is the strong emphasis on open-source models, driven primarily by regulatory and data sovereignty requirements rather than purely cost considerations. The Authority works with sensitive citizen data that cannot be sent to external commercial APIs, necessitating on-premise or private cloud deployment. This constraint has pushed the organization to develop deep expertise in operationalizing open-source models. The discussion reveals a sophisticated understanding of the tradeoffs involved. While acknowledging that commercial models may have a slight quality edge (described as potentially being "a couple of percentage points better"), the speakers argue that the total system performance depends far more on factors beyond the base model—including fine-tuning, system integration, UX design, and the broader "compound AI system" architecture. This perspective aligns with the "Hidden Technical Debt in Machine Learning Systems" paper from Google (2015), which suggests the model itself represents less than 5% of a production ML system. The open-source approach also provides flexibility for experimentation, customization, and avoiding vendor lock-in. The organization can test multiple model variants, fine-tune for specific Swedish tax domain knowledge, and maintain control over the entire pipeline. However, the transcript also reveals honest acknowledgment of challenges—open-source models typically lag commercial frontiers by several months, and building the infrastructure to run these models securely requires significant engineering investment. ## Infrastructure and Platform Architecture The case study discusses an emerging concept called "AI Varan" (AI Workshop), a proposal from the Swedish Tax Authority in collaboration with the Swedish insurance agency. This initiative aims to create shared AI infrastructure that public sector agencies can use to develop, deploy, and scale AI applications securely. The concept is not about training frontier models but rather providing the secure computing environment, development tools, and deployment capabilities needed for practical AI applications in government. This reflects a platform-oriented approach to LLMOps where infrastructure investments are amortized across multiple agencies and use cases. The discussion touches on the challenge of "value lineage"—understanding how a centralized investment in a capability like text categorization creates value across multiple business domains. For example, if ten different departments can reuse a text categorization engine, the business case becomes compelling, but if only one uses it, the investment doesn't justify itself. The infrastructure must support the unique constraints of public sector work, including data segregation between organizational silos (which exist "by design" due to legal and security requirements), while still enabling knowledge sharing and component reuse. This creates an interesting architectural challenge of building distributed systems with shared platform capabilities while maintaining strict data boundaries. ## Organizational Structure and Ways of Working The Swedish Tax Authority adopted the SAFe (Scaled Agile Framework) around 2019, representing a significant organizational transformation. The framework provides structures for value streams, Agile Release Trains (ARTs), and portfolio management aligned with agile principles. The organization has undergone multiple reorganizations to better align business areas with development capabilities and improve portfolio steering. The AI strategy role sits separately from IT, serving as a bridge between technical development and business leadership. This positioning enables support for AI project prioritization and decision-making across the organization. Development teams work flexibly across different business silos, creating solutions that can be scaled and adapted for various domains while maintaining architectural independence. The discussion reveals tensions common in large organizations between centralized and decentralized approaches. While business domains operate in silos (often by regulatory design), development teams aim to create platform capabilities that can serve multiple domains. This requires careful management of dependencies, data contracts, and value attribution across organizational boundaries. ## Regulatory Compliance and EU AI Act Preparation A significant portion of the discussion addresses how the organization is preparing for the EU AI Act, participating in proof-of-concept regulatory sandboxes. These workshops involve legal experts and technical teams analyzing specific AI project ideas through the lens of AI Act requirements, including AI definition assessment, risk profiling, role determination, and compliance obligations. However, the discussion reveals a critical gap—the current approach feels "waterfall" and "design-heavy," requiring fairly complete project specifications upfront rather than supporting iterative, agile development cycles. The speakers express concern that lawyers are leading these efforts without sufficient engineering perspective on how modern software development actually works. They advocate for a "shift-left" approach where compliance considerations are built into development processes from the start rather than treated as a final gate. The conversation draws parallels to GDPR compliance and security-by-design principles, arguing that with proper documentation, risk assessment, and legal basis established early in development, compliance can be maintained continuously through small, incremental changes rather than large batch approvals. The example of Tesla/SpaceX's approach to regulatory compliance—making continuous small changes with daily regulatory submissions rather than large batch approvals—illustrates a more agile approach to compliance that could be adapted for AI Act requirements. ## Technical Debt and Engineering Excellence Throughout the discussion, there's a recurring theme about the importance of engineering excellence versus research capabilities. The speakers note that European public sector organizations often over-invest in research relative to engineering, while successful tech companies typically invest at least 10 times more in engineering than research. The critical capability gap isn't necessarily in understanding cutting-edge AI research but in having engineers who know how to build robust, scalable AI systems in production. This perspective challenges common narratives about needing to compete with frontier model development (like the Stargate project's $500 billion investment). Instead, the speakers argue that Sweden and Europe should focus on being world-class at applying AI—building practical applications, fine-tuning models for specific domains, and creating the engineering infrastructure to deploy AI safely and effectively at scale. The discussion of "distillation" and creating smaller, specialized models from larger frontier models represents this applied engineering approach. The case study also touches on data engineering patterns, noting that many perceived barriers to data sharing are actually solvable technical problems rather than fundamental limitations. Examples from regulated industries like energy (with unbundling requirements) and multi-national corporations (like Enel managing data across 40 countries with different regulations) demonstrate that sophisticated data engineering can enable secure, compliant data operations even in highly regulated environments. ## Broader Ecosystem and Future Directions The Swedish Tax Authority's work exists within a broader ecosystem of AI development in Swedish public sector. The discussion references the Swedish AI Commission's report (released before Christmas, presumably 2024) containing 75 proposed actions with concrete budgets for advancing AI in Sweden. The commission emphasizes research investment, which the speakers partially challenge by arguing for more focus on engineering capabilities and practical infrastructure. The case also touches on emerging topics like DeepSeek's R1 model, which achieved OpenAI o1-level reasoning performance using open-source approaches at dramatically lower cost. This development validates the open-source strategy for organizations like the Swedish Tax Authority, suggesting that the gap between open-source and commercial models may be narrowing in ways that favor practical deployment of capable models on-premise. Looking forward, the organization faces questions about how to scale AI capabilities across government, balance centralized infrastructure with domain autonomy, navigate the EU AI Act's requirements in agile development processes, and maintain engineering excellence in an environment that traditionally emphasizes research and policy over practical implementation. The case study provides valuable lessons for other public sector organizations navigating similar challenges in operationalizing LLMs responsibly and effectively.