Building and Evaluating European Portuguese Language Models

Overview

Duarte Carmo’s project addresses a specialized LLM development problem: how to create and measure models that perform well in European Portuguese, the Portuguese variety used in Portugal. The central challenge is not simply producing another Portuguese-language model. It is establishing what “good” means for this language variety, assembling sufficiently representative and useful training data, and creating evaluation infrastructure that can distinguish genuine improvements from broad-model capability or benchmark artifacts.

The work combines data engineering, lightweight machine learning, model pretraining experiments, and evaluation design. Carmo constructed successive versions of the Bagasu dataset from the multilingual FineWeb 2 corpus and additional PDF and Wikipedia collections. He supplemented basic domain filtering with a fast European-Portuguese-versus-Brazilian-Portuguese classifier and an educational-value score. The experiments suggested that high-quality educational text improves Portuguese benchmark performance more quickly than indiscriminate corpus growth. Comparisons with Amalia and other open-weight models further suggested that smaller models can be competitive or superior on selected European Portuguese tasks, but the evidence does not establish overall production superiority or a clear business case.

Problem and Use Case

European Portuguese is a lower-resource language variety relative to English and Brazilian Portuguese in many web-scale datasets. A domain suffix such as .pt is a useful initial signal, but it is an incomplete proxy for language variety and content quality. Portuguese-language material can be hosted outside Portugal, while Portuguese websites may contain Brazilian Portuguese, multilingual content, duplicated pages, low-information text, or content that is unsuitable for pretraining.

There was also a measurement problem. Before dedicated evaluation work became available, it was difficult to compare models consistently on European Portuguese. Generic multilingual benchmarks could obscure differences between Portuguese varieties, and a model could appear capable because it had strong general reasoning or knowledge rather than because it handled European Portuguese naturally. Carmo therefore treated evaluation as a first-class part of the project rather than as a final demonstration step.

The intended users are not specified as a single production customer. The potential use cases include Portuguese public-sector and research models, language technology, education-oriented systems, legal question answering, and other applications where European Portuguese quality, local cultural knowledge, or data governance could matter. However, the presentation explicitly leaves open the more demanding question of why a business would adopt a Portugal-specific model when a general model such as Qwen may be cheaper, stronger, or easier to deploy.

Data Pipeline and Dataset Construction

The first Bagasu release was created by extracting European Portuguese candidates from FineWeb 2, a multilingual web-text collection. The initial heuristic selected URLs ending in .pt, producing approximately 10 billion tokens. Each item was also assigned an educational score intended to represent how informative or useful the page was for model training, along with a broad content category such as arts, business, or games. Wikipedia-like educational material received a higher score than lower-information sources such as some sports-news content.

The .pt heuristic was intentionally simple and had clear limitations. To improve language-variety identification, Carmo used research on distinguishing European Portuguese from Brazilian Portuguese and built a smaller classifier optimized for corpus-scale processing. The referenced academic classifier was substantially larger and processed roughly 100 rows per second in the comparison described. Carmo’s classifier was approximately 65 MB, ran in a browser, and processed around 10,000 rows per second. This made it practical to apply across terabytes of text, where a slower or larger model would create an unnecessary data-processing bottleneck.

Bagasu version 2 applied this classifier to FineWeb 2 rather than relying only on domain names. The resulting collection contained approximately 21.3 billion European Portuguese tokens. The language classifier also enabled the discovery of high-value sources that did not have .pt in their domain name, including educational or knowledge-oriented websites. This illustrates an important LLMOps pattern: a relatively small auxiliary model can improve the quality and coverage of a much larger training-data pipeline, provided that its precision, recall, and domain behavior are understood.

Bagasu version 3 expanded the inputs with FineWeb-derived PDF data and FineWeb-derived Wikipedia data. The collection reached approximately 30 billion tokens and moved toward a larger proportion of high-educational-value material, described as roughly 40% under the project’s scoring approach. The dataset exposes fields such as source, European Portuguese score, educational score, category, and character length. The metadata supports filtering, ablation studies, corpus audits, and later investigation of which source types contribute to model quality. The project also recognizes that document count and character count can differ substantially, particularly for PDFs, so both measures are useful for understanding corpus composition.

Training and Evaluation Workflow

The project evaluated small pretrained models under controlled comparisons. Training experiments varied corpus composition, model size, and educational-score thresholds while tracking performance on Portuguese benchmarks during pretraining. The reported results showed that larger models generally performed better, consistent with scaling-law expectations, but data quality changed the rate at which performance improved. Models trained on higher-quality educational text rose faster on Portuguese high-school examination tasks than models trained on broader mixtures containing more low-value material.

A fixed-compute comparison reinforced the same point. A model trained using data with an educational score of at least two achieved a PT Core score of 12.8 and performed better than the other tested data mixtures. The tradeoff was that filtering at this threshold retained only about 20% of the broader Bagasu collection in one analysis. This means that quality filtering is not free: aggressive selection can improve learning efficiency while reducing the quantity and diversity of available text. Bagasu v3 was an attempt to address that constraint by adding more high-quality documents rather than relying only on stricter removal.

The evaluation suite included sentiment classification, linguistic acceptability, reading comprehension, Portuguese high-school examinations, temporal reasoning, and legal question answering. These tasks capture different dimensions of a language model’s behavior, including grammar, comprehension, local knowledge, educational content, and domain-specific knowledge. Carmo also created PT Core, a single aggregate score inspired by compact evaluation projects such as Karpathy’s nanochat work. Aggregation makes model tracking easier during experiments, although it necessarily hides task-level differences and depends on the selection, weighting, and stability of its component benchmarks.

The workflow also uses leaderboards and comparisons across models such as Amalia, Qwen, Gemma, and other open or closed systems. This provides a practical way to monitor progress, but leaderboard results should be interpreted cautiously. Some compared models are reasoning models that generate long internal or visible reasoning traces, while Amalia was described as not being tuned as a reasoning model. A reasoning model may spend considerably more inference-time compute on a question, so a raw accuracy comparison can mix base-model quality with test-time computation. The comparisons also involve different training data, alignment methods, parameter counts, tokenizer choices, and evaluation procedures.

Results and Tradeoffs

The clearest result is that lack of data is not necessarily the primary blocker for European Portuguese. The project reached approximately 30 billion tokens without presenting the effort as an exhaustive collection exercise and estimated that 50 billion or more could be feasible. Synthetic data was mentioned as another possible way to address specific gaps, although no synthetic-data production system or measured benefit was presented.

The experiments also indicate that educational value is a meaningful training-data signal. High-quality subsets improved benchmark performance earlier and, under one fixed-compute comparison, produced the strongest PT Core score. This does not prove that the educational score is universally valid: it is a heuristic, and the benchmark suite may reward the same kinds of text that the score favors. Over-filtering could remove conversational language, regional variation, current events, or other data needed for robust real-world behavior. A production corpus would therefore need audits for duplication, contamination, privacy, copyright, harmful content, and representational balance, none of which are described in detail here.

Amalia was presented as valuable partly because its release included Portuguese-focused datasets and evaluations, even though its European Portuguese pretraining share was described as only about 5% of the data. On the reported comparisons, several models smaller than Amalia’s 9 billion parameters performed better on selected European Portuguese tasks, particularly some Qwen and LFM variants. Amalia nevertheless performed well on at least one temporal-reasoning benchmark relative to some competitors. These findings support testing model size and specialization rather than assuming that a larger Portugal-specific model will automatically be the best option.

Production Readiness and Open Questions

The project demonstrates components of an LLMOps lifecycle: corpus ingestion, automated filtering, metadata generation, scalable classification, dataset versioning through Bagasu releases, training ablations, continuous evaluation during pretraining, aggregate metrics, and cross-model benchmarking. The low operating cost reported for filtering—renting a machine for about €30 per month and completing the process in a few days—also shows how inexpensive experimentation can be when the main task is data processing rather than frontier-scale pretraining. Some model training and evaluation used personally funded compute and borrowed GPU capacity, but the work does not provide a full cost model for repeatable production training, serving, monitoring, or governance.

There is no evidence here of a deployed customer-facing application, online inference service, latency target, uptime objective, human-feedback program, safety monitoring system, or rollback process. The case is therefore best understood as a research and model-development pipeline rather than a proven production LLM service. Before deployment, teams would need to evaluate real user tasks, dialect coverage, hallucination and refusal behavior, data licensing, privacy, security, inference cost, and maintenance as language and model ecosystems change.

The most important unresolved issue is product differentiation. A model trained specifically for European Portuguese may be attractive when local accuracy, sovereignty, transparency, or domain adaptation is more important than general capability. Yet a general multilingual model may already perform better, have stronger tooling, and benefit from broader reasoning and instruction tuning. The next useful step would be benchmark design tied to concrete adoption decisions—for example, a Portuguese public-service workflow, educational assessment, or legal-information task with measurable accuracy, safety, latency, and cost requirements. Without such application-level evidence, the results demonstrate promising data and evaluation practices but do not by themselves establish that a specialized model should replace a general-purpose alternative.