A language model built for a specific linguistic community

AMALIA is an important example of what it means to build an open language model not as a generic commercial product, but as a public research infrastructure for a specific linguistic community. Its central goal is clear: European Portuguese should not be treated as a secondary variant inside models mostly trained on English and, within Portuguese, heavily influenced by Brazilian Portuguese. AMALIA was designed to make European Portuguese a first-class language variety in both training and evaluation.

The project addresses a structural weakness of today’s large language model ecosystem. Many open-weight and closed models perform well in English and reasonably well in high-resource languages, but they often fail to capture the linguistic, administrative, educational and cultural details of smaller or underrepresented language varieties. Machine-translated benchmarks are not enough, because translation can preserve the surface of a question while losing the actual cultural and linguistic difficulty. AMALIA responds to this problem with a full technical pipeline: targeted data collection, rigorous filtering, continued pretraining, supervised fine-tuning, preference optimization, native benchmarks and open release.

Data as infrastructure: turning Arquivo.pt into a training corpus

The most important design decision behind AMALIA was the use of Arquivo.pt, the Portuguese Web Archive, as a primary source of European Portuguese data. The team collected publicly available WARC archives from Arquivo.pt, amounting to 195 TB in raw form. This is not merely a matter of scale. A national web archive captures public language in context: institutions, media, education, culture, policy, public services, everyday writing and local expressions. For a language-specific model, this kind of corpus is more valuable than an undifferentiated multilingual web scrape.

Raw web data, however, cannot be used directly for model training. AMALIA therefore relied on a multi-stage processing and filtering pipeline inspired by FineWeb2 and implemented with DataTrove. The pipeline started with URL filtering, including the removal of “.br” domains to reduce Brazilian Portuguese content. It also used blacklists to remove sensitive or inappropriate material. HTML pages were processed with Trafilatura to extract the main text, after which short lines, duplicate lines and remaining boilerplate were removed.

The team then applied language identification, quality filters adapted to European Portuguese, repetition filters, encoding repair and personal data removal. Public IP addresses, email addresses and phone numbers were removed to improve compliance with privacy requirements. Deduplication was performed using MinHash, reducing the risk that repeated pages or templated content would dominate the training distribution. Finally, the data was classified by quality using EuroFilter and divided into high, medium and low quality splits. The high and medium quality portions, totaling about 5.8 billion tokens, became the European Portuguese component of the training mix.

This process is a central lesson of AMALIA. A language model is not built only by choosing a neural architecture. It is built by deciding what counts as legitimate, representative, clean, reusable and culturally meaningful data.

Continued pretraining from EuroLLM-9B

AMALIA did not start from a blank model. It modified the final pretraining phase of EuroLLM-9B, a European multilingual model designed to support all official EU languages and additional European languages. This was a pragmatic and efficient choice. Instead of spending enormous resources on training a model from scratch, the team used an existing European multilingual base and continued its training with a carefully designed mixture.

The pretraining mix combined four major components. First, the original EuroLLM training mixture, including around 40 billion tokens from multiple European languages, with some mathematics and code. Second, an additional 60 billion tokens with improved code coverage. Third, 1.4 billion synthetic long-context tokens. Fourth, the 5.8 billion tokens extracted and filtered from Arquivo.pt.

A key architectural change was the extension of the maximum sequence length from 4K to 32K tokens. To support this, the team used RoPE scaling, increasing the relevant scaling parameter from 10,000 to 1,000,000. In practical terms, AMALIA was trained to handle much longer contexts. This matters for public-sector use, education, legal documents, long reports, technical manuals and archival material, where the ability to process extended documents is often more important than producing short conversational replies.

The continued pretraining phase took 80 hours on 256 NVIDIA H100 GPUs. This shows that building a serious national or language-specific model still requires substantial compute, but it also shows that strategic continued pretraining can be far more attainable than trying to reproduce the scale of closed frontier systems.

Supervised fine-tuning: instructions, dialogue, reasoning and safety

After pretraining, AMALIA underwent supervised fine-tuning to improve instruction following and conversational behavior. The training mixture was organized into four categories: instruction following, conversational reasoning, mathematical problem-solving and safety. This structure reflects a mature view of model development. A model must not only know language. It must follow instructions, hold a dialogue, solve problems and refuse unsafe requests.

The SFT mixture combined synthetic data, manually curated data and open licensed datasets from Hugging Face. Synthetic generation relied heavily on the PersonaHub approach, using diverse personas to generate varied instruction-following and conversational examples. European Portuguese data was produced through machine translation with a dedicated pt-PT machine translation model and Gemma 3-27B, followed by question and answer generation. The use of larger and varied models for data generation helped reduce the risk of narrow sampling bias.

A particularly important part of the data mixture was the manually curated Portuguese Linguistic Instructions dataset. Although small, at 200 entries, it was created by a Portuguese linguistics expert and covered phonetics, orthography, wordplay, idiomatic expressions and grammatical classification. This kind of data is crucial because large models often fail on precisely the features that define a language variety: idioms, register, cultural references, morphology and variant-specific vocabulary.

The team also performed careful filtering of synthetic data. All reasoning traces were removed, self-referential artifacts from other models were eliminated, and the Deita Quality Scorer was used to remove low-quality synthetic examples. Global deduplication ensured that only one entry per unique user prompt was retained.

The SFT stage ran for 14,000 steps, approximately 4.25 epochs, using AdamW, cosine learning rate scheduling, weight decay, bfloat16 mixed precision and checkpoint selection based on validation performance. Training took 76 hours on 64 H100 GPUs.

Preference training with DPO

The third major stage was preference training using Direct Preference Optimization. The initial strategy followed a process inspired by recent open model development practices: 200,000 prompts were sampled from the SFT dataset, 32 candidate responses were generated by the AMALIA-SFT model for each prompt, and a reward model, ArmoRM, was used to select the highest and lowest scoring responses as chosen and rejected answers.

This improved overall performance, but it also revealed a common problem in preference optimization: a reward model can favor fluent, general responses even when they fail strict instructions or weaken mathematical reasoning. The AMALIA team therefore adjusted the preference mixture with domain-specific strategies. For persona datasets, original answers were used as chosen responses and AMALIA-SFT generations as rejected responses. For mathematics, stronger chosen answers were generated with Qwen 3-32B and weaker rejected answers with Qwen 3-0.6B. Additional Portuguese data was added to protect European Portuguese capabilities.

Safety was also built into the preference stage. The team incorporated safety datasets, including one tailored to the Portuguese cultural context, so that alignment would not be purely generic or English-centric. In total, the DPO mixture contained 478,000 preference pairs. Training was completed in one epoch on 64 H100 GPUs.

Native evaluation, not only translated benchmarks

AMALIA’s strongest contribution may be its evaluation strategy. The team did not rely only on translated versions of standard English benchmarks. It built a pt-PT benchmark collection that included translated standard tasks, human-authored or native Portuguese material and four new datasets targeting European Portuguese.

PT-PT Completions tests whether a model can choose European Portuguese expressions over Brazilian Portuguese alternatives. PT Exams uses questions from official Portuguese national high school exams, including multiple-choice and open-ended questions across six subjects: mathematics, Portuguese, history, geography, biology/geology and philosophy. ALBA evaluates linguistic competence across categories such as language variety, culture-bound semantics, discourse analysis, wordplay, syntax, morphology, lexicology, and phonetics and phonology. P3B3 measures whether a model, when explicitly instructed to use European Portuguese, still drifts toward Brazilian Portuguese.

This evaluation design matters because model quality is always relative to what we measure. A model that performs well on machine-translated MMLU may still fail to write naturally in European Portuguese. AMALIA’s benchmark suite therefore, aligns evaluation with the actual linguistic and cultural goal of the project.

Results and broader significance

AMALIA achieved state-of-the-art or highly competitive results among fully open models of comparable size on several benchmarks, with especially strong performance on European Portuguese tasks. The DPO variant improved performance on Portuguese exams, language generation and instruction following, while also showing strong safety results, with low attack success rates on safety benchmarks. Most importantly, AMALIA showed much lower bias toward Brazilian Portuguese than many strong multilingual models, confirming that targeted data and native evaluation can change model behavior.

The broader significance is clear. AMALIA is not only a Portuguese achievement. It is a European blueprint. It shows that open language models for specific linguistic communities can be built through public research collaboration, national web archives, open datasets, transparent evaluation and carefully designed post-training. It also shows that open-source AI is not just about publishing weights. Full openness requires code, data documentation, training procedures, benchmarks, model cards, licenses and reproducible evaluation.

For Greece and other European countries, the lesson is direct. A fully open Greek language model should not be a superficial adaptation of an English-centric system. It should be built on Greek public knowledge, web archives, legislation, educational material, cultural corpora and domain-specific datasets processed through transparent pipelines. Tools such as GlossAPI can play the role of a national AI-ready data pipeline, turning PDFs, public documents and cultural material into clean, reusable corpora. AMALIA demonstrates that linguistic sovereignty in AI is not an abstract slogan. It is a technical stack, a data governance model and a public-interest strategy.

Source of this article: blog.glossapi.gr