.png){kind=icon}
.png){kind=icon}
Return
From raw text to fairseq RoBERTa: A modular snakemake-based framework enabling language-specific BPE tokenization
.png){kind=icon}
PRE
AI
DOI：10.1016/j.simpa.2026.100824.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Large-scale language model training requires robust and reproducible data preprocessing. While fairseq provides efficient training routines for RoBERTa models, preparing high-quality, language-specific data remains complex. We present modular Snakemake-based workflows for large-scale language model preparation, covering filtering, GPT-2 BPE tokenization, and fairseq-compatible data generation. The pipelines support new and existing tokenizers, enable scalable HPC parallelism, and include utilities for converting trained models to the Huggingface format. Bundled with a fairseq fork supporting GPU clusters and Cloud TPUs, the framework has been used to train GottBERT, GeistBERT, ChristBERT, PortBERT, SindBERT, and HalleluBERT, and generalizes into a reusable preprocessing infrastructure.
Keywords:
Natural language processing
Language models
Data preprocessing
Text tokenization
Open source software
RoBERTa
fairseq
Snakemake
Hugginface