.png){kind=icon}
.png){kind=icon}
Return
Alternating Bayesian Mixed Integer Optimization for Context Window Extension in Large Language Models Without Tuning Model Parameters
.png){kind=icon}
PRE
AI
DOI：10.1007/s40305-026-00672-w.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Extending the context window of large language models (LLMs) is crucial for effectively processing long texts, yet existing models typically struggle due to their restricted input lengths. Existing methods for extending context windows mainly include training-based and training-free approaches. However, training-based methods require significant computational resources, while training-free methods typically achieve limited performance. In this paper, we propose a new method to effectively extend the context window of LLMs. This method consists of three key components: First, we conduct an experimental analysis of Rotary Position Embedding (RoPE), a primary positional encoding method used in LLMs, to identify the critical characteristics that influence the context window size and performance of LLMs. Second, based on the analysis results, we formulate a mixed integer black-box optimization model and utilize its optimal solution to determine the best adjustment strategy for RoPE. Third, we develop an alternating Bayesian mixed integer optimization algorithm to solve the model. Extensive experiments show that our method successfully extends the context window of LLMs and outperforms most existing training-based and training-free methods across various tasks.
Keywords:
LLM context window extension
Mixed integer black-box optimization model
Alternating Bayesian mixed integer optimization algorithm
Rotary positional embedding
Journal
J
IF:
1.1
Papers:
64
Citations:
487