Hierarchical Rotary Positional Encoding for Long Contexts
An ongoing hierarchically-anchored rotary positional encoding for long-context transformers, validated by from-scratch pretraining of 160M/420M LLMs.
Where is this token?
Scroll to watch it unfold. Each beat is one documented step of the project.
- 01
Global position
Each token still has a position in the whole sequence.
- 02
Chunk index
Positions are grouped into anchored chunks.
- 03
Offset in chunk
Within a chunk, a local offset is kept.
- 04
Rotation
The rotary encoding combines these coordinates instead of one global phase schedule.
- 05
Outcome
A schematic of the idea; exact formulas stay in the research.
- 06
Standard RoPE
One phase schedule for every position, the baseline this work replaces.
An illustration of how the system works, drawn from the write-up below. It is not recorded output.
The idea
Transformers rely on positional encodings to sense word order and distance. The popular RoPE scheme packs local order, long-range distance and structure into one fixed global schedule — which recent analyses link to long-context retrieval failures and numerical fragility. This is an ongoing project that instead factors a token's position into hierarchical coordinates — a coarse chunk index, a within-chunk offset and optional structure — each driving its own band of learnable phases, while leaving some channels unrotated as a content bypass.
Why it's promising
The design is set up to contain RoPE exactly as a special case, so training only departs from that strong baseline when it actually helps — and it is meant to control the numerical error that plagues long-context models in BF16. It's being validated by pretraining 160M- and 420M-parameter models from scratch. (Full formulation is held back while the work is under submission.)
Tech stack & key skills
Core tools, methods and skills demonstrated in this project: