3 hasil untuk "Wen, Tiansheng"
Menampilkan 1–3 · 0,20 detik · RSS
-
Preprint Teks lengkap
Scaling Attention via Feature Sparsity
Scaling Transformers to ultra-long contexts is bottlenecked by the $O(n^2 d)$ cost of self-attention. Existing methods reduce this cost along the sequence axis through local windows, kernel approximations, or token-level sparsity, but these approaches consistently degrade accuracy. In this paper…
-
Preprint Teks lengkap
CSRv2: Unlocking Ultra-Sparse Embeddings
In the era of large foundation models, the quality of embeddings has become a central determinant of downstream task performance and overall system capability. Yet widely used dense embeddings are often extremely high-dimensional, incurring substantial costs in storage, memory, and inference…
-
Preprint Teks lengkap
Route Experts by Sequence, not by Token
Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed number of experts to all tokens, ignoring their varying complexity. Prior adaptive routing methods introduce additional…