Título:
Memory Tokens: Large Language Models can generate reversible sentence embeddings
Otros títulos:
Coordinador:
Director:
Compilador:
Autor:
Sastre, Ignacio
Rosá, Aiala
Rosá, Aiala
Tutor:
Tipo de documento:
Preprint
Editor:
Palabras clave:
Descriptores:
Año de publicación:
2025
Contenido:
Resumen:
In this work, we observe an interesting phenomenon: it is possible to generate reversible sentence embeddings that allow an LLM to reconstruct the original text exactly, without modifying the model’s weights. This is achieved by introducing a special memory token, whose embedding is optimized through training on a fixed sequence. When prompted with this embedding, the model reconstructs the fixed sequence exactly. We evaluate this phenomenon across English and Spanish datasets, sequences of up to approximately 240 tokens, and model scales ranging from 100M to 8B parameters. Notably, Llama 3.1 8B successfully reconstructs all tested sequences. Our findings highlight an interesting capability of LLMs and suggest potential applications in memory-based retrieval, compression, and controlled text generation.
Descripción:
metadata.articulos.dc.description.uri:
Editorial:
EN:
Financiadores:
Beca Maestría ANII POS_FMV_2023_1_1012622.
Citación:
Sastre, I. y Rosá, A. Memory Tokens: Large Language Models can generate reversible sentence embeddings [Preprint] Publicado en : Proceedings of the First Workshop on Large Language Model Memorization (L2M2), Vienna, Austria, August 2025. pp. 183–189.
Citación:
metadata.articulos.cc.license.name:
ISBN:
e-ISBN:
ISSN:
ISMN:
Otros identificadores:
Cobertura geográfica:
Cobertura temporal:
Departamento académico:
Grupo de investigación:
Licencia:
Licencia Creative Commons Atribución - No Comercial - Sin Derivadas (CC - By-NC-ND 4.0)
Colecciones:
| Ficheros | Descripción | Tamaño | Formato | ||
|---|---|---|---|---|---|
| SR25.pdf | Preprint | 371.17 KB | Adobe PDF |