Ask HN: Is anybody building an alternative transformer?

Various alternative architectures to Transformers are being explored, with MAMBA showing promise through faster inference and lower compute costs, performing on par with transformers up to 7B parameters. Researchers are investigating recurrent architectures, state-space models, and efficient attention mechanisms, while debating the future direction of foundation models.

LM2: Large Memory Models

A novel Large Memory Model (LM2) architecture enhances Transformers with an auxiliary memory module, significantly outperforming existing models in multi-hop inference and numerical reasoning tasks. The model demonstrates a 37.1% improvement over RMT and 86.3% over Llama-3.2 on the BABILong benchmark while maintaining strong performance on general tasks.

AI Architecture

Ask HN: Is anybody building an alternative transformer?

LM2: Large Memory Models