AI/Benchmark/Needle-in-a-Haystack

• Needle-ina-Haystack Benchmark

  • Chroma/Paper/25/07/Context Rot - Increasing Input Tokens Impacts LLM Performance
    • includes a nice overview of alternative benchmarks

    • NIAH is fundamentally a simple retrieval task, in which a known sentence (the “needle”) is placed in a long document of unrelated text (the “haystack”), and the model is prompted to retrieve it. While scalable, this benchmark typically assesses direct lexical matching, which may not be representative of flexible, semantically oriented tasks.
      

    •