Conference/ORCwAI/Pt3/1500 Eval Engineering Lili Jiang

• The End of Machine Learning Engineering as We Know It – Person/Lili Jiang (15 minutes)

  • My Notes

    • high level

      • REALLY great talk

      • Action items

        • TODO find lili jiang's blogs, etc and subscribe to Person/Lili Jiang
    • worked for quora
      • key question - merging questions about cookie recipes
    • machine learning engineer is now becoming an eval engineer
    • also worked at Waymo - led the AI/Eval team
    • "in other words, the eval code is similar in complexity with the code under eval"
    • 15:06 eval metrics are increasingly looking like eval algorithms
    • 15:06 comparison to feature engineering in the modeling step
    • 15:07 how do we build a flywheel to identify these evaluation metrics?
      • time with Coverbase LLM Vendor/Management
    • 15:11