AI/LLM/Ops/Platform

• LLM Ops Platform (Prompt Ops)

  • Platforms that help with LLM operations, including prompt engineering, evaluations, versioning, and deployment
  • Also known as "Prompt Ops" tools
  • Key capabilities typically include:
    • Prompt management and versioning
    • Prompt engineering workflows
    • Evaluation and testing frameworks
    • A/B testing and experimentation
    • Regression testing
    • Dataset management
    • Prompt deployment and rollback
    • Collaboration tools for non-technical stakeholders

  • Platforms

    • PromptLayerAI - Your workbench for AI engineering with prompt management, evaluations, and observability
    • BraintrustAI - AI observability platform with prompt engineering and batch testing
    • HumanloopAI - LLM evaluation platform with prompt management
    • PromptFooAI - Testing framework for detecting prompt injection attacks and security vulnerabilities
    • LangSmith - LLM observability and evaluation platform