LangSmith/Q/Which LangChain tools manage each part of a LangSmith setup?
Which LangChain tools manage each part of a LangSmith setup?
- The parts: prompts in the Prompt Hub, structured outputs, assistants, deployments, resource tags, tracing projects, datasets and evaluators.
- Short answer: no single tool covers everything. The LangSmith/CLI covers prompts, tracing projects, datasets and evaluators. The LangSmith/MCP/Remote mostly reads. The Terraform provider covers deployments, resource tags and evaluators, and nothing about prompts or datasets. Assistants are reachable only through the LangGraph SDKs, the Agent Server API and the UI: no LangChain command line tool, MCP server or Terraform resource manages them.
- For the assistant lifecycle and a Git-based cross-deployment design, see LangSmith/Report/26/10/Assistants as Code in LangSmith Deployments.
- Sources read on 2026-10-08, pinned: LangSmith CLI v0.3.0 at
596db1e, the Terraform provider v0.0.16 at 0896d0f, LangGraph at 40a2e6d, and the LangChain docs at 1da22ee. Prompts in the LangSmith/PromptHub
- LangSmith SDK:
client.push_prompt and client.pull_prompt (docs). - LangSmith CLI:
langsmith prompt lists, gets, creates, deletes, pulls and pushes Prompt Hub repos, and lists their commits (prompt.go). This group is in v0.3.0 but not yet in the CLI docs page. langsmith hub is a different thing: it manages agent and skill repos (hub.go). - LangSmith MCP:
list_prompts and get_prompt_by_name read prompts; push_prompt only returns instructions and changes nothing (standalone server docs). - UI: the Playground.
- Not covered: the Terraform provider has no prompt resource.
Structured outputs
- A structured output lives inside a prompt as its output schema, so the prompt tools above carry it. No tool manages schemas on their own.
- LangSmith SDK: push a
StructuredPrompt, a template plus a schema, with push_prompt, alone or chained with a model (docs). - UI: add an output schema to a prompt in the Playground (docs).
- LangSmith CLI:
langsmith prompt push sends a whole manifest, so it can carry a schema, but it has no schema-specific options. - More in this garden: LangSmith/PromptHub/How To/Define Structured Output and Models for PromptHub in Git.
- Assistants live on each deployment's own Agent Server, not on the LangSmith API, so tools that call the LangSmith API do not reach them.
- LangGraph SDK (Python and JavaScript):
create, update, delete, search, count, get, get_versions and set_latest. create takes an optional assistant_id and an if_exists setting, so a re-run can leave an existing assistant alone (assistants.py). - Agent Server API:
/assistants on each deployment's hostname, which the SDK wraps. - UI: create, edit and version assistants by hand (docs).
- Not covered: the LangSmith CLI, the LangGraph CLI, the LangSmith MCP servers and the Terraform provider. A deployment's own
/mcp endpoint exposes its agents as tools to call, not to manage (docs).
Deployments
- LangGraph CLI:
langgraph deploy builds an image and creates or updates a deployment from it; deploy list, deploy delete, deploy logs and deploy revisions list round it out. All are marked beta (deploy.py). - Terraform provider:
langsmith_deployment builds from a GitHub repository, with data sources for revisions. Changing name replaces the deployment: Terraform deletes it and creates a new one (deployment_resource.go). - Control plane API:
/v2/deployments on the api.host hosts, for CI scripts (docs). - UI: create a deployment and its revisions.
- Not covered: the LangSmith CLI and the LangSmith MCP servers.
Resource tags
- Terraform provider:
langsmith_tag_key, langsmith_tag_value and langsmith_tagging, plus the langsmith_tag shortcut that owns one key and one value. A tagging can point at a deployment, prompt, dataset, tracing project, evaluator and several other types (tagging_resource.go). Prefer the separate key and value resources when other people add values to the same key, since destroying the shortcut destroys the key (docs). - REST API:
/api/v1/workspaces/current/tag-keys and /taggings. The docs show it with plain HTTP calls, not an SDK method (docs). - UI: Settings, then Resource tags.
- Not covered: the LangSmith CLI's typed commands and the LangSmith MCP servers.
langsmith api can call the tag endpoints directly. - Creating keys and applying any key but
Application takes workspaces:manage, which only a Workspace Admin has by default (docs).
Tracing projects
- LangSmith CLI:
langsmith project lists, inspects and deletes them, and trace, run and thread read what is in them (project.go, project_delete.go). - LangSmith MCP:
list_projects, fetch_runs and get_thread_history read them (docs). - LangSmith SDK:
client.create_project, and tracing creates a project on first use. - Terraform provider: only a
langsmith_project data source that finds a project by name, which is enough to tag it or attach a run rule. No resource creates one (docs).
Datasets
- LangSmith SDK:
create_dataset, create_examples and the rest (docs). - LangSmith CLI:
langsmith dataset lists, gets, creates, deletes, exports and uploads; langsmith example manages examples (dataset.go). - LangSmith MCP: reads datasets and examples;
create_dataset and update_examples only return instructions (docs). - Terraform provider: no dataset resource. A
langsmith_run_rule can add matching runs to a dataset that already exists.
- LangSmith CLI:
langsmith evaluator lists, gets, deletes, uploads code evaluators and creates LLM-as-judge rules; evaluator rule manages what attaches them to projects and datasets (evaluator.go). - Terraform provider:
langsmith_evaluator for LLM-as-judge and code evaluators, and langsmith_run_rule to attach one to a tracing project or dataset (evaluator, run rule). - LangSmith SDK:
evaluate() runs offline evaluations against a dataset. - LangSmith MCP:
list_experiments reads results; run_experiment only returns instructions.
What this sweep left out
- Community tools outside
langchain-ai, and the JavaScript SDK beyond the assistants client. The query below covers them.
Deep research query