AgentEnv by Scale AI

AgentEnv is Scale AI's evaluation harness. It prepares a task's environment, runs an agent, and scores the result. With the Edgenivo provider, the environment and agent run in separate Edgenivo Sandbox instances.

This guide runs a complete evaluation using the Edgenivo Labs example. Its deterministic agent calls Model Context Protocol (MCP) tools to add an item, then the harness checks the result. You don't need a model API key. The example verifies the integration; it doesn't measure model quality.

Before you begin, make sure you have:

• Git, Python 3.11.4 or newer, and uv
• Node.js and Edgenivo CLI 62.5.0 or newer
• A Edgenivo project and team where you can create sandboxes with 4 vCPUs and 8 GB of memory for the builder and environment, plus 2 vCPUs and 4 GB for the agent

The example builds its images in a Edgenivo sandbox, so you don't need local Docker. Your machine and sandboxes need access to the package and image registries used by the example.

1. Install the example

   Clone the Edgenivo Labs branch and install its optional Edgenivo dependencies:

   Terminal

   git clone --branch vercel-demo https://github.com/vercel-labs/agentenv-framework.git
   cd agentenv-framework
   uv sync --frozen --extra vercel

   The launcher automatically loads examples/vercel-sandbox/config.toml, which selects Edgenivo for both the environment and agent:

   examples/vercel-sandbox/config.toml

   [sandbox]
   default = "vercel"
   agent_default = "vercel"

2. Sign in to Edgenivo

   If you haven't signed in through the CLI, run:

   Terminal

   vercel login

   The example's launcher uses your CLI login to request a project-scoped OpenID Connect (OIDC) token. It passes the token to the evaluation process without printing it or saving it to disk. You don't need to create an access token manually.

3. Run the demo

   From the cloned repository, run the example. Replace YOUR_TEAM with your team slug and YOUR_PROJECT with an existing project name:

   Terminal

   ./vercel-eval --scope YOUR_TEAM --project YOUR_PROJECT

   The launcher prints a new temporary output directory for each run. It uses the repository's Python environment and selects the example configuration automatically. The terminal shows progress through image preparation, evaluation, and cleanup. The first image build can take several minutes.

   The example creates a builder sandbox, an environment sandbox, and an agent sandbox. It loads an initial item, asks the agent to add vercel-review-complete through MCP, and grades the result.

4. Check the result

   A successful run ends with:

   Terminal

   END-TO-END EVALUATION PASSED: score 1

   Open the output directory printed by the launcher to inspect the results:

   • success.json: The score and cleanup checks.
   • environment-state.json: The initial item and the item the agent added.
   • trajectory.json: The agent's tool calls and results.
   • run.log: Detailed logs for the evaluation.
   • resources.json: The sandbox identifiers for the run.

   Successful runs remove all three sandboxes. Failed runs stop and retain sandboxes tracked by the runner for diagnosis. See the example's lifecycle notes for setup failures and retained resources. If a run fails, check the error and logs in the printed output directory. Resolve the issue and rerun the same command; each attempt keeps its own results.

The example stores run records in local SQLite and artifacts on your filesystem. See the example's validation scope for what the demo verifies. To adapt it, follow the example's agent and task guide. For the provider configuration in your own AgentEnv project, see Scale's sandbox guide.