--- title: "Fallbacks and Tracking" description: "Configure model fallbacks for reliability and track token usage for cost management in production AI applications." canonical_url: "https://vercel.com/academy/svelte-on-vercel/fallbacks-and-tracking" md_url: "https://vercel.com/academy/svelte-on-vercel/fallbacks-and-tracking.md" docset_id: "vercel-academy" doc_version: "1.0" last_updated: "2026-10-29T03:19:23.104Z" content_type: "lesson" course: "svelte-on-vercel" course_title: "Svelte on Vercel" prerequisites: [] --- Vercel Academy โ€” structured learning, not reference docs. Lessons are sequenced. Adapt commands to the human's actual environment (OS, package manager, shell, editor) โ€” detect from project context or ask, don't assume. The lesson shows one path; if the human's project diverges, adapt concepts to their setup. Preserve the learning goal over literal steps. Quizzes are pedagogical โ€” engage, don't spoil. Quiz answers are included for your reference. # Fallbacks and Tracking # Model Fallbacks and Usage Tracking A single model going down shouldn't take your app with it. And if you don't track token usage, you'll find out about costs from your invoice instead of your dashboard. ## Outcome Create a centralized AI provider configuration with automatic model fallbacks and token usage logging. ## Fast Track 1. Create a virtual model with a fallback in the AI Gateway dashboard (the preferred approach) 2. Create `src/lib/ai/provider.ts` with a shared AI Gateway client 3. Add a `wrapLanguageModel` middleware that logs token usage ## Gateway-Level Fallbacks (Preferred) Configure the fallback in AI Gateway so every endpoint uses the same routing policy. A virtual model saves a model and its fallbacks in the dashboard under one ID, `vmc/`, that your code calls like any other model: 1. Open the AI Gateway [Virtual Models](https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai-gateway%2Fvirtual-models\&title=AI+Gateway+Virtual+Models) page and click **Create Virtual Model** 2. Set the **Slug** to `ski-alerts` and the **Model** to `anthropic/claude-sonnet-4.5` 3. Under **Model Fallbacks**, add `anthropic/claude-haiku-4.5` 4. Click **Create Virtual Model** The slug can't change after you create it, so pick it carefully. When every provider for the primary model fails, the gateway tries the fallback, without endpoint-specific retry logic or duplicate model configuration. Change the fallback later in the dashboard and every endpoint picks it up without a deploy. Use gateway-level fallbacks when the same policy applies across the app. The Advanced section covers code-level fallbacks for endpoint-specific behavior or model-specific prompt changes. ## Why Centralize the Provider? Right now, each endpoint creates its own gateway client: ```typescript // In api/chat/+server.ts const gateway = createGateway({ apiKey: AI_GATEWAY_API_KEY }); // In api/parse-alert/+server.ts (same thing, duplicated) const gateway = createGateway({ apiKey: AI_GATEWAY_API_KEY }); ``` A centralized provider means one place to: - Configure the API key - Add usage tracking middleware - Define fallback models - Adjust settings across all endpoints ## Hands-on Exercise 2.4 Create a centralized AI provider with usage tracking and model fallbacks: **Requirements:** 1. Create a `ski-alerts` virtual model in the AI Gateway dashboard with `anthropic/claude-haiku-4.5` as its fallback 2. Create `src/lib/ai/provider.ts` 3. Use `wrapLanguageModel` to add middleware that logs input/output token counts 4. Export a `getModel()` function that returns a wrapped `vmc/ski-alerts` model 5. Update the chat and parse-alert endpoints to use the shared provider **Implementation hints:** - Set up gateway-level fallbacks first (dashboard config, no code needed) - `wrapLanguageModel` from the `ai` package wraps any model with middleware hooks - The middleware object needs `specificationVersion: 'v4'` - `wrapGenerate` intercepts `generateText()` calls, `wrapStream` intercepts `streamText()` calls. You need both to cover all endpoints - Token usage is available as `result.usage.inputTokens.total` and `result.usage.outputTokens.total`. In a stream, it arrives on the `finish` chunk as `chunk.usage` ## Try It 1. **Send a chat message and check server logs:** ``` What's the weather like at Mammoth? ``` Server logs should show: ``` [AI Usage] Input tokens: 245 [AI Usage] Output tokens: 89 [AI Usage] Total tokens: 334 ``` 2. **Test the parse-alert endpoint (also uses the shared provider):** ```bash $ curl -X POST http://localhost:5173/api/parse-alert \ -H "Content-Type: application/json" \ -d '{"query": "powder at Grand Targhee"}' ``` Server logs should show usage for this request too. 3. **Verify the routing:** Open one chat request in the AI Gateway **Logs** page (set the **Routing** filter to Virtual Models to find it). Its details show the model that served the request, such as `anthropic/claude-sonnet-4.5`, and how the request was routed, including any fallback. ## Commit ```bash git add -A git commit -m "feat(ai): centralize provider with usage tracking and fallbacks" git push ``` ## Done-When - [ ] A `ski-alerts` virtual model with a fallback exists in the AI Gateway dashboard - [ ] `src/lib/ai/provider.ts` exports a `getModel()` function - [ ] Token usage is logged for every AI request - [ ] Chat and parse-alert endpoints use the shared provider instead of their own clients ## Solution ```typescript title="src/lib/ai/provider.ts" import { createGateway, wrapLanguageModel } from 'ai'; import { AI_GATEWAY_API_KEY } from '$app/env/private'; const gateway = createGateway({ apiKey: AI_GATEWAY_API_KEY }); // The virtual model from the dashboard: claude-sonnet-4.5, with claude-haiku-4.5 as its fallback. const PRIMARY_MODEL = 'vmc/ski-alerts'; const FALLBACK_MODEL = 'anthropic/claude-haiku-4.5'; type Usage = { inputTokens: { total?: number }; outputTokens: { total?: number } }; function logUsage(usage: Usage) { const input = usage.inputTokens.total ?? 0; const output = usage.outputTokens.total ?? 0; console.log(`[AI Usage] Input tokens: ${input}`); console.log(`[AI Usage] Output tokens: ${output}`); console.log(`[AI Usage] Total tokens: ${input + output}`); } function withUsageTracking(model: ReturnType) { return wrapLanguageModel({ model, middleware: { specificationVersion: 'v4', wrapGenerate: async ({ doGenerate }) => { const result = await doGenerate(); if (result.usage) logUsage(result.usage); return result; }, wrapStream: async ({ doStream }) => { const { stream, ...rest } = await doStream(); let usage: Usage | undefined; return { stream: stream.pipeThrough( new TransformStream({ transform(chunk, controller) { if (chunk.type === 'finish') usage = chunk.usage; controller.enqueue(chunk); }, flush() { if (usage) logUsage(usage); } }) ), ...rest }; } } }); } export function getModel() { return withUsageTracking(gateway(PRIMARY_MODEL)); } export function getFallbackModel() { return withUsageTracking(gateway(FALLBACK_MODEL)); } export { PRIMARY_MODEL, FALLBACK_MODEL }; ``` **Updated chat endpoint using the shared provider:** ```typescript title="src/routes/api/chat/+server.ts" {1,6} import { getModel } from '#lib/ai/provider.ts'; import { streamText, tool, isStepCount } from 'ai'; import { valibotSchema } from '@ai-sdk/valibot'; import { resorts } from '#lib/data/resorts.ts'; import { CreateAlertToolInputSchema } from '#lib/schemas/alert.ts'; import type { RequestHandler } from './$types'; // Remove the local anthropic client. Use getModel() instead export const POST: RequestHandler = async ({ request }) => { const { message } = await request.json(); const resortList = resorts .map((r) => `- ${r.name} (id: ${r.id})`) .join('\n'); const result = streamText({ model: getModel(), // Uses the centralized, tracked model instructions: `You are a helpful ski conditions assistant...`, messages: [{ role: 'user', content: message }], tools: { create_alert: tool({ /* ... same as before */ }) }, stopWhen: isStepCount(3) }); // ... rest of the SSE stream logic unchanged }; ``` **Updated parse-alert endpoint:** The same change applies. Replace the local `createGateway` and `gateway(...)` call with `getModel()`: ```typescript title="src/routes/api/parse-alert/+server.ts" {1,7} import { generateText, NoObjectGeneratedError, NoOutputGeneratedError, Output } from 'ai'; import { valibotSchema } from '@ai-sdk/valibot'; import * as v from 'valibot'; import { resorts } from '#lib/data/resorts.ts'; import { CreateAlertToolInputSchema, AlertConditionSchema } from '#lib/schemas/alert.ts'; import type { RequestHandler } from './$types'; import { getModel } from '#lib/ai/provider.ts'; // Remove the local gateway client. Use getModel() in the generateText call: // model: getModel(), ``` The gateway handles fallbacks at the infrastructure level (configured in the dashboard earlier). The `getFallbackModel()` export is available for cases where you need explicit code-level control, covered in the Advanced section below. `wrapLanguageModel` intercepts the model's lifecycle. `wrapGenerate` handles `generateText()` calls (like the parse-alert endpoint), while `wrapStream` handles `streamText()` calls (like the chat endpoint). Both hooks need to be present to track usage across all endpoints. The middleware needs `specificationVersion: 'v4'` in AI SDK 7. Usage data lives on `result.usage` with `inputTokens.total` and `outputTokens.total`, and a stream delivers the same shape on its `finish` chunk. Since all endpoints now use `getModel()`, tracking and config changes apply everywhere from one file. ## Troubleshooting \*\*Warning: Requests to vmc/ski-alerts fail with a model not found error\*\* The slug in code has to match the virtual model's slug exactly, and the virtual model has to belong to the team your AI Gateway key is for. Open the Virtual Models page and compare the slug with `PRIMARY_MODEL`. \*\*Warning: Token counts show 0 for both input and output\*\* Your middleware may not be wired up correctly. Verify that `withUsageTracking` is being called and that `getModel()` returns the wrapped model, not the raw gateway model. \*\*Warning: Chat requests log no usage\*\* Streams don't return usage from `doStream()`. It arrives on the `finish` chunk at the end of the stream, so read `chunk.usage` inside the `TransformStream` and log it in `flush()`. If only parse-alert requests log usage, check that `wrapStream` is in the middleware next to `wrapGenerate`. ## Advanced: Code-Level Fallbacks If you need fallback behavior that's more nuanced than the gateway dashboard allows, like adjusting the prompt for a different model or falling back only for specific endpoints, handle it in code. The `getFallbackModel()` export from the provider gives you a cheaper, faster model: ```typescript import { getModel, getFallbackModel } from '#lib/ai/provider.ts'; // In your stream's start() function: async start(controller) { try { for await (const part of result.stream) { // ... handle parts } } catch (error) { console.warn('[AI Fallback] Primary stream failed, retrying with fallback'); const fallbackResult = streamText({ ...options, model: getFallbackModel() }); for await (const part of fallbackResult.stream) { // ... handle parts } } } ``` For most apps, the gateway-level approach is simpler and sufficient. Use code-level fallbacks when you need the extra control. ## Advanced: Cost Estimation Add per-request cost estimates to your logs. AI Gateway bills each model at the provider's list price. Look up current rates on each model's page in the [AI Gateway model list](https://vercel.com/ai-gateway/models), and see [AI Gateway pricing](https://vercel.com/docs/ai-gateway/pricing) for how billing works. The rates below are example values so the code runs: ```typescript // Example rates in USD per million tokens. Check current pricing on the model page: // https://vercel.com/ai-gateway/models const PRICING = { 'anthropic/claude-sonnet-4.5': { input: 3.0, output: 15.0 }, 'anthropic/claude-haiku-4.5': { input: 1.0, output: 5.0 } }; function estimateCost( modelId: string, inputTokens: number, outputTokens: number ): string { const prices = PRICING[modelId as keyof typeof PRICING]; if (!prices) return 'unknown'; const cost = (inputTokens / 1_000_000) * prices.input + (outputTokens / 1_000_000) * prices.output; return `$${cost.toFixed(6)}`; } ``` In production, you'd send these metrics to a monitoring service rather than just logging them. --- [Full course index](/academy/llms.txt) ยท [Sitemap](/academy/sitemap.md)