Connect promptfoo

Run a small local evaluation against a service text model with promptfoo's OpenAI-compatible provider.

AELDER · SERVICE BASE URL

https://aelder.dev/v1

LOCAL CONFIGURATION

Let my agent set this up

Copy this task to an agent already running on your computer. Enter your API key locally when asked.

Help me connect promptfoo to AELDER using this guide: https://aelder.dev/docs/promptfoo/
Service Base URL: https://aelder.dev/v1
First inspect my installed version, active profile and configuration. Explain any mismatch with the guide before changing anything.
Use the documented local configuration or CLI method. Do not automate the GUI, install software or change permissions without asking.
Preserve existing accounts, providers, models, defaults, permissions and unrelated settings. Propose the smallest change, show a redacted diff, and back up any files before editing.
Have me enter my API key locally using the tool's secret storage or an environment variable. Never ask me to paste a key into chat, print it, or include it in a command, diff, log or commit.
Use the exact model ID I choose from my account's model list. Do not invent model IDs or assume every model supports every API format or tool feature.
Validate syntax and configuration loading before making any generation request. If there is no dry-run, say so.
Explain any proposed paid test, the model and what it sends, and ask for approval before running it. Do not send project files or private conversation history as a test.
Finish with the settings changed, checks performed and steps to restore the backup.

Use your existing promptfoo installation. This setup explicitly selects OpenAI-compatible Chat Completions and does not require Promptfoo Cloud.

1. Prepare a private key variable

Define SERVICE_API_KEY in a private environment or an ignored local secret file. Enter the key locally, without putting its value into shell history, YAML or a setup chat. The configuration below reads the named variable; it does not fall back to another OpenAI key when that variable is missing.

Choose an exact text model ID from the authenticated models endpoint. Copy the service Base URL above, including /v1.

2. Save a small evaluation

Create a separate service-check.yaml file:

prompts:
  - 'Reply with exactly OK.'
providers:
  - id: openai:chat:YOUR_MODEL_ID
    config:
      apiBaseUrl: https://aelder.dev/v1
      apiKeyEnvar: SERVICE_API_KEY
tests:
  - assert:
      - type: equals
        value: OK
sharing: false

Replace YOUR_MODEL_ID with the exact ID. The AELDER Base URL is already set. Keep /v1 in that URL and omit /chat/completions; promptfoo adds the request path. Keep the openai:chat: prefix even for a model from another vendor.

For an existing evaluation, merge only the provider entry and retain its prompts, tests and other providers. This example uses a local string comparison, so it needs no separate model to judge the output. Model-based assertions in other evaluations can make additional calls with their own provider settings.

3. Run and inspect

From the directory containing the file, run:

promptfoo eval -c service-check.yaml --no-cache

If your key is in a private environment file, add --env-file followed by that file’s path. Running the evaluation makes paid model requests; --no-cache forces a fresh request.

Inspect the result and service dashboard. An unexpected text answer is an evaluation failure, which differs from a network or authentication failure. For API failures, check the Base URL, named key variable and exact model ID. The YAML persists locally; promptfoo reads it again on the next run.

See the official OpenAI provider reference and configuration reference.

Need the key? Open your AELDER dashboard. For direct API calls, see the API reference.