Skip to content

Add Github Copilot CLI harness #199

Description

@bai-uipath

Motivation

Add GitHub Copilot CLI as a harness next to claude-code, codex, antigravity, opencode and pi, so the skills suites can be scored on it. A quick spike (Copilot CLI 1.0.88, BYOK against the Azure OpenAI deployment codex already uses, no GitHub login) completed a read/bash/write task headlessly and loaded the UiPath plugin as-is.

Feasibility: 7/10 (8/10 if Azure-hosted models are enough and Bedrock can wait).

Scope is the Copilot CLI only. The cloud Copilot coding agent (issue in, PR out) is out of scope: we can't sandbox it or control its model.

What the spike showed

Area Status Notes
Headless run Works copilot -p ... --allow-all --no-ask-user --no-auto-update --output-format json streams JSONL: model calls, tool start/complete with args and results, assistant messages, final result with session id
Skills / plugins Works --plugin-dir <skills repo> loads the Claude Code plugin layout unchanged (28 skills); a skill load is a skill tool call carrying the skill name
Per-call tokens Works, two sources JSONL has no tokens; the OTel file exporter (COPILOT_OTEL_FILE_EXPORTER_PATH) has per-call input/output/cache/reasoning tokens with timestamps; OTel input includes cached tokens
Model routing Partial BYOK supports OpenAI-compatible, Azure and Anthropic APIs; Azure verified; no Bedrock provider
Run limits Gap No max-turns flag, only --max-ai-credits

Proposed approach

  • Subprocess + JSONL harness following the OpenCode/Pi pattern, not the preview github-copilot-sdk (0.1.x).
  • BYOK only, via COPILOT_PROVIDER_* env vars; cost from the rate card on OTel tokens, same as OpenCode/Pi.
  • Pin the npm version in the Docker image and always pass --no-auto-update.
  • Enforce max_turns by counting model.call_start and stopping the process, as OpenCode/Pi do.
  • Multi-turn / dialog via --session-id / --resume.
  • Usual harness checklist: AgentKind, config, Dockerfile layer, evalboard badge, HARNESS_PARITY.md row, smoke task, tests.

Open questions

  • Bedrock: put a LiteLLM proxy in front (reuse the existing LITELLM_BASE_URL route), or accept Azure-only for now?
  • GitHub-hosted models: worth supporting at all? They need a Copilot seat and bill in AI credits rather than tokens.
  • Two built-in skills always load alongside ours; can they be disabled, or does activation scoring need to filter them?
  • Release cadence is fast (a new version published the day of the spike); how often do we bump the pin?

Effort

Roughly the size of the Pi harness PR (#159: ~3.4k lines across 42 files, about half tests).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions