Motivation
Add GitHub Copilot CLI as a harness next to claude-code, codex, antigravity, opencode and pi, so the skills suites can be scored on it. A quick spike (Copilot CLI 1.0.88, BYOK against the Azure OpenAI deployment codex already uses, no GitHub login) completed a read/bash/write task headlessly and loaded the UiPath plugin as-is.
Feasibility: 7/10 (8/10 if Azure-hosted models are enough and Bedrock can wait).
Scope is the Copilot CLI only. The cloud Copilot coding agent (issue in, PR out) is out of scope: we can't sandbox it or control its model.
What the spike showed
| Area |
Status |
Notes |
| Headless run |
Works |
copilot -p ... --allow-all --no-ask-user --no-auto-update --output-format json streams JSONL: model calls, tool start/complete with args and results, assistant messages, final result with session id |
| Skills / plugins |
Works |
--plugin-dir <skills repo> loads the Claude Code plugin layout unchanged (28 skills); a skill load is a skill tool call carrying the skill name |
| Per-call tokens |
Works, two sources |
JSONL has no tokens; the OTel file exporter (COPILOT_OTEL_FILE_EXPORTER_PATH) has per-call input/output/cache/reasoning tokens with timestamps; OTel input includes cached tokens |
| Model routing |
Partial |
BYOK supports OpenAI-compatible, Azure and Anthropic APIs; Azure verified; no Bedrock provider |
| Run limits |
Gap |
No max-turns flag, only --max-ai-credits |
Proposed approach
- Subprocess + JSONL harness following the OpenCode/Pi pattern, not the preview
github-copilot-sdk (0.1.x).
- BYOK only, via
COPILOT_PROVIDER_* env vars; cost from the rate card on OTel tokens, same as OpenCode/Pi.
- Pin the npm version in the Docker image and always pass
--no-auto-update.
- Enforce
max_turns by counting model.call_start and stopping the process, as OpenCode/Pi do.
- Multi-turn / dialog via
--session-id / --resume.
- Usual harness checklist:
AgentKind, config, Dockerfile layer, evalboard badge, HARNESS_PARITY.md row, smoke task, tests.
Open questions
- Bedrock: put a LiteLLM proxy in front (reuse the existing
LITELLM_BASE_URL route), or accept Azure-only for now?
- GitHub-hosted models: worth supporting at all? They need a Copilot seat and bill in AI credits rather than tokens.
- Two built-in skills always load alongside ours; can they be disabled, or does activation scoring need to filter them?
- Release cadence is fast (a new version published the day of the spike); how often do we bump the pin?
Effort
Roughly the size of the Pi harness PR (#159: ~3.4k lines across 42 files, about half tests).
Motivation
Add GitHub Copilot CLI as a harness next to claude-code, codex, antigravity, opencode and pi, so the skills suites can be scored on it. A quick spike (Copilot CLI 1.0.88, BYOK against the Azure OpenAI deployment codex already uses, no GitHub login) completed a read/bash/write task headlessly and loaded the UiPath plugin as-is.
Feasibility: 7/10 (8/10 if Azure-hosted models are enough and Bedrock can wait).
Scope is the Copilot CLI only. The cloud Copilot coding agent (issue in, PR out) is out of scope: we can't sandbox it or control its model.
What the spike showed
copilot -p ... --allow-all --no-ask-user --no-auto-update --output-format jsonstreams JSONL: model calls, tool start/complete with args and results, assistant messages, finalresultwith session id--plugin-dir <skills repo>loads the Claude Code plugin layout unchanged (28 skills); a skill load is askilltool call carrying the skill nameCOPILOT_OTEL_FILE_EXPORTER_PATH) has per-call input/output/cache/reasoning tokens with timestamps; OTel input includes cached tokens--max-ai-creditsProposed approach
github-copilot-sdk(0.1.x).COPILOT_PROVIDER_*env vars; cost from the rate card on OTel tokens, same as OpenCode/Pi.--no-auto-update.max_turnsby countingmodel.call_startand stopping the process, as OpenCode/Pi do.--session-id/--resume.AgentKind, config, Dockerfile layer, evalboard badge,HARNESS_PARITY.mdrow, smoke task, tests.Open questions
LITELLM_BASE_URLroute), or accept Azure-only for now?Effort
Roughly the size of the Pi harness PR (#159: ~3.4k lines across 42 files, about half tests).