Refine API contract audit evaluation cases - #964
ArBridgeman merged 4 commits into
Conversation
|
Hi @Rimsha2535 , I refined the evaluation cases in this PR. The main principle is that For the signature/docstring case, the prompt now asks the reviewer to:
This makes the evaluation reproducible and checks that the skill performs an API-contract audit rather than producing a general documentation review. |
097f8c2
into
feature/942-add-api-contract-audit-skill
|
* Add api-contract-audit skill for type hint checking * Update test/resources/skills/api-contract-audit/eval_cases.yml Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com> * Update test/resources/skills/api-contract-audit/eval_cases.yml Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com> * Update test/resources/skills/api-contract-audit/eval_cases.yml Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com> * Refine API contract audit evaluation cases (#964) * Improve API contract skill evaluation cases * Explain evaluation case expectations * Keep API audit evaluation library agnostic --------- Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com> Co-authored-by: Janaarthanan Selvarajan <janaarthanan.selvarajan@exasol.com>



Follow-up to #962 and related to #963.
This stacked PR refines the evaluation cases by:
The branch is based on Rimsha’s PR branch so this PR contains only the follow-up evaluation-case changes.