Skip to content

Refine API contract audit evaluation cases - #964

Merged
ArBridgeman merged 4 commits into
feature/942-add-api-contract-audit-skillfrom
feature/963-improve-skill-eval-cases
Sep 25, 2026
Merged

ArBridgeman merged 4 commits into
feature/942-add-api-contract-audit-skillfrom
feature/963-improve-skill-eval-cases

Conversation

@jana-selva

Copy link
Copy Markdown
Contributor

Follow-up to #962 and related to #963.

This stacked PR refines the evaluation cases by:

  • using concrete PyExasol APIs in the signature/docstring case;
  • requiring evidence from Python introspection helpers;
  • making must_include and must_not_include expectations more specific.

The branch is based on Rimsha’s PR branch so this PR contains only the follow-up evaluation-case changes.

@jana-selva

Copy link
Copy Markdown
Contributor Author

Hi @Rimsha2535 , I refined the evaluation cases in this PR.

The main principle is that must_include should require observable evidence of the audit, rather than broad words such as “runtime” or “annotation” that could appear in a generic answer.

For the signature/docstring case, the prompt now asks the reviewer to:

  • inspect two concrete public functions from the target Python library;
  • name those functions in the report;
  • use inspect.signature(), inspect.get_annotations(), typing.get_type_hints(), and inspect.getdoc() where applicable;
  • report the actual mismatch, severity, and source-file references.

This makes the evaluation reproducible and checks that the skill performs an API-contract audit rather than producing a general documentation review.
The must_not_include entries are intended to prevent the skill from proposing code changes, treating one source as authoritative, or reporting a mismatch without evidence. The broader documentation and test work remains scoped to issue #963.

@jana-selva
jana-selva deployed to manual-approval September 25, 2026 08:26 — with GitHub Actions Active
@jana-selva
jana-selva deployed to manual-approval September 25, 2026 08:26 — with GitHub Actions Active
@ArBridgeman
ArBridgeman merged commit 097f8c2 into feature/942-add-api-contract-audit-skill Sep 25, 2026
32 checks passed
@ArBridgeman
ArBridgeman deleted the feature/963-improve-skill-eval-cases branch September 25, 2026 11:38
@sonarqubecloud

Copy link
Copy Markdown

ArBridgeman added a commit that referenced this pull request Sep 25, 2026
* Add api-contract-audit skill for type hint checking
* Update test/resources/skills/api-contract-audit/eval_cases.yml

Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com>

* Update test/resources/skills/api-contract-audit/eval_cases.yml

Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com>

* Update test/resources/skills/api-contract-audit/eval_cases.yml

Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com>

* Refine API contract audit evaluation cases (#964)

* Improve API contract skill evaluation cases

* Explain evaluation case expectations

* Keep API audit evaluation library agnostic

---------

Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com>
Co-authored-by: Janaarthanan Selvarajan <janaarthanan.selvarajan@exasol.com>

This branch was successfully deployed

1 active deployment
manual-approval — e1a8e847 Deployed Sep 25, 2026 by jana-selva via Merge Gate / Approve Running Slow Tests? #2824
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants