Skip to content

Automatically Generated Security Regression Tests - #43

Open
officialfrancismendoza wants to merge 2 commits into
openhackai:mainfrom
officialfrancismendoza:feat/officialfrancismendoza/041-auto-generate-security-regression-tests
Open

officialfrancismendoza wants to merge 2 commits into
openhackai:mainfrom
officialfrancismendoza:feat/officialfrancismendoza/041-auto-generate-security-regression-tests

Conversation

@officialfrancismendoza

Copy link
Copy Markdown

Summary

Implements autonomous security regression test generation for verified vulnerabilities, helping prevent previously discovered and remediated security issues from being reintroduced. Solves (#41).

Changes

  • Generate executable regression tests from verified vulnerabilities.
  • Validate tests against vulnerable and remediated implementations.
  • Integrate generated tests into existing remediation workflows where supported.
  • Support configurable test generation and graceful failure handling.
  • Add automated tests and documentation.

Motivation

Extends OpenHack's vulnerability discovery and remediation capabilities into persistent security regression prevention, enabling developers to catch recurring vulnerabilities through CI/CD without requiring repeated AI-powered scans.

@greptile-apps

greptile-apps Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 3/5

[Medium impact] The PR adds an optional generated-test workflow to scans and saved reports.

Fix the disabled coverage-plugin failure and normalize reserved test paths before merging.

Fix All in CodexFindings

  1. P1 Disabled coverage stops pytest ▶
  2. P1 Generated patches conflict ▶
Fix with agent prompt
### Issue 1
openhack/regression/frameworks.py:704
If a project installs `pytest-cov` but disables it with `addopts = -p no:cov`, importing `pytest_cov` still succeeds. This command then adds `--no-cov`, but the disabled plugin cannot register that option. Pytest exits before running any tests, so otherwise valid regression tests cannot be validated. Check whether the plugin is enabled before adding its option.

### Issue 2
openhack/regression/pipeline.py:495-506
Accepted test paths are not reserved across findings. Two IDOR findings for the same resource get the same suggested filename because `available()` checks only the original revisions. Both tests can be accepted, but their patches both add the same file, so users cannot apply both. Track accepted repository paths and reject or rename later collisions.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds opt-in security regression tests for sandbox-verified IDOR findings in pytest projects.

  • Verified IDOR findings can produce tests for the app’s ownership rules.
  • Candidate tests must distinguish vulnerable code from its fixed version.
  • Scans and saved reports can show regression-test results and follow-up checks.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Verified IDOR finding] --> B[Extract security property]
    B --> C[Generate pytest file]
    C --> D[Check path and test code]
    D --> E[Run against vulnerable code]
    E --> F{Controls pass and security assertion fails?}
    F -- No --> C
    F -- Yes --> G{Fixed revision available?}
    G -- No --> H[Save pending test and patch]
    G -- Yes --> I[Run against fixed code]
    I --> J{All tests pass and runs finish cleanly?}
    J -- Yes --> K[Save validated test and patch]
    J -- No --> L[Report failure or retry setup]
Loading

Reviews (2) · Last reviewed commit: "Updated with fixes from Greptile feedbac..." · Reviewed by Greptile

Comment thread openhack/regression/workspace.py Resolved
Comment thread openhack/regression/frameworks.py Resolved
Comment thread openhack/regression/frameworks.py Resolved
Comment on lines +476 to +487
def available(rel: str, stored_code: Optional[str] = None) -> bool:
path = repo_path(rel)
if vulnerable.exists(path):
return False
if remediated is not None and remediated.exists(path):
# The developer may have committed our pending test alongside
# the fix; re-validating that identical file is fine.
existing = (remediated.root / path).read_text(encoding="utf-8", errors="replace")
return stored_code is not None and existing == stored_code
return True

suggested = adapter.suggest_test_path(profile, invariant, lambda rel: not available(rel))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Generated patches conflict

Accepted test paths are not reserved across findings. Two IDOR findings for the same resource get the same suggested filename because available() checks only the original revisions. Both tests can be accepted, but their patches both add the same file, so users cannot apply both. Track accepted repository paths and reject or rename later collisions.

Prompt To Fix With AI
This is a comment left during a code review.
Path: openhack/regression/pipeline.py
Line: 476-487

Comment:
**Generated patches conflict**

Accepted test paths are not reserved across findings. Two IDOR findings for the same resource get the same suggested filename because `available()` checks only the original revisions. Both tests can be accepted, but their patches both add the same file, so users cannot apply both. Track accepted repository paths and reject or rename later collisions.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code Fix in Cursor Fix in Conductor

Comment thread openhack/tui.py Resolved
Comment thread openhack/sandbox/container_runner.py Resolved
Comment thread openhack/regression/frameworks.py Resolved
Comment thread openhack/regression/pipeline.py Resolved
# --no-cov is added (only when pytest-cov is installed, as it is
# otherwise an unknown option).
script = (
'if python -c "import pytest_cov" 2>/dev/null; then set -- "$1" --no-cov; fi; '

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Disabled coverage stops pytest

If a project installs pytest-cov but disables it with addopts = -p no:cov, importing pytest_cov still succeeds. This command then adds --no-cov, but the disabled plugin cannot register that option. Pytest exits before running any tests, so otherwise valid regression tests cannot be validated. Check whether the plugin is enabled before adding its option.

Prompt To Fix With AI
This is a comment left during a code review.
Path: openhack/regression/frameworks.py
Line: 704

Comment:
**Disabled coverage stops pytest**

If a project installs `pytest-cov` but disables it with `addopts = -p no:cov`, importing `pytest_cov` still succeeds. This command then adds `--no-cov`, but the disabled plugin cannot register that option. Pytest exits before running any tests, so otherwise valid regression tests cannot be validated. Check whether the plugin is enabled before adding its option.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code Fix in Cursor Fix in Conductor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant