We noticed most "AI code review" tools just comment, which get ignored pretty often
Source Reddit Cybersecurity
Discussion
This report discusses the limitations of AI code review tools that rely solely on comments in pull requests (PRs) to flag security issues. Comments from automated tools often blend into the noise of multiple bot messages and are frequently ignored or skimmed over, leading to security issues being merged without proper attention. The authors highlight a real example where sensitive information was exposed before authorization checks, which might have been missed if only commented on. Their solution is to fail the GitHub check itself for issues above a severity threshold, preventing merges until fixed, and providing instant fixes. This approach aims to increase the visibility and enforcement of security findings in code reviews.
r/cybersecurity (opens in a new tab)u/RyzeBlaziken0 points0 comments
Working on a security scanner for AI-generated PRs, and one pattern keeps showing up: the review comment itself isn't the bottleneck; the re-reading of it is. A comment sits in a PR thread next to five other automated bot comments (lint, coverage, dependency bot, style bot...) and blends in. We found this concretely auditing a real open-source signing flow: a page returned a recipient's name and email in the response before the authorization check that was supposed to gate it. If a bot had left a comment about that, it's exactly the kind of note that gets skimmed and merged past under deadline pressure. There's no sense of urgency. The fix we landed on for our own tool (Scanity) was to stop leaving comments and instead fail the GitHub check itself, so the PR can't merge at all above a severity threshold, same as a failing test. At the same time, Scanity also provides the fix for it instantly.
Curious whether others have data on comment-based bot feedback actually changing merge behavior vs just adding noise to the PR.
Check it out: https://scanity.dev/