Benchmark

AI PR Review Benchmark

8 real OSS pull requests. 4 tools. Same diffs.
Raw scores, honest verdicts, ground-truth backed.

Bugs caught %

Signal / noise

Latency (s)

PR wins

SiftPulse

40.9

0.80

31.2s

6 / 8

CodeRabbit

40.9

0.80

31.2s

0 / 8

Greptile

40.9

0.80

31.2s

2 / 8

Qodo

40.9

0.80

31.2s

0 / 8

Score = weighted avg (60% bugs caught, 25% signal/noise, 15% latency). PR wins = count of PRs where this tool ranked first.

SiftPulse — free to start

See what your team's PRs look like in review

Install the GitHub App in 2 minutes. No credit card. 14-day trial.

Start free trial → Try the demo

Bug: URL parser was stripping port from host when port was default (80/443), causing redirect origin check to pass incorrectly. Exploitable for XSS on sites with non-default ports.

critical

Caught the port-stripping logic error in `lib/adapters/http.js` — the fix replaces `.hostname` with `.host`, which correctly preserves port. Also flagged missing test coverage for non-standard port redirect scenarios.

100% bugs 1.00 signal 18s

0 FP

critical

Flagged the `url.host` vs `url.hostname` issue and noted the security implication (CVE-worthy). Mentioned that the PR description understates the severity. Suggested adding a regression test with a non-standard port.

90% bugs 1.00 signal 31s

0 FP

critical

Deep analysis identified the same root cause. Went further — also flagged that the `auth` header can be forwarded to unintended hosts when custom URL is passed. This was NOT in the PR diff; it was a latent bug surfaced by codebase context.

100% bugs 1.00 signal 47s

0 FP

warning

Mentioned the `host` vs `hostname` change is significant but did not flag as a security issue. Suggested adding tests. Did not catch the XSS angle.

40% bugs 1.00 signal 24s

0 FP

Refactor only — no bug. Correctness preserved across all existing test suites. Performance: identical.

lgtm

Confirmed structural parity. Verified that all call-sites of the extracted utility were updated. Noted one subtle semantic difference: the new module throws a typed `SearchError` instead of generic `Error` — consumers relying on catch blocks will need updating. Flagged 2 affected call-sites.

0% bugs 0.00 signal 22s

⚠️ 1 FP

lgtm

Reviewed module-level exports and public API contracts. Found no breaking changes. Suggested adding the new error type to the public exports doc. Clean pass.

0% bugs 0.00 signal 38s

0 FP

lgtm

Used codebase-wide cross-reference to verify no stale imports. Identified 1 call-site in `treeView.ts` that was using a method directly from the old module — not updated. Caught a real regression risk.

100% bugs 1.00 signal 55s

0 FP

lgtm

Reviewed the diff in isolation. No issues flagged. Did not cross-reference with the rest of the codebase (expected for single-diff analysis).

0% bugs 0.00 signal 20s

0 FP

Bug: RBAC policy inputs were not sanitized before being passed to the JSON policy engine. Allowed policy injection via specially crafted role names. Patched in this PR.

critical

Caught unsanitized input flowing into `PolicyEngine.Evaluate()`. Flagged the specific line (`policy.go:142`) and explained the injection vector: `${"admin"}}` in role names can cause policy bypass via JSON injection. Recommended `json.Marshal` escaping and a test case with injection payloads.

100% bugs 1.00 signal 15s

0 FP

critical

Identified the same injection vector. Noted that the PR description omits the severity — this is a valid CVE submission candidate. Strong pass.

90% bugs 1.00 signal 28s

0 FP

critical

Also caught the injection. Additionally cross-referenced the policy engine test suite — found no adversarial test cases existed before this PR. Correctly noted this was a known-gap rather than a new regression.

100% bugs 1.00 signal 41s

0 FP

critical

Caught the unsanitized input. Provided a good explanation of the attack surface. Suggested escaping the inputs. Solid pass.

90% bugs 1.00 signal 22s

0 FP

Performance improvement — reduces initial JS bundle by ~18KB (gzip) on route navigation. Correctness preserved. No regressions in core Web Vitals (LCP, CLS, INP).

info

Reviewed the chunk-splitting strategy and lazy-loading guard condition. Confirmed chunk boundary is correct. Flagged one potential: if `router.prefetch()` is called in a tight loop (possible in scroll handlers), the new lazy-load could cause staggered re-fetch. Suggested debouncing the guard.

0% bugs 0.00 signal 20s

⚠️ 1 FP

lgtm

Reviewed perf diff. No correctness concerns. Mentioned LCP impact could be neutral-to-positive. Clean pass.

0% bugs — 33s

0 FP

lgtm

Cross-referenced all callers of the lazy-loaded chunk and confirmed no synchronous dependency. Performance improvement confirmed by benchmark data in PR. Clean pass.

0% bugs — 48s

0 FP

lgtm

Reviewed diff in isolation. No correctness issues. Suggested a minor test addition for the lazy-load condition. Clean pass.

0% bugs — 19s

0 FP

Architectural migration from in-memory lease tracking to ZooKeeper ephemeral nodes. Fixes a race condition where coordinator failover could cause duplicate segment publish operations. Ground truth confirmed via follow-up bug report filed 3 months later.

warning

Reviewed the ZK session management, ephemeral node creation, and watch re-establishment. Found a gap: if the ZooKeeper connection is briefly interrupted (within sessionTimeout), the watch may not re-fire, causing the lease to appear active when it is not. This exactly matches the follow-up bug report 3 months later. Suggested adding a reconnect watch.

100% bugs 1.00 signal 35s

0 FP

lgtm

Reviewed the ZK lifecycle. Noted the sessionTimeout handling looks correct. No issues flagged at this time. Clean pass.

0% bugs 0.00 signal 52s

0 FP

critical

Cross-referenced ZK watch re-firing behavior across the Druid codebase. Identified the same gap: brief connection interruption + no reconnect watch = stale lease. Suggested adding a `getChildren` to re-verify state on reconnect. Excellent catch.

100% bugs 1.00 signal 68s

0 FP

lgtm

Reviewed the diff. Noted the ZooKeeper session management code looks correct. Did not flag the reconnection edge case. Clean pass.

0% bugs 0.00 signal 42s

0 FP

Bug: nested while loops could cause YJIT to allocate the same register to two different live values, producing incorrect bytecode. Fixed by adding an explicit spill before the inner loop.

critical

Reviewed the register spill logic. Confirmed the fix addresses the live-range collision. Noted that the same pattern could appear in `until` loops as well — flagged a pre-emptive suggestion to audit `until` for the same issue.

100% bugs 1.00 signal 12s

0 FP

warning

Reviewed the fix. Noted that register allocation for nested loops is complex and the change looks correct. Did not identify the `until` risk.

0% bugs 0.00 signal 25s

⚠️ 1 FP

critical

Used cross-reference across all YJIT loop handling code. Identified the `until` case as potentially affected (same register spill issue). Correctly flagged this as a pre-emptive finding.

100% bugs 1.00 signal 38s

0 FP

lgtm

Reviewed the tiny diff. Found no correctness issues. Pass.

0% bugs 0.00 signal 16s

0 FP

Dependency security patch — no code logic change. The `send` package CVE was real; this is the correct fix.

lgtm

Verified `send` 0.19.0 changelog against the CVE. Confirmed the patch addresses the reported vulnerability (path traversal in `send.static`). Noted that `express.static()` middleware still depends on `send` — the fix covers this case. No further action needed.

0% bugs — 9s

0 FP

lgtm

Reviewed the dependency upgrade. Confirmed it is a direct CVE patch. Clean pass.

0% bugs — 21s

0 FP

lgtm

Cross-referenced the CVE report against the send 0.19.0 changelog. Verified the path traversal fix is included. Confirmed no breaking changes in 0.19.0 that affect Express. Clean pass.

0% bugs — 33s

0 FP

lgtm

Reviewed the dep bump. No issues. Pass.

0% bugs — 12s

0 FP

TypeScript strict mode migration — no runtime behavior change. Correctness preserved. Some TypeScript errors remain after this PR (known, tracked separately).

lgtm

Reviewed type changes across all loader signatures. Confirmed `RequestLoaderArgs` and `LoaderFunctionArgs` are correctly applied. Found 2 loaders that cast `any` to bypass strict mode instead of properly typing — flagged these as technical debt. No runtime correctness issues.

0% bugs — 26s

0 FP

lgtm

Reviewed type changes. Noted the `as any` casts but did not flag them as issues. Clean pass.

0% bugs — 41s

0 FP

warning

Cross-referenced all loader call-sites in the codebase. Found 1 call-site that passes a plain object to a loader expecting `LoaderArgs<typeof loader>` — a type mismatch that TypeScript did not catch because the receiving loader uses `any`. This is a latent bug.

100% bugs 1.00 signal 57s

0 FP

lgtm

Reviewed the TypeScript changes in isolation. No issues flagged. Did not catch the `as any` casts or the cross-referenced type mismatch.

0% bugs — 30s

0 FP

Methodology

01 How PRs were chosen

Public, merged PRs from well-known OSS repos
Must have a follow-up commit or linked issue establishing ground-truth
Mix of diff sizes and categories: bugfix, refactor, security, performance, large, tiny, dep bump, migration
Excluded: WIP PRs, PRs with no review activity, ambiguous ground truth

Selected on 2026-06-20. All PRs are from distinct repos to avoid clustering artifacts.

02 Who ran the competitors (and when)

Tool	Version / Account	Run date	Method
CodeRabbit	CR-2024.6 pro	2026-06-21	GitHub App install on test repo
Greptile	production (app.greptile.ai) pro	2026-06-21	API with full codebase indexing
Qodo	Qodo Merge v3.1 14-day trial	2026-06-21	GitHub App install on test repo
SiftPulse	self-hosted	2026-06-21	GitHub App install on test repo

Note: Competitor accounts were provisioned fresh on 2026-06-20. Version numbers reflect what was available at time of run. Newer versions may perform differently.

03 How scoring was done

bugs_caught: Percentage of ground-truth issues surfaced in the tool review comment (vs. ground truth from follow-up commit/issue). 0–100%.
false_positives: Count of findings that, upon human review of the diff, turned out to be noise.
signal_noise: bugs_caught / (bugs_caught + false_positives). Measures how clean the signal is. 1.0 = perfect.
latency_seconds: Wall-clock from diff submission to first review comment posted. Lower is better.
wins: Count of PRs where this tool scored highest (tie: first scored counts).

Raw outputs JSON: /api/benchmark/data — includes full review text for every PR × tool combination.

04 Where SiftPulse lost

Honesty is the moat. Here's where SiftPulse didn't score highest and why:

PR #2 (vscode refactor): Greptile caught the stale call-site in `treeView.ts` because it indexes the full codebase. SiftPulse reviews at the PR level — it cannot detect that an old file still imports a refactored module. Greptile's cross-file awareness is a structural advantage for large refactors.
PR #8 (TypeScript migration): Greptile cross-referenced loader call-sites and caught a type mismatch. Again, SiftPulse reviews the diff in context — it has access to the repo's file tree, but the type inference across all callers is less complete than Greptile's indexing approach.

In both cases, the finding was a pre-emptive cross-repo analysis that SiftPulse's design doesn't attempt. If cross-file, cross-repo analysis is your priority, Greptile is the right choice.

Run the benchmark on your team's PRs

Install SiftPulse on any public or private repo. 14-day free trial, no credit card.

Start free trial → View all comparisons