Taking the Design Instead of Buying the Reviewer — Claude Code Review and the PR Agent I Built Myself#

2026-10-25

Two Stories from the News Brief#

Last week, while putting together the AI news brief, I covered two code review stories side by side. One was a post from Anthropic’s Claude Code team about how they redesigned code review in their own repository, along with the general availability of Claude Code Review, the public version of that reviewer. The other was GitHub adding several security mechanisms to the PR (pull request, a request to merge a code change) process.

While writing the brief, one question kept coming back. Would things get better if I brought these into my company?

Since July this year, I have been running an agent I built myself that reviews and merges PRs across my company’s repositories. So the real question was less “should I adopt this?” and more “is it good enough to throw away what I built and switch?”

Here is the answer up front. I decided not to bring in the products. Instead, I decided to borrow their design. Code Review is a product that only reviews, and what I need goes beyond review to merging, conflict resolution, and deployment. And since I can change my harness (the framework of rules and tools an agent works inside) however I like, I only need to pick out the good ideas and build them in.

Borrowing was not simple copying, though. Some ideas clashed with the assumptions of my environment and had to change shape. Others, once I thought them through, were better left behind. This essay is a record of those judgments.

I start by describing what my PR agent does. Then I look at Claude Code Review and GitHub’s security mechanisms and compare all three side by side. Finally, I go over what I chose not to borrow and what I borrowed and adapted.

The PR Agent I Run#

Why I built it#

Once you start handing code to agents, code comes out fast. The bottleneck moves to what comes next. Someone has to read the PR, wait for CI (Continuous Integration, the system that automatically runs tests and checks every time code is pushed) to finish, resolve conflicts in later PRs after earlier ones merge, press the merge button, and confirm the deployment. When people do this, people become the bottleneck.

I built my PR agent to shrink that bottleneck. The goal is to minimize the moments when a person has to step in.

What one cycle does#

The agent runs on Hermes, an open-source agent. It lives on a single cloud server, and command execution is isolated inside a Docker container. A cycle runs every 10 minutes, and each cycle does the following.

1. Review: It reviews PRs that are new or have changed. Reviews are handed out two or three at a time to read-only subagents (helper agents that the parent agent delegates work to).
2. Merge: It merges PRs that passed review, oldest first.
3. Conflict resolution: When a PR picks up a conflict because an earlier PR merged, the agent resolves it itself. If the right resolution isn’t clear, it leaves the conflict alone and sends the PR back to its author.
4. Post-merge CI handling: When tests that run after a merge fail, the agent opens a fix PR for failures that need no judgment and notifies the original author about the rest.
5. Deployment: If anything was merged, it triggers a deployment to the development server.

One choice stands out. When merging into the shared development branch, the agent does not wait for CI to finish. A PR that passes review is merged right away, and CI runs after the merge. In exchange, the release that leads to production deployment can only be merged when the post-merge CI is green, enforced through branch protection (a GitHub setting that places merge conditions on a specific branch). Merge fast, then filter once at the door to production.

Rules born from incidents#

Many of the agent’s rules came out of incidents.

• Block only product defects: Early on, the agent sent PRs back even for shallow tests, and authors and the agent passed the same PR back and forth several times. Now it blocks only defects with proven incorrect behavior and leaves test depth and matters of taste as non-blocking comments. It raises all of its criteria in the first review, blocks only newly introduced defects in later rounds, and limits send-backs to three rounds.
• Subagents are read-only: I once handed the job of fixing post-merge CI failures to a subagent, and it went as far as opening a fix PR on its own. The result was fine, but since then only the parent agent does any write work.
• Never merge an unfinished PR: A PR whose description said “I am fixing an execution error” once passed review and got merged. The author was still in the middle of the fix, and with the shared branch red, the release was blocked for nearly half a day. Now, when a description contains in-progress wording like “fixing,” the agent turns that PR back into a draft.

In short, my agent is less a reviewer and more an operator that runs the pipeline from review all the way to deployment. That turns out to be the biggest difference in the comparison that follows.

What Claude Code Review Is#

The following is based on Anthropic’s blog post, launch announcement, and official Code Review documentation, as of October 2026.

How the Anthropic team reviews code#

On October 15, Addy Osmani, Morgan Westlee Lunt, and Qing Wang of the Claude Code team published how their team reviews code. About 90% of the code in the Claude Code repository is written by Claude. When code comes out that fast, a process where a person reads every diff (the changed lines) cannot keep up. So the team changed the question. Instead of asking “did a person read every diff?”, they ask “what could go wrong with this change, and is a person or a machine the right check to catch it?”

The core of it is risk-tiered approval.

A feature flag is a switch that lets you ship code while turning the feature itself on and off separately. If something goes wrong, you flip the switch off instead of rolling back the deployment, which is why such changes count as low risk.

The tier is set by the change, not by the author. A classifier reads the change and assigns its tier. Paths that owners have marked as sensitive, along with every CI workflow file, always get a human. And an agent’s approval can never fill a slot that requires a person.

Review also happens in layers. Before a PR is opened, Claude runs the tests and build and shows the output, and a subagent with fresh context reviews the diff. Once the PR is open, several agents hunt for bugs at the same time, and a separate group of agents votes on each finding, and only the ones with enough agreement get posted. Findings that are only about style are dropped.

Recurring feedback gets fixed at the source. When the same finding keeps coming up, it moves into a review rules file, a lint rule (an automated check that catches mistakes and style violations in code), or a skill. To keep reviews from looping forever, after the third round the reviewer asks the author to rethink the approach. That rule came from a PR that went through more than fifty rounds of review and fixes overnight.

They also shared results. From June to September, merged PRs more than doubled while the share of PRs that had to be reverted fell by more than 40%. The team treats revert and incident rates as guardrails (lines that must not be crossed). If those rates rise, they reason, human time was not saved, only moved somewhere else.

Code Review as a product#

On the same day, Code Review, the public version of this reviewer, became generally available. It is offered on Claude’s Team and Enterprise plans.

• How it works: You install it as a GitHub App, and reviews run on Anthropic’s infrastructure. Several agents examine the diff and the surrounding code in parallel, a verification step filters out false positives, and the remaining findings are deduplicated, ranked by severity, and posted to the PR.
• How results appear: It comments on the lines with problems and sorts findings into three severities: bugs to fix before merging (Important), minor points (Nit), and problems that were already there rather than introduced by this PR (Pre-existing). That last category keeps authors from being called out for someone else’s bug.
• What it cannot do: Its repository access is read-only, and it never pushes commits. It neither approves nor blocks PRs, and it does not build or run code. The decision to merge stays entirely with people.
• Rules files: It reads the repository’s CLAUDE.md and REVIEW.md. It reads REVIEW.md only from the base branch, never from the PR itself, so a PR cannot rewrite its own review rules on the spot.
• When it runs: Per repository, you choose between once when the PR opens, on every push, or only when someone comments @claude review.
• Cost: There is no fixed price per review; it is billed by token usage. Anthropic’s support documentation says each review averages $15–25. According to the launch announcement, most reviews now finish in 6–10 minutes, down from about 20 minutes during the preview.

The Security Mechanisms GitHub Added#

The same week, GitHub added several security mechanisms around the PR process. Since the terms may be unfamiliar, I explain them one at a time. Prices are as of October 2026 and are per active committer (someone who pushes code) per month.

Secret Protection: keeping secrets out of code#

Secrets are strings that must not leak, such as API keys, passwords, and access tokens. Secret Protection ($19 per month) does two things. One is push protection: if someone accidentally commits a file containing a key and pushes it, GitHub rejects the push. The other is finding keys that already sit in past commits and alerting you. Starting October 13, Anthropic API keys are also blocked by default in repositories with secret scanning turned on.

Code Security and AI Scan: finding vulnerabilities#

The core of Code Security ($30 per month) is CodeQL, GitHub’s analysis engine. CodeQL does not run your code; it follows the paths data flows through. For example, if it finds a path where user input reaches a SQL query without validation, it flags it on the PR. Unlike an LLM review, it decides by fixed rules, so the same code always gets the same result.

AI Scan, which became generally available on October 15, uses AI to check files CodeQL does not yet cover, such as PHP, Bash, Terraform, and Dockerfiles. Its results are advisory and by design cannot block a merge. It does not run on PRs from forks (copies of a repository under another account) or on PRs from Dependabot, the bot that automatically opens dependency update PRs. Using it on private repositories requires Code Security or Advanced Security.

Agentic autofix: fix it, recheck it, hand it to a person#

Agentic autofix, which became generally available the same day, hands code scanning alerts to an agent. The agent looks through the relevant files, proposes a fix, and reruns CodeQL to confirm the alert actually closes before opening a draft PR. Whether to merge is decided by a person who reviews it. However, it is only available to GitHub Enterprise Cloud customers, so my team, which is on the Team plan, cannot use it.

Actions workflow lockfiles: preventing swaps of parts someone else built#

This feature needs a few concepts explained first.

GitHub Actions lets you define automated work such as “test when a PR is opened, deploy when it is merged.” The steps are written in a workflow file. Each step either runs a command directly or pulls in a published building block that someone else made. These building blocks are called actions: an action that checks out the repository code, an action that logs in to a cloud provider, an action that runs deployment commands, and so on.

When you pull in an action, you usually add a version at the end, as in uses: actions/checkout@v5. That @v5 is a tag. A tag is a label attached to the code at a particular point in time (a commit), and the person who made the action can move it to a different commit at any time. A commit, on the other hand, has a 40-character fingerprint (SHA), and changing even a single character of its content changes the fingerprint. To use an analogy, pulling by tag is like ordering “please send me the v5 edition,” while pulling by fingerprint is like ordering “an exact copy of this printed book.”

This difference has led to a real incident. Between March 14 and 15, 2025, an action called tj-actions/changed-files, used by more than 23,000 repositories, was compromised. The attacker moved every tag of the action, from v1 through v45.0.7, to a malicious commit, and the malicious code printed the secrets available in CI into the run logs. The affected repositories had not changed a single line of their own code. Only the repositories that had pinned the action by fingerprint instead of by tag were safe.

With a deployment action, it would be even more alarming. If someone swaps the tag of an action that receives production server credentials and runs deployment commands, those credentials could go to the attacker at the next deployment. Since nothing changes in the repository, no PR review can catch it, whether by a person or an agent.

Workflow lockfiles, released in public preview on October 14, record in a .github/workflows/actions.lock file which commit each tag points to. At run time, GitHub uses the recorded commit instead of resolving the tag again. So even if a tag moves, you keep running the old code, and because upgrading a version means regenerating the lockfile, the change shows up in a PR. They are available on the Free and Team plans as well. As GitHub itself notes, though, locking only freezes the current version; it does not judge whether that version is safe.

When I checked, my team’s workflows were no exception to this problem.

Side by Side#

Putting the three together makes the differences clear. GitHub’s mechanisms are less reviewers than checkers that sit next to the review, so the comparison centers on Code Review and my agent.

The reason I decided not to switch to Code Review is in the first two rows. Even with Code Review in place, someone still has to merge, resolve conflicts, and deploy. Hand off only the review, and my agent has to stay around for the rest, while review decisions and merge decisions split across two systems.

Cost matters too. For a team that opens a few hundred PRs a month, running it just once per PR adds up to several thousand dollars a month. Running it on every push multiplies that. And the $10,000 credit Anthropic offers to Enterprise organizations does not apply to the Team plan.

That does not mean my agent is ahead on every count. There are clearly rows in the table where the product leads.

• It verifies findings a second time. My agent blocks a PR as soon as a single subagent calls something a “defect.”
• It separates the agent from people. My agent works under my account, so GitHub’s records alone cannot tell who did what.
• It has rule-based checks and supply chain protection. CodeQL and lockfiles do things LLM review structurally cannot. An attack that changes nothing in the repository, like the tag swap above, cannot be caught by any reviewer.

On the other hand, the row where I thought my agent was ahead also came with a caveat. I come back to that later.

So the conclusion did not stop at “keep what I have.” What remained was bringing the places where the product leads into my own harness. But as soon as I tried, the things that could not be carried over as-is showed up first.

What I Chose Not to Borrow#

Paths a person always reviews#

The Anthropic team always has a person look at CI workflow files. At first I wanted to adopt this principle as it is. Workflow files handle production deployment keys, and like post-merge CI, they are safety mechanisms themselves. Whether by mistake or by attack, a broken workflow file does the most damage. So I considered a setup where the agent only reviews PRs that change workflows, and merges them after I confirm.

But this collides head-on with the reason I built the agent. My agent exists to shrink the human bottleneck, and adding a human confirmation step makes me the bottleneck again. As I see it, Anthropic’s principle assumes a team with several human reviewers. In my environment, the way to reach the same goal was not to bring people back in, but to make the review thicker. I describe how in the next section.

Publishing review rules in the repository#

Code Review keeps its review rules in a REVIEW.md inside the repository. The upside is that repository owners can edit the rules directly and share the same file with other tools. At first I also wanted to move my agent’s per-repository rules into each repository.

Thinking it through, the downsides were bigger.

• Anyone can change the rules. If the agent reads rules only from the base branch, a PR cannot change its own rules on the spot. But if someone opens a separate PR that just loosens the rules, the agent reviews and merges that PR too. In a setup where no person looks, weakening the rules can pass on the agent’s review alone.
• Format rules become easy to game. A rule like “write the rollout plan in the PR description” is easy to satisfy in form only, once it is public. A judgment rule like “block product defects,” on the other hand, cannot be dodged even when public. A defect is still a defect.
• The upside is small. With me running the harness directly, “repository owners can edit the rules” is not much of an advantage.

So I decided to keep the review rules in the harness on the server, but split them into per-repository files that are read only when reviewing PRs for that repository. Alongside that, I am using another Anthropic idea: moving the reasons the agent often sends PRs back into the repository’s writing guides (CLAUDE.md, AGENTS.md) as “this is how it should be written.” The AI that writes PRs reads those files, so PRs come out right from the start, and the only thing made public is the writing standard.

The product itself#

For the reasons above, I am not bringing in Code Review. I considered calling it only for risky PRs as a second opinion, but decided to apply the improvements below first and pilot it only if they fall short.

Of GitHub’s mechanisms, I put Code Security on hold. AI Scan results are advisory and cannot block merges, and secret checks and action pinning were more urgent for my team. Agentic autofix is not available on our plan.

What I Borrowed and Adapted: The Improvement Plan#

What follows is a plan that has not been applied yet.

1. Review risky files more thoroughly, and explain why in the merge notification#

Where Anthropic requires approval from two people for high-risk changes, I substitute machine checks, two agent reviews, and post-merge monitoring.

• Define the risky files. The starting set is CI workflow files, DB schema change files, and authentication and authorization code.
• Run machine checks first. Not an LLM but a script that runs on fixed rules scans the diff. It checks whether a new external action came in by tag rather than by fingerprint, whether secrets are newly used, whether permissions widened, whether deployment triggers changed, and whether a required check picked up “pass even on failure.” A script gives the same result for the same input every time and cannot be fooled by instructions hidden in a PR description.
• Review only the risky files a second time. A fresh subagent looks again at just the risky-file portion, and passes it only if every signal the machine checks found can be justified, with a “why is this needed?” answer drawn from the PR description and diff. Both reviews must pass before merging. If the justification is missing, the PR goes back to its author, not to me.
• Watch the first run after merging. When a workflow change is merged, the agent checks the next run of that workflow and alerts if it fails or if zero tests ran.

And the merge notification records the reasoning. I don’t confirm in advance; I read it afterward and roll back if something looks off.

✅ [backend] #1234 "Add build cache to deploy workflow" merged
⚠️ 1 risky file: production deploy workflow
- Changed: added a cache action to the image build step
- Machine checks: 1 new external action, pinned to a commit SHA. No change to secrets / permissions / triggers
- Second review: passed. Cache keys are split per branch, so results from other branches don't mix
- First run after merge: to be checked next cycle

2. Verify blocking findings a second time#

This borrows Code Review’s verification step. When a subagent raises a blocking finding, a fresh subagent checks just that one finding against the code. A finding that cannot point to an actual location in the code is downgraded from blocking to non-blocking. Since this runs only when a blocking finding appears, the added cost is small, and authors spend less time filing rebuttals and waiting for the next cycle because of false positives.

3. Actually check the impact on other repositories#

In the comparison table, I wrote that my agent “can see several repositories.” Since Code Review sees only the repository the PR belongs to, I thought this was a clear advantage on my side. For example, if a backend PR changes the shape of an API response, the agent could also check whether the places in the frontend that use that API would break.

When I checked the server, that was only half true. The agent has permission to access every repository, but when I split the repositories into two groups, each agent ended up with local copies, the ones used for searching, of only its own group. The subagents that do the reviewing were blocked from creating new copies, and the review instructions said nothing about checking other repositories. It was an advantage that existed in principle but was never actually used.

So I am giving both agents copies of the remaining repositories, updating them before each review, and adding a rule to the review instructions: “for a PR that changes an API or data format, find and check the other repositories that use it.” Code Review, which sees only the PR’s own repository, cannot do this, so if I use it properly, it becomes a clear strength on my side.

4. Log pre-existing bugs separately#

Code Review marks problems that were already there separately. My agent already follows the principle of “block only this PR’s defects,” so it almost never blocks a PR over an existing bug. And since it mostly looks at what changed, it rarely discovers existing bugs in the first place.

The gap is what happens after one is found. Right now it ends up as a single “follow-up” line in a review comment, and nobody picks it up. A bug that has been found should be fixed, but folding the fix into the current PR means the author touches code they don’t know, and the PR grows larger and harder to review and roll back. So existing bugs will not block the PR; instead, the agent notifies the original author or adds them to the task list to be fixed separately. Only when this PR causes that bug to newly execute or makes it worse does the agent treat it as this PR’s defect and block.

5. Measure the revert rate#

I track how long PRs wait to merge, how long the shared branch stays red, and how many conflicts get resolved. All of these measure how smoothly the flow moves. What Anthropic watched alongside its speed gains was the revert rate. To judge whether merging without waiting for CI is safe, I need that number.

6. Outside the agent: pinning actions and checking for secrets#

Some risks cannot be caught by review. One is the tag swap described earlier. I will pin external actions by commit fingerprint and try lockfiles in one repository first. PRs that bump a pinned version will go through the risky-file review in item 1.

The other is secrets. Because my team moved CI on the shared branch to after the merge, a commit containing a key can land on the shared branch before it gets caught. A key that lands there stays in the commit history, so it has to be revoked and reissued. So I will either block it at push time with Secret Protection or, at minimum, have the agent scan the PR’s changes with a secret detection tool before merging.

Closing Thoughts#

To sum up, I decided not to bring in Claude Code Review and to borrow its design instead. Not because the product is bad, but because the scope of work I need is different. Code Review is a reviewer that attaches to teams where people decide what gets merged; my agent is an operator that steps in where people step back and carries the work through to the merge.

This conclusion comes with conditions: the goal is to automate merging, I can change the harness myself, and the team does not have many human reviewers. For a team that only needs to automate review, a team that must keep agents and people clearly separated in the record for audits, or a team with nobody to run a harness, Code Review may well be the better choice.

What I took away from this review is a habit of reading product docs as design documents. A product from a team that solved the same problem first contains the answers that team earned through its incidents: a step that filters findings by vote, a category that marks problems as already present, a round cap, and the revert rate as a guardrail. But those answers rest on that team’s assumptions. Where my assumptions differed, as with the paths a person always reviews, I had to find a different shape that serves the same purpose.

I reached a similar conclusion in my earlier essay, A Hand-Assembled Agent Stack vs. OpenAI’s Finished Product. As finished products multiply, the reason for building your own shifts from “there was no other way” to “I can change it however I need.” Once the improvements above are in place, I plan to write up their effect again, revert rate included.

References#

• Anthropic — How the Claude Code team does code review
• Anthropic — Code Review is now generally available
• Claude Code Docs — Code Review
• Claude Code Docs — Code Review costs and usage
• Claude Help Center — Set up Code Review for Claude Code
• GitHub Changelog — AI Scan for pull requests generally available
• GitHub Changelog — Agentic autofix for code scanning alerts general availability
• GitHub Changelog — GitHub Actions workflow lockfiles in public preview
• GitHub Changelog — Secret scanning blocks Anthropic keys by default
• GitHub — Secret Protection and Code Security pricing
• OSV — CVE-2025-30066, the tj-actions/changed-files compromise
• AI News Brief — October 18, 2026
• A Hand-Assembled Agent Stack vs. OpenAI’s Finished Product — Looking at dots and ChatGPT Space