PR triage automation with risk scoring

PR triage automation uses risk scoring to prioritize code reviews based on change impact, contributor history, and affected systems rather than simple rule-based labels.

Pull request backlogs grow faster than review capacity. Most teams adopt some form of automated triage—labels, assignments, CODEOWNERS matches—but these binary rules leave critical questions unanswered: Which three PRs should the security engineer review first when she has thirty minutes before standup? Simple automation doesn't prioritize; it only categorizes.

Risk scoring fills that gap. Instead of "this PR touches authentication code" you get "this PR scores 72/100 because it modifies JWT validation, expands API surface area, and ships from a contributor with two merged commits." The difference is actionable priority that turns merge decisions from gut calls into reproducible workflows.

Why static rules break down quickly

Most teams start with simple heuristics: "Flag PRs that touch authentication code" or "Escalate any change over 500 lines." These rules work for about two weeks. Then someone refactors the auth module into smaller helpers, and your 50-line change that completely rewrites the session logic flies under the radar.

Static rules fail because codebases evolve. The authentication logic you hardcoded as src/auth/ gets split into src/session/, src/identity/, and src/tokens/. Your line-count threshold becomes meaningless when the team adopts auto-formatters that inflate diffs. The real risk patterns hide in context: what changed, who changed it, how recently that code last broke production, and what dependencies it affects.

A useful risk scorer needs feedback loops that capture when it guessed wrong, and it needs to aggregate signals that attackers or well-meaning engineers can't trivially game.

Risk signal taxonomy: choosing inputs that survive adversarial pressure

Start by bucketing signals into three categories:

Structural signals come from the diff itself. Lines changed in authentication modules, new dependencies added, changes to access control logic, and modifications to cryptographic implementations all carry inherent risk regardless of author intent. These resist gaming because the code diff is immutable once committed.

Concrete structural patterns to detect:

New HTTP endpoints or API routes
Changes to permission decorators or authorization checks
SQL query construction patterns
Secrets handling (environment variable access, encryption calls)
Database schema migrations

Contextual signals depend on repository state. Does this PR expand your public API surface? Does it alter database schemas? Does it touch files flagged in past security incidents? Context signals require indexing your repo history and CI telemetry, but they scale better than manual inspection.

Calculate blast radius by analyzing import graphs. If shared/validation.ts is imported by 40 files versus 3, changes there carry more weight. For most languages, static analysis tools expose this—jdeps for Java, go mod graph for Go, or AST parsers for JavaScript. Compute betweenness centrality scores: files that sit on many import paths act as choke points.

Track fragility per file: test flakiness rates, recent production incidents, and rollback frequency. If checkout.py caused two outages in the last month, any PR touching it deserves eyes-on review even if it's a one-line change. Store this metadata in a simple JSON file or database; update it from your incident postmortems and CI failure logs.

Authorship signals reflect contributor patterns. A contributor with sixty merged PRs and zero reverts carries less risk than a first-time external contributor, not because of trust but because established patterns create predictability. Track review-to-merge latency, revert rates, and security-flagged commit ratios per author. Also flag ownership proximity: PRs where the author has never touched the modified files score higher than veterans tweaking their own code.

The tradeoff: structural and contextual signals cost more to compute (you're running static analysis and diffing against policy rules), but authorship signals introduce fairness questions. Weight authorship lightly—perhaps 15–20% of total score—so new contributors aren't permanently penalized.

Score calculation: additive vs multiplicative models

You can combine signals additively (sum weighted factors) or multiplicatively (cascade modifiers). Each approach changes what kinds of PRs float to the top.

Additive scoring treats each signal independently:


risk_score = (

    0.4 * structural_risk +

    0.35 * contextual_risk +

    0.15 * authorship_risk +

    0.1 * velocity_risk

)

This produces stable, interpretable results. A PR touching auth code and expanding API surface gets flagged, but a PR with zero structural risk stays low-score even if the author is new. Additive models work well when you want transparency: engineers can see exactly why a PR scored high.

Multiplicative scoring applies cascading modifiers:


base_risk = structural_risk

if touches_sensitive_paths:

    base_risk *= 1.5

if external_contributor:

    base_risk *= 1.3

risk_score = base_risk

Multiplicative models amplify combinations. A routine dependency update from a new contributor might jump ahead of an internal refactor, which better reflects real-world risk. The tradeoff: harder to debug when a PR's score feels wrong, and small tuning changes can swing rankings dramatically.

In practice, hybrid models work best. Use additive scoring for your base calculation, then apply a narrow set of multiplicative flags—"modifies production secrets," "external contributor without signed CLA," "bypasses required CI checks"—that genuinely multiply risk.

Here's a concrete implementation showing both approaches:


def score_pr(pr_data):

    # Additive base

    risk = 0.0

    

    # Ownership: author unfamiliar with files?

    for file in pr_data['changed_files']:

        if pr_data['author'] not in get_past_authors(file):

            risk += 0.15

    

    # Blast radius: imports per changed file

    for file in pr_data['changed_files']:

        import_count = count_importers(file)

        risk += min(import_count / 50, 0.3)

    

    # Fragility: recent incident history

    fragile_files = load_fragile_files()  # from incident log

    overlap = set(pr_data['changed_files']) & fragile_files

    risk += len(overlap) * 0.25

    

    # Breaking changes: API signature diffs

    if has_breaking_changes(pr_data['diff']):

        risk += 0.4

    

    base = min(risk, 1.0)

    

    # Multiplicative modifiers

    if modifies_secrets(pr_data['diff']):

        base *= 1.8

    if pr_data['author_type'] == 'external':

        base *= 1.3

    

    return min(base, 1.0)

The specific weights matter less than consistency. Tune them based on your team's incident history. If most outages trace to deployment configuration rather than application logic, weight operational sensitivity higher.

Caching and staleness: when to recompute scores

Pull requests evolve. A PR starts with three lines changed, gets feedback, balloons to 400 lines, then shrinks back to fifteen after the author extracts a refactor. Recomputing risk scores on every push is expensive; stale scores mislead reviewers.

Store scores in a lightweight database keyed by PR ID and commit SHA. When new commits land, queue a background job to recalculate. Surface the score's timestamp in your UI—"Risk score: 68 (computed 14 minutes ago)"—so reviewers know if they're looking at current data.

Cache invalidation rules:

Hard invalidate when new commits land or the PR base branch changes
Soft invalidate after 4–6 hours for long-lived PRs, since repository context (like new CVEs in dependencies) can shift
Never invalidate authorship signals mid-PR; these should reflect the contributor's track record at PR open time

For high-velocity repos, batch recomputation: collect commits every five minutes, dedupe by PR, and score in parallel. In Goatfied's agent loop, this fits naturally—the validation step can invoke your scoring logic, and because diffs are small and reversible, you're never stuck waiting on a massive recalculation. The compile-first reliability model means your risk checks run before code reaches human review.

Routing decisions based on composite scores

Once you have a risk score per PR, automate routing decisions. Define thresholds that trigger different workflows:

Low risk (0–30 points): Auto-merge after tests pass if the author has write access. This handles dependency bumps, documentation, and isolated feature toggles. If 40% of your PRs are trivial changes that score under 30, removing them from review queues frees reviewers to focus on the work that matters.

Medium risk (31–60 points): Require standard code review from a teammate. One approval suffices. Post the risk breakdown as a PR comment for transparency.

High risk (61–80 points): Require review from a senior engineer or domain expert plus specialized review if specific dimensions are elevated—operational risk over 40 needs SRE eyes, attack surface over 50 needs security review.

Critical risk (81+ points): Block auto-merge, mandate two approvals including one from security or platform team. Trigger a security checklist that reviewers must acknowledge: "Does this PR validate user input? Does it modify authentication logic? Are secrets handled correctly?"

Use GitHub's CODEOWNERS or GitLab's approval rules to enforce this programmatically. Avoid blocking merges based solely on scores. Scores inform priority, not permission. If a PR scores 95 but the author explains it's a well-tested config change, the reviewer should merge. Treat scores as sortable metadata, not binary gates—your compile, lint, and test gates already handle the blocking logic.

Inject scores into three places:

1. PR list views in GitHub/GitLab/Bitbucket via browser extension or webhook-driven labels

2. Slack/Teams notifications with score in the first line: "🔴 High-risk PR #847 (score: 81) awaits review"

3. Required reviewer assignments where PRs above threshold 70 auto-assign your security rotation

False negatives vs false positives: tuning for your team's risk appetite

Every scoring model produces misranks. A typo fix in a README might score moderately if it comes from a new contributor; a subtle race condition in payment processing might score low if it's a one-line change from a trusted engineer.

False negatives (high-risk PRs ranked low) hurt more than false positives (low-risk PRs ranked high). Engineers can quickly dismiss an over-flagged PR, but they won't review something that looks safe and isn't. Tune your model conservatively: when in doubt, score higher.

Calibration tactics:

Seed with historical incidents. Pull your last 20 security-flagged commits and run them through your scoring model retroactively. Did they score in the top quartile? If not, adjust weights.
Weekly spot checks. Sample ten PRs from each score band (0–25, 26–50, 51–75, 76–100) and manually inspect. Track how often the scores align with your security engineer's gut reaction.
Override logs. Let reviewers manually bump or reduce a PR's score with a reason. Tag a change as risk-override: low with required justification. Aggregate these overrides monthly to identify systematic blind spots. If 30% of high-risk PRs get overridden, your model needs tuning.

Your goal isn't perfect ranking; it's to surface the three most critical PRs in any 20-PR batch 80% of the time. That threshold lets security engineers trust the system enough to rely on it daily.

Instrumenting failure patterns

Your scoring system will miss things. The goal is to notice quickly and adjust. Instrument these feedback loops:

Post-incident tagging: When you write a postmortem, tag the causative PR. Build a weekly report: "PRs scored <30 that caused incidents." Review these misses as a team. Often you'll spot patterns—maybe all of them touched configuration files your scorer didn't weight heavily, or involved new contributors during crunch time.

Review duration correlation: Track time-to-first-review for each PR, grouped by risk score. If your highest-scored PRs sit untouched for days while low-scored ones get immediate attention, your scores aren't aligned with actual human priorities. Tune accordingly.

False positive audits: Flag PRs that got escalated but merged cleanly after cursory review. Too many false positives train engineers to ignore your alerts. Sample a few each week: did the scorer overreact to a refactor-only change? Was a dependency update flagged unnecessarily?

Export scoring data alongside deployment outcomes. Correlate risk scores with rollback rates, incident reports, and security findings. After 50+ scored PRs, review the distribution:

Are 80% of PRs landing in low risk? If not, thresholds are too conservative.
Do high-risk PRs actually correlate with post-deployment issues? If not, adjust dimension weights.
Are certain file paths consistently over- or under-scored? Add path-specific rules.

Every merged bug is feedback. When a low-scored PR causes a production incident, that's training data. Your system should learn from its mistakes faster than your team accumulates new code.

Handling the edge cases that matter

Large refactors: A 2,000-line change that renames a module will score high on blast radius but might be low-risk if it's purely mechanical. Add a "refactor" label that authors can apply, then discount line count for labeled PRs but keep the fragility and breaking-change checks. Track these separately and review them after merge to verify the label was accurate.

Hotfixes under pressure: In an incident, you'll merge PRs that score high. That's fine. Build an escape hatch: a risk:override-emergency label that bypasses all checks but requires post-merge audit and documentation. Log them separately and review them after the fire is out. The scorer's job isn't to block critical fixes—it's to surface risk so you make informed trade-offs.

New projects with no history: For fresh repos, lean on ownership and breaking-change signals. Fragility scores need time to accumulate. Start conservative; relax thresholds as you build confidence.

Generated code: Dependency updates, schema migrations, build artifacts. Exclude these files from scoring or apply a separate model that focuses on whether the generator itself changed.

Integrating with Goatfied's agent loop

Risk scoring identifies what needs review, but it doesn't prevent risky patterns from reaching the PR stage. Integrate compile-time and pre-commit validation to catch issues earlier.

In Goatfied's agent loop (plan → constrain → edit → validate → retry), the constrain phase enforces project-specific security rules before code generation. Define constraints like "never construct SQL queries via string concatenation" or "all new endpoints must include rate limiting middleware." When the agent plans edits, these constraints act as hard boundaries—the generated code won't compile or pass lints if it violates them.

Pair this with the validate phase that runs tests, security scanners, and policy checks before the agent considers a change complete. If a PR emerges from this pipeline, you've already eliminated common vulnerability classes. Your risk scoring then focuses on architectural decisions and change coordination, not catching basic mistakes.

Example constraint for secret handling:


# .goatfied/constraints.yaml

- name: no-hardcoded-secrets

  pattern: '(password|api_key|secret)\s*=\s*["\x27]'

  scope: all

  severity: error

  message: Use environment variables or secret management

Any agent-generated code violating this fails validation before reaching your repository. Layer risk scoring on top of these gates: a PR that passes all constraints but modifies high-centrality files or ships from a new contributor still needs human review, just for different reasons.

Quarterly calibration and iteration

Plan to revisit your scoring model quarterly. As your codebase evolves, yesterday's critical paths become maintenance mode and new features become load-bearing. Run analyses on precision (of PRs scored high-risk, what percentage actually caused issues?) and recall (of incidents traced to code changes, what percentage were scored high-risk?). Track drift: are certain modules consistently over- or under-scored relative to outcomes?

Adjust weights, add new signals (deployment frequency, time of day, feature flag coverage), or retire signals that stopped correlating with real risk. The operational checklist isn't about achieving perfect scores—it's about making risk visible, consistent, and actionable so your team makes better merge decisions faster.

Risk scoring isn't about perfect prediction. It's about building a system that surfaces the right PRs for review 80% of the time and learns from the 20% it missed.