tjklug.com

AI in Operations

CI/CD Is a Security Control Now. Your Pipeline Is Your Patch Clock.

Published Read 24 minWords 5,297

A critical flaw is disclosed in the language runtime that two companies both run. The fix ships the same day as a new patch release. At the first company, a bot opens a pull request moving the base image to the patched version within the hour. The tests run. The engineer who owns the service approves it, it goes to a small slice of traffic, the error rate holds, and it’s everywhere the same day. Nobody skipped a step, because the emergency path and the normal path are the same path.

At the second company, the next release train leaves in two weeks, regression is manual and the deploy belongs to one engineer who is out until Thursday. Leadership gets to choose between shipping untested and staying exposed.

October 21, 2026 was the live test. The PHP project released fixes for two flaws, quickly named PHP2Shell, “that can allow unauthenticated remote code execution” on any PHP installation serving web requests. By watchTowr’s count, PHP runs 70.8% of the websites whose server-side language is known. The release notes left no room: “There are no complete mitigations, updating is the only safe choice.” AWS reported an exploitation attempt on its sensors dating to October 17, “four days before public disclosure.” Cloudflare’s CEO posted that its network saw “thousands of attempts to exploit it” in the 10 hours before the announcement, “tens of thousands in the 45 minutes post announcement, hundreds of thousands within 2 hours, and millions now.” CISA added both flaws to its Known Exploited Vulnerabilities (KEV) catalog on October 26, with a due date of October 29.

Those two companies are invented. The real ones are in the data from that week. Where a hosting provider or a Linux distribution pushed the update, PHP was patched within days. Where the owner had to act, or the version was past end of life, it mostly wasn’t. The difference was whether a path for the fix existed before the advisory did.

For years CI/CD was sold as velocity: ship features faster, beat the competitor to market. That made it optional. A company could look at its monthly release train and decide its customers were fine with that. It isn’t optional anymore, and the reason has nothing to do with features.

In September I argued that a verification floor and a safe deploy path come first in any move to agentic engineering. This post is the other half of that argument. The same foundation is now a security control: it decides how long a known hole stays open. If it’s behind, catching up is a business objective, with an owner, a budget and dates.

Two Clocks

Every vulnerability starts two clocks.

The exploit clock runs from the day a flaw becomes public to the day someone is using it in real attacks. Attackers set it.

The patch clock runs from that same day to the fix running everywhere in your production environment. That one is yours. It’s the sum of how long it takes to learn you’re affected, get the change made, prove it doesn’t break anything, ship it and confirm it took.

Your exposure is the gap between them. You can’t slow the first clock. Everything in this post is about the second.

What the Data Actually Says

Start with what holds up.

  • Exploited vulnerabilities are the leading way in. Verizon’s 2026 Data Breach Investigations Report, covering more than 22,000 breaches, says “Exploitation of vulnerabilities is now the most common initial access vector for breaches. It has risen to 31%,” up from 20% the year before. Phishing is at 16% and credential abuse, the previous leader, at 13%.
  • The fast cases are measured in hours. After the PHP2Shell fix shipped, AWS says “additional proof-of-concept exploits appeared within hours, and we saw them used against our sensors almost immediately.” Rapid7’s third-quarter review of 43 newly exploited high and critical flaws reports that “on average, exploitation occurred just three days after public disclosure.” Attempts aren’t break-ins. The confirmed PHP2Shell break-ins so far are on firewalls and gateways.
  • Some are attacked before a fix exists. Those are zero-days, and Google’s threat intelligence group says they were “the majority (62%) of all observed exploited vulnerabilities from January 2026 to August 2026.” For PHP2Shell, AWS and Cloudflare document attempts before the fix. Nobody has documented a break-in before it. No pipeline helps with the days before a patch exists.
  • The volume doubled. The CVE Program, which assigns the public ID each vulnerability gets, published 72,812 records in the first three quarters of 2026 against 35,448 in the same quarters of 2025. Its third-quarter report puts the increase down to “AI-driven vulnerability discovery and requests.”

Two cautions before anyone puts “months to minutes” on a board slide. That phrase is real. The president of Microsoft Security wrote this week, in a product announcement, that “the time from discovery to exploit is shrinking from months to minutes.”

First, the trend numbers depend on which list you count and when you count it. Microsoft’s researchers, working from CISA’s catalog, put the mean time from a CVE’s publication to its listing as exploited at 89 days for CVEs published in 2024, 57 days for 2025 and 20 days for 2026 through September, “with a median of only 3 days.” Recent years always look fast on that measure, because a flaw published this year hasn’t had time to be listed late.

Second, volume is not risk. Google calculates that “only 0.23% of all disclosed vulnerabilities in 2026 (roughly 1 in 431) were ever observed in active exploitation.” VulnCheck tracked 442 CVEs it attributes to Anthropic’s AI-driven discovery program and found “only 2 of the 442 have been exploited in the wild.” Both figures will rise as late exploitation shows up. CISA’s catalog shows 298 additions in 2026 through November 17, against 221 by the same date in 2025. The CVE count doubled. The exploited list grew by about a third.

The honest summary: the window has not collapsed from months to minutes across the board, and almost none of the new volume is ever exploited. What changed is the shape. Exploitation is the leading way in, the fastest cases move in hours, a large share is hit before disclosure, and there’s twice as much to sort through to find the few that matter. You don’t know which kind you’re holding when the advisory lands.

Where AI Comes In

The short version: AI is cutting the cost of turning a published fix into a working exploit, some attackers now do it with no person in the loop, and how much of the speed-up it explains hasn’t been measured. You don’t need that last point settled to act, because AI is also changing how much there is to patch.

AI is making it cheaper to turn a patch into an exploit. Anthropic’s November 9 system card describes a test in which its strongest model “is given a vulnerable build of V8,” the JavaScript engine in Chrome, “and a patch that fixes the vulnerability.” It “achieved full arbitrary code execution 90.0% of the time (369 out of 410 attempts).” That’s a lab result from the company that sells the model, with safeguards off. On November 17 the research firm Calif disclosed a critical Java flaw it found with an AI agent: “an hour to discover the vulnerability, and two more hours to develop the first working exploit. The total token cost was less than $200.” Google’s Project Zero changed its disclosure policy on November 13 for this reason, writing that “zero patch-to-exploitation time is a reasonable, though imperfect, model of attacker behavior.”

Attackers are doing it, and this quarter it crossed a line. In September, Google’s threat intelligence group wrote that it had “not yet observed threat actors deploying fully autonomous pipelines against targets in the wild.” Its November 16 report says that has changed: it “identified threat actors using LLM frameworks to fully automate vulnerability discovery and exploitation without a human in the loop for the first time.” (LLMs are large language models, the technology behind AI assistants.) The framework it found “compares code before and after patch announcements,” drafts exploits “and automatically executes them without human intervention.” The finding is one exposed attacker server, which its own configuration calls a pilot.

PHP2Shell shows both sides. Three teams reported the flaws, working with AI tools. Wiz says that with frontier models “a single researcher completed in a week what previously would have taken a world-class team months of work.” Afterward, Google notes, came “hundreds of claimed but non-functional POCs” (proof-of-concept exploits), “likely generated with the help of LLMs.” Working exploits stayed scarce.

How much of the speed-up is AI hasn’t been measured. Google now says plainly that LLMs are “shrinking the patch window for organizations.” The UK’s National Cyber Security Centre, in its annual review on November 3, says “it is highly likely defenders and maintainers of systems will face a race to shorten patch deployment timelines.” Nobody has published a number for AI’s share. The direction isn’t in doubt.

What isn’t in dispute is the patch queue. Project Zero again: “We are finding several vulnerabilities a week in targets where we once found a few vulnerabilities a year.” Microsoft says its October update alone “addressed more than 1,100” CVEs for Windows. Every one of those is a patch someone has to apply.

The planning assumption is simple. More vulnerabilities will be found in what you run, building an exploit from a published fix is getting cheaper, and at least one attacker has automated the whole loop. The day a fix is published is the day to plan around.

Your Patch Clock Is Running Slow

PHP2Shell produced something rare: a measurement of how fast the internet patches, as far as version numbers show it.

Bitsight rescanned 9.47 million hosts it had seen running a vulnerable version. On supported PHP versions the curve was steep: “By October 22 it was 55%, by October 24 it was 78%, and on October 27 it reached 85%.” On versions PHP no longer supports, “only 6.4% of 5.34 million hosts are patched.” Across everything, 33.5%. Its conclusion: “Where a provider manages PHP for its customers, patches arrive in days. Where the site owner is responsible, or the server runs a branch that no longer gets fixes, they often don’t arrive at all.” Censys counted over 30 million PHP services still reporting affected versions at the end of October and drew the same lesson: “we should also continue to pay more attention to how readily those patches can actually be deployed.”

The operators who published their timelines show what that looks like from the inside. The host alwaysdata blocked the risky request type hours before the fix, accepting broken uploads for its customers, then had every version it offers patched 2 hours 28 minutes after the release. Its account: “Language updates are routine operations for us, but in this case, we wanted to patch all the versions we offer.” At fortrabbit it took seven hours. “We had to wait until those were updated to be able to build our own images,” its post-mortem says of the official PHP images. “Next time it should not wait on others.” Neither is a textbook pipeline. Both had a practiced path.

It isn’t only PHP. The same Microsoft research, on its own customers’ data, found “the median vulnerability is patched in about two weeks on Windows workstations but in more than a month on Windows Servers,” and “beyond a year on network appliances.” Verizon’s report, on 2025 scanner data from more than 13,000 organizations, found “only 26% of the CISA KEV vulnerabilities had been fully remediated, a considerable drop from last year’s 38%,” with a median of 43 days from detection to a full fix. Verizon’s reading: “There appears to be a ceiling, and the data suggests diminishing returns at current resource levels.” The first-week rate, it says, “barely moved despite three years of additional process development, tooling investment and mandate pressure.” That’s an argument against buying more tools to work the same queue harder. It doesn’t prove this post’s remedy either: Verizon’s data can’t tell an automated path from a manual one.

A patch that was applied is not the same as a fix that is running. Alpine Linux wrote up its own miss last month. GitLab fixed a critical flaw on October 12. Alpine “upgraded to GitLab 19.3.4 instead of 19.3.5 (which had the fix) by mistake,” and an attacker tried the exploit “a week after the fix was released.” A second upgrade still didn’t close the hole, because the fix lived in a patched copy of Ruby that only GitLab’s official packages carried: “Our image used standard Ruby, so it didn’t get the patch.” Alpine rebuilt the instance from scratch. The team had patched, twice. It had no way to prove the fix was in what it shipped.

Why the Pipeline Is the Patch Clock

An emergency patch is just a deployment with a deadline. It goes through the same steps as any other change, and it can’t go faster than those steps allow unless you skip them. DORA lists this among the outcomes of continuous delivery: “Short lead times for both regular and emergency changes, with a goal of using your regular process for emergency changes.” Microsoft’s Windows team put the same point to its customers this month: “The time between an update becoming available and reaching your devices is part of your security posture.”

Break the patch clock into its parts and each one maps to a delivery capability:

Part of the patch clock What shortens it Recent evidence
Knowing you’re affected A current inventory of every dependency in every service After a certificate authority’s keys were compromised, Cloudflare says “within four hours we had a full analysis” of what trusted them
Getting the fix proposed Automated dependency updates GitHub’s Octoverse 2026: developers “merged more than 30 million Dependabot pull requests”
Proving it doesn’t break An automated test suite you trust A small host, on PHP2Shell: a version change “on a deadline at 4am turns a security fix into an outage”
Shipping it Frequent, automated builds and deploys Docker says its build system ran “16,567 package and image builds” in the 48 hours after the PHP release
Limiting the damage if it’s wrong Staged rollouts Azure’s review of a September 30 incident: paused updates “resumed within a short period” across its gateways, not in stages
Undoing it A tested rollback In DORA’s 2025 survey, 21.3% of respondents recover from a failed deployment in under an hour

The first row is vulnerability management, and volume has changed what that job is. With CVE counts doubling, “patch every critical” isn’t a policy anyone can carry out. Rank by evidence: known exploitation and internet exposure go to the front of the line, on a deadline. Everything else rides the routine update path, which only works if you have one.

The last row is the one that gets cut first, and it’s what makes fast patching safe to approve. If a bad deploy costs a week, every urgent patch turns into a negotiation about risk, and the negotiation takes longer than the patch. In the DORA survey, about 43% of respondents say recovery takes more than a day.

Now put the delivery numbers next to the clocks. In the same survey of 4,867 people, 16.2% say they deploy on demand. About 24% deploy less than once a month. About 44% take more than a week to get a change from commit to production. For those teams, a three-day patch window isn’t a matter of trying harder. Their normal process can’t produce it, so they’ll either miss it or go around their own safeguards to hit it.

It’s Becoming a Compliance Clock Too

Regulators have started writing patch clocks down, and the last month shows them being applied.

  • US federal agencies. CISA’s Binding Operational Directive 26-04, issued in June, sets deadlines by risk: 3 or 14 days for known-exploited vulnerabilities, with mitigation counting. Of the 43 flaws CISA has listed since October 12, 29 carry a three-day deadline. In 2025, 226 of 245 carried 21 days. PHP2Shell was listed October 26 and due October 29, and CISA’s alert told anyone who couldn’t update to “take PHP-based systems offline until they can be patched or updated.” Agencies must meet the timelines in full by December 7. The directive binds agencies, not their vendors, but it tells agencies to review their contracts.
  • Cloud vendors selling to the US government. FedRAMP’s vulnerability rules become mandatory December 7. Its November 10 notice says a provider that hasn’t migrated or agreed a corrective plan by March 8, 2027 “will have its FedRAMP Certification revoked and be banned from FedRAMP for one year,” and that “the recent Black October” has only raised the urgency.

None of these says “adopt CI/CD.” Each sets a number of days. The pipeline is how you hit the number without heroics and show afterward that you did.

Where Fast Patching Falls Down

A faster pipeline doesn’t solve all of this. Its limits should shape what you build.

It only covers what you build and ship. Your delivery pipeline patches your code, its dependencies and the runtime underneath them. It doesn’t patch the firewall, and most of CISA’s 43 recent listings are vendor products and appliances, not anything an application build ships. Palo Alto’s first fixes for PHP2Shell shipped on October 25, four days after PHP’s, and its advisory confirms “active exploitation” of exposed management interfaces. A security firm told The Record it found 24 compromised firewalls, and “most of the attacks took place between October 23 and 24,” before a vendor fix existed. Check Point confirmed compromises of unpatched gateways “approximately one week after a security fix became available.” Appliances need the same principle through different tooling. Until the vendor ships, the only control is taking the management interface off the internet.

It does nothing for the window before disclosure. Cloudflare saw thousands of PHP2Shell attempts before the announcement. A fast pipeline shortens the exposure once the fix is out, but it can’t start before then. Until it can, the first line is mitigation. Cloudflare had a blocking rule live at 3:02 a.m. UTC on October 21, hours before the release, and CISA counts mitigation toward its deadlines. It’s a real control. It isn’t the fix: PHP published partial measures and said they “do not make an installation safe.”

The pipeline is itself an attack surface. On October 21, attackers pushed a malicious release of LiteLLM, a widely used AI library, through its own release pipeline. It was live for about an hour and three-quarters. The maintainers note that a release approval rule had been “relaxed earlier this week to accommodate the temporary push for more frequent releases” (they add that the attacker could have removed it anyway). Starting November 10, a campaign used “4,837 compromised accounts” to open “probe pull requests in 61,764 repositories,” and “at least 1,835 repositories ran the payload,” which steals build secrets and deletes its own logs.

The policy is fast for fixes, slow for routine updates, and a person opens the fast lane. Package managers are adding waiting periods for new releases: Composer in version 2.11 on October 29, Rust’s Cargo in 1.100 on November 12. Composer’s applies “to every new version, including releases that fix security vulnerabilities,” because an attacker who controls a package “can publish a security advisory for the package themselves.” Skipping the wait in a real emergency is a logged decision by a named owner. Harden the pipeline like the production system it is: pin third-party actions, scope build credentials tightly and treat workflow files as security-sensitive code.

A fix is a change, and changes cause outages. PHP2Shell’s own emergency release wasn’t clean. fortrabbit reported that two of the patched versions “showed a regression: a crash” in one component, and PHP left a third, related flaw for a later release. A patch is a production change, so staged rollout and rollback are part of the control.

A green build isn’t proof. Patch-level dependency updates fit the low-risk tier from earlier this month: green checks, a staged rollout and a named owner. Anything touching a protected path still goes to senior review.

One more caution, on the evidence itself. Most of it is correlational, and nearly every source here sells security, cloud or AI products. None of it is a controlled study of remediation time with and without delivery automation, and a study of 978 open source projects found tests and CI only slightly raised the odds that a project merged security updates at all (58% with both, 55% with neither). Some practitioners think speed is the wrong goal. The security chief of Chainguard, which sells pre-patched software images, in a survey report last month: “Patching faster isn’t viable for most folks.” What the evidence supports is narrower: a fix can’t reach production faster than a change can, and where patching is automated it is measurably faster.

Make It a Business Objective

If these foundations are behind, catching up is a company objective, not an engineering hygiene item.

This work usually stalls the same way. “Improve CI/CD” sits in the engineering backlog, competes with features every sprint and loses, because nobody outside engineering can see what it buys. The risk it carries shows up everywhere else. In a survey of 1,300 security leaders that Chainguard commissioned, “93% of leaders worry their remediation speed can’t keep up with AI-assisted exploits.” Moody’s, as reported by Infosecurity Magazine, has published patch times, built on Bitsight’s data: a median of 87 days for known exploited vulnerabilities. A risk that credit raters and regulators are measuring shouldn’t be funded from whatever capacity is left after the roadmap.

So write it down the way you’d write any other objective.

State it in business terms. Not “adopt CI/CD.” Something a board member can check: every internet-facing system can take a critical security fix from advisory to production within a set number of days, through the normal release process, and we can show it. Start from outside reference points and adjust them to your baseline: three days from the day a flaw on an exposed system is listed as exploited (CISA’s shortest tier), test feedback in under ten minutes (DORA’s bar for developers) and recovery from a bad deploy in under an hour (what 21.3% of DORA’s respondents report).

Give it one owner. A named executive, usually the CTO. The security leader decides what gets prioritized, the CEO sponsors it and the CPO signs up for the roadmap trade-offs. If it belongs to “engineering,” it belongs to nobody.

Fund it. A named team with protected capacity and its own budget line, not a percentage of everyone’s spare time. FedRAMP’s notice makes the budget point outright: providers “must have sufficient budget, planning, and engineering capability for the continuous operational improvements necessary to keep responding to this accelerating threat.” Most of the work is unglamorous: a regression suite, a one-command deploy, a dependency inventory, automated update pull requests, a staged rollout and a rollback that’s been tested. Agents can do a large share of it. Earlier this year I built 1,500 regression tests across a fifty-repo legacy platform in three days. The scarce input isn’t typing anymore. It’s someone with the authority and the time to decide this comes first.

Set milestones with dates. The timeline below is [ILLUSTRATIVE]. Reset it from your own baseline.

  1. First 30 days: measure. Inventory every production service, what it depends on and which runtimes are past end of life. List the vendor appliances too, with who patches each one. Pull the last five critical patches and record how long each took and where the time went. Then run a drill: replay a real past advisory against your oldest internet-facing service and time it end to end. That number is your patch clock. If you’ve never run it, you don’t know yours.
  2. By 90 days: the exposed systems. Every internet-facing service has automated security update pull requests, a test gate that runs on its own, a deploy any engineer on the owning team can run and a rollback that’s been exercised. Anything on an unsupported runtime has a dated plan to get off it.
  3. By 180 days: hit the number. Known-exploited vulnerabilities on exposed systems are fixed or mitigated inside the target window, with staged rollout in place. The drill runs every quarter, because in a quiet quarter it’s the only measurement you’ll have.
  4. By 12 months: everything else. All production services are on the same path, and the emergency process is the normal process.

Report it to the board. Four numbers, quarterly: time from advisory to production for known-exploited vulnerabilities, the count of those open past their deadline, the share of services on the automated path and the latest drill result.

One caveat worth stating directly: don’t let this turn into a tool purchase. A scanner tells you what to fix. It doesn’t shorten the path from a merged fix to production, and no scanner builds that.

What This Means For You

If you’re a CEO, ask one question this week: how long would it take to ship a one-line dependency fix to every production system, and when was that last timed? If nobody knows, that’s the finding. Then treat the gap like any other strategic exposure. Name the owner, fund the team, set the dates and ask for the number every quarter.

If you’re a CTO, run the drill before you propose anything. Bring the baseline, the target and the list of what stands between them. Build the path so the emergency route and the normal route are the same route. Set the policy to fast for fixes and slow for routine updates, and harden the pipeline itself, because it now holds the keys to production. Then stop describing this work as developer productivity. It’s the control that lets the company answer a customer or an auditor who asks how fast it can patch.

If you’re a CPO, this competes with your roadmap, and some quarters it should win. Release trains, change freezes and manual sign-offs are product decisions with a security price, now measured in days of exposure. If your customers sell to government, the timelines above reach you through their contracts. The capacity isn’t lost. The pipeline that ships a fix in hours is the same one that ships features in hours.

CI/CD was sold as the way to ship faster than the competition, and a company could decide to live without that. For the software you build and ship, it now sets how long a known hole stays open. Time your patch clock.

Sources

PHP2Shell (October 2026): PHP: Security advisory for PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 · AWS: PHP2Shell exploitation prior to public disclosure · Matthew Prince (Cloudflare) on X, October 21, 2026 · CISA: Critical vulnerabilities in PHP require immediate patching · Wiz: PHP2Shell · watchTowr: PHP2Shell rapid reaction · Bitsight: PHP2Shell scanning, exploitation and patching · Censys: Thoughts on PHP2Shell · Palo Alto Networks advisory · Check Point Research: PHP2Shell exploitation on Check Point gateways · The Record: Limited reports of PHP bug exploitation emerge · Cloudflare: WAF protection for the PHP vulnerabilities · alwaysdata: PHP2Shell post-mortem · fortrabbit: PHP2Shell post-mortem · Managed Hosting Partners: How we patched PHP2Shell · Docker: PHP2Shell and eMudhra patched

Exploitation data: Verizon 2026 Data Breach Investigations Report (the full report requires registration; the executive summary does not) · Rapid7: Q3 2026 threat update · Microsoft: Exploited vulnerabilities, a data-driven analysis (November 2026) · VulnCheck: Glasswing November update · Google GTIG: Vulnerability discovery and exploitation trends in the AI era · CVE Program metrics · CVE Program report for Q3 2026 · CISA Known Exploited Vulnerabilities catalog

AI and exploitation: Google GTIG: Agentic threat activity crosses a significant threshold (November 2026) · Google GTIG: From prompting to autonomy (September 2026) · Google Project Zero: Disclosure policy update (November 2026) · Anthropic: Claude Fable 5.2 and Mythos 5.2 system card · Calif: Java4Shell · Microsoft Security: Security for the agentic era · UK NCSC Annual Review 2026: AI impact on the cyber threat

Patching and delivery: Microsoft Windows: A new standard for vulnerability management in the AI era · Alpine Linux: GitLab security incident · Cloudflare: eMudhra patching retrospective · GitHub Octoverse 2026 · Azure status history, incident 7Q30-010 · DORA 2025 State of AI-assisted Software Development · DORA: Continuous delivery · DORA: Test automation · Mohayeji et al.: Dependabot’s impact on vulnerability mitigation (author’s copy; published in Empirical Software Engineering, 2025, doi:10.1007/s10664-025-10638-w)

Supply chain and pipeline security: LiteLLM: Security incident, October 21, 2026 · Aikido: Compromised GitHub accounts push CI secret stealer · Composer 2.11 release · Rust 1.100.0 release

Rules and raters: CISA Binding Operational Directive 26-04 · FedRAMP notice NTC-0014 · FedRAMP notice NTC-0017 · Chainguard: The AI exploit gap · Infosecurity Magazine: Moody’s on time to patch