Future AI Pulse
Quick answer: AI can help draft a detection change, but the important question is what that change would have done to real cases. Block’s security case study describes retaining query evidence so proposed changes can be replayed against historical telemetry. The practical lesson is to compare behavior against a baseline before promoting a rule.
What Block reports
In an OpenAI case study published October 14, 2026, Block says its team can replay a proposed detection change against seven months of telemetry in under an hour. The account attributes that workflow to stored raw query evidence and describes humans validating results.
These are results reported in a vendor customer story, not an independently reproduced benchmark. The article does not establish that another organization will obtain the same speed or that every proposed detection change is correct. Future AI Pulse has not inspected Block’s telemetry or tested its system.
A baseline makes the change reviewable
For your own authorized defensive workflow, start by writing down what the existing rule is intended to detect. Record the behavior it catches, common benign matches and the cases where evidence is incomplete. Then describe the specific change under review.
Run the current and proposed versions against the same permitted historical sample. A reduction in alerts is not automatically an improvement: it may remove noise, or it may hide useful signals. Inspect which cases changed rather than relying only on the total.
A compact review table can use five fields: case identifier, baseline outcome, proposed outcome, evidence reference and reviewer explanation. Keep the source evidence accessible to authorized reviewers so they can verify the explanation without trusting a fluent summary.
Use AI to organize disagreement
A useful assistant can group changed cases, identify missing fields and draft a comparison. It should not silently redefine what counts as a successful detection. Supply the criterion before asking it to evaluate the result.
Ask it to separate evidence from interpretation. If a case cannot be judged from the available record, an explicit unresolved label is more useful than a confident guess. Review both newly triggered cases and cases the proposal would stop triggering.
Review this authorized historical comparison. Detection purpose: [specific behavior]. Baseline and proposed outcomes: [permitted case table]. Group cases where the two versions disagree. For each group, cite the case identifiers and evidence references. Separate observed fields from your interpretation. Flag missing evidence and unresolved classifications. Do not treat fewer alerts as proof of better detection. Return a review memo only; do not change production rules.
This is an original planning prompt. It is not Block’s internal prompt, an executable query or a workflow we have tested.
Keep historical success in perspective
A replay answers a question about the chosen historical sample. It does not show every future behavior the rule might encounter. Preserve a separate validation set when feasible and document the sample’s time range, sources and known gaps.
Also distinguish a rule evaluation from a full operational deployment. Before adopting a change, review ownership, rollout conditions and a way to restore the previous version. If the evidence is insufficient, record the unanswered question and keep the proposal under review.
Measure review effort as well as runtime
Track how long the comparison takes to prepare, how many cases require manual inspection and whether reviewers agree about the result. An assistant that writes a fast memo but produces untraceable explanations can simply move work to a later stage.
The aim is a reproducible record of why the change was accepted or rejected. Related reading: our security agent handoff guide covers the distinction between evidence, recommendations and authority.
Illustrative server photograph: Kevin Ache / Unsplash, under the Unsplash License. Not Block infrastructure.
Sources
Source checked October 15, 2026, America/Chicago. Written with AI assistance. The proposed review method is editorial guidance, not a reported hands on test.