Serious AI incident detection and regulator reporting

Description

Serious-incident reporting for AI systems has shown up in regulation faster than most other AI-governance obligations, and the reason is that the regulators writing AI law have been watching the aviation and pharmaceutical regimes for a template that was already legible. Both of those industries have decades of experience with mandatory adverse-event reporting on a short clock, and the EU AI Act borrowed the shape almost directly: a defined trigger (death, serious harm to health, serious and irreversible disruption of critical infrastructure, infringement of fundamental rights), a defined clock (15 days from the provider's awareness of the incident, shorter for some categories), and a defined recipient (the national market-surveillance authority in the member state where the incident occurred). California SB 53 added a parallel US regime targeted specifically at frontier-model providers, with a similar reporting-on-a-clock shape but a different scope. The emerging US federal regime under executive orders sits somewhere between the two and has not yet stabilized. A working incident-reporting program has four operational layers. The detection layer is the monitoring infrastructure that flags candidate incidents from product telemetry, customer complaints, regulator inquiries, and internal red-team exercises; the harder-than-it-looks part is distinguishing a serious incident from a routine model error, because the regulatory definition turns on consequence to a natural person rather than on internal model performance. The classification layer is the triage decision that decides which candidates cross the reporting threshold, who owns the decision, and how the decision is documented (regulators evaluate the program partly by reviewing close-call decisions where the operator decided not to report). The notification layer is the regulator-facing report itself: incident description, affected systems, affected persons, suspected root cause, immediate mitigations, and follow-up commitments. The remediation layer is the corrective-action tracker that closes the loop, because a 15-day report is the beginning of the regulator's interest in the incident, not the end. The thresholds and timelines are concrete. EU AI Act Article 73 requires providers of high-risk AI systems to report serious incidents to the national market-surveillance authority within 15 days of becoming aware, reduced to 2 days where the incident involves death or where widespread harm has occurred or is suspected, with widespread infringement of fundamental rights triggering immediate notification; high-risk-system obligations are applicable 2026-08-02. California SB 53 requires frontier-AI providers (developers of foundation models trained using above-threshold compute) to report critical safety incidents to the California Office of Emergency Services within 15 days, effective 2026-01-01, with separate trigger categories for catastrophic-risk-relevant capabilities, unauthorized model access, and incidents demonstrating significant operational misuse. The FTC's Section 5 enforcement posture treats failure to report material AI-related consumer harms as a candidate unfairness or deception predicate, even outside a specific reporting statute. The 15-day clock is short relative to most enterprise incident-response cycles and tight relative to legal-review cadences, which is why standing up the classification layer before an incident occurs is the operationally critical step. The genuinely uncertain piece is what counts as a serious incident under each regime. The EU AI Act's serious-incident definition is statutory but the national market-surveillance authorities have not yet published implementing guidance on borderline cases; California SB 53's critical-safety-incident definition is similarly statutory but new; and the EU and California regimes will need to be navigated separately even where the same incident is reportable under both, because the recipients, formats, and clocks differ. The pattern most operators are converging on is a single internal classification taxonomy mapped to multiple regulator-specific notification templates, on the assumption that the regimes will not harmonize on the substantive trigger any time soon.

Applicability

Applies when: AI role is ai-provider or ai-deployer.

How predicates are evaluated

Required by (3 regulations)

• EU AI Act

  Article 73 requires providers of high-risk AI systems to report any serious incident occurring in the EU market to the national market-surveillance authority of the member state where the incident occurred. The general clock is 15 days from awareness, reduced to 2 days for incidents involving death of a person or widespread harm, and immediate notification for widespread infringement of fundamental rights. Serious incident is defined to include death, serious harm to health, serious and irreversible disruption of critical infrastructure, and infringement of obligations under EU law intended to protect fundamental rights.

  Regulation (EU) 2024/1689 of the European Parliament and of the Council (AI Act); Article 3(49) (serious incident definition); high-risk-system obligations applicable 2026-08-02

• California SB 53

  Developers of frontier foundation models trained using above-threshold compute must report critical safety incidents to the California Office of Emergency Services within 15 days of awareness. Reportable categories include incidents involving catastrophic-risk-relevant capabilities, unauthorized access to model weights or training infrastructure, and operational evidence of significant misuse of the model. Providers must also publish a safety and security protocol describing how they identify and mitigate catastrophic risks.

  California SB 53 (Frontier AI Models Transparency Act); effective 2026-01-01

  Source →

• FTC Act

  Section 5 prohibits unfair or deceptive acts in or affecting commerce. The FTC has signalled in guidance and enforcement actions that failure to disclose material AI-related consumer harms is a candidate unfairness or deception predicate even outside a specific reporting statute, and consent decrees in AI-enforcement matters have required ongoing reporting on safety incidents (Rite Aid 2023, Amazon Alexa 2023).

  Federal Trade Commission Act § 5; 15 U.S.C. § 45; in force since 1914 with subsequent amendments

Evidence formats

• incident-detection SOP covering telemetry signals, customer-complaint intake, and red-team escalation paths
• per-incident report template aligned to EU AI Act Article 73 and California SB 53 notification fields
• regulator-notification calendar with per-jurisdiction clocks (EU 15 day / 2 day, California SB 53 15 day) and named owners
• post-incident remediation tracker linking each reported incident to root-cause analysis and corrective actions
• incident-classification log including close-call decisions where the operator decided a candidate did not meet the threshold
• staff training log for engineering, support, and legal teams on incident recognition and escalation triggers