AI for risk management: analyzing information is not predicting incidents

# Why are analysis and prediction confused?

The confusion often starts with a convincing demo. A model receives a hundred reports, identifies recurring themes, and drafts a summary. Because the response seems intelligent, it is easy to take the next leap: “if it understands past incidents, it can anticipate the next one.”

That leap is not justified.

Summarizing existing information is a retrospective task. Estimating the probability of a future event is a predictive task. The first can rely on language and search; the second needs to demonstrate a valid relationship between observed variables and subsequent outcomes.

In addition, an infrequent incident may be poorly documented, change as processes evolve, or depend on conditions that were never recorded. A lack of reports does not prove that there is no risk either: it may reflect underreporting, inconsistent categories, or incentives not to document.

# Three distinct uses of AI in risk management

The same solution can combine all three, but it should not obscure the transitions. If a system goes from summarizing a report to recommending that an operation be stopped, that involves a different authority, a different risk, and a higher standard of evidence.

# What can AI do well with risk reports?

Language systems are useful when an organization has information scattered across reports, minutes, logs, procedures, and authorized emails.

# Classify and normalize

AI can suggest common categories for descriptions written in different ways. “Minor leak,” “loss of containment,” and “controlled spill” could be grouped for review, provided the taxonomy is approved by specialists and the original text is preserved.

Normalization helps with search, but it can also erase important differences. That is why the proposed category should be accompanied by evidence and allow for correction.

# Summarize with traceability

A good risk summary should not hide its source. Each important claim should link to the report, page, or passage that supports it. If there is not enough evidence, the right response is to say so, not to fill in the story.

# Find controls and pending actions

AI can locate mentions of responsible parties, dates, corrective measures, and controls. It can also detect documents missing an expected field. This does not verify that an action was carried out: it identifies what is documented and what is missing in the dataset analyzed.

# Compare procedures with documented practices

A system can compare a report with a list of controls or a current procedure and flag differences for review. Compliance decisions belong to the authorized function and must take versions, exceptions, and operational context into account.

# Prioritize a review queue

When the volume makes it impossible to read everything right away, AI can sort files according to defined criteria. Prioritization helps distribute attention; it is not an automatic statement about severity, fault, or cause.

# What does a real prediction require?

A responsible prediction starts by defining the event. “Predict threats” is too broad. “Estimate the probability of an unscheduled shutdown of a certain piece of equipment within seven days” is a question that could be evaluated if suitable data is available.

At a minimum, you need:

• an observable and consistent target variable;
• a defined time window;
• data from before the outcome, with no leakage of future information;
• sufficient coverage of normal operations and events;
• documentation of changes to processes, equipment, and criteria;
• a separate validation set;
• metrics suited to the cost of missing an event or overreacting;
• monitoring to detect degradation.

A model can have good overall accuracy and still fail on important cases. If incidents are rare, always predicting “nothing will happen” can seem accurate. That is why metrics are needed to distinguish how many events it detects, how many alerts are false, and how much advance notice it provides for actionable information.

# False positives and false negatives

Every predictive signal involves errors.

An false positive occurs when the system alerts about an event that does not happen. This can lead to unnecessary inspections, alert fatigue, and loss of trust.

An false negative occurs when the system does not alert and the event does happen. In safety, continuity, or compliance settings, its cost can be much higher.

There is no universal threshold. The organization must decide on the balance with input from specialists, operations, safety, and leadership. It must also establish what action corresponds to each level and whether that action could introduce a new risk.

# What changes in mining and industrial settings?

In mining, manufacturing, logistics, or infrastructure, information combines documents, sensors, maintenance, human observations, contractors, and environmental conditions. A language model can help with the documentary side, but it does not replace engineering, geology, safety, maintenance, or operational leadership.

Reasonable use cases to explore include:

• summarizing incident and near-miss reports;
• linking corrective actions to owners and dates;
• comparing descriptions against an approved taxonomy;
• preparing a case file for a review meeting;
• identifying controls that are mentioned or missing;
• identifying recurring topics for training;
• translating and standardizing reports while preserving the original;
• supporting searches for current procedures.

By contrast, claims such as “AI will prevent accidents” or “AI will anticipate threats” should not be made without a specific system that has been validated for the context and is subject to operational controls. A document tool does not become a safety system just by adding a dashboard.

# How do you design a low-risk pilot?

The first pilot should improve an analysis task without taking control of operations.

# Step 1. Choose an existing decision

Example: preparing for the weekly review of corrective actions. The team already knows who makes the decision and what information they need. AI can reduce the time spent searching and organizing without inventing a new process.

# Step 2. Define authorized sources

Specify which versions of procedures, reports, catalogs, and data may be used. Avoid mixing drafts, expired documents, or information from other units without context.

# Step 3. Create an evaluation set

Select representative cases and have specialists document the expected answers, ambiguities, and cases where no conclusion can be drawn. This collection will be more valuable than an improvised demo.

# Step 4. Require evidence

Every summary, classification, or recommendation should show the source excerpt. A useful interface lets users open the source and record a correction.

# Step 5. Define the limits of autonomy

During the pilot, the system can make suggestions, organize information, or draft text. It must not close actions, modify procedures, determine causes, impose sanctions, issue a critical alert, or control equipment without the appropriate validation and authorization.

# Step 6. Measure usefulness and risk

Measure time saved, coverage, errors, disagreements between specialists, incorrect citations, omitted information, and actions the team actually used. Include adversarial tests: incomplete documents, conflicting versions, scanned tables, and uncommon terms.

# Which metrics are useful for document analysis?

Not every implementation needs a predictive metric. A document assistant can be measured by:

• extraction accuracy by field;
• proportion of claims with a correct citation;
• relevant documents retrieved;
• critical errors and omissions;
• time to prepare a review;
• percentage of outputs corrected;
• frequency of appropriate use of “insufficient evidence”;
• specialist satisfaction, assessed alongside real-world observation.

The metric must match the use. A polished summary does not make up for an incorrect reference; a fast search is not useful if it relies on an outdated procedure.

# Which metrics are useful for prediction?

If the project actually includes a predictive model, it needs statistical and operational metrics. These may include sensitivity, specificity, alert precision, probability calibration, lead time, and intervention cost.

It should also be compared against a baseline. Does the model improve on a simple rule, the maintenance program, or current judgment? Without that comparison, an isolated metric may impress without demonstrating added value.

NIST recommends that measurements be representative of the context of use, that limitations and uncertainty be documented, and that evaluation continue after deployment. A single test does not guarantee ongoing performance.

# Governance: who is accountable for the output?

Before the pilot, assign people responsible for:

• approving sources and versions;
• defining taxonomies and criteria;
• validating outputs in their domain;
• authorizing operational changes;
• manage access and retention;
• receive error reports;
• decide when to suspend the system.

The technology provider can operate components, but should not inadvertently become the owner of the risk. The organization retains authority over processes, personnel, security, and compliance.

# Signs of an overblown proposal

Be wary when a proposal:

• promises prediction without defining the event;
• doesn't ask about data quality, coverage, or age;
• uses “accuracy” without explaining the metric;
• doesn't distinguish language analysis from predictive modeling;
• doesn't account for false positives and false negatives;
• doesn't identify who validates or who decides;
• presents a demonstration as evidence of performance;
• doesn't include monitoring, correction, or a safe exit.

A responsible proposal may be less spectacular. It will also be more useful in deciding whether an investment makes sense.

# Frequently asked questions

# Can generative AI predict accidents?

Not on its own. It can analyze text and generate hypotheses, but prediction requires a defined problem, representative data, and validation. Even a validated model provides probabilities, not certainties, and does not replace safety controls.

# Can historical reports be analyzed with Claude or ChatGPT?

Yes, as long as the product, contract, configuration, and information are appropriate. Access, sensitivity, retention, purpose, and versions should be reviewed. For a recurring workflow or large volume, an API integration with your own controls may be appropriate.

# Can an incident summary serve as a risk model?

It can serve as an input for analysis, not as a predictive model. It summarizes what has been documented and may reveal descriptive patterns. It does not automatically estimate the likelihood of future events.

# What should the system do when data is missing?

It should indicate what is missing, lower its confidence, or request a review. Filling gaps with plausible-sounding language is especially dangerous in a risk context.

# What is the best first use case?

A frequent, well-defined, and reversible task: searching current procedures, preparing case files, or tracking actions through documentation. This makes it possible to learn about data and adoption before considering predictive uses.

# Conclusion

AI can improve risk management by making evidence more accessible: organizing reports, finding controls, summarizing case files, and directing human attention. That benefit can already be significant.

Predicting an incident is a different problem. It requires data, statistics, validation, and an explicit conversation about errors and consequences. Confusing these two capabilities creates expectations the system cannot meet.

The first question should not be “What can AI predict?” It should be “What decision do we want to improve, with what evidence, and who will be responsible?”

# Sources consulted

• NIST AI Risk Management Framework: Generative AI Profile
• NIST AI RMF Core
• NIST AI RMF Trustworthiness Characteristics
• NIST: AI system evaluation and uncertainty quantification