Audit trails and evidence integrity: how AI procurement tools maintain compliance

AI procurement tools generate compliance claims at speed: "This device meets requirement X, as evidenced by datasheet Y, page Z, section W." But how do you know the evidence is real? How do you verify that the AI didn't hallucinate a reference, cite an outdated document, or misinterpret a specification?

This is the evidence integrity problem. It matters more for AI-generated claims than for human-generated ones, because AI can produce plausible-sounding citations to documents that don't exist.

What a proper audit trail looks like

For every compliance claim in a tender submission, the audit trail should contain:

1. The requirement: The exact text of the tender requirement, with reference number and source document page
2. The claim: The compliance statement made in the submission (compliant, partially compliant, non-compliant)
3. The evidence source: The specific document (datasheet, certificate, test report, regulatory filing)
4. The evidence location: Page number, section heading, and relevant text excerpt
5. The confidence score: How strongly the evidence supports the claim (0-100%)
6. The match method: How the requirement was matched to the evidence (semantic match, exact match, human override)
7. Timestamp: When the match was generated and when it was last verified
8. Reviewer: Who approved the match (human name, not "AI system")

Evidence integrity verification

A robust AI procurement system verifies evidence integrity at three levels:

1. Document existence verification

The cited document actually exists in your document repository. The AI can't reference a datasheet that hasn't been uploaded. This sounds obvious, but general-purpose LLMs routinely cite documents that don't exist. A RAG architecture prevents this by constraining the AI to only cite documents in your verified corpus.

2. Content accuracy verification

The cited page and section actually contain the text the AI claims. The system should retrieve and display the exact passage so a human reviewer can verify. If the AI says "datasheet page 12, section 3.2 states operating frequency 2-12 MHz," that specific text should be retrievable and viewable.

3. Currency verification

The cited document is current. A CE certificate that expired last month is worse than no certificate — it suggests either negligence or a compliance gap. Evidence integrity verification includes checking: certificate expiry dates, standard version currency (is the cited IEC 60601-1 the current edition?), and document version (is this the latest datasheet revision?).

Why this matters for audits

Medical device companies face regular audits: ISO 13485 quality management audits, Notified Body surveillance audits, and tender authority compliance audits. In each case, the auditor may ask: "Where does this compliance claim come from?"

With a proper audit trail, the answer is immediate: "This claim is supported by document X, page Y, verified on date Z, with confidence score of 94%." Without it, the answer is: "Someone on the team checked it manually... we think."

AI-generated audit trails are actually stronger than manual ones. Manual compliance checking rarely records the evidence source at this level of detail. The team knows the product is compliant, but they can't point to the exact paragraph that proves it. AI systems, properly designed, make every claim traceable by default.

Red flags in vendor audit trail implementations

• "AI-generated summary" without source citation: The AI summarized a document but doesn't point to the original text. This is a hallucination risk.
• Confidence scores always above 90%: If every match is "high confidence," the scoring is meaningless. Real spec matching produces a distribution of scores.
• No document version tracking: The system cites "Product Datasheet" without specifying which version. When the datasheet is updated, old claims become unverifiable.
• No human review step: Every AI-generated claim should be reviewed and approved by a named individual before submission. The AI generates; the human verifies and takes responsibility.

MedStrato's evidence integrity approach

Every match in MedStrato generates a complete evidence chain: requirement → claim → document → page → section → excerpt → confidence score → timestamp → reviewer. Documents are version-controlled, certificates are expiry-date tracked, and the entire audit trail is exportable as a standalone compliance report for auditors.