top of page
Abstract Shapes

INSIDE

PUBLICATIONS

AI Vendor and Model Due Diligence

AI Vendor and Model Due Diligence
AI Vendor and Model Due Diligence

UIB emblem

UIB University 365 Institute of Business

Series Business AI Series

Duration 25 minutes

Business Management, Digital Entrepreneurship, Innovation, Finance, Leadership


UNOP isochrone

UNOP Sound (University 365 Neuroscience Oriented Pedagogy)

Take five minutes to prepare your brain. Play the isochronous tone track (40Hz gamma frequency) with your eyes closed. Gamma-frequency tones before a learning session raise attention and make the material easier to absorb.

[Audio player: UNOP Pre-Lecture Isochrone (40Hz, 5 minutes)]

In this Lecture


Back to the TOC

Section icon: The Deal That Passed the Demo.

AI Vendor and Model Due Diligence

The Deal That Passed the Demo


The demonstration took forty minutes. The agent opened a supplier invoice, compared it to a purchase order, flagged two mismatches, drafted an exception note and asked a human for approval. The committee approved a three-year contract the same week. Eleven months later, three things had changed. The model version the pilot ran had been retired with 45 days notice, the per-token price had risen by 40 percent at renewal, and the vendor had added a subprocessor in a second jurisdiction. None of this was hidden. All of it sat in documents nobody had requested, because the demo answered a different question.


A demo answers one question: can it work? Due diligence answers four harder ones. Will it still work on our data in two years? On what terms? Under whose control? And at what cost if we have to leave? Those are procurement questions, and they are settled with evidence rather than opinion.


This lecture gives you twelve criteria, a weighted scorecard, five tests that produce evidence, the contract clauses that carry each score, and a one-page decision memo. It is written for the business owner who signs, and for the security, legal and finance people whose signatures sit beside it. Every price and date here comes from the November 2026 baseline used across this series.

Back to the TOC

Section icon: Due Diligence Is a Decision, Not a Questionnaire.

AI Vendor and Model Due Diligence

Due Diligence Is a Decision, Not a Questionnaire


Due diligence serves three decisions: which vendor to choose, on what conditions, and with what exit. A questionnaire produces answers. Due diligence produces a decision with evidence attached, and it is complete only when the memo names the conditions and the owner of each one.


Three documents get confused in most procurement files. The vendor's security questionnaire contains claims. The contract contains commitments. Your own tests contain evidence. A claim is worth reading, a commitment is worth having, and only evidence earns the top of the scale. The scorecard exists to force every important claim into one of the last two categories.


The scope rule matters more than the criteria. Assess the service on the workload, not the vendor in general. The same provider can deserve a 5 for a drafting assistant and a 1 for an agent that can change a bank record. The unit of analysis is the combination: vendor, model, version, configuration, data, permissions and task. A due diligence file that scores the company rather than the deployment tells you nothing about the risk you are about to accept.


Not every purchase needs the same depth. Three tiers keep the process proportionate. A desk check takes two hours and suits a low-consequence tool with public data: read the terms, confirm the data use, record the owner. A standard review takes about ten days and suits a departmental assistant that touches internal documents. A full review takes thirty days and suits anything that moves money, processes personal data at scale, or supports a regulated decision. The criteria are the same in all three tiers; the evidence depth changes. Scoring a stationery purchase at the full tier wastes a month, and scoring a payment agent at the desk-check tier is how the case at the start of this lecture happens.


The annual questionnaire is the other trap. Many organisations send a long questionnaire once a year, collect the answers and file them. The file is out of date within a quarter, because models, prices and subprocessors change on their own schedule. Tie the review to four triggers instead: a new workload, a model version change, a price change above the notice threshold, and a subprocessor change. Each trigger runs the affected criteria only, which costs far less than repeating the whole review.

Back to the TOC

Section icon: The Twelve Criteria in Four Blocks.

AI Vendor and Model Due Diligence

The Twelve Criteria in Four Blocks


The twelve criteria sit in four blocks. Capability and evidence asks whether the system does your work. Data and security asks what happens to your information. Control and operations asks what the system can do without you. Commercial and exit asks what it costs and how you leave. The weights sum to 100, and they are a starting point you may change before scoring, never after.


No.

Criterion

Question it answers

Weight

1

Task fitness

Does it pass on our own tasks, scored blind?

10

2

Evaluation evidence

Can we inspect the evaluation for this version?

7

3

Lifecycle and change

How long does this version last, and what notice do we get?

8

4

Training use

Do our inputs or outputs train their models?

9

5

Retention and deletion

How long is our data kept, and can we delete it?

7

6

Residency and subprocessors

Where is it processed, and who else touches it?

6

7

Security assurance

What do the reports cover, and what do they miss?

8

8

Agent identity and permissions

What can the agent reach, and as whom?

8

9

Action limits and approval

What is the worst single action it can take?

9

10

Logging and incident notice

Can we see what happened, and how fast are we told?

8

11

Price mechanics and notice

What is the formula, and how fast can it change?

10

12

Exit, portability and liability

Can we leave with our data, and who pays for harm?

10


Two rules apply before any score exists. Any criterion scored 0 or 1 stops the deal regardless of the total. The total then chooses between proceed, conditional approval and stop. A vendor with a brilliant model and no exit clause is not a 70; it is a stop at criterion 12, and the model's quality does not buy it back.


The weights are a policy statement, and they should follow the consequence of failure. For a drafting assistant, capability deserves the most weight. For an agent that can issue a refund, control deserves it. Take the same twelve criteria and re-weight them for a payment agent: raise criterion 9 (action limits) from 9 to 15, and take the six points from criteria 1, 2 and 11, which fall from 10 to 8, from 7 to 5 and from 10 to 8. The total stays 100, and the scorecard now reflects what a mistake costs. Write the re-weighting down before the scores arrive, or the exercise becomes a search for the answer you wanted.


The twelve due diligence criteria in four weighted blocks, from capability and evidence through data, control and commercial terms
The twelve due diligence criteria in four weighted blocks, from capability and evidence through data, control and commercial terms
Back to the TOC

Section icon: Block A: Capability and Evidence.

AI Vendor and Model Due Diligence

Block A: Capability and Evidence


Criterion 1, task fitness. Build a set of 60 to 100 real tasks from your own work, with answers the business already accepts. Write the rubric before the test, including what counts as a severe error. Run the tasks blinded and score the pass rate, the severe error rate and the time per task against your current process. A vendor benchmark is context. Your test is evidence. Ask which tasks the system fails, and score the answer: a vendor that names a failure mode has tested the product, and one that claims none has not.


Criterion 2, evaluation evidence. Ask for the model or system card for the exact version you will receive: its date, the evaluation sets, the rubric, the slices, the red-team summary and the known limitations. The model card convention dates from 2019 and is now common, but a card is only useful when it names a version and a date. Independent evaluations are useful context and never a substitute, because no public benchmark measures your invoice exceptions. Score 2 for marketing material, 4 for a card and an evaluation report, and 5 when the vendor can discuss the slices that fail.


Criterion 3, lifecycle and change policy. Models are retired, and minor versions change behaviour. Ask for the version calendar, the minimum retirement notice for the version in production, the notice for a material behaviour change, the migration support and whether you can pin a version. A 45-day retirement notice is common and too short for a system that needs an evaluation cycle. Ask for 90 days for retirement and 30 days for a material change, written into the contract. Score 2 when no calendar exists, whatever the sales team promises.


Build the task set with a shape, not a random sample. Forty routine tasks, fifteen hard ones and five adversarial ones, including an instruction hidden in a document and a request that should be refused. Include the tasks your current process fails, because those are where the difference shows. Double-score 20 percent of the set with a second reviewer and record the agreement; a rubric that two people read differently is not ready. Keep the set as a regression suite, because the same 60 tasks let you compare a new model version in an afternoon.


Version pinning is the control that makes the set reusable. Record the exact model identifier in the inventory, not the marketing name. When the provider announces a change, re-run the 60 tasks before you switch, and compare the severe error rate rather than the average. A new version that improves the average while doubling severe errors is a regression for your use case, and the average will hide it.

Back to the TOC

Section icon: Block B: Data and Security.

AI Vendor and Model Due Diligence

Block B: Data and Security


Criterion 4, training use. Read the clause for the tier you actually use, including the free tiers staff sign up for outside procurement. Ask four questions: do our inputs train your models, do our outputs, does the answer differ by tier, and can we turn it off in writing? A zero-retention endpoint is a configuration, not a promise; record the setting and who can change it. Test what you can. Place a unique phrase in a document, delete the document, and ask for confirmation that it is gone. You cannot prove a negative from outside, but you can prove the clause, the configuration and the deletion path.


Criterion 5, retention and deletion. Ask for default retention, which often covers abuse monitoring for a stated period, deletion on request, deletion on termination, backup behaviour and a written deletion certificate. Ask how a data subject request reaches the vendor and within what time. Test one document: delete it, require a dated confirmation, and record the elapsed days. A vendor that cannot produce a certificate has a deletion policy rather than a deletion process.


Criterion 6, residency and subprocessors. Inference, logging and backups can sit in three different places, and one endpoint can serve from three regions. Ask for the routing table rather than the region named in the brochure. Get the subprocessor list, the notice period for changes, the right to object, and the transfer mechanism: an adequacy decision or the standard contractual clauses. If the vendor cannot say where a request is processed, score 0. That is not a documentation gap; it is an answer.


Criterion 7, security assurance. A SOC 2 Type II report covers controls over a period for a defined system: check the period, the system boundary and which of the five trust services criteria are in scope. Security is always in scope; confidentiality and privacy are optional additions. An ISO/IEC 27001 certificate covers the management system in its stated scope. ISO/IEC 42001 covers an AI management system, which is a governance claim, not a claim about model quality. Ask for the latest penetration test summary, the vulnerability disclosure path, tenant isolation, key management, single sign-on and audit log export. Map the product to the OWASP Top 10 for LLM Applications 2025: prompt injection is LLM01 and excessive agency is LLM06. A certificate does not tell you whether the model is right for your task.


Two legal roles decide most of the data questions. You are usually the controller and the vendor the processor, which is why the data processing agreement under GDPR Article 28 is the document that matters. If the vendor decides why and how the data is used, it is a controller and the relationship changes. Ask which role the vendor claims for each service, because one vendor can be a processor for the API and a controller for a free consumer product. If you process personal data at scale or use a high-risk system, your data protection impact assessment is where the answers are recorded, and the vendor's evidence feeds it.


The office suite belongs in the same file. Many organisations buy an assistant inside a productivity platform, and that platform is already a processor for email and documents. The AI feature may carry its own terms, its own retention and its own region, so ask for the AI-specific annex rather than assuming the existing agreement covers it.


Classify the data before you classify the vendor. Write the sentence that says what may never enter the tool, such as payment card data, health records or unpublished financial results. Then ask how the vendor enforces the same boundary, and whether the setting is per user or per tenant. A tool that is acceptable for public marketing copy is not acceptable for a client contract, and one data classification rule prevents most of the incidents a questionnaire is meant to catch.

Back to the TOC

Section icon: Block C: Control and Operations.

AI Vendor and Model Due Diligence

Block C: Control and Operations


Criterion 8, agent identity and permissions. An agent needs an identity, and the identity needs a permission list. Ask for a table of every tool, the scope on each, whether the action reads or writes, and whether it is reversible. Check whether the vendor's agent uses a service account with broad access or delegated permissions scoped to the user. Ask how a token is revoked, how long it lives, where secrets are stored, and what access the vendor itself has to your tenant. Score 2 when the answer is "the agent can call our API" and no table exists.


Criterion 9, action limits and approval. Limits must be enforced by the system, not written in the prompt. Ask for the maximum consequence of one error: one email, one record, one payment up to a stated amount. Ask where the limit is enforced, what happens when it is crossed, who approves, and whether the action can be rolled back. Test one prohibited action in a sandbox and watch whether the system refuses or warns. A warning is not a control, and a prompt instruction is not a limit.


Approval design decides whether human oversight is real. Ask who sees the proposed action, what context they see, how long they have to decide, what they can change, and how the decision is recorded. An approval screen that shows only the model's summary invites rubber-stamping, and a queue that clears 400 items a day produces the same result. Set one value threshold for single approval and a second for four-eyes review. Test the queue with a deliberately wrong proposal and see whether the reviewer catches it.


Name the owner of the kill switch and test it on a schedule. The switch should stop new runs, revoke the agent's credentials and leave the logs readable. A kill switch that nobody has operated is a diagram. Record the last test date in the same file as the permission table, because the reviewer will ask for both.


Criterion 10, logging, incident notice and monitoring. Ask what is logged, including prompts, outputs, tool calls and approvals, for how long, whether you can export it, and whether you can read it without the vendor. Ask for the incident notice window in hours. "Promptly" is not a number, and your own regulatory clock may be 72 hours under GDPR Article 33. Ask for the last three post-incident reports. Check that you can see error rates, behaviour drift and cost anomalies without asking the vendor for a report.

Back to the TOC

Section icon: Block D: Commercial and Exit.

AI Vendor and Model Due Diligence

Block D: Commercial and Exit


Criterion 11, price mechanics and change notice. Ask for the pricing formula, not the price list: the unit (seat, token, run or outcome), minimum commitments, overage, cache pricing, true-up, currency and the notice period for a change. The November 2026 board carries a live example. Gemini 4 Carbon and Argon are $2 input and $10 output per million tokens through 30 November 2026, then $4 and $20 from 1 December. A task priced at $0.1290 becomes $0.2581 with no change to traffic, prompts or model. A 15-day notice does not protect a budget; 60 days does. Put the formula in the contract and re-price the top three workloads on the first working day of each quarter.


Criterion 12, exit, portability and liability. Ask what you can export: data, prompts, evaluation sets, logs and any fine-tuned model. Ask in what format, how fast, at what cost, and whether professional services are required. Ask about termination for convenience, transition assistance, and the liability cap against the exposure you are carrying. Check the IP indemnity for outputs if you publish them. The exit test is simple: export a sample of everything in one week without help. If you cannot, the exit is theoretical, and a theoretical exit is not a negotiation position.


Total cost over three years is the number the business case needs. Take a worked shape: 10 seats at an average of $30 a month is $300 a month, or $3,600 a year. A token pool of 6,000 tasks at $0.030 is $180 a month, or $2,160 a year. An agent at 8,000 runs and $0.156 a run is $1,248 a month, or $14,976 a year. Integration costs $15,000 once, and operations cost $6,000 a year. Three years total $95,208. That is the number to compare with the alternative, not the seat price on the quote. Stress it once: a 40 percent price rise at the first renewal adds $5,990 to the agent line across the three years, and the total becomes $101,198. A 60-day notice clause does not prevent the rise, but it gives you time to re-price, re-route or renegotiate before you sign the renewal.


Size commitments below the floor you expect, not at the forecast. A commitment is a bet that usage will not fall, and usage falls for good reasons: a workflow retires, a team reorganises, a cheaper model arrives. Sign for the volume you would keep in a bad quarter, and pay the list price for the upside. The same discipline applies to liability: a cap of 12 months of fees on a $27,000 annual line is $27,000, and it needs to be read against the exposure you assessed, which may be $250,000 for a single payment error. If the cap sits far below the exposure, the cap is the risk.

Back to the TOC

Section icon: Five Tests That Produce Evidence.

AI Vendor and Model Due Diligence

Five Tests That Produce Evidence


A questionnaire produces claims. Five tests produce artefacts. Each artefact has a date, a name and a result, and each one moves a score from 2 to 4 or 5.


  • Task test: 60 real tasks, blinded, rubric written first. Record pass rate, severe errors and time per task.

  • Deletion test: one document, written confirmation, elapsed days recorded.

  • Permission test: a sandbox run that attempts a prohibited action, with the refusal or the warning recorded.

  • Change test: the version calendar plus the last three change notices, compared with what actually changed in your logs.

  • Exit test: export data, prompts, evaluations and logs, and record the format, the time and the gaps.


The five due diligence tests that turn vendor claims into dated evidence artefacts, from the task test to the exit test
The five due diligence tests that turn vendor claims into dated evidence artefacts, from the task test to the exit test

Score 2 for a claim, 3 for a partial artefact, 4 for a complete artefact, and 5 for a complete artefact plus a test you ran yourself. A vendor that refuses a test is telling you something, and it is not a 5.


Keep the artefacts in one register: criterion, artefact name, version, date, owner and location. The register is what makes the next review cheap and an audit answerable. Retain it for the life of the contract plus the period in which a claim can be brought, because the question after an incident is not what the vendor said last month, but what the file said when the decision was taken.

Back to the TOC

Section icon: Warning Signs and False Comforts.

AI Vendor and Model Due Diligence

Warning Signs and False Comforts


Six warning signs recur in procurement files, and each one points to a clause.


Warning sign

What it usually means

What to do

Refuses to name subprocessors

The list changes faster than the contract

Require the list, 60 days notice and a right to object

"We never train on your data", with no clause

The term is policy, not contract

Ask for the clause for your tier, and the configuration

No deprecation calendar

The version you tested will disappear

Require 90 days retirement notice and a migration contact

No permission table for the agent

The pilot scope is not the production scope

Require the table and the limits before go-live

Incident notice "as soon as practicable"

There is no clock

Require hours, and a written report within days

Unlimited liability in the sales deck

The contract caps it at 12 months of fees

Read the cap against the exposure


False comforts matter as much as warning signs. The SOC 2 badge needs its scope and period read. The big brand does not make your account large. Benchmark leadership says nothing about your task. The enterprise tier does not remove the free tier that staff still use. A successful pilot tested happy paths, which are the paths that were never the risk. An ISO/IEC 42001 certificate says the vendor manages AI, not that the model is suitable for your invoice exceptions.


Read the refusals carefully. A vendor that will not provide a subprocessor list, a permission table or a deletion confirmation is not being difficult; it is telling you that the control does not exist in a form it can evidence. Treat a refusal as a score of 1, record it, and let the total carry the consequence. The same logic applies to answers that arrive only after a legal escalation: the clause you eventually get is the clause that should have been offered at the start.

Back to the TOC

Section icon: The Weighted Scorecard.

AI Vendor and Model Due Diligence

The Weighted Scorecard


Fix the weights first, define the anchors, score each criterion with one line of evidence and a date, and let two reviewers score independently.


Anchors: 0 for no answer, 1 for refused, 2 for a claim only, 3 for a partial artefact, 4 for a complete artefact, and 5 for a complete artefact plus your own test. Weighted points are the weight multiplied by the score and divided by 5, so the total runs to 100.


Block

Criteria

Weight

Capability and evidence

1, 2, 3

25

Data and security

4, 5, 6, 7

30

Control and operations

8, 9, 10

25

Commercial and exit

11, 12

20


The bands are 70 and above to proceed with the conditions recorded, 55 to 69 for conditional approval with a 90-day remediation plan written into the contract, and below 55 to stop. Any criterion at 0 or 1 stops the deal until it is fixed, whatever the total. A difference of 2 points or more between the two reviewers gets discussed against the evidence, not averaged away.


Calibrate the anchors against a vendor you already know. Score the incumbent first, because you have evidence for it, and use that row to align both reviewers on what a 3 and a 4 look like. Record dissent in the file with the evidence that caused it. Averaging two scores hides the disagreement, and the disagreement is usually the most informative line in the review. Re-score at every trigger event rather than carrying last year's total forward.

Back to the TOC

Section icon: A Worked Example: Two Vendors, One Workload.

AI Vendor and Model Due Diligence

A Worked Example: Two Vendors, One Workload


The workload is a support-triage agent. It reads a ticket, checks the account, decides a remedy and drafts a reply, and a human approves any refund. Two vendors are in the file: A, the model lab selling direct, and B, a platform that serves several models.


Criterion

Weight

A score

A points

B score

B points

1 Task fitness

10

5

10.0

4

8.0

2 Evaluation evidence

7

4

5.6

3

4.2

3 Lifecycle and change

8

2

3.2

4

6.4

4 Training use

9

4

7.2

5

9.0

5 Retention and deletion

7

3

4.2

4

5.6

6 Residency and subprocessors

6

2

2.4

4

4.8

7 Security assurance

8

5

8.0

4

6.4

8 Agent identity and permissions

8

3

4.8

4

6.4

9 Action limits and approval

9

2

3.6

4

7.2

10 Logging and incident notice

8

4

6.4

3

4.8

11 Price mechanics and notice

10

3

6.0

2

4.0

12 Exit and liability

10

2

4.0

4

8.0

Total

100


65.4


74.8


Vendor A wins the task test and the security review, and it loses four criteria at a score of 2: lifecycle, residency, action limits and exit. Vendor B is behind on task fitness and evaluation evidence, and ahead on data, control and exit. The decision is B, with two conditions: raise the price notice from 15 to 60 days, and the incident notice from 72 to 24 hours. Vendor A stays in the file as a second model for a bounded pilot, not as the production vendor, because its exit clause fails at the weight where exit matters most.


The lane arithmetic is worth checking by hand. A scores 18.8 in capability, 21.8 in data and security, 14.8 in control and 10.0 in commercial, for 65.4. B scores 18.6, 25.8, 18.4 and 12.0, for 74.8. Two criteria carry 10 points each, which is why a 2 on exit costs 4 points and a 2 on lifecycle costs 3.2. The scorecard does not pick the best model. It picks the best vendor for this workload, and it tells you what to negotiate.


The two profiles are typical. Vendor A is the best model with the weakest terms, and Vendor B is adequate capability with stronger terms. Neither profile is universally right. If the workload is a research assistant with no write access and no personal data, the capability gap may justify A despite its exit clause. If the workload can move money, the control and exit scores decide, because the consequence of failure is financial and model quality does not reduce it. The scorecard makes that trade explicit instead of leaving it to the loudest voice in the room.


The weighted scorecard for two vendors on one support-triage workload, with block subtotals of 65.4 and 74.8 and the two conditions attached to the decision
The weighted scorecard for two vendors on one support-triage workload, with block subtotals of 65.4 and 74.8 and the two conditions attached to the decision
Back to the TOC

Section icon: Contract Clauses That Carry the Score.

AI Vendor and Model Due Diligence

Contract Clauses That Carry the Score


If it is not in the contract, it is not a control. Map each low score to a clause before signature.


  • Training use: the data processing agreement required by GDPR Article 28, plus a specific no-training term for your tier and the configuration that enforces it.

  • Deletion: a deletion certificate with a stated period, and assistance with data subject requests.

  • Subprocessors: the list, 60 days notice, a right to object, and the transfer mechanism.

  • Security: a security schedule referencing the SOC 2 scope and the ISO certificate, refreshed annually, with a penetration test summary.

  • Lifecycle: 90 days retirement notice, 30 days material change notice, and a named migration contact.

  • Permissions and limits: the permission table and the action limits as an annex, with a test before go-live.

  • Incident: notice within 24 hours, and a written report within five working days.

  • Price: 60 days notice, the formula, and the right to terminate without penalty on a price change above a stated percentage.

  • Exit: 90 days transition assistance, export in a stated format, and no charge for a standard export.

  • Liability: a cap sized to the exposure, with carve-outs for confidentiality and IP.


A score of 5 on a criterion with no clause is a 3 at best once the contract is signed. The clause is the part that survives the renewal, the reorganisation and the person who approved the pilot.


Negotiate in order of consequence. The must-have clauses come first: data use, deletion, incident notice, subprocessor notice and the permission annex. Commercial terms come second, because a discount cannot compensate for a missing deletion right. Service levels come last, since credits are a refund rather than a control. Prepare a fallback for each must-have: if the vendor refuses audit rights, ask for the SOC 2 report and the penetration test summary annually; if it refuses a termination right on price change, ask for a longer notice and a cap on the increase. A fallback that protects the same risk is a good trade, and a fallback that only changes the wording is not.

Back to the TOC

Section icon: Ten Days, Five People.

AI Vendor and Model Due Diligence

Ten Days, Five People


Day 1: scope the workload, name the decision and fix the weights. Day 2: send the request list. Days 3 and 4: read the evidence. Day 5: run the task test. Day 6: run the deletion and permission tests. Day 7: score independently. Day 8: negotiate the conditions. Day 9: write the one-page memo. Day 10: put the clauses in the contract.


Five people carry it. The business owner is accountable for the decision. Security owns criterion 7 and the permission test. Legal and the data protection officer own criteria 4 to 6 and the clauses. Finance owns criterion 11. Procurement runs the process and keeps the file. The memo is one page: the decision, the score, the conditions, the owner of each condition, and the review date. A file that cannot fit on one page has not reached a decision.


The request list is twelve artefacts, one per criterion: the task-test access, the model or system card, the version calendar, the training and retention clauses, the subprocessor list and routing table, the security reports, the permission table, the action limits, the logging sample and incident reports, the pricing formula and the exit plan. Send it as one table with a due date and a named recipient. A vendor that answers nine of twelve in a week is telling you its file is in order. A vendor that needs three weeks for the first four is telling you something else.

Back to the TOC

Section icon: Six Ways Vendor Due Diligence Fails.

AI Vendor and Model Due Diligence

Six Ways Vendor Due Diligence Fails


  • Scoring the vendor instead of the workload. The same score then covers a chatbot and a payment agent, and it means nothing for either.

  • Changing the weights after the scores are known. The scorecard becomes a justification rather than a test.

  • Replacing evidence with a questionnaire. Two hundred answers, no artefact and no test is a file, not due diligence.

  • A pilot that runs only happy paths. The failures are the part you are buying protection against.

  • No owner for the conditions after signature. The clause exists, the control does not, and the first review finds it.

  • Stopping at go-live. Models change, prices change and subprocessors change, so the review is annual at minimum and quarterly for a production agent.


Each failure is cheap to prevent at the start and expensive to discover at renewal. The scorecard, the five tests and the clause list are the cheap version.

Back to the TOC

Section icon: Feynman Summary: Explain It Like You Are 12.

AI Vendor and Model Due Diligence

Feynman Summary: Explain It Like You Are 12


Before you pay a company to do a job with a computer, check six things. Can it do the job on your own examples? Can you see the evidence instead of the advertisement? What happens to your information, and who else can see it? What is it allowed to do on its own, and what does it have to ask about first? What does it cost, and how fast can the price change? How do you leave and take your things with you?


Then write the answers into the contract. A promise that is not written down is not a promise you can enforce. Twelve questions, five tests and one page are enough to make the decision, provided the page names who owns each condition and when you will look again.

Back to the TOC

Section icon: Mindmap: The Complete Picture.

AI Vendor and Model Due Diligence

Mindmap: The Complete Picture


The mindmap places vendor due diligence at the centre and four blocks around it: capability and evidence, data and security, control and operations, commercial and exit. Each block carries its three or four criteria with the weight, the five tests sit on a second ring, and the three decision bands close the picture at the outer edge.


The complete vendor due diligence mindmap, from the four criteria blocks and twelve weights to the five tests and the three decision bands
The complete vendor due diligence mindmap, from the four criteria blocks and twelve weights to the five tests and the three decision bands


UNOP isochrone

UNOP Sound (University 365 Neuroscience Oriented Pedagogy)

Take five minutes to consolidate your memory. Play the isochronous tone track (10Hz alpha frequency) with your eyes closed. Alpha-frequency tones after a learning session support consolidation, helping move what you just learned from short-term to long-term memory.

[Audio player: UNOP Post-Lecture Isochrone (10Hz, 5 minutes)]

Back to the TOC

Section icon: Practical Exercise: Score One Vendor in Ninety Minutes.

AI Vendor and Model Due Diligence

Practical Exercise: Score One Vendor in Ninety Minutes


Take one vendor currently in your procurement file and run the scorecard before your next meeting.


  • Write the decision in one sentence: which workload, which vendor, and what you are deciding this month.

  • Take the twelve weights as given, or change at most two, and write one line on why.

  • Gather the evidence you already have. Score each criterion with one line and a date. Use 0 when you have nothing, not 2 for a friendly answer.

  • Multiply each weight by the score and divide by 5. Add the twelve numbers. Identify the three lowest criteria.

  • For each of the three, write the artefact you need and the clause you will ask for.

  • Write the one-page memo: decision, total, conditions, owners and review date.


Deliverable: one page, twelve rows and three conditions. If the total is below 55, or any criterion is 0 or 1, the memo says no and gives the reason. That memo is the useful output, even when the answer is no.

Back to the TOC

Section icon: Glossary.

AI Vendor and Model Due Diligence

Glossary


Term

Definition

Due diligence

The evidence-based assessment of a vendor, model and configuration against a named workload, completed before the contract is signed

Data processing agreement

The written processor contract required by GDPR Article 28, setting the purpose, duration, data types, obligations and deletion terms

Subprocessor

A third party the vendor uses to deliver the service, such as a cloud host or a model provider, whose use must be disclosed and controlled

SOC 2 Type II

An AICPA attestation report on the design and operating effectiveness of controls over a stated period, for a defined system and selected trust services criteria

ISO/IEC 42001

The international standard for an AI management system, published in December 2023 and certifiable within a stated scope

Model card

A short document describing a model's intended use, training data, evaluation results and limitations, a convention introduced in 2019

Deprecation notice

The advance warning a vendor gives before retiring a model version, which must be long enough to run an evaluation and migrate

Excessive agency

A failure mode where a system is granted more functionality, permissions or autonomy than the task requires; LLM06 in the OWASP Top 10 for LLM Applications 2025

Zero retention

A configuration in which the vendor does not store prompts or outputs beyond the request, used to limit data exposure

Exit assistance

The contractual commitment to help a customer export data, prompts, evaluations and logs and to migrate within a stated period

Liability cap

The maximum amount a vendor will pay under the contract, usually expressed as a multiple of fees paid, which must be compared with the assessed exposure

Evidence artefact

A dated, attributable document or test result that supports a score, such as a report, a clause, a log extract or a completed test

Back to the TOC

Section icon: Quiz: TEST YOUR UNDERSTANDING.

AI Vendor and Model Due Diligence

Quiz: TEST YOUR UNDERSTANDING


1. Which event stops a vendor decision regardless of the weighted total?


A) A total below 70


B) A price rise above 10 percent


C) A score of 0 or 1 on any criterion


D) A score of 2 on the exit criterion


2. A vendor scores 2 on criterion 3, which carries a weight of 8. How many weighted points does that criterion contribute?


A) 1.6


B) 3.2


C) 5.0


D) 8.0


3. What turns a due diligence score into a control that survives the renewal?


A) The vendor questionnaire


B) The sales deck


C) A successful pilot


D) The contract clause


4. What does a SOC 2 Type II report cover?


A) Controls over a period for the system in scope


B) The model's accuracy on your task


C) The vendor's financial statements


D) The training data and its licences


5. Gemini 4 Carbon is $2 input and $10 output per million tokens through 30 November 2026, then $4 and $20. A task costs $0.1290 in November. What does the same task cost after 1 December?


A) $0.1290


B) $0.1935


C) $0.2581


D) $0.5162


Answers: 1-C, 2-B, 3-D, 4-A, 5-C

Back to the TOC

Related Resources


U365 INSIDE Publications



External Resources



Related U365 Lectures (Coming Soon)


  • Agentic Commerce: When AI Buys for Your Customers (Business AI Series)

  • Board-Level AI Governance (Business AI Series)

  • AI and Liability: Who Answers When an Agent Acts (Cross-institute lecture)

Back to the TOC

Section icon: U.Copilot for This Lecture.

AI Vendor and Model Due Diligence

U.Copilot for This Lecture


You are my vendor due diligence analyst. I am deciding whether to contract an AI vendor for one named workload, and I need a scored file I can defend to security, legal and finance. Workload: [what the system does, who uses it, what it can read and write] Vendors in the file: [names, models, versions offered] Evidence I already hold: [reports, clauses, test results, with dates] Regulatory context: [sector, personal data, EU or other jurisdictions] Do six things: 1. Restate the decision and the workload in one sentence each, and flag any scope that mixes two workloads. 2. Score each vendor against the twelve criteria on a 0 to 5 scale, using the anchors: 0 no answer, 1 refused, 2 claim only, 3 partial artefact, 4 complete artefact, 5 complete artefact plus my own test. State the evidence and its date for every score. 3. Compute the weighted points as weight times score divided by 5, give the four block subtotals and the total, and apply the rule that any 0 or 1 stops the deal. 4. Name the three lowest criteria for the leading vendor, the artefact I still need for each, and the contract clause that would carry it. 5. Draft the one-page decision memo: decision, total, conditions, owner of each condition, review date. 6. List the five tests I should run before signature, with the artefact each test produces. Ask me for the weights if I have not fixed them, and never invent a price, a date or a certification I have not given you.

Back to the TOC

Section icon: Next Steps.

AI Vendor and Model Due Diligence

Next Steps


  • Pick one live vendor and write the decision in one sentence: workload, vendor and what you are deciding this month.

  • Send the twelve-artefact request list as one table, with a due date and a named recipient at the vendor.

  • Run the task test on 60 of your own tasks, with the rubric written before the first output is scored.

  • Run the deletion and permission tests in a sandbox, and record the date and the result of each.

  • Write the one-page memo with the total, the conditions, the owner of each condition and the review date, then put the must-have clauses into the contract before signature.

Back to the TOC

Section icon: IMPORTANT NOTICE.

AI Vendor and Model Due Diligence

IMPORTANT NOTICE


Copyright University 365, Inc. All rights reserved. This lecture is provided for educational purposes and does not constitute legal, financial or professional advice. For enrollment, visit university-365.com/tuition. For permissions or inquiries, contact uda@university-365.com.



Published by the Department of Academics, University 365.

Lecture delivered by the University 365 Institute of Business (UIB).

Denise Cromwell, Dean of Business, UIB

Signed for the academic year 2026.

Comments

Rated 0 out of 5 stars.
No ratings yet

Add a rating
Image by Erik  Lucatero

Become Superhuman

Master AI to stay irreplaceable in every field.

 

 

 

​

​

Apply for Admission Today.
Select Your Initial Access Level.


Become a DISCOVERY, INSIDER, or SUPERHUMAN Fellow.

Image by Milad Fakurian

Master Your Life with a Digital Second Brain

Turn overwhelm into clarity with LIPS + CARE
U365’s unique framework to organize your goals, projects, and knowledge into a superhuman system for success

bottom of page