Before You Buy That AI Tool: How Credit Unions Can Vet an AI Vendor

Almost every demo you sit through this fall will include the word "AI." The trouble is that the word covers very different things, with very different risks. A tool that sorts transactions by rule is not the same as a tool that writes emails to members, and neither is the same as a tool that can move money on its own. If you vet them all the same way, you'll either waste time on the safe ones or miss the dangerous ones.

This guide gives you three things: a plain way to tell the types apart, a set of questions to ask, and a decision tree for deciding how deep to dig.

Why vendor vetting is your job, not just the examiner's

There's no NCUA rule written specifically for AI. The agency's long-standing guidance on third parties, a 2007 letter to credit unions, says the review you do should reflect how critical the service is and what the vendor's expertise and risk controls look like. On September 11, the NCUA joined the FDIC, the Federal Reserve, and the OCC in proposing updated third-party risk guidance, so the expectations are moving, though the proposal is non-binding and still open for comment.

There's a second reason. A 2025 report from the Government Accountability Office noted that the NCUA lacks the authority to examine technology service providers the way other regulators can. In practice, much of the checking falls to you.

Step 1: Work out which type of AI you're looking at

Ask the vendor to describe, in one sentence, what the tool does and how it decides. Then place it in one of four groups.

1. Rules and automation. The tool follows if-then logic someone wrote down. It's often called AI in marketing, and it's usually the lowest risk because you can read the rules.
2. Predictive models (machine learning). The tool learns patterns from past data to score or rank things, such as a likelihood of default, fraud, or leaving the credit union. The risk is in the data it learned from and whether it treats groups of members fairly.
3. Generative AI. The tool writes text, summarizes documents, or answers questions in plain language. The risk is that it can sound confident and be wrong, and that member information may be sent to a third party to produce the answer.
4. Agentic AI. The tool takes steps on its own, such as opening a case, sending a message, or changing a record. The risk is that mistakes happen at speed, without a person in the loop.

Federal banking regulators treat the last two as a special case. In April, the new model risk guidance that replaced SR 11-7 (OCC Bulletin 2026-13) set generative and agentic AI outside its scope, saying they are novel and evolving. That means those categories have the least official guidance and deserve the most care from you.

A useful tell: ask "what is the worst thing this tool could do if it were wrong, and who would find out first?" A good answer is specific. A vague one tells you something too.

Step 2: Ask these questions

Use the full set for anything in groups 2 to 4. For group 1, the first two sections are usually enough.

What it is and what it touches

• Which part of your product uses AI, and which part is ordinary software?
• Is it a model you built, or are you using a third party's model underneath? Which one, and who else sees our data?
• What member data does it read, and does any of it leave our environment?
• Is our data used to train or improve your models, for us or for anyone else?

How well it works

• Is this feature in production at credit unions like ours, or is it on the roadmap?
• Where does that accuracy or time-savings figure come from, how large was the sample, and did anyone independent check it?
• How do you test for errors, bias, and drift over time, and can we see the results?
• What are the known limits, and what should it never be used for?

Control and accountability

• Where does a person review the output before it reaches a member, a filing, or a decision?
• Can we explain to a member, or an examiner, why the tool produced a given result?
• Can we turn it off, and what happens to our work if we do?
• Who is responsible if it makes a mistake, and what does the contract say about that?

Security and continuity

• What independent assurance reports can you share, such as a SOC 2 report, and what do they cover?
• Who are your subcontractors, and how do you oversee them?
• How and when will you tell us about an incident or a material change to the model?
• What happens to our data when the contract ends?

Step 3: Use the decision tree

This tree tells you how deep to go. Start at the top and follow the first answer that fits.

In plain text, the same logic reads like this:

1. No member data and no effect on member decisions: light review. Confirm the security report and contract terms.
2. Shapes credit, pricing, eligibility, or fraud holds: high scrutiny. Require explainability, bias test results, a named human reviewer, and counsel review of lending use.
3. Acts on its own (agentic): high scrutiny. Require limits on actions, a kill switch, full action logs, and human sign-off for money or records.
4. Generates content (generative): medium-high scrutiny. Require a data-flow map, no training on your data, and human review before content reaches members.
5. Scores or predicts (machine learning): medium scrutiny. Require data lineage, performance and drift reporting, and an owner who reviews results.
6. Follows fixed rules: light-medium scrutiny. Confirm the rules are documented and changes are logged.

Two overrides apply to every branch. If the vendor is critical to daily operations, move up one level of scrutiny. If the vendor can't answer the questions in plain language, treat that as a finding, not a delay.

Step 4: Write it down

Whatever you decide, keep a one-page record for each tool: what type of AI it is, what data it sees, who owns it, which questions were answered and how, and when you'll review it again. The point isn't paperwork. It's that when a board member, an auditor, or an examiner asks why you chose this tool, you can answer calmly.

A note on red flags

Be cautious when a vendor offers a big number with no source, won't say which third-party model sits underneath, says the AI "learns from all our clients" without explaining how your data is protected, or treats a request for a human review step as an inconvenience. None of these means walk away. Each means ask a follow-up question before you sign.

This is general information, not legal advice. Your counsel and compliance team should confirm what applies to your credit union.

Sources

• NCUA Letter to Credit Unions 07-CU-13, Evaluating Third Party Relationships (NCUA)
• Agencies seek comment on proposed third-party risk management guidance (Federal Reserve)
• Artificial Intelligence: Use and Oversight in Financial Services, GAO-25-107197 (GAO)
• Model Risk Management: Revised Guidance, OCC Bulletin 2026-13 (OCC)
• AI Risk Management Framework (NIST)