The Patterns of Errors From AI Contracting Systems

I saw a lot of impressive systems, and I remain optimistic about these tools. I think they’re going to make our work easier and free us to focus on the most interesting tasks. I also think they’ll eventually make legal services less expensive, solving some of the Western world’s justice gap.

However, these tools have serious problems. And it’s crucial that we understand them. In particular, we need to understand the type of errors they make. A great example came up in my testing.

Reproducing Typical Errors about Reciprocal Indemnities

I uploaded a SaaS agreement on vendor paper and asked AI tools to review it on behalf of the customer. Just about every system red-flagged that the indemnity is not reciprocal or mutual. The customer indemnifies the vendor against third party claims but not vice versa. The AI systems recommend the vice versa: the same indemnity protecting the customer. And often the AI drafts reciprocal language (sometimes badly, sometimes well). But there’s a problem.

Why would you ever want a reciprocal or “mutual” indemnity?

We don’t need reciprocal indemnities. We need indemnities addressing the key risks of third party lawsuits resulting from the deal. So if you’re the customer, you don’t need an indemnity against claims by your end-users just because the vendor has one. If you’re not worried about those claims, the indemnity doesn’t work much? Maybe you’re worried instead about third party IP or data claims. Or maybe you’re worried that the vendor’s subcontractors will sue you for compensation that the vendor didn’t pay. If so, you want indemnities against those claims, not a reciprocal indemnity about end-user claims.

But that’s not what the AIs recommend.

On the other hand, maybe you’re not worried about third party suits at all. If so, why waste your powder on indemnities at all? Just to be reciprocal? Instead, how about swapping the vendor’s requested indemnities for something of real value, like a lower price or higher limit of liability?

Where AI Does Better

AI does better with more straightforward issue-spotting and redlining. Your AI has good odds of catching and fixing problematic termination for convenience terms, for instance – or an overly low or high limit of liability, or one party’s unilateral right to amend the contract.

The systems also catch and recommend fixes for “non-reciprocal” language where balance between the two parties does make sense. If the limit of liability protects the other party but not you, you probably want a reciprocal limit. And you can count on a lot of these systems to catch and fix that issue.

Subtle Issues

AI doesn’t do as well with subtle issues, which unfortunately litter most contracts. I chose the reciprocal indemnity for that reason – because it’s subtle: easily missed – but also because of its stark outcome: low value terms, important terms left out, opportunities lost, and wasted negotiation capital. But the indemnity issue isn’t unique. Contract-drafting AIs rack up a lot of these more subtle errors.

The AIs get these issues wrong because a lot of contract-drafters get them wrong, including experienced lawyers. So their training data suggests we need reciprocal indemnities.

Even curated systems – AI with human “editing” – can get this wrong. If a lot of lawyers get it wrong, it’s reasonable odds that the AI company’s human curators did wrong too.

Unfortunately, subtle errors don’t necessarily have subtle costs. Choosing a reciprocal indemnity instead of one you need could cost orders of magnitude more than the deal’s value.

What’s the solution?

First, pick your AI vendor with care.

Learn all you can about how it trained its AI and about the role of human experts.
Confirm quality.
We talk more about that in our new course on the topic (described bel0w).

Second, learn the stuff yourself. So far, AI has not offered a substitute for your expertise.

Conflict of Interest

This article needs a conflict of interest statement because I have several biases – three in particular:

As you’ve no doubt realized, I’m encouraging you to attend our course on the topic (described below), and raising this issue helps.
Through Tech Contracts Academy, I provide training, and that’s just what this article implies you need.
I’m considering offering products and services – e.g., playbooks and related consulting – meant to solve this problem, and this article helps seed that market.

So take my views with a grain of salt. But please think it through.

Now Available in our On-Demand Library: Best AI and Other Software for Contract Professionals, c. 1 hour recording, with 100 days’ access. [Our live webinar was on May 22, 2025.] This course benefits everyone engaged in technology contracting. But as a bonus, it offers California CLE credit in the Technology in the Practice of Law subfield. (See here for more CLE information.)

Article updated July 28, 2025.

THIS ARTICLE IS NOT LEGAL ADVICE. IT IS GENERAL IN NATURE AND MAY NOT BE SUFFICIENT FOR A SPECIFIC CONTRACTUAL, TECHNOLOGICAL, OR LEGAL PROBLEM OR DISPUTE, AND IT IS NOT PROVIDED WITH ANY GUARANTEE, WARRANTY, OR REPRESENTATION. LEGAL SITUATIONS VARY, SO BEFORE ACTING ON ANY SUGGESTION IN THIS ARTICLE, YOU SHOULD CONSULT A QUALIFIED ATTORNEY REGARDING YOUR SPECIFIC MATTER OR NEED.