Debugging an AI Underwriting Artifact: Missing Prompt and Comps Files Behind the Errors

When an AI underwriting tool starts throwing errors, the instinct is to blame the scoring logic and start rewriting it. At one private lender that would have been the wrong move. The errors came from two files that were simply not in the tool.

The client is an executive at a private real estate lender, using the Claude underwriting build we set up for the team. This is how a review call turned a vague failure into a specific cause. If you run lending for investors, our page for mortgage loan officers and private lenders covers the wider work.

The situation

The lender's underwriting Artifact in Claude was producing errors. We held a review call on the build and the weekly tasks that feed it.

Before touching any logic, we looked at what the Artifact actually contained compared with what the build requires.

What was broken

The root cause turned out to be missing inputs. Two things had never made it into the lender's Artifact:

• The master deal analysis prompt, the instruction set that tells Claude how to run a deal from start to finish.
• The weekly comps knowledge base files, which hold the current market comparables the analysis leans on.

What we built

The repair had four pieces.

• An updated master prompt. A revised version with the appraisal logic removed goes to the lender by email.
• One central home for the files. A single shared cloud folder is set up to hold every skill file, so there is one place to look and one place to update.
• A joint validation session. We set a working session with the lender to go through the fixed Artifact together instead of handing it over and waiting for the next error.
• A request for live deals. The executive was asked to send real deals before the session, not made-up examples, so the team could judge whether the outputs were accurate and realistic.

What changed

No result was reported yet, because the validation session had not happened when the summary went out. What changed was our understanding of the failure. It went from unexplained errors to two identified missing components, with a fix underway and a validation session booked to check it.

A second change looks ahead. The lender planned to drop in updated loan program guidelines in the following weeks, so unified guideline documents for its short-term loan and DSCR rental loan programs were prepared in advance for ingestion. When the guidelines change, the update path is already defined.

Debugging an AI tool: where to look first

If your own AI tool starts failing, check what is loaded before you touch the logic. List the files and prompts the tool is supposed to have, then compare that list with what it actually holds. A missing file produces errors that look like logic problems.

Keep every file in one shared folder, and treat that folder as the source. And when you test, use live deals. Fabricated examples are easy for any tool to pass and tell you nothing about how it will behave on real work.

Frequently asked questions

Why would an AI underwriting tool throw errors?

One common cause is missing components. In this case the master deal analysis prompt and the weekly comps knowledge base files were absent from the lender's Artifact, so the tool lacked instructions and data it needed. Check that everything is loaded before you rewrite any scoring logic.

Why keep all skill files in one shared folder?

It gives you a single source to update and a simple way to check that nothing is missing. We set up one centralized folder for this lender so that a fix to a file can reach the Artifact the same way every time.

Why test an underwriting tool on live deals instead of made-up ones?

Real deals expose problems that invented examples hide. We asked the lender for live deals so the team could judge whether the tool's outputs were accurate and realistic, which is the standard the tool has to meet in actual use.

How do you prepare for updated loan guidelines?

Get the guidelines into unified documents ahead of time. Here, the guideline documents for the lender's short-term loan and DSCR rental programs were prepared so they could be ingested when the update arrived, without rebuilding the tool.

If your AI tools fail and nobody can say why, book a 30-minute AI follow-up and reporting audit. We trace what each tool is loading and where the gaps are.