Impressive Finance Report? Be Careful! Your AI Tools Might Fail Next Month.

Full transcript

Lightly edited for readability. The audio is the original.

0:00AI in finance: 11 minutes versus 2 weeks

A few weeks back, I handed an AI a real piece of my finance work. Not a demo version, the actual monthly job, with all the mess still sitting in the data. It took about 11 minutes, and it came back good enough that I sat there thinking I just got my whole afternoon back.

And then I spent the next 2 weeks trying to get it to do that same job again, the same way, without me sitting there watching every step. And to be honest, I'm still not completely done. 11 minutes to 2 weeks. And that's the most useful thing I've learned about AI this year.

The problem is you almost never hear anyone talk about it, because the demo, even my own, only shows you the 11 minutes. So today we're going to walk through the three places those 2 weeks go. By bringing it out into the open, you can budget for it properly instead of promising your CFO a date you can't hit.

Welcome to Excel at AI in Finance, the show for finance pros who want to use AI to get home sooner and be the one their company can't afford to lose. I'm Mike, I run F9 Finance, and I'm a senior finance leader working in the trenches. All right, let's get into it.

1:15The month-end reconciliation I handed to an AI

So let me make this real with the job I mentioned up top. I've changed a couple of details, but the timeline is exactly what happened. Back in March, I went after our monthly reconciliation between the billing system and the general ledger.

It's about 4,000 lines a month. It should take an analyst a full day and a half, plus a chunk of my Thursday reviewing whatever came out the other side. I didn't ease into it either. I pulled March out of both systems, described the matching the way I'd describe it to a new analyst, told it what to flag, and let it run.

11 minutes later, it had cleared about 3,400 of the 4,100 lines. It handed me back 700 exceptions, already grouped by reason, with a note on each group. What really got me was the 9 lines it flagged that matched perfectly on the amount, but sat 40 days apart on the dates. That's exactly the thing a tired analyst scrolls right past at 6 in the evening.

I'll be honest about how that felt, because I think most of you have had this exact moment. It felt like the job was finished. I was already thinking about what I'd do with the rest of the day.

So I started listing what was left, and my list had two things on it: point it at April, and show an analyst how to run it. That list was completely wrong. It missed the point, and I had no idea.

2:30What was really left: exceptions, drift, and plumbing

Here's what was really left. The first was those 700 exceptions, and they were never one problem. There were partial payments split across two months. There were credit memos applied against a different invoice than the one they were issued for. There was a customer whose billing record carries a legal entity name that doesn't exist anywhere in our ledger.

Then there were the two non-dollar entities. Billing converts at the transaction date rate, and the ledger converts at month end. So every line looks like a mismatch, when none of them actually are. Every one of those needed a decision from me, and then a rule that would still be right in November.

The second one showed up in April. Same prompts, same two extracts, but the exception list came back grouped 6 ways instead of 9, because it had folded timing differences and partial payments into one bucket. On paper, the exception count dropped about 30% month over month.

If somebody had shown me that in a meeting, I'd have assumed the team had been cleaning things up. The team had cleaned up nothing at all. The line had just moved.

The third one is the boring one, but it ate the most calendar by a mile. Both extracts have to land in the same place on the same day, and in April the billing export added a column, and the whole thing fell over. Nobody told anybody it had broken. I found out because I went looking for it.

So that's the 2 weeks. 2 weeks of evenings and one Saturday I'd rather have spent doing almost anything else. It's been running every month since May, and an analyst runs it now instead of me.

There's one piece I still do by hand, which is the FX exceptions on those two non-dollar entities, and I left it that way on purpose. The rate call is a judgment call. It's about 15 lines a month, and I'd rather own it than automate it badly.

Now, here's why I'm telling you the long version instead of the 11-minute one, which is obviously the better story. The 11-minute version is the one that gets posted. It's the screenshot, the LinkedIn post, the slide, and the vendor deck. All of it is true, and all of it stops right at the point where the real work starts.

And if that's the only version you ever see, sooner or later you'll sit in a meeting and put a date on something you don't understand yet. I've done it, and it's a bad afternoon.

11 minutes to 2 weeks on one job. And 2 weeks is the good case, because that was my own data, in systems I control, on a process I've run for years. The builds that reach into a system I don't own take months. And it's the same curve every time. So let's back up and look at the three places all that time goes.

5:51Why AI hits 80% on finance work so fast now

Here's what changed, and it changed quickly. Two years ago, the hard part was getting an AI to do anything useful with finance work at all. You fought the tool. You rewrote the prompt 9 times, and you got back something 60% right and structurally strange. And you told yourself that was still faster than doing it by hand.

Well, that's over. On a lot of finance tasks, now you can hand them out of your data and a decent description of the job. You'll get something roughly right in the time it takes to make a coffee. 80% on the first try. No fight. And that 80% is real, on your data, with your mess on it, which is exactly where it gets you.

That first run feels like the finish line, when it's a lot closer to the starting gun than you'd like to admit. What you've proved in those 11 minutes is that the model can do the task. What you haven't touched is whether the work can be produced again next month, the same way, by a process instead of by you sitting at the keyboard. And in finance, that second thing, the exceptions, has always been the actual job.

6:53Reason 1: finance runs on exceptions

So, three places the rest of the time goes. The first one is that 80% is the average case, but finance runs on exceptions.

Think about why that first pass works as well as it does. Most of your rows look like each other. Same shape, same pattern, nothing unusual. In most finance data sets, that's most of the rows, which is why the number keeps landing around 80%.

The trap is that the leftovers aren't more of the same task. The 80% was one problem you solved once. And the last 20% is 40 small problems that don't share a solution. You saw it in my reconciliation, where 3,400 cleared and 700 didn't.

And it happens on any data set. Point one at a year of vendor spend, and you'll get 500 of 600 categorized in about 4 minutes. It's the 100 that are left that change the math on you, because they could be 100 different reasons. And fixing one doesn't touch the next one.

Take three of them. You've got the same company sitting under 4 different legal names after an acquisition. You've got a credit memo sitting there as a negative and getting binned as a refund. And you've got the cost center that was restructured in March, so the first quarter maps one way and the rest of the year maps another.

Every one of those three needs a decision from somebody who understands the business. Then it needs a rule that's still right in November, when a slightly different version of the same thing turns up. That's the part that eats the calendar.

And the cost isn't only one time. Every one of those 100 is a place where the number can be wrong and still look completely fine. A misclassified vendor doesn't throw an error. It quietly moves 200 grand from one line to another. The report still reconciles, and your commentary explains a movement that isn't real.

And the exceptions are always the interesting rows: the acquisition, the restructure, the one-off, the intercompany. In almost every data set I've worked with, the 20% the model can't place is where a disproportionate share of the dollars and the risk are sitting. So it handled the part that was never going to hurt you and handed back the part your CFO asks about.

So the fix here is the cheapest 15 minutes you'll ever spend. Before you get excited about that first pass, go and pull the 20 ugliest records in your data. Run just those, the ones you'd hand to your most senior person. That's your real timeline, because a clean sample only ever tells you your best case.

9:18Reason 2: the same prompt won't give the same answer twice

The second one is the big one, and it's that the same input won't give you the same output twice. This is the one that catches finance people out, and it comes down to instinct more than knowledge.

Every tool you've ever used in this job is deterministic. An Excel formula gives you the same answer today, next Tuesday, and in 4 years when somebody opens the file to argue with you about it. A Power Query refresh applies the same steps every time. That reliability is so complete that you've never once had to think about it, the same way you don't check whether gravity is still working.

But a language model doesn't behave like that, and that's by design. Run the same prompt against the same file on Tuesday and again on Thursday, and you'll very often get two answers that are both defensible but not identical. Slightly different grouping, a materiality cutoff it picked for itself, and picked differently this time.

In most jobs, that's a rounding error, but in finance, it's a problem, because our work is comparative. This month against last month, against actuals, against plan, this quarter against the same quarter last year. And some of our executives have the weird intuition of knowing exactly when a number changed, even if they haven't seen it for 3 months.

These comparisons are our product. So if the method quietly moved underneath you, the comparison is broken. And the part that should worry you is that nothing looks wrong. There's no error message and no red cell. Just a number that no longer means what the last one meant, sitting in a deck right next to the number it's compared against.

My 6 buckets instead of 9 was the mild version of that. The expensive version is the variance commentary. Your June run is excellent. It calls out the 5 things that mattered, and your CFO says something nice about it in the meeting.

Then the July run uses the same prompt and the same file structure, but it applies a materiality threshold it invented for itself, a little higher than last month's. So two items that were called out in June are silent in July. They didn't go away. They fell under a line that moved without anybody moving it.

Nothing failed, and the commentary reads beautifully. But somebody is going to ask why headcount was a driver last month and isn't one this month, and you'll be holding a deck you can't explain.

I always get asked whether turning the temperature down fixes this. It helps at the margin, but it doesn't reach the part that hurts you, because the damage lives in the decisions rather than the wording. Where the cutoff sat, how things got grouped. Those are judgment calls, and a judgment call made fresh every month is a judgment call that moves.

There's a cheap test for this that takes about 20 minutes and saves you months. Before you trust any of it, run the same job 3 times on the same file on 3 different days, and put the 3 outputs side by side. If they match, you found something you can build on. If they don't, you've just learned which decisions you need to take away from the model. And you learned it at your own desk instead of in a board meeting.

That's why getting from a demo to production gets measured in months. Making a model produce good work is fast now. Making it produce the same work is a design problem, and a better prompt won't solve it.

So the fix is that anything you can pin down outside the model, you pin down outside the model. The materiality threshold becomes a number in a settings file. The groupings become a list. The account map becomes a table. And the category names become fixed strings, spelled one way.

The model should be writing the words. It shouldn't be picking the thresholds. Every decision you move out of the model is one that can't drift on you in month 4. And month 4 is somehow always when somebody's watching.

12:40Reason 3: plumbing and proof

The third one is that the last 20% is plumbing and proof, and neither of those turns up in anybody's plan. This is the time that has nothing to do with AI, which is exactly why it gets missed.

Ask yourself these about anything you've built. Where does the input come from every month without you pasting it in? What happens when the file lands late, or somebody renames a column the way my billing export did in April? Who gets told when it breaks? And 6 months from now, when somebody asks how that number was produced, can you show them?

All of that is the difference between a thing that ran once and a thing that runs, and none of it involves the model. Count it out. Your 2-minute demo needs exactly one thing to work, which is you at the keyboard with the file open. One dependency, and it's sitting in the chair.

But the production version needs a lot more than that. An input that arrives on a schedule, and something sensible that happens when it isn't. An owner who gets told. An output that lands somewhere permanent, and a record of what ran and against which version of the rules. That's 6 things, and not one of them is the AI.

That list is your 2 weeks, and this is old news, really. Anybody who's built a Power Query chain that had to survive January knows this list by heart. AI didn't remove a minute of it. It just made the first 80% fast enough to hide the rest. So now everybody's timeline is built on the only part they could see.

Then there's the proof half, which matters more in our job than almost anywhere else. If you can't explain how the number was made, you can't defend it. And a number you can't defend doesn't go in the pack, no matter how good it is.

I'd rather hand my CFO a slower number I can walk them through line by line. That beats a better number I have to describe as "the AI did it."

So the fix is that you write down what happens when it breaks before you write down what happens when it works. Open a page and put the failure cases on it first. If that page is blank, what you've got is a demo, and you should call it a demo when somebody asks you for a date.

14:40What the last 20% proves

Getting to 80% proves the model can do the task. The last 20% is where you prove the work can be produced again next month, the same way, by a process instead of by you. That second part is what finance has always paid for, and AI just made the first part fast enough that it feels like the whole job.

That's why the curve bends the way it does. The first 80% is one problem you solved once, with you watching it. The last 20% is every exception, every bit of drift, and every piece of plumbing arriving at once.

And it's worth sitting with for a second, because the exceptions and the drift and the plumbing are exactly the work your team keeps pushing into next quarter, which is why the person who's willing to do it is worth what they're worth.

15:213 questions before you promise a delivery date

So, all that said, what do you do with this on Monday? There are three questions worth running before you put a date on anybody's calendar. Be strict with yourself on all three, because they're all easy to answer optimistically.

The first one is whether you've run the 20 ugliest records in the data set, or only the clean sample. If it's the clean sample, all you've got so far is a best case. And the best case is the part that already worked in 11 minutes.

The second one is what could come back different if you run this again next month, and what you've pinned down so that it can't. Thresholds, groupings, the account map, the wording of the categories. If the answer is that you just eyeball it, understand what you've signed up for. You've signed up to eyeball it every month for the rest of your career, and to be the single point of failure on your own automation.

The third one is what happens the month it breaks and you're on a beach. If the honest answer is that it quietly stops, and the first person to notice is whoever asked for the report, well, you've built a demo with good manners.

And here's the hard part. If you can't answer all three, what you've got is a very good afternoon. There's a real distance between a very good afternoon and something you can hand to somebody else.

That gap is where almost every stalled AI project in finance is sitting right now. They stall there because the last 20% never got put on anybody's plate. It never got put on anybody's plate because the first 80% made it look like there wasn't much left to do.

But I want to leave you on the other side of that, because the flip is good news. The exceptions, the pinning down, the plumbing, all of that runs on finance skill. It's the judgment you already have and spent 15 years building.

AI made the fast part free, which means the part you're already good at is the part that just got scarce. The finance pros who do that last 20% don't just get their nights back. They become the ones their company can't afford to lose.