Claude and Microsoft Copilot Giving You Garbage? Check Your Brief!

Full transcript

Lightly edited for readability. The audio is the original.

0:00Why AI in finance gives you bad output

Say I hand you a brand new analyst on Monday morning. Smart, keen, first job out of school. And all I say to him is, "Write the variance commentary." Then I walk away.

Tuesday, he comes back with something generic and a little bit wrong. Would you sit there and conclude that analysts don't work? Well, of course you wouldn't. You'd think, well, that was my fault, I gave him nothing.

The crazy thing is that is what almost every finance pro I know has done with AI. Same handoff, same bad result, and then the exact opposite conclusion. So today, the three things missing from that handoff and what to put back in before you try again.

Welcome to Excel at AI in Finance, the show for finance pros who want to get home sooner and be the ones their company can't afford to lose. I'm Mike, I run F9 Finance, and I'm a senior finance leader working in the trenches. Each week, I share the AI tools, workflows, and mindsets that get you where you want to be. All right, let's get into it.

1:02My first Copilot for Excel attempt, and 2 months of blaming the tool

First time that I tried Copilot for Excel, I was so excited. This brand new tool had just come out from Microsoft. They were singing all the praises, and it was my two favorite things: we had AI and we had Excel together. There's a reason this is called the Excel at AI in Finance podcast.

So I opened up a data set, and I'm like, I'm just going to have it build this cool report file. So I gave it a bunch of revenue data, and I said, "Hey, here's some data from the last 3 months. I'm really trying to extract some of the trends from this. I want to understand why this particular line is not working. Can you help me out with that?"

That's what I said. I said, here's the data, I want to understand why this isn't working. Go.

I watched the tool start to run, and things were moving all over the place. It was processing. It was listing out steps. Excel started to change, new tabs came in, and I thought, wow, this is going to be really interesting.

The chat window stopped running and I thought it had frozen mid-step. I was looking at unformatted garbage. It hadn't understood anything. It had pulled out trends and outliers in the data, but it had nothing to do with the business.

It was clunky and it was all over the place. There was no consistency. There was no consistent formatting. I didn't know what I was even supposed to get out of the report. Not only was this something that I wouldn't be ready to hand to my boss, it was something that I didn't even want to be looking at.

And immediately I went, wow, I'm glad Microsoft's excited about this, but this tool is a piece…

The thing is, I did everything wrong. I gave it data. I didn't give it any context about what it was supposed to do. How was it ever going to find trends? I didn't tell it anything about formatting or the standards I expected.

I just gave it the data and said, pull trends out of this. And I never gave it an example of what good would look like. Can you imagine asking an analyst to go from raw data straight to a polished report? It would never happen.

So I decided that the tool was crap and I didn't come back to it for 2 months, until I saw other people having success. When in reality, it was me that was the problem.

Now, I've watched some version of my experience happen enough times that I stopped thinking of it as bad luck with a prompt or just something that I was doing. And across everyone I've talked to, it's the same three things missing every time. So let me walk you step by step through all three, because once you can name them, you can put them back. And putting them back takes about 10 minutes, if that.

4:06A bad AI output is a fair result of a bad brief

A bad AI output is a fair result of a bad briefing. The model did the job it was given, and the job it was given was one sentence long.

Now, you've hired people. I've hired people. You've trained people. We know what a real handoff looks like. A new analyst gets weeks of context before anyone starts judging them.

Now, nobody's being lazy. The box looks like a search bar, and search bars take one line. So one line is what people type. It's exactly what I did.

Somehow we think that AI can just read our minds, but you have to treat it like an intern or an analyst on their first day. So there's three things missing from that one line we're feeding it: context, standards, and the job itself, the job we give it. And this is the order, because the first one is the biggest.

5:02Missing thing 1: context, and what a new analyst gets that the model doesn't

Starting with context, a new analyst does not get told, "Write the commentary," and then get left alone. They get the last 6 months of the commentary. They get an account map, and they get a warning about which numbers the CFO really reads and will be paying attention to. None of that is in the data file that I just handed over to Copilot. I just gave it data and, again, expected it to read my mind.

So how do you know when you haven't given enough context? Well, the symptom for this is you see generic output. This is something like it coming back saying, "Revenue increase due to higher sales." Duh. You've given it the numbers. It has nothing else. So when it comes back and it's vague, it's indescriptive, it's not reflective of the business, you're lacking context.

So let's go back and think about when I first did Copilot to when I came back to Copilot, having learned a lot more about AI, and I started again. The first time, I gave it data. I gave it the numbers and I said I wanted trends. That's all I told it.

When I came back to Copilot a few months later, I worked with a similar data set, and I told it that I wanted to evaluate if we should end one of our product lines. I then gave it a link to our website and said, understand the business. So already I've given it all of the context about what world it's working in. Then I said I was preparing an analysis and I needed to send a one-page recommendation to a vice president in operations. Now it knows about the business and it knows who I'm sending it to.

Now, I still wasn't fully advanced in using Copilot or AI tools at that time. So I hit send with that. It had the data, it had those two pieces. And what came back wasn't perfect, but it was so much better.

It was formatted instead of a chaotic Excel file. It did data on one tab and then gave me a really nice, clean, print-ready report on another. It used the business context and it found trends that actually mattered. It started looking at cost of sales. It started looking at gross margin. It started looking at some of the other features of the product to understand if this was a trend that would improve or if this was something we really needed to end.

So just that little bit of context, still not perfect, changed it by about 80%. And now it was a matter of small tweaks or some additional feedback to Copilot, versus something that came back completely unusable.

So my biggest tip for context: before you get started, or before you judge the output, ask what you would have to hand a smart new hire to get a good version of the output. Then go and hand the model exactly that. Otherwise, you're setting yourself and the model up for failure.

7:40Missing thing 2: standards, and why more instructions never fix them

After context, next up is standards. Every finance team has a house standard, but almost nobody has written it down. How long commentary runs. How far you chase a driver. Whether you name the business unit or the person.

Now, you absorbed all of that in your first few months, and you've never had to say it out loud since. But the model hasn't had that opportunity, and that's where standards come in. When you don't give a model a standard, it invents the standards. Anywhere that you leave open to interpretation, the AI is going to fill it in based on the average, based on what's most likely.

Now, we've talked about this before on the podcast, but as a reminder, these large language models are just statistics machines, and they're trying to show what is the next most likely thing to happen. When you give them context, when you give them your business standards, you're taking that decision away, and they know exactly what to do. Otherwise, they're going to guess, and everything is going to move towards the average.

You can throw all the instructions you want at this, but it's really hard for the AI tool to nail every single little decision it has to make along the way with just some written-out instructions. So what you need to do is stop describing what you want and start showing it what good looks like. You've got all of this prior work laying around. Share some of that with the AI tool, just like you would do with a new analyst or a new intern.

Go back to my Copilot example. When I first did this, I just gave it data. It had no context for what I was looking for. And what I got back was average: a rough Excel file with some numbers, but not laid out, not clear, not concise.

Now, after this, a few months later, I was working on a related project. I wasn't quite happy with the output I was getting. I'd come a long ways. I was giving it all my context, but it wasn't quite getting it.

So what I did is I went back and I found a similar report to what I was working on at the time. And that similar report had gotten really good feedback from my leadership. So I just attached that report to the tool, and I said, "This went really well, and this work I'm doing now is similar. Replicate this view using the new data."

And what came back was 95% of the way there, because the tool knew exactly what I needed. So what you need to do is stop describing what you want and attach the best version you've ever sent, and tell it to do work like this.

10:05Missing thing 3: the job you picked

Finally, number three, the job you choose. I got this wrong myself for a long time. The tasks people hand over first are the ones that hurt most. Write the commentary, build the model, make the recommendation. Those are the highest-judgment jobs you own, which makes them the worst possible first test.

The best things to throw at AI are reformatting, data cleaning, cross-checking, reading 40 pages of a document for one clause. Let's look at two jobs that you could throw at it in the same afternoon and watch the difference.

Board commentary. You ask it to build board commentary, you spend 40 minutes arguing with it and then rewriting half of it. Now, separately, you hand it a stack of contracts. In a minute, you have every renewal date, every termination clause in a table. And you can check every single row against the source.

Now, the second one is boring. Nobody makes a video about it. I'm not even making a video about it. But it's also where the hours come back.

The real test is if you can check it. If you can check AI against a total, a source file, a signed document, then you are in a perfect spot to save time. If the only way to check it is to completely redo it, well, then you've picked the wrong job.

11:17The reversibility test for what to hand over

The other thing to think about is reversibility. So as I think about that board deck, or an external reporting, or something that's doing a sales pitch, that is really hard to reverse. Once it's out, it's done.

Now, contrast that with management reporting or a first pass of a forecast. That is easily reversible, and there's time to review it. So that's where you can start giving a little bit more of the scope to AI.

But the job you choose is one of the most critical pieces, because you could pick a wrong job and no amount of context or business standards will save it. So pick a job where you can check the answer against something. If verifying an answer takes longer than doing it, pick a different job.

12:01The output is a mirror of the brief

The output is a mirror of the brief. You handed over a task with no context, no example of good, and no check on whether it was even the right job to hand over. A new analyst given that same sentence would have come back with the same thing. And you'd have spent a week fixing the handoff instead of writing them off.

That week is all the model is asking for too. And this goes well past whatever you were trying to automate that afternoon. The people who work out how to hand work over properly are going to be handing over more of it every year, to models and to people. That is a skill with a price on it.

12:403 questions to run before your next prompt

Okay, so what do you do with this on Monday? Three questions. Run these questions before you type a single word into that chat box again.

Question one: what would I have to hand a new hire to get a good version of this, and have I handed the model any of it? If you gave it the ask and some data and nothing else, well, you haven't really tested AI yet. You tested a one-line brief.

Question two: do I have an example of good I can point at? Not describe, but point at. If you cannot find one past output you were proud of, sit with that, because it means the standard you were holding the model to is one you've never written down.

Question three: can I check the answer against something that already exists? A total, a source file, a signed contract. If the only way to check it is to redo the work myself, well, I picked the wrong job.

Now here's the hard part. If you cannot answer all three, then what came back was never a verdict on AI in Finance. It was a verdict on a sentence.

And there are a lot of finance pros right now who have written off the fastest thing that ever landed on their desk on the strength of one bad sentence on one bad Tuesday. The finance pros who get past that first bad output don't just get their time back. They become the ones their company can't afford to lose.