Are AI Tools Worth It for FP&A? Most CFOs Can't Prove It

Full transcript

Lightly edited for readability. The audio is the original.

0:00Why AI ROI has been missing for 16 years

Since 2007, a firm called Rexer Analytics has been asking data scientists how their work is going. 10 surveys over 16 years. And in the latest one, only 32% said their models usually get deployed. And ROI, the thing they ranked as the most important measure of success, only 41% of them even measure it.

And over that time, the software changed from SPSS to R to Python, and the complaint about getting the business to use the work kept coming back, survey after survey. So today: why that gap has lasted 16 years, what it means for AI and finance, and why the people best placed to close the gap are sitting in finance.

Welcome to Excel at AI in Finance, the show for finance pros who want to get home sooner and be the ones their company can't afford to lose. I'm Mike, I run F9 Finance, and I'm a senior finance leader working in the trenches. Each week, I share the AI tools, workflows, and mindsets that get you where you want to be. All right, let's get into it.

1:01Six AI projects a week and no business case

Before we get into the survey, let me give you a version of this I've seen up close. Any names and details have been changed to protect those involved. I keep watching companies get their hands on AI tools, and this is true for a lot of clients of mine, and one I'm thinking of in particular.

They got their hands on AI tools, and they went crazy. This team was churning out at least six different AI projects a week. They brought me in because token spend had started to go up as the models were getting better, and the CFO of the company was getting really concerned that they weren't seeing a return.

They hadn't really found a way to streamline any positions. It didn't seem like work was getting faster. A lot more was being produced, but nothing seemed to be getting better or changing.

I sat down with the business, and I started asking them to share their business cases. They stared at me in a way that told me they hadn't really considered the business cases. What they'd started doing was just throwing the cool technology at things to impress people. They started throwing the cool technology to start building projects that sounded really cool and demonstrated the technology.

One specific use case I remember that really jumped out to me was that they built an AI tool that could customize the view of the dashboard you were looking at. Essentially, it was churning through tokens to click dropdowns to use an automation tool. Nobody was asking for this.

Nobody really needed this, and it wasn't even performing all that well because it didn't have a rich business context. So burning tokens without adding any value.

In the end, after a review, we found that 95% of the projects not only were demonstrating no ROI, but nobody had even asked for them. All the benefit was in that last 5%, which turned out to be using only 2% of the entire token spend.

Where was finance in these discussions? Well, a few of the tools were being built by finance, but finance from an analysis and business case standpoint? Absolutely nowhere. Everyone was racing to use the technology, and nobody really thought about where it was going. That's why when I read these Rexer Analytics surveys, it jumped out to me just how long running this problem is, even before we started working with AI.

3:51The Rexer data science survey, 2007 to 2023

Now, you're probably rolling your eyes and saying, oh man, not another survey about AI not working. Every few months, these are coming out. We're hearing about how they never pay off and they get treated as news for a few days.

What really interests me about this survey is that it's been going on since before AI was even officially in the landscape, generative AI the way we know it today. What was even more interesting is they're talking about exactly the same problem, but with a lot more data, a lot more knowledge to share in some of the more recent AI surveys.

Now, this comes from Rexer Analytics. It's a small analytics firm outside Boston. From 2007 to 2023, they ran 10 surveys, all with the same core questions.

This group was made up of data scientists, not directly finance, but it's still very relevant. And almost entirely, except for the 2023 survey, this was before ChatGPT. I highly encourage you to read the full survey from 2023 (it was 328 people) if you have a chance, because there's a lot you can learn, and I'll put a link to it down in the show notes.

Survey after survey, the technology keeps getting better, but the same two things keep going wrong. The business doesn't use the tools and nobody's tracking what they actually returned. And those two things have carried on into the AI world while most people are treating them like new problems. That's where this really becomes insightful, because we can learn from years of experience what hasn't been working and how we can fix it.

5:18Why AI adoption stalls: the people problem older than AI

The first thing I really want you to take away is that having 16 years of surveys all the way back to 2007 tells us that the problems we're facing are much older than AI. They're business problems. They're people problems. They have nothing to do with the technology.

Back in 2007, the top complaint surfaced was that data scientists had trouble explaining their work to the people who were supposed to use it. In 2013, the problem became a lot more about deployment and measuring performance. In 2015, buy-in was noted as the biggest barrier to deployment, always pointing back to the people involved.

By 2023, the surveyors were honestly saying that only 32% of the models get deployed. No matter what software it is, over time, the same problems kept coming up: that people aren't using it, aren't understanding it, and aren't finding ways to drive value.

So the next time a project stalls, ask one question first. Who in the business was supposed to do something differently because they had the tool? And did anybody actually tell them?

6:27Measuring AI ROI starts with a baseline

So let's say you've gotten someone to use your tool and you've gotten them to understand what they need to do differently with it. Well, even with that, are you understanding what the return on investment is supposed to be? And are you tracking towards that?

I think back to the tool where you could ask that AI chatbot to adjust the filters in the dashboard you were looking at. They got people to use it. People were using it. It was adjusting their filters all day, but nobody was looking for an ROI.

In fact, in the most recent survey, only 41% of companies were even tracking the ROI of the project they developed. These were data scientists. Their entire role was to develop tools that added value, and less than half were being asked to track or catalog the value that they were even driving.

You have to start by having a baseline. What does the process cost today in time and money? And what's the expectation of what the process will cost in time and money in the future?

7:27Usage-based AI costs make ROI tracking urgent

The really fascinating thing as we think about ROI is AI has very dynamic costs. If we were running a Python script or building a dashboard, those tended to be fixed costs. We'd pay for a Tableau license. We'd pay for a Power BI license.

There could be some computer storage space based on what we're serving, but that was largely fixed and we knew what the bill was going to be. With AI tools, they charge you every single time they're accessed. And that gives you a very different risk on the cost side, that we don't always know exactly what the cost side will be.

That makes it even more critical to make sure that we're driving value in the form of time or money. But just like decades before, this is rarely being tracked as people race to roll AI tools out for the sake of just doing AI.

Before any AI project goes live, you have to understand what is supposed to move: the time, the money, or something else. What is that baseline today? And specifically, when are you going to come back and check on the value?

8:30The 3 fixes the survey names are finance jobs

Now, the most recent survey dove into what would help close the gap. Where did people see the opportunities? The biggest thing was stopping to plan early in the project, understanding what the goal was, what the business outcome was, getting buy-in to change how the business runs, and helping management understand the work and the way that an AI or automation tool could plug in.

Now, if you think about what I just described, it's a business case. It's what finance has been doing for decades on capital projects, on mergers and acquisitions, on business investments. And that's why finance is so well positioned to own and track this process.

It's a dynamic cost, so finance already needs to be involved. I'm sure you've heard the stories about companies blowing through their entire AI budget for the year in the first 4 months, or about companies taking away Claude Code licenses because spend had just gotten out of control.

So it's back to the basics for us. Early planning, business cases, holding people accountable to the business cases, stopping the projects that are getting off track, doing budget reviews, having operations, data, whatever teams they are, committing to a number, and then coming back to check on how it's going. We're talking about the regular management reviews and due diligence that finance is already doing.

Moving forward, ask to be in the kickoff for AI projects and bring the business case template that you're already using for capex, just applied and adjusted for AI cases. This problem has been around at least since 2007. We know it's been around since before then. So it's critical to get ahead of it and stop managing in new ways for AI when the ways that have always worked will work just fine.

16 years of surveys. The software keeps changing and the models keep getting better. What stayed the same is the stretch after the model gets built. Somebody has to get the business to use it and somebody has to value what it returned.

Writing down the starting number and coming back later to count the result, well, that's something finance has been doing for decades. The AI return on investment gap is a finance job, and it has been as long as we've been tracking ROI.

The good news is this skill is on your team. The hard part is committing to that number before the project starts, making sure everyone can see it, and coming back for it later.

10:583 actions to close the AI ROI gap

Okay, so what do you do with this tomorrow? Three things, about an hour in total, and you don't need anyone's permission to start.

Number one: find one AI or automation project running at your company right now. Ask the person who owns it one question. What is this supposed to move?

Number two: find out what that is today. Write it down with today's date next to it and send it to the project owner. That's your baseline.

And number three: put a meeting in the calendar 90 days after it goes live. Invite the person who built it and look at the results together against the baseline.

The first two take a conversation and a spreadsheet. The third one means putting your name next to a result you don't control yet, in front of the team that built it. And some of these counts will come back at zero. That's the one that gets skipped every time. And it's the one that closes the gap.

The finance pros who do that counting don't just keep their jobs. They become the ones no one can replace.