Skip the AI Proof of Concept [AI Today Podcast] AI transcript and summary - episode of podcast AI Today Podcast: Artificial Intelligence Insights, Experts, and Opinion

Go to PodExtra AI's episode page (Skip the AI Proof of Concept [AI Today Podcast]) to play and view complete AI-processed content: summary, mindmap, topics, takeaways, transcript, keywords and highlights.

Go to PodExtra AI's podcast page (AI Today Podcast: Artificial Intelligence Insights, Experts, and Opinion) to view the AI-processed content of all episodes of this podcast.

View full AI transcripts and summaries of all podcast episodes on the blog: AI Today Podcast: Artificial Intelligence Insights, Experts, and Opinion

Episode: Skip the AI Proof of Concept [AI Today Podcast]

Author: AI & Data Today
Duration: 00:06:54

Episode Shownotes

Here’s a hint as to what is separating the AI failures from successes: skip the proof of concept. When it comes to AI projects go right for pilot projects. In this episode of AI Today hosts Kathleen Walch and Ron Schmelzer discuss AI Pilots vs. Proof of Concepts. AI Pilots

vs. Proof of Concepts A proof-of-concept is a project that is a trial or test run to illustrate if something is even possible and to prove your technology works. Continue reading Skip the AI Proof of Concept [AI Today Podcast] at Cognilytica.

Full Transcript

00:00:01 Speaker_02
The AI Today podcast, produced by Cognolitica, cuts through the hype and noise to identify what is really happening now in the world of artificial intelligence.

00:00:10 Speaker_02
Learn about emerging AI trends, technologies, and use cases from Cognolitica analysts and guest experts.

00:00:22 Speaker_01
Hello and welcome to the AI Today podcast. I'm your host, Kathleen Malch.

00:00:26 Speaker_00
And I'm your host, Ron Schmelzer. And we have heard from a whole bunch of you on these past few podcasts where we've been talking about process stuff. Some of you are like, oh, I totally agree with you. This has happened in my organization.

00:00:37 Speaker_00
Hey, comment on our posts because we We post on the various socials, but we're most active on LinkedIn. So we really like seeing your posts, comment on our posts. Happy to share them there.

00:00:47 Speaker_00
Of course, you can do them in the reviews to give us a five star review, of course, and then you can comment on what you like about an episode. But, you know, definitely engage with us on LinkedIn. We're really very active there.

00:00:56 Speaker_00
And many of you have been sharing your own personal expertise with some of the things we talked about, whether it's the over promise under deliver, whether it's the fact that, you know, people are moving fast and breaking things or doing stuff without planning.

00:01:08 Speaker_00
you know the whole issues around you know what does it really mean to think big and start small this is the stuff i want to hear about because at the end of the day it's your problem it's your project you're trying to do it's your project you're trying to introduce ai and solve some sort of problem with and hopefully you're accomplishing it and and being a success not a failure if you're listening to our podcast hopefully you're on the closer to the side of success than failure

00:01:30 Speaker_00
So we're going to continue with that a little bit, because we actually still have another thing that we haven't really dove too deeply on. We have talked about this on previous podcasts, but I think there's still a lot of confusion.

00:01:40 Speaker_00
We talked about in this last few podcasts, we mentioned many times, you guys should not be doing proof of concept. You should be doing more pilots.

00:01:48 Speaker_00
And we didn't really adequately define and discuss what does it really mean to do a pilot and what it really means to do a proof of concept, which you shouldn't be doing. What's the difference? Is there a difference?

00:01:58 Speaker_00
The answer is, of course, there's a big difference, especially if we're recommending you do one thing and recommending you not do something else. Clearly, there's a difference here.

00:02:05 Speaker_00
So we want to spend a little bit more time talking about the pilots and why you should really be avoiding this idea of the proof of concept when it comes to AI.

00:02:16 Speaker_01
Right. And, you know, the reason we bring this up too is because we're not saying, oh, this is just a terminology issue. And so some people call it a proof of concept and some people call it a pilot, but we're really talking about the same thing.

00:02:27 Speaker_01
So don't worry, it's kind of interchangeable. They're not. And we've seen, you know, we talk a lot about how we see AI projects fail. We've, you know, seen thousands of AI projects at this point.

00:02:39 Speaker_01
And what we did is we said, well, yes, they are failing, but why are they failing? And so we synthesized and said, there's common reasons why AI projects fail.

00:02:48 Speaker_01
And that's why we continue to share that with our listeners so that you can understand and not fall into these same traps yourself, and that you know how to get out of it if you think that your organization is going there.

00:03:00 Speaker_01
So a proof of concept is a project that is a trial or a test run. just to illustrate if something is even possible, right? Just to prove that it actually is possible and that your technology works.

00:03:14 Speaker_01
So proof of concepts are run in very specific and controlled environments, and they're not really in real-world environments and not really using real-world data. So they are going to have the best data that you have.

00:03:28 Speaker_01
You know, it's all going to be clean. It's going to be deduped. It's going to be, you know, that really high-quality data.

00:03:35 Speaker_01
people that are running these proof of concepts are usually very closely aligned with the project so that they know exactly what's going on and that they're running it in a certain way, right, because they've built it and so they want to prove that it's possible.

00:03:49 Speaker_01
But what we found is that that You know, the problem with these proof of concepts is that they don't actually prove if the specific AI solution will work in the real world.

00:04:00 Speaker_01
It only proves that it works in these limited circumstances with that good clean data that you've labeled or, you know, cleaned and made sure that it's available and maybe spent a lot of time and money to clean.

00:04:13 Speaker_01
So your technology may work great in that proof of concept, but then fall apart once you put it into production in real world scenarios. And we provide a lot of examples in our CPMAI training and certification.

00:04:24 Speaker_01
And so if you, I know a lot of our listeners are CPMAI certified. And if you haven't taken it, we encourage you to go to Cognolitica.com slash CPMAI and get certified yourself. And we talk about these in much greater detail and walk through why.

00:04:37 Speaker_01
we see these problems.

00:04:38 Speaker_01
Ron and I also present a lot, sometimes in person, sometimes virtually, and I encourage you to follow us on LinkedIn so that you know when we are talking because sometimes we do talk about, you know, these common reasons for AI project failures and how to not have your project bail.

00:04:53 Speaker_01
But whenever we talk about that proof of concept, we really say it isn't proving anything, that it's just showing that it can work in these very limited situations, and that when it gets into the real world, it may not work at all.

00:05:06 Speaker_01
And so if you run that proof of concept, then you might need to start over and actually run the pilot, which is going to be in the real world to see if things run as needed. So that's what the proof of concept is.

00:05:18 Speaker_01
The pilot, on the other hand, is when you are running it, it can be a small, You know, you can, you could say a smaller project.

00:05:26 Speaker_01
You don't need to like, you know, scale it out right away, but it's on the real world data and it's using your real world environment and your real world system and real world users are doing it, not just the people that have built it.

00:05:39 Speaker_01
And that's when you can see if things are working as expected or if they're failing in some spectacular ways.

00:05:44 Speaker_00
Yeah. One interesting thing about pilots versus proof of concept is, I think, a psychological thing. So somehow with proof of concepts, people feel like they don't need to be as rigorous. They're like, oh, this is a proof of concept.

00:05:57 Speaker_00
I'm going to throw it away. I'm just going to test something out. We're going to spend a little bit of money, not a lot of money. We're going to we're going to have the vendors do the proof of concept for us, which actually happens a lot.

00:06:07 Speaker_00
They're like, go build us something as a proof of concept. Why would you ask a vendor to do it? Of course, they're just going to put something together that makes their tool look good so that you're going to buy it.

00:06:16 Speaker_00
And of course, you end up buying this tool and it doesn't do what you want it to actually do in the real world, which of course is the part that matters. So the proof of concept was really irrelevant. It was just another dog and pony show.

00:06:27 Speaker_00
So when we see like, especially vendor produced proof of concepts, we know that's truly a waste of time because no one's taking it seriously.

00:06:35 Speaker_00
They're using it for some sort of, I don't know, sales evaluation, but it has nothing to do with what you're actually going to be using with. And of course, that's the part that matters.

00:06:42 Speaker_00
Why spend all that money if it's not going to actually work in your real world? So proof of concept is this weird psychological thing where people feel not as invested in it, whereas a pilot

00:06:52 Speaker_00
all of a sudden they're like, whoa, I got to take this thing seriously. It's going to be real people, real customers, real employees, real data, the real world, real things can go wrong.

00:07:01 Speaker_00
Like, yeah, well, that's kind of what you were supposed to be doing in your proof of concept anyways. It's just you didn't take the proof of concept seriously, but you're taking the pilot seriously. That's why these things really are different things.

00:07:13 Speaker_00
And we say there's no point to a proof of concept because it all is a dog and pony show. It doesn't matter who's doing it, whether you're doing it for yourself or whether you're having a vendor do it.

00:07:21 Speaker_00
So the thing about this is that, for some reason, pilots are pushed in terms of their length. We had an interview with somebody at one of our events a long time ago, and they told us that their AI project was taking 18 to 24 months.

00:07:37 Speaker_00
We're like, that's ridiculous, or 12 to 18 months, whatever it was. Absolutely ridiculous. They're a name-brand company that you would be very familiar with in a very large industry, a Fortune 1000 company. They're a C-level person.

00:07:49 Speaker_00
And they're telling us that their AI projects are 18 to 24 months, or 12 to 18 months, whatever it is. And we're like, this is ridiculous. I mean, it's just absolutely ridiculous. I don't know what you think is going to happen in those 12 to 18 months.

00:08:03 Speaker_00
A lot of different things. What are you going to do if you discover that all of a sudden, 18 months from now, you don't have the data you need?

00:08:08 Speaker_00
or that the problem can't actually be solved that way, or you're doing too much, too much complexity, you bought the wrong product, all that sort of stuff, right?

00:08:16 Speaker_00
What the world of agile, and I'll put that in quotes, taught us is that moving towards smaller iterative sprints helps you be more successful because you can deal with those ongoing changes as they happen.

00:08:28 Speaker_00
Doing so-called waterfall or the predictive approach is harder because you need to plan for everything in advance, and if you have a long project, that won't work. Now, it turns out that They're not diametrically opposed.

00:08:39 Speaker_00
You can actually do bits and pieces of each. And it turns out that having a hybrid approach actually is a better practice, because some things should be planned in advance. Building a hospital, developing an expensive aircraft, things like that.

00:08:53 Speaker_00
You can't be iterative and agile and one day deciding you need 12 windows, and the next day you need 20 windows. You can't do that, move the door around. It just can't happen on an airplane or in a skyscraper.

00:09:04 Speaker_00
So there are some things that do require that predictive style, that plan-ahead style, but then there are other things where you don't know the real world, and so you do need to plan for iteration.

00:09:14 Speaker_00
Either way, having these long-term projects is a recipe for failure. So those pilots that we talk about need to be very short, and they need to be in the real world. They need to prove that the technology actually does solve a problem.

00:09:28 Speaker_00
So you might think, well, that sounds like a proof of concept. It's like, no, we're not proving a concept. we're proving a solution. It should be called proof of solution. I don't know. That's a weird idea.

00:09:37 Speaker_00
But a pilot is basically the proof that the thing you're trying to do solves the problem in the way that you think it will solve it.

00:09:44 Speaker_00
And if you can't prove that it will solve the thing that you want to solve, even in a very narrow sense, then just hitting the fire button and going full steam ahead with the larger project is most definitely

00:09:56 Speaker_00
firing blind, that you are probably not going to achieve the success you want. Almost every major project that's implemented, not just AI, but especially with AI, has gone wrong.

00:10:08 Speaker_00
The moment that if you can't prove it in the small case, you're not going to prove it in the big case. A vendor-based proof of concept proves nothing. Your own internal research team doing some toy project, that doesn't prove anything either.

00:10:19 Speaker_00
As we say, skip the proof of concept, go right to the pilot, and focus on small iterative pilots you can do in very short amounts of time.

00:10:29 Speaker_01
Exactly. And I think that that, you know, really is important to understand. We understand too, you know, we say that it's people, process, and technology. And the technology is usually the easiest part to implement.

00:10:41 Speaker_01
And a lot of, because it is so easy, a lot of people just try and apply more technology to problems. But the people and the process, I mean, you know, change is hard. People don't enjoy change, right? There's a whole industry around change management.

00:10:56 Speaker_01
So it can be difficult.

00:10:58 Speaker_01
And depending on how your organization operates right now, so we're not saying that this is easy, but it really is important to understand that when given the choice, when you're saying proof of concept or a pilot, always go for the pilot and push hard for the pilot at your organization, because the proof of concept really just won't prove

00:11:18 Speaker_01
much. And that pilot is where you're going to learn, you know, how people are interacting with these systems, where they're failing. Maybe the data that you, you know, you have in the real world is very different than the data that it was trained on.

00:11:32 Speaker_01
Or you have no idea until you get it out there, right? You know, we can say projects can fail in spectacular ways. You can't always kind of guess ahead as to what some of these problems could be.

00:11:43 Speaker_01
So that's why when it's actually out in pilot, you can understand all this. And then, of course, when it is out there, you need to still be following best practices, step-by-step approaches.

00:11:53 Speaker_01
And so we are big advocates of the CPMAI, the Cognitive Project Management for AI methodology or framework. Sometimes the word methodology trips people up, but really we say it's just a step-by-step approach.

00:12:05 Speaker_01
for how to go about running and managing your AI projects. Many of our listeners are CPM AI certified.

00:12:11 Speaker_01
And so if you would like to get certified yourself, go to cognolitica.com slash CPM AI, where you can really learn best practices for running and managing AI projects. You can get ahead in your career.

00:12:23 Speaker_01
You can bring this to your organization and really hit the ground running. A lot of organizations have adopted this and been very successful in their AI projects.

00:12:33 Speaker_01
So we say, if you don't want to be a failure statistic, then follow this step-by-step approach so that you can know how to run and manage AI projects. Again, we'll link to that in the show notes, but it's Cognolitica.com slash CPM AI.

00:12:45 Speaker_01
And if you haven't done so already, subscribe to AI Today. We have some great interviews lined up and we also have some more podcasts on various topics. Like this episode and want to hear more?

00:12:56 Speaker_01
With hundreds of episodes and over 3 million downloads, check out more AI Today podcasts at aitoday.live.

00:13:04 Speaker_01
Make sure to subscribe to AI Today if you haven't already on Apple Podcasts, Spotify, Stitcher, Google, Amazon, or your favorite podcast platform. Want to dive deeper and get resources to drive your AI efforts further?

00:13:17 Speaker_01
We've put together a carefully curated collection of resources and tools. handcrafted for you, our listeners, to expand your knowledge, dive deeper into the world of AI, and provide you with the essential resources you need.

00:13:29 Speaker_01
Check it out at aitoday.live slash list. This sound recording and its contents are copyright by Cognolitica. All rights reserved. Music by Matsu Gravas. As always, thanks for listening to AI Today, and we'll catch you at the next podcast.