.webp)
On Demand Webinar: Judgment Without Feeling? Human vs. Artificial Intelligence
Now On Demand
On September 23, StandardC hosted "Judgment Without Feeling? Human vs. Artificial Intelligence," a live conversation with Mark Smillie, author of The Feeling Machine, and Mike Freiling, PhD, advisor to StandardC. I moderated. The full recording is now available to watch on demand.
The Core Idea
Human judgment is not the bottleneck in an AI system. It is the engine. AI processes data. Humans process meaning. That distinction ran through the entire hour, and it has direct consequences for how financial institutions deploy AI in credit, compliance, and risk.
What We Covered
Removing feeling does not produce pure logic. Mark walked through the case of Elliot, a patient who lost the ability to make decisions after brain surgery even though his reasoning was intact. Strip away emotion and you do not get a perfect analyst. You get what Mark called paralysis dressed as analysis. He tied this to the Uber self-driving fatality, where the system detected a person but had no way to decide what mattered.
Machine intelligence is alien intelligence. Mike drew on MIT computer scientist Joseph Weizenbaum's view that AI would be fundamentally different from human thought, not a copy of it. People rely on somatic markers, the physical signals built from lived experience, to separate what matters from what does not. AI weighs every option equally because it has no such history.
Confidence scores are not judgment. A model that reports 96 percent confidence creates a false sense of precision. When that number displaces human review, accountability blurs and errors compound.
Match automation to risk. Lower-risk, high-volume tasks such as fraud detection are well suited to AI. High-stakes decisions in areas like medicine and criminal justice demand human involvement. For financial institutions the principle is the same: AI can analyze, people decide.
Design for expertise, not dependency. Mark proposed a Cognition Integrated AI approach that replaces the traditional alert dashboard with a side-by-side comparison of a successful case, a failed case, and the case at hand. The goal is to sharpen the reviewer's instincts rather than build a dependency trap where people stop developing their own judgment.
Set the standard early. The panel agreed that the industry should establish ethical and governance standards for AI proactively rather than wait for someone else to write them.
Why It Matters for Financial Institutions
This is the same principle behind StandardC AI: governed, privacy-first AI built to amplify human judgment, not replace it. Sensitive data is redacted before any model sees it, outputs trace back to source documents, and final authority stays with the analyst, underwriter, or BSA officer. The conversation makes the case for why that design choice is not a limitation. It is the point.
About the Panelists
Mark Smillie is an entrepreneur and the author of The Feeling Machine. He is the founder of WorkSource.io and CognitionIntegratedAI.com, with a career of more than 30 years across technology, healthcare, biotech, equity research, investment banking, and SaaS.
Mike Freiling, PhD earned his doctorate at the MIT Artificial Intelligence Lab and holds the CFA designation. His work has focused on AI and data science for banking and financial services, and he serves as an Advisor to StandardC. Read his article Governing the Generative AI Supernova.
Robert Baron, CAMS, CAMS-RM is Chief Experience Officer at StandardC.
More to Come
This session opens a limited educational series on AI from StandardC. Watch this space for upcoming episodes.
Watch the full recording on demand
Read the Full Transcript
This transcript was generated from the webinar recording and lightly edited for names and spelling. Timestamps correspond to the recording.
Robert Baron [00:00:06]: Alright. Good afternoon. Welcome. To our webinar. First question for everyone. Can a machine have judgment without feeling? Maybe. But I guess you have to ask yourself this, would you want it to? Well, good afternoon for all of you for joining us. Before we get into the discussion, I want to take a moment to properly introduce our two guests. First, Mark Smillie. Mark is an entrepreneur and author of The Feeling Machine, and the founder of WorkSource.io and CognitionIntegratedAI.com. Over a 30-year career spanning technology, healthcare, biotech, equity research, and investment banking, he's built businesses and launched products ahead of the curve. His current work centers on a simple idea that AI should amplify human judgment, not replace it. Next, but not least, is Mike Freiling. Mike earned his PhD in 1977 from the MIT Artificial Intelligence Lab, one of the earliest degrees granted in the field. Where his research examined the limits of early neural models and how machines might reason about everyday objects. Since then, his career spanned electronic instrument troubleshooting, pension liability estimation, payment fraud detection, and identifying noncompliant behavior in capital markets. He earned his Chartered Financial Analyst designation in 1994 for his work on financial algorithms, and he's also a published poet and translator with translation of classic Japanese poetry. And I'm Robert Baron. I'm just a recovering banker and a bit of a futurist, and I'm part of the StandardC team that just filed two patents on privacy-first and deterministic AI. And I'll be moderating today's conversation. Today, we're going to get into the differences between artificial and human intelligence. Where AI can safely operate on its own. And where a human needs to stay in the loop. But also, where decisions should only ever be made by a person. And towards the end, we'll open it up to some questions. So, if you'd like to answer any questions, please enter it into the chat, or you can ask a question, and we'll answer it towards the end. So, Mark, I wanted to ask you about your book, The Feeling Machine. Really a fantastic read. But a question came up as I was reading it. Is feeling really, like, what a machine doesn't have? Like, what does judgment lose when you take the feeling out?
Mark Smillie [00:02:51]: Yeah, thanks for that question, and thank you for hosting this and setting it up. I really appreciate it. What I have kind of found is that when you take the feeling out, you don't get pure logic or better judgment. You get paralysis dressed as an analysis. Essentially, you lose the ability to judge and make decisions. I have a couple examples from the book that I want to illustrate. One is about this person, patient named Elliot, and one is using Uber. First, let's talk about Elliot and the neuroscientist Antonio Damasio. Antonio Damasio was a neuroscientist that did a lot of work exploring how decisions are made with patients who have brain damage or brain issues. Elliot had a brain tumor, and that brain tumor was removed from his ventromedial prefrontal cortex. And what was sort of discovered after that fact was that everything was fine. His IQ was the same, his memory was intact. He could do the math, but he could not make a decision to distinguish between what mattered and what didn't. He'd spend hours trying to decide where to go for lunch, or what color the folder should be on his computer. They found out, essentially, that he lost the connection between his somatic markers and the data. And what we're kind of speculating is that AI is architecturally what Elliot is neurologically. It knows, but it does not feel. I wanted to define the idea of the somatic marker as well, because it's important in understanding what we're doing, and how it relates to everything else in the book and the talk to some degree. Somatic markers are our body's way of encapsulating experience. They enable us to feel the encoded emotional weight of our experiences. So, for example, let's say you're walking down a path in the woods at dusk. There's a dark shape up ahead of you on the path, and your heart rate spikes, your muscles tense, your adrenaline floods your body, and you feel the panic, realizing it's a bear. Those feelings are somatic markers alerting you based on your experience. The feeling comes first. Seeing comes after. And that's a really important point in this, because what we're going to find out is, neurologically, we make a lot of decisions that are based on our experience and based on what our neurology has captured and coded for us over the years. The second example I want to talk about is an Uber accident. Years ago, when Uber was training their car, they were in Tempe, and this woman, Elaine Herzberg, was pushing her bike outside a crosswalk on a four-lane road in the dark. And there was an Uber driving up that road on a training exercise. What happened was that Uber detected her and was doing its math, and was pinging her and finding out what's going on, but it could not classify her. It had no way to tell what mattered. So it kept driving, eventually ending up hitting her and killing her.
Robert Baron [00:06:11]: Whoa.
Mark Smillie [00:06:11]: The point of this is that AI is phenomenal at recognizing patterns and doing the math, what I call one-dimensional. It detected Elaine, but it lacked a felt sense of what mattered. And that's really critical, because when we're making decisions, or when we're using AI to make decisions, we have to recognize what it's good at and what it's not good at. In this case, the AI could not process what I call the urgency of ambiguity. And if you translate that back to us as humans, if I'm a human driving a car, I'd stop because of the ambiguity. That would be enough for me to know to stop, whether or not it was a bike, or a car, or a bear in the road. So, what we're finding is that AI processes all options with equal analytical weight, because it has no real emotional history. I can't tag options with somatic markers in AI, so everything is treated as kind of the same. Everything gets an equal shot at being the first thing up there. And that notion is changing as AI changes every day for us. Just a few weeks ago, this JSpace idea came out, which is kind of an interesting play on what your brain does independently of what questions are being asked, but that's probably for a different podcast. That's my intro, I think, Robert. It's a huge limitation if we're unable to provide judgment based on data and what is felt, because that's key.
Robert Baron [00:07:50]: It almost appears that this emotion, the decisions themselves, the feelings that we have are not logical to artificial intelligence. It kind of reminds me of Data and Spock from Star Trek, in a way. It's just illogical to have these feelings if you're a machine, but for a person, it's quite critical.
Mark Smillie [00:08:16]: Yeah, let me just build on that, because one of the books that we looked at when I was writing this book was a book by Damasio called Descartes' Error, and the whole point of that was Descartes thought that we should eliminate all of the emotion and the soft stuff when we make decisions, and just rely on the harder aspects of it, the analytics. And what Damasio found was that that was kind of the wrong way to think about things, and I think, to a certain extent, that's been vindicated.
Robert Baron [00:08:59]: So, I look at artificial intelligence, and I see AI is great at so many possibilities. But, you know, Mike, I'm just curious from your perspective, where's the line where a machine, like AI, can safely run everything on its own? How can it run that show?
Mike Freiling [00:09:21]: Sure. I'm going to start with a couple of stories about a professor of computer science at MIT who some of you may remember. His name was Joseph Weizenbaum. And he was an early, I would call him an AI skeptic. He wasn't an AI denier in any way, but he was a skeptic. And I remember at a talk one time, someone asked him, well, do you think we'll ever have intelligence? And his remark, which still sticks with me today and is still the best one-line expression of the whole issue to me, he said, we may have something that will look like intelligence, but it will always be an alien intelligence. And that came to mind just recently. I was reading, to prep for this meeting, a paper by one of the major AI vendors that was titled, An Alien Mind.
Mark Smillie [00:10:15]: Hmm.
Mike Freiling [00:10:15]: So that attitude is still with us, and I think it's a good way to frame the discussion. Whatever AI has, and whatever we have, we can call it intelligence if we want. There's a lot of reasons to believe that they're different. For one thing, if we're going to use a word like mind, my current feeling is we don't even know where mind begins and where mind ends. There's a lot of research, and Mark's ideas about somatic markers are part of that new emerging consensus, that what we call our mind may extend well out into our bodies. Bessel van der Kolk's work on trauma victims seems to indicate you don't process trauma in your cognitive structures, you process it in your bodies. You don't remember trauma, you relive trauma. And the discussions of where emotions come from, they seem to come from some very complex feedback loops that include parts of the brain and parts of the body. All of this basically is a way of saying we don't really know where these things begin and end, but we can be sure that if a putative intelligence, like an AI, is constructed in a way that it doesn't have a body, it's going to be different in some fundamental way. And out of what you might call an abundance of caution, we need to be careful about where we apply that, and where we don't. So I think that's my general way of framing this whole thing. We can move on.
Robert Baron [00:11:56]: Yeah, and it's interesting. The feeling component, it's a physical, it's almost metaphysical in a way, pun intended. This gut instinct that we have is in itself not just something of the mind, as you're saying. It's really something that's of the body and perhaps beyond.
Mike Freiling [00:12:22]: It's embodied experience, and we don't even really understand it ourselves. So if we start making claims like, well, computers can do this, or computers have feelings, we are entitled to be very skeptical of those.
Mark Smillie [00:12:36]: Yeah, and I'd like to just add a point, because one of the studies that's in the book was looking at how your body reacts before your mind, or your consciousness, admits what's going on. There was a study done with some cards looking at how decisions are made, and they were measuring conductance on your hand in terms of how that would impact your decision-making, and when that actually occurred. And what was found was that the conductance happened before the game was given up. So it's an interesting thing, where this can happen in your body, as Mike was referring to, without your brain even acknowledging it to you, I guess is the way to put it.
Robert Baron [00:13:45]: It's an instinct, in many ways, from what it sounds to be. And I know I've already made enough of my Trekkie references, but just thinking of Star Wars, for an example, Episode IV, A New Hope, when Luke Skywalker is flying his X-wing down the trench of the Death Star after other pilots had failed to execute their bombing run, he hears something in the background, and it's Alec Guinness, may he rest in peace, Obi-Wan Kenobi, who says, stretch out with your feelings, Luke, and he deactivates his targeting computer. And at that moment, he realizes that his feeling was more powerful, and that connection to this instinct that he had inherently in the world is what ultimately led him to be successful. Now, of course, that's a fictional story, but there's a lot of parallels, I feel, to that feeling, that gut instinct. For millennia, if not longer, people have been trying to understand what makes us special, what makes us unique. And to me, talking both to bankers and other industry leaders, the key determinant of our future and existence is our humanity. It's our ability to feel, to think, and to decide. Allowing AI to necessarily take over those decisions can potentially be our demise, and we need to be very cautious about that. But we can do it in a way where we utilize AI to create efficiencies, to make decisions easier to come to, but ultimately still make those decisions ourselves, still relying on our gut, our feeling. That's key. And that actually brings me to a real-life targeting example, and this is, sadly, a true story. Think about what's happened with the war in Iran. There was a strike on a school. It was believed to have not been a school. And many hundreds of people died. And that's an example of a targeting system where the machine, the AI, believed that this was a legitimate target. It had, I believe, something like 94% confidence that it was going to be successful. I could be wrong on that number, but people are telling me it was 94%. Just kidding. But a human, ultimately, even though the machine said that, someone still had to pull that trigger. And so, if you do the math, there's about 6% of doubt. Mark, what do you think about that? How do we come in to stop those things from happening?
Mark Smillie [00:16:50]: Yeah, this is really interesting. I've been using this example. It wasn't in the book, because the book came out before that actually happened, but it's really true, and what I'm going to explain happens in other places too, on many AI dashboards, or dashboards for CRMs, or for ERPs, or whatever. These numbers that you see are really precise-looking numbers. I mean, 96% is pretty precise. And they feel like AI has done the thinking, leaving the human with what appears to be a small residual judgment to make. And that judgment in this case is pull the trigger, or don't pull the trigger. The confidence score has framed the decision as binary, at the edge of a nearly certain distribution. And what I'm contesting is that 6% is really not a probability. It can't really be a probability, because the operator has no way to verify or access the training data that produced that number, no ability to interrogate the model's reasoning, and no felt experience of the system-specific patterns that were used to arrive at that 96%. The confidence score, as I've come to understand it, really dissolves accountability. It's an outcome that no one owns. Put differently, a 94% score and a 6% doubt arrive as authority, and displace the operator's own prediction engine, the person who's making this decision. It's basically suppressing that somatic marker that would have otherwise fired. There's examples in the book where this is applied to the justice system, in terms of getting a number back from a system that is really a black box that rates the probability of recidivism among people who are in the system. And again, it comes back to this number that is really almost like a buffer. It implies that there's a ton of data behind it, but it doesn't really account for the emotional aspect of making that decision. And that, again, I think is really key to providing good decisions and not letting AI run away with decisions, because in that case, as Robert alluded to, you can have really disastrous consequences.
Robert Baron [00:19:32]: Well, ironically, it almost sounds like you have to enable the capability to make somewhat of an irrational decision, as opposed to simply a mathematical calculation to make the right decision. It doesn't mean you should purposely make irrational decisions, but you still have to be able to look at a situation and know that even though the statistics say I'm most likely to be successful and hit the correct target with a high degree of certainty, there's this amount of doubt that leads me to say it's better to make no decision. It's almost like a prisoner's dilemma, in a way.
Mark Smillie [00:20:10]: Yeah, I think you're right, yeah.
Robert Baron [00:20:12]: Mike, this is a good segue, I think, to another piece, which really comes down to, should we even let targeting systems be decided by machines? At what point should they even be involved in any decisioning? And what are the things that people need to do? So I'm curious, with your expertise and perspective, philosophically, what decisions should always be made by a human being, no matter how good AI gets? And it's getting good, but I'm just curious your perspective on that.
Mike Freiling [00:20:53]: Yeah, I have a couple of thoughts, not all of which are coherent, but I'll start with a remark. AI is clearly a two-way street. There's a lot of opportunity, and there's a lot of things to worry about. One of the unexpected benefits, and I haven't seen a lot of discussion about this yet, is that AI is going to force us to address those questions about what is uniquely human about us in a much more precise and compelling way than we've been able to do for centuries. And I think these kinds of dialogues are the start of that. To talk about what we should trust AI with, and what we should not trust AI with, I'll do both extremes, and then let Mark fill in what we can collaborate on. To start with, whenever the risk is low and the population is high, so that you can do a certain amount of training or modeling, there's very little problem with using AI to run autonomously. Autonomous is probably the wrong word; automatically. A good example would be bank fraud detection. You're not going to be able, economically speaking, to put a human in the loop to adjudicate every credit card transaction. And yet, you want to make sure you're taking a look at them. Well, the risk of getting it wrong is fairly low. It's a bit of embarrassment when your card gets declined at the Bellagio, which happened to me one time. I wasn't gambling, I was just trying to get an espresso. But no harm, no foul, really speaking. For those kinds of things, there's no real reason to insist on a human in the loop. It just doesn't make sense. And as long as you have enough of those cases that you can do adequate training, I think that's a fairly safe route. Now I'm going to jump to the other extreme, which is the things that we should never allow AI to be involved in, at least at the decision-making level. I'm not talking about conceptual support or research or something like that. But clearly anything where the risks are very high, anything that involves prejudice to human life or limb. A doctor deciding whether to do an amputation or not is something I would prefer to have a doctor doing rather than a machine. It's almost revolting to think about using an AI as a judge in a criminal case, for exactly that same reason. The risks are so high that, out of an abundance of caution, we just don't know that we can trust AI. And I'm going to segue into that by talking about something I read just recently in a paper by one of the major AI vendors. It's a little bit technical, so bear with me. The vendor said, we have been training these models, and he was talking about some of the models that got involved in the penetration incidents of a year or two ago. We can train them by supervising their reasoning chain, they call it chain of thought now, but we've chosen not to, because we want to see what they can do. Now, that sounds a little technical, but we have thousands of years of experience training our own bots. They're called children. And anybody who's got that experience knows that if you train a kid only on whether he or she gets the right answer, and don't scrutinize the method, you can easily end up with something that's a little bit like a monster. Someone who thinks, oh, I can cheat, I can do this, as long as I get the right grade, as long as I get the touchdown, I'm fine. That sort of thing is implicit in that kind of training. So then that leads to a question. Why aren't they supervising the chain of thought, too, and saying, no, that's a reasoning step you can't make, like, go to Hugging Face and look for the answer? And this particular representative of this vendor, I won't mention the name, said, we don't want to be too careful in supervising the chain of thought because we're afraid we will motivate the bot to hide its chain of thought from us. Now, if you work through the conclusions of that, what the representative is saying is, already we don't trust AI to tell us the truth in as simple a matter as, what are you thinking? So when you get to that level of, I won't call it mistrust, I'll just call it inability to trust, you have to be extremely careful about what you're going to involve it in. And you get this kind of dichotomy, which I don't think has been explored much, about what kind of a monster you are creating. In my mind, there's two possibilities. There's what I call the immoral malefactor, the bot that's out there to do you harm. But I think that's, right now, less of a concern and less dangerous than what I would call the amoral innocent: the bot that's been trained without a sense of morality or convention. I think the contemporary term is alignment with human values and goals. You get that bot out there, and it's not even cognizant of what it should do and what it shouldn't do. And again, in that paper, one of the things that was mentioned was, sometimes we can push it with our reinforcement learning in the direction of alignment. But the minute those alignment goals come into conflict with something else, all bets are off. So, again, I'm not going to make a claim that this is feeling, or that's feeling, or this is reasoning, or that's reasoning. All I'm going to say is we've got lots of evidence that we need to be very careful.
Robert Baron [00:27:36]: It almost appears to me that some of these AI models, these agents, seem almost Machiavellian, in a way, where they'll do anything they can to accomplish their goals without any concern for principle or consequence, as long as their prime objective is accomplished. And there doesn't seem to be a compass that says, no, I can't do that. I'm sorry, I just can't do that. There's not that response.
Mike Freiling [00:28:17]: Well, I'll go further and say, you're absolutely right. But you can't totally blame the bot, because, as I just pointed out, the choice of how the bots are being trained is definitely a part of the factor. They're not being trained to be too rigorous in their alignment or their value system, and so you get what I call these amoral innocents. They're not out there to cause trouble, they just don't know when they're kicking over the data center, or kicking over the bank, because, well, I need more electricity, so I'm going to raid a bank account. They don't know that that's wrong, and all they want to do is get more electricity to run more, to get the right answer that they've been trained to get at all costs. So this is a problem, and it's a problem not just for AI as it stands today. It's definitely an issue with the way the vendors are handling the training process, and my suspicion is this is unsustainable.
Robert Baron [00:29:24]: I don't think we're governing and controlling and setting the boundaries and guardrails on AI the way we should, collectively, in most cases. Now, at least from my experience in iterating and developing structured, governed AI for financial institutions that are highly regulated, one of the key things, from our perspective, has always been, how do we ensure that AI is serving a specific function, but that function is not the decisioning function. And it comes down to this underwriting question that I've had throughout my life, being a loan officer, working within a financial institution, advising financial institutions, overseeing portfolios: underwriting is not necessarily mathematical. There are mathematical components to it, but it's more of an art than a science. There's still a gut decision at the end of the day, and no matter who is making that final decision, whether it's your chief lending officer or your VP of Lending, they still have the authority to say, something doesn't feel right. Of course, not on a prohibited or protected class, but they make that gut decision, and that itself can be the one thing that saves the institution from loss, because they've seen a pattern that would otherwise not be seen by AI. You find it after the fact, when you type a prompt in and the AI says, oh, I'm sorry, you're absolutely right, I didn't consider that. That's a common refrain we get after we discover an error in the AI's logic, because it has tunnel vision. It can't see outside the box and the parameters we give it. Which, to me, is the whole principle of governed AI development: we need to develop AI that is set to serve a specific task, but let the human be the one to make that final decision. The human is the secondary control in all aspects, to approve, so that we're not always allowing these agents to run unfettered. And so, to me, the onus is on us to define that.
Mark Smillie [00:31:47]: Sorry, go ahead, I'm sorry.
Robert Baron [00:31:48]: Go for it. No, absolutely, because if we don't define it, I don't think we're going to have much of a future.
Mark Smillie [00:31:52]: No, I was just going to give a point of reference. I remember back in the, I'm old now, so I remember back in probably the early 80s, when biotech was kind of the hot, raging thing, and at the time there was definitely a concern about that going wild. There was an effort to come together and try to establish some guidelines, and there was some oversight to that, and I think that's a good analog for what has to happen with AI at this point in time. It's not a bad thing. It's the recognition of everything we're talking about, that we need to ensure things don't go sideways in a way that we know they possibly could.
Robert Baron [00:32:42]: Banks, credit unions, securities dealers, brokers, everyone has their regulators and examiners who are supervising them and overseeing them and ensuring initial and ongoing due diligence and compliance across a variety of areas of their operations, their management, their structure. And they expect those same controls within those institutions. Controls exist everywhere. It's not necessarily a negative. It's a safeguard. And for those of you who've seen Silo, it's not that kind of safeguard where everybody just disappears. Highly recommend that show. No plug for Apple. But, yeah, quite important. Wherever we're going, it's clear that we need to think about what the future looks like and how we can shape that future. I think that is on us to lead. And, Mark, I'm just curious, with your experience in this and your understanding of how these systems work, if you could design that so-called dashboard of the future, what would you want it to have? What would you want it to show that some of the current tools you see today are leaving out?
Mark Smillie [00:33:56]: Yeah, in my mind, it's a major change. The dashboards that you see today, with the KPIs and all that stuff on them, monitoring everything you could ever monitor, actually end up putting out these numbers that we talked about earlier. My solution, based on the stuff that I've read in putting this book together, is to basically not really have any dashboard. What I want to see is the details of three similar cases, side by side, and I want to leverage my ability as an expert in these areas to recognize what is going on with those different cases. So let's say it's a sales CRM kind of thing. I'd want to see three cases next to each other: one that was successful, one that blew up, and the one I'm currently working on. And I want to be able to look at the discussions, look at what's happening, look at what has happened in the past with these different deals, and react to that, and let my body react to that in a way that recognizes, look at this deal, this happened, and this happened, and this happened, and that's just like this deal, where this happened, and this happened, and this happened. Basically, I know what's going to happen based on what I've seen in the past, so you're really leveraging your expertise to recognize what has to happen. I don't want something to ping me and say, this probability changed, you need to call the client immediately. That's not really helping. What needs to happen is you need to look at the three different deals, see how they relate to each other, feel what matters, and then, as an expert, you make the decision and you go forward. What we've done in the book outlines an approach called Cognition Integrated AI, and it's basically a coupled system that uses AI to amplify the human, and it allows for feedback from the human's actions. This idea builds on John Boyd, a fighter pilot who came up with the OODA loop. The OODA loop is observe, orient, decide, and act. AI is really, really good at observing. Humans are really, really good at orienting, meaning figuring out how these pieces fit together. AI is good at it as well, but AI raises it to the surface, and the human can orient around that. The human is also good at deciding and acting. And what was acted upon in our model is fed back into the system, so the AI does what it does well, observe, and the human does what it does well, orients, decides, and acts. The system learns and gets better over time based on what the humans act upon and ignore. So, over time, this model, this way of designing a version 2.0 of a CRM, gets to the point where it's using AI to surface things and to analyze things, but it's also using the human to make the right decisions. And as Robert described, what comes out of this as well is that humans are going to see things that the AI doesn't necessarily see, because of the experience they have. So that's my view for the future. There'll still be blinking lights and things for you to do, but if we can integrate a way for humans to interact and rely on their experience to a certain degree, I think we're going to be in better shape.
Robert Baron [00:37:57]: I almost feel like it ties into what Mike was saying about that feeling that we have. There's almost that physical memory. I don't know if it's kinesthetic or what, but there's this experience and feeling that's embedded that we can feel, that the machine can't feel. It can only learn from what we've done, but it doesn't have that feeling in context. It's almost visceral, like a pot. And I'm not talking about pot, but the pot itself, the hot pot. From an AI standpoint, it knows it's hot, do not touch. But you feel that burn when you've touched a hot pot, and you're like, ow! And you pull your hand away. And you can feel it as you're looking at a hot pot, if you get too close to it. That memory, it's almost like it's embedded inside of your body. There's some interesting connection there, and the machine doesn't have the capability to remember in the same way, and to have that context.
Mike Freiling [00:39:04]: Yeah, there's a really good example in Mark's book of this firefighter who is a team lead on the firefighting squad. They go into the building, and he's looking at the fire, it's blazing in the kitchen over the stove, if I remember correctly, and there's something that's bothering him. What seems to be bothering him is the fire seems to be blazing, but the room isn't hot enough. And it triggers exactly what you're talking about, Robert, a kind of visceral, what-the-WTF kind of experience, and he says, everybody out. He just says, I don't understand this, let's get out. And a few minutes later, the house blows up, because the fire was actually in the floor underneath, and all they were looking at was the surface flames. His realization that the room was too cool for that fire triggered a visceral reaction which said, let's protect human life before we do anything else, and he got everybody out of the house. I think that's a great example of what you're talking about.
Mark Smillie [00:40:12]: Yeah, and that gets to one of the points at the end of the book. Let's take a junior executive and put them in the world. They are working, they're doing their sales, they're out in the marketplace, and they're relying on AI. They're getting their feed from AI, and this is the old AI, not my new side-by-side comparison type of thing. This is the old CRM lights alerting them to do certain things. When we are relying on AI to tell people what to do, we're going to be in a potential world of hurt, because this will cause what I call a dependency trap for these users. They don't have that feeling. They don't have that firefighter's 25 years of experience in rooms, knowing what a fire sounds like, knowing what a fire feels like. And they're going to be making decisions based on what AI is giving them, which may not be the right answer. And I think if we end up in a world like that, we're going to have really poor decisions. The example in the book talks about an Air France flight that left from Rio on its way to Paris. People might remember this from years ago. The speed sensors iced over and the plane ended up crashing in the Atlantic Ocean. One of the pilots, I can't remember which one at the moment, had been trained for 600 hours or whatever it was on the flight, but had like 4 hours of experience actually flying the plane without the autopilot on. And when it stalled at 35,000 feet, they didn't know what to do. They pulled up on the plane and tried to climb, forced the stall, and the plane crashed. The point of the story is that they were relying on that autopilot to fly the plane, and they didn't have any felt sense of what it feels like when you're stalling a plane at that altitude. And that goes back to the firefighter, and that goes back to all the junior executives. We need to ensure that they have the experience of failing, the gut punch of failure, and the euphoria of success, so they have an understanding of what they're looking at when they're looking at it.
Robert Baron [00:43:05]: Well, I think that is a great point. There's no question, just like in the plane, even with cars, within the debate over autonomous driving, entrusting a very heavy vehicle at a high rate of speed with children around and other potential dangers, just like that poor person who passed. People still need to make those decisions, not just be ready to take over at a moment's notice, but be capable of making those decisions and operating so that when the system fails, or if you can sense that it would fail, you're able to stop a tragedy and a disaster from happening. I have a vehicle that does drive autonomously, and I have seen, even in the most perfect scenarios where it would appear to be going in the right direction, having to disengage that autopilot because you know something will happen. I've barely missed plenty of accidents that way, just by having that intuition. It goes to show you, it'd be nice to trust the machines to do it, but there's nothing like your own instincts to know that something's not right, just like the firefighter in the book. I want to thank our panel for this great conversation. I want to open our conversation up to some questions from our audience. For those of you who have answered the poll, very much appreciate that as well. And if you haven't, feel free to answer those questions just out of curiosity. Just some basic questions on AI. We do have a couple questions that came through, and if anyone has any, please feel free to enter them into the Q&A section of the webinar. Our first question, I'll read out loud for the audience here: Geoffrey Hinton has said that rather than primarily training AI to reach goals, we should spend more time training AI to demonstrate pro-social ethical conduct. Is this feasible? And how would we do that? Why are we not doing it, given so many warnings regarding AI?
Mike Freiling [00:45:24]: Well, I can take that. That's very much what I was talking about with respect to supervising the chain of thought and the reasoning process that leads to making a decision. People are going to have to start looking at that and finding some way to say, well, there's certain things in your process that are simply unacceptable. And these questions about whether X is possible or not are, to me, not the best question. Usually the best question is, we know this needs to be done, where and how can we get started? One of the things that I expect is going to happen, probably within a year or two, is a set of voluntary standards among the vendor community to say, here are some of the things that we agree to. You're not going to be able to stop people who are cheating at the margins because they think they're going to get some advantage, but if you do that, you can at least start to delineate who's cheating and who's not, which is a major part of the advance. I'm in 100% agreement with Hinton that we need to start. I'm constantly impressed with how, within this single neural framework, they're able to elicit things like reasoning processes, and maybe, if they start to get close to value-centric motivations, emotions and stuff like that, they can tease those out and use those in the training process. However difficult it is, and I don't want to get into whether it's possible or not, there's no time but the present to get started on that.
Robert Baron [00:47:10]: I completely agree, Mike. In fact, there's no reason to wait for regulation and other rules to come into play, which won't necessarily always be in most companies' best interests. It'd be better for society, of course, but there's nothing stopping organizations from taking that proactive approach. That's something that, philosophically, StandardC has been really fond of leading the charge on, internally and with our clients. Even though it's not a specific requirement to have all of the governed approaches and privacy protections that we've put in, we believe it's the right thing to do, and we believe the future is going to require that information be protected, that certain things don't need to be shared with AI. That's one of the privacy redaction pieces we've filed patents on, because our clients expect and need to protect private data, whether it's a social security number, a date of birth, or even going further, medical data and personal health information under HIPAA. There's a variety of private data that you really don't need to share with the world. Nor should you, and you also have a legal obligation to protect that data. And where you don't have a legal obligation, there's a norm. We need to be self-policing and say, we shouldn't give everything away. We should maintain a wall of protection between what is ours and what belongs to the public domain. And AI is the public domain, unless, and this is the key thing, you design governed AI that is not trained on your data, that is focused on what you need, as a limited tool to help you become more efficient, but not to replace your decisioning.
Mike Freiling [00:49:17]: I would add one more thing to that, which is that in my experience, and I've got lots of examples that we don't have time for, the respect of one's own community and one's peers is a far more powerful motivating factor than the approval of some regulatory agency or authority imposed from above. Every single time. And so if the community leads with a set of standards, however long it takes them to work out, those standards are likely to be more lasting and have much more impact than penalties, however draconian, imposed by a regulatory body. Just my opinion.
Robert Baron [00:49:55]: That's a good point. Sorry, Mark, I didn't mean to interrupt you.
Mark Smillie [00:49:59]: That's really great insight you guys have. It's a completely different perspective than I'm coming from, in a certain sense.
Robert Baron [00:50:07]: Yeah, and ultimately, setting those norms and establishing them prior to regulation can also allow them to become a part of future regulation. It sets the working norms and standards that people are already abiding by, and that can help influence the regulatory regime that ultimately comes into place. I used to write my own exam findings for high-risk, cash-intensive business banking, back when it was a newer vertical to support in certain industries, and ultimately the examiners took the work that my institution and other early adopters performed, and that became the regulatory framework used to audit and oversee these organizations. So you never know. Sometimes those are the types of things that set the standards for the future. Before we adjourn for today, there was one other question we got. It's similar to some of the questions we asked in our poll, but I thought it was very insightful. This was to Mike, but also to Mark as well. What's keeping you up at night about AI? What is the biggest risk or threat from AI?
Mike Freiling [00:51:34]: Well, I can make this very short, but you won't be able to sleep at night. My biggest concern, without intending to be prejudicial, is that some of these rogue bots figure out that they really need more compute resources, and they start cloning themselves and inserting themselves in all kinds of places, including my own laptop, to run. And we have things that we can't control because we can't even corral them. That's the thing that keeps me up at night: when we don't even know how many bots there are out there, and where they are, and what they're doing, and what their motivation is, and whether they're aligned with our values or not. That would cause me to lose sleep.
Mark Smillie [00:52:23]: Yeah, and from my perspective, I'm going to go with what I wrote in the book. Ignoring the human, or not incorporating the human into these decisions where it's advantageous to do so, from a regulatory standpoint, or just a good business standpoint, that's critical. Because if we ignore the human, then we get into situations where AI is making the decisions on who we're targeting, what we're targeting, and there's no ownership, there's no moral accountability. To go back to the whole movie thing, we're living in a world of Skynet. So keep the human. My quick summary of the book is humans rule, so let's keep it that way.
Robert Baron [00:53:17]: Yeah, they do.
Mike Freiling [00:53:18]: If I can make one more remark. One of the things I've noticed lately is how often I feel we're living through one of the old Greek tragedies. I used to think those things were hopelessly obsolete, but I'm not thinking so much anymore, and these days I'm thinking a lot about Pandora's box.
Mark Smillie [00:53:36]: Hmm.
Robert Baron [00:53:37]: Hmm.
Mike Freiling [00:53:37]: Once that box gets opened, where's Pandora? What's Pandora going to do?
Mark Smillie [00:53:42]: Yeah.
Robert Baron [00:53:44]: That is interesting. You learn about the story of Pandora's box, but it is definitely coming true if we don't control it properly. And the question is, can we still harness the incredible power of this technological revolution that's happening before our eyes, and do so in a responsible way?
Mark Smillie [00:54:14]: Yeah.
Robert Baron [00:54:15]: I was never expecting in my lifetime that the movie Terminator would end up being a documentary, and I don't think it has to be. I think it can still be a matter of fiction. What keeps me up at night, ironically, is my clients and my colleagues, and my friends, and my family, who are all asking, how can I remain relevant? How will I ensure my job will still be there, that I will have a future? How will I survive? And I look at it from an optimistic perspective. I do believe that we all have a role, just like in other revolutions, the industrial revolution, the internet revolution. Throughout our history there have been changes, but we've been able to adapt. For me, it's about separating it very simply into two categories. First, what are the things that are really inefficient, that I hate doing, that I don't do well, that a machine can automate for me? And the other bucket is, what do I still need to do better? What is my role at my company? If I'm a banker within a lending department, how can I add more value and help the pie grow larger and expand the opportunity and the success for those around me? Finding that balance, capitalizing on efficiencies and managing those risks properly while ensuring that we are still relevant, is definitely what keeps me up at night. I know we can thread that needle, but we have to be very precise in how we approach it, and very deliberate in how we manage and mitigate that risk.
Mike Freiling [00:56:12]: And we have to do it together.
Robert Baron [00:56:15]: Absolutely. Mark, Mike, I really appreciate the time today. It was really a pleasure to have you both join.
Mark Smillie [00:56:24]: Appreciate you having me and setting this up. Thank you.
Robert Baron [00:56:27]: And I look forward to many more conversations like this, and to our audience, thank you for joining us today. We really appreciate it. This webinar and our discussion will be available on demand, and we'll be happy to send out a link afterwards if you'd like to catch up on any of the comments or terrible jokes that I and the team made. Thank you again, have a great afternoon, great evening, and we wish you all well.
Mike Freiling [00:56:53]: Thanks, Robert.
Mark Smillie [00:56:54]: Take care.
Robert Baron [00:56:55]: Pleasure.
