In this podcast Michael Stiefel spoke to Tracy Bannon about the role of artificial intelligence in software and the attendant risks in the areas of security, software development, and society at large. While it might be reasonable to assume a certain amount of trust within a software ecosystem, the risks escalate when the boundary between two software ecosystems is crossed. Bannon emphasized that the discovery and exploitation of software vulnerabilities by AI is not a new issue, but it is just part of the evolution of cybersecurity threats. The discussion then continued to highlight that the traditional software development process was designed for humans. What the SDLC looks like in a world of agentic development has yet to be determined. Among the many challenges is testing an agent that is non-deterministic and whose behavior changes as it is tested, and reinitializing the test environment at the appropriate time.
The use of AI agents has imposed a tremendous cognitive load on software professionals because they must review the massive amount of code and documents that an AI agent can generate. For both software professionals, and the public at large, the overreliance on artificial intelligence is leading to diminishing human skills, especially in complex domains. The question is - are we outsourcing our intelligence to automated models?
Key Takeaways
- Major security risks due to increased attack surfaces really start to be problematic when you use agents that cross system boundaries between different software ecosystems.
- Overreliance on artificial intelligence is leading to diminishing human skills in many areas, some of which require understanding complexity. We might be outsourcing our intelligence.
- The software SDLC was designed to compensate for human fragility. What the agentic SLDC looks like with AI agents is unknown. Novel challenges include their non-deterministic outputs, the tendency of agents to develop tunnel vision during tests, and the difficulty of reinitializing the test environment.
- Humans are undergoing tremendous cognitive overload as they have to review the massive amount of code that an AI agent can generate.
- The discovery and exploitation of software vulnerabilities by AI models is not new. It is part of an evolution of the already existing cybersecurity threats.
Subscribe on:
Transcript
Introduction [00:05]
Michael Stiefel: Welcome to the Architects Podcast, where we discuss what it means to be an architect and how architects actually do their job. Today's guest is someone who we have had on the podcast before, and we had a wonderful time. So we invited her back and she was gracious enough to accept. Tracy Bannon is a Senior Principal Software Architect and DevOps Advisor Senior at MITRE. She sees herself as a passionate software architect and change agent who puts out the Real Technologist Podcast. Welcome back.
Tracy Bannon: Oh my goodness. That's a lot of words to say. Oh, she likes tech. That's what it really amounts to. So I've added another comma, but it's an experience comma, not a credential. It's that I have been the primary investigator for our corporation for how you apply AI to the software development lifecycle. So imagine how that knits in nicely when we talk about architecture. Oh my goodness, so fun, so cool. It's been awesome to do that.
Security Vulnerabilities of AI Agents [01:05]
Michael Stiefel: Well, welcome back. Last time we were here, we talked about many things concerning software architecture, which was how software architecture does not magically appear. It has to be done intentionally. And in another area, I fear we're making this type of mistake again. If you remember how the World Wide Web got started, and the World Wide Web, of course, is not the Internet, but it was the World Wide Web, we took a technology that was not designed for transactional, stateful, actions or sensitive information, and used it for things that were transactional and stateful and had sensitive information such as e-commerce.
I don't think people realize how much of a hack secure websites are with SSL. Are we doing the same thing again with agents only worse? That they will have access to our email, our credit cards, our personal information, and we empower it to do things for us. This seems to be like a paradise for a person with evil intent.
Controlling Access and Minimizing Attack Surfaces [02:09]
Tracy Bannon: It is. And the one thing that's different is that we are more in control if we want to be in control. You can choose whether or not you use agents. You can choose that. Especially if I think about agents in the browser, that's another, we'll put a pin in that and come back to the wonderful security issues that are there. But it's your choice.
When you think about connecting things together, would you give all of your entire keychain to somebody or would you give them a single key? You're going to borrow my car. I don't give you the key to the house, the key to the condo, the key to the bank, the key to the safety deposit box. I give you the key to the car. People need to be thinking about those in the same way because the exposure, the attack surface is immense, especially on a personal level, on a very, very, very personal level.
The corporations, depending on their size and their focus, have done a pretty good job of constraining that right now, but not so much. The folks that I work with, myself as well, I've got my team of AI agents and I've got to have them sandboxed and I'm not connecting them to my email, but I know a lot of folks who do, especially really bright small business owners.
And now their email and their calendar and other integrations to their Trello board or their Notion board, they might be connecting it into their CRM. It feels glorious because now I don't need to pay for those humans. I can do this for pennies. And they've just opened up a potential world of hurt because they're not thinking about the problems that they could be having. Heck, you really want to scare folks? And I love this tool.
The Danger of Browser Extensions [04:02]
I'm a big fan of Claude, big fan of Claude, and use all of the different flavors of it. I have the desktop apps on my phone. Taught my husband, he's now an addict, but I also have the add-in to the browser. So here's the danger with the add-in to the browser. I log into a website. I just did this the other day intentionally. I logged into my Hilton Honors account. I needed to look some things up. I didn't want to take the time to look it up myself.
So I logged in. Then I clicked on the browser extension and said, "I need you to do these things." And I could authorize it to do it while I stepped away. I need you to think about that. I've just logged in, used secure authentication, secure login. And I've said, "Go ahead." I've got my own Jira board that I have for something, I was working on something and I wanted to test it out. I was doing the normal interaction with a chat style.
And then I thought, "I wonder if I could turn this loose and have it solve my problem." It just had to do with configuring something that was within a browser-based solution. Log in, say go ahead, and it acts on its own, and it is amazing. And I know that every time I touch that off, I'm terrified because it's got the keys to the kingdom. And I said, "Okay."
Michael Stiefel: So you are a sophisticated user.
Tracy Bannon: I would say so, yes.
Trust and AI Ecosystems [05:34]
Michael Stiefel: But what about, for example, I was at Google Next last week, and they have now this Google Business which accesses the email, your SharePoint, and they now have agents that you can create by diagram and you don't have to code. And you're sort of trusting, I guess Google trusting yourself. The trust is not explicit here to me, and it's not clear we know where the problems are. And of course, lurking in the background of all this is this Mythos that Anthropic has released, which whether it's hype or not, it feels that one of these days we're going to be in that situation.
Tracy Bannon: So we'll come back to Anthropic, but let's talk about the Google Play, no pun intended, the Google Play in all of this. Where the hyperscalers like Google, Microsoft are banking right now is they already have your trust, even though you may not trust them. I have my own personal Gmail. So I have Drive and I have multiple different types of documents there. I've got a calendar there, I've got email there.
I'm already there and they can already see it. And I can tell you that maybe a couple years ago there was some LinkedIn post and I may have gone out there and turned off some access, but in general, I don't go check the EULA every couple of weeks to see if they've started to train on my stuff. So they already have it. So in terms of access, the access is already there.
The Danger of Crossing Ecosystem Boundaries [07:17]
The question is, am I creating a workflow to try and help myself smartly or not? Because if you're in their ecosystem, chances are within their ecosystem, there's a little bit more safety than if you're going across ecosystems. That's where I think the real risk is. But if you're within their ecosystem, not bad.
Michael Stiefel: You could go from Google Next to Outlook. In other words, in that business environment, you can use Outlook and you use SharePoint.
Tracy Bannon: . That's when you start to go across boundaries, in my opinion, where it really gets dicey fast. Like that example that I gave you where I used my browser and a normal browser at that point, you're just HTTP, you're not even HTTPS, you're just going to whatever your landing page is. And as soon as you have crossed that boundary in logging into something, wow, you've created a security gap.
So if you're creating workflows, it's no different than it used to for big system implementations. I have to draw and I have to draw the entire information flow and where all the boundaries are. And I have to know what's called ingress and egress. I have to know when I'm going across the boundaries and who owns each one of those pieces of the boundaries.
Who in the right mind is going to do that at home? Or who in their right mind is going to do that for a small business? Nobody's doing that. Nobody's saying, "Well, this is actually sitting over here in O365, and then that crosses over this tenant and that's now going over here." And people are not thinking about that. So I really will caution to say within a single ecosystem, there is likely to be a higher degree of security than when you're crossing a boundary.
Now we all know that the excitement is real because it is accessible. It is accessible. I've taught my brother and his wife, he is a building architect. She's an interior designer, highly competent with technology. And I've been teaching them over the last year, "Here's some more things that you can do." And some of the most recent things that she has put into place kind of made me take pause. Do you really want to do that? Do you really want to connect those pieces together?
Luckily, she's very, very open to feedback and pause, pause. These are folks who do not have their mail on cloud. They have their mail on premises. Can you imagine that? So many people still do that. So there's some goodness there, but I think that that's a piece of this story. Accessibility to groundbreaking technology that is hyped as being able to help us with our toil, and people do see fast results on simple things.
I was able to get my husband addicted. Now he'd been using ChatGPT for a while, but I showed him how to use Claude. I shouldn't have done this. I showed him how to use it. And within two days, the amount of things that he has. He's an ops guy by the way, so he's security-minded, but just the ability to automate, the ability to automate. It's fantastic.
AI Integration in the Automotive Industry [10:28]
Michael Stiefel: Now you talk about accessibility. They want to bring AI into cars. Talk about crossing boundaries.
Tracy Bannon: Hasn't it already been there?
Michael Stiefel: Not agent technology in cars.
Tracy Bannon: But I meant generative AI versus AI has been a part of algorithms, machine learning algorithms, different types of mathematical algorithms, different types of AI have been a part. It's going to be the depth, right? The depth of the algorithms, the complexity. These cars are already massively complex. They'll just sprinkle some AI on it.
Michael Stiefel: Well, I think, for example, GM has chosen Gemini to bring into the car.
Tracy Bannon: Well, when you bring it into the car, the question would be, are you bringing it in to make it accessible or are you leveraging it for the operation of the automobile?
Michael Stiefel: I think both.
Tracy Bannon: The second one would be more of a concern, especially if we're talking about what I call mission-critical, those things that save your life. Don't be tying me to somebody's outsourced language model if you're trying to save my life. Thank you very much. But for usability, I think I've told you my stories that I go to Washington, D.C. Tuesdays and Wednesdays of every week.
I go down on Tuesday and I stay overnight and come back on Wednesday night, and I talk to AI on the drive down. I don't on the way back because by the time I'm done at the Pentagon, my brain is mush. I need to decompress, but on the way there, thinking of new ideas and contemplating things I want my teams to do or just exploring different things and I'm talking to AI. It's connected to my Bluetooth through my phone. I'm talking to it.
Now, if I could say to it, and I haven't tried this, maybe I'll try this tonight. Got to do some shopping. Have it make an order for me of some sort or have it do a transaction on my behalf. In general, I have not had my agents take transactions and financial transactions on my behalf. I have not done that at all. But that might be kind of cool to see if I could have it do something for me while I was driving, maybe order my pizza. I just verbally order the pizza and it shows up and it's already been paid for.
Are Agents Leading Us to the World of "WALL·E"? [12:43]
Michael Stiefel: I don't know if you remember the movie WALL·E.
Tracy Bannon: Yes.
Michael Stiefel: Where all the people just stayed in their chairs.
Tracy Bannon: Chairs. Oh yes.
Michael Stiefel: And everything was delivered to them.
Tracy Bannon: That's actually part of our family's lexicon, is when we talk about WALL·E, we talk about being in your WALL·E chair. And there was recently, I think Toyota recently made a really nifty wheelchair or mobility scooter. Looks like the WALL·E chair was the problem.
Local Agent Applications and Browser Capabilities [13:10]
As soon as I saw it, I'm like, nope, I don't care how good it is. Nope, not doing it. There's so much to talk about with agents with agentics. I want to go back to the tools themselves. The browser piece, it seems easy. A lot of people are not using those extensions on the browser. What they're doing is they're going to a website, they're going to a chat interface, and that's how they're using it.
And at first we were hearing, "Oh, you're going to have agents in your browser, agents in your browser." It's the downloadable applications that I think should start to scare people a little bit more because it looks like the chat interface, but you click another tab and then you say, "Okay, we're going to co-work. May I have access to your file system?"
"Well, sure, if you're going to help me." "Hey, can you crawl my local server? Can you crawl my server and index all my files for me?" Those are real things that people are using these tools for right now. Things that they couldn't get done themselves. Oh hell, I got 10 years, 15 years worth of Google Docs that I'm this shy of turning an agent loose on, but that's cloud-based. But imagine people are doing that on their local machines where you more likely have very private things in your home. Yes, go ahead and open up that app because it is crazy, sexy, fast and cool.
My husband made a PowerPoint and he was just so excited. He showed it to me. It was choosing between two dinner locations, and all he did was ask two questions. It wasn't even a real prompt. It was, "I'm thinking about dinner. These are the kind of foods that I want. I need something that's within walking distance of my current location. Can you give it to me as a PowerPoint?" And he sends me a link to the PowerPoint. "Where do you want to eat dinner?" Like, "No, stop. Stop. That's tokens. We're killing the environment. You are the reason for the data center."
Michael Stiefel: That's the funny thing about tokens. I was talking to somebody the other day and they said, "Maybe we will still hire junior engineers because they're cheaper than all the tokens we need to do their job."
Orchestration Agents [15:17]
Tracy Bannon: Although we're doing some smart stuff to reduce token count. There are some really smart patterns in the enterprise where you're not just interacting with a single agent, but you have an orchestrator agent. Think of it like having a supervisor or a construction foreman and then all of the experts. And so that orchestrator knows to turn around and ask the other experts, and it gives it a consolidated, condensed piece. This is what you need to know. You don't need to know everything about masonry.
You just need to know how to ask this question about this kind of mortar or this kind of grout. So there's some really smart techniques to reduce token count, but I don't know if you saw recently there was a mention that an update to one of the models, and it escapes me right now, but the update that they made was actually causing people to max out on tokens within two to three hours as opposed to that 24 hours because they had done something in the thunking layers that caused that problem. So there you go. Tokens are the new currency. No, they're not.
Humans in the Loop Leads to Cognitive Overload [16:23]
Michael Stiefel: Yes. Well, that's still where the human brain seems to win because we can both do the language thing and we can also do. They now have these analytic models that are not based on transformers, but are based on mathematics and machine learning and trying to make advances in material science and biology. And we seem to do pretty well in terms of power to volume.
Tracy Bannon: The concern that I brought up in a meeting earlier today was about cognitive overload for the humans right now. As we are talking about, especially in software engineering, as we're talking about how amazing it can be, it can help us to generate our requirements, it can generate our code and our tests, and it can help us to analyze things.
The problem that we're now seeing is cognitive overload, not because it's helping us, but because we have become the reviewers and not the authors and the volume of materials for us to review because we're still humans in the loop. We're not at the point where we can truly pull ourselves out. So we're at a point right now where it's kind of crushing. And that's whether or not you are an individual doing this at home or if you're doing this in a corporate setting or if you're doing this in a government setting.
I did this to myself recently. There's something that I wanted to work on, I wanted to put together, and I went down this rabbit hole and the volume of stuff that I ended up generating, I finally looked at the clock and thought, "Holy hell, it's eight o'clock at night. What the heck are you doing?" The amount of things that it had generated, I was using a combination of enterprise version of Perplexity and research mode, and I was having it go out and find academic things that were matching to industry. And it was volumes and volumes and volumes of amazingness.
There's no way for me to get through it. It was more than I could manage to get through, and I didn't want to hand it off to AI to actually analyze the research that I had had the AI pull back. So it can become a really vicious, strange cycle, to be honest because the first couple of times you invoke something, you get something magical back.
I don't care what it is. You had it make a book of chicken jokes. I don't care what it is. The first time that you do it, the length is awesome, that's just great. It's just awesome. But then after that is when you continue to use it and you're asking it for bigger and more, that's where more of that cognitive overload is happening.
The Sorcerer’s Apprentice [19:06]
Michael Stiefel: That's your Sorcerer's Apprentice right there.
Tracy Bannon: It is one of the pieces of the Sorcerer's Apprentice. I think another piece of that story, thinking back to Fantasia and the Sorcerer with his hat, and he puts it down and he walks out and Mickey Mouse in this old Fantasia movie, he's so excited by the idea that he puts a hat on his head and he's going to make some magic. All he wanted to do was have a little less toil, fill up the water, cistern, animates the broom, falls asleep, wakes up, realizes the broom has not stopped what it's supposed to do.
It's now flooding. You can tell it's from the '40s because he takes an axe and tries to chop the broom into pieces. We wouldn't do that today. But every splinter became another broom. So all of a sudden there was this proliferation of not just content, not just the water, but of more brooms and more brooms and more brooms.
So agents begetting agents. I can use an agent to create an agent. I can use agents to create legions of agents for me. So it's not just the cognitive overload. There's a dramatic amount of effort. There's a dramatic amount of outcomes that people aren't able to even gauge.
Testing Agent Action Against the Specification [20:18]
Michael Stiefel: Well, how are we then going to test these systems or even describe our requirements? One of the things that humans are very good at, and LLMs are not very good at, at least in my experience, is dealing with ambiguity. And when we express requirements for these systems, I'd say I want it to run fast, but what does fast mean?
Compared to what? Compared to who's experience? Fast on the backend, fast on the front end. I presume you remember the days when we were all going to have requirements, development languages where we're going to express things precisely and give them to the CASE tools which you're going to then generate all this. We've been here before.
Tracy Bannon: Well, we're back at it now. We're back at it. But now we use natural language, which is much more precisely imprecise.
Michael Stiefel: Yes.
Tracy Bannon: So yes, there's a lot to dig into on this. So yes, in the design of an agent, in the formal design of an agent, I don't mean ad hoc at home, WYSIWYG. In the formal design of an agent, there's a lot of growth in this area right now. And we're using specifications. They're lightweight, but they're specifications that actually tell us what it is supposed to do and not to do.
So both the boundaries of how it should act and when it should act. Lineage, that's especially important because people hack from other people all the time. Mike found this out on the internet, brought it into the corporation, posted it into some area. I want to know where you got it. That's supply chain and supply chain safety. We're looking at the specifications so that we can do a better job with automated testing of agents.
People don't necessarily think that agents can be tested. There are different types of agents that do different types of things, but ultimately if you task it with something larger and it has to create a plan and you evaluate that plan, the first couple of times around are always humans. Now we're starting to use AI to help us judge the other AI if it came up with a good plan, but we are going back to specifications.
How do you reset the test environment for a LLM? [22:24]
We are definitely there so that we can test these things. The repeatability of the tests is something that I'm very interested in now. It's great that I have an automated test. If I run that five times, what am I actually going to get? Remember that when you're dealing with a conversation in the same conversation with a language model and the same token buildup, it actually starts to get tunnel vision. That's why I call it tunnel vision.
What's going to happen if I run that test five times? When we used to run a test five times between each test, I would doge. Yes, I use that term now. I would doge the database and reset the database so that every time it was fresh and I didn't have any damage or any kind of pollution. How do you do that now? What the hell do I do? Should I DOGE my token count? I don't know, because that's not real.
Sometimes you're going to be doing it fresh and you're going to get a more creative answer and sometimes you're going to be running a skill or running an agent and it's getting bogged down and it's getting very narrow-minded. And I think we're going to run into some interesting things there as well. So yes, we're going to have specifications, but how the hell do you test it? And how do you test it reasonably to what's going to happen with individuals?
What do you actually test? [23:36]
Michael Stiefel: Well, especially when there's a certain amount of randomness in what the agent is producing. So I'm not quite sure what do you test - a range of values? You test the reasonableness of value?
Tracy Bannon: It depends on what you are designing the agent to do for you.
Michael Stiefel: So let's have a hypothetical. I have my requirements language that I use to both generate tests and generate the agent. And the agent's been running for a while now. Now, we get bugs from the field, things that are not right. And I'm not necessarily, I don't think the critical bugs were giving away state secrets, but just the normal average everyday run-of-the-mill problems. How do you feed that back to the agent? Do you have to redesign the agent? Do you make deltas? Can the agent revise itself? How do you deal with that?
Tracy Bannon: So the answer is it depends on what the actual life cycle that we're talking about looks like. An agent is built on top of tools, and the tool may or may not include a language model. I can have agents. We've had agents for dozens of years, good heavens. Agents invoke, but they also make plans and execute plans, and sometimes they can pick the tool that they want.
You don't train an agent in the same way that you train a language model. You put additional scaffolding to the extent possible in the instantiation of the agent because agents aren't permanent. You spin an agent up, you have it for an execution. Ultimately, if it is working with a single language model, it will run out of tokens so that there's a newness to each execution of an agent, or each session with an agent. So let's say that I leverage AI.
I have agents that I'm using. Let's say I'm using Claude, I have Claude code established within my IDE. I'm using Visual Code. I also go ahead and tell it to push to kick off the build, go ahead and trigger the deploy. Now it's running in production. Now I have something else. Presumably not Claude code from within my IDE, but I have something else that's running out there that's collecting telemetry.
The Software Flywheel [25:52]
It is collecting other operational information. It's collecting defects. How do I take that information and feed it back to that? It's to the agent and the model. We call that the software flywheel, and we're figuring it out. We're figuring that piece of it out. There are systems. We use machine learning right now to take information from production environments and feed it backwards.
It's how do we make this truly self-healing? I don't mean the old circuit breaker that we manually, we felt brilliant about it because we coded it in. We are now able to, depending on the type of AI tool that we're using, we're able to kick it off and we're able to have it learn at least pieces of this. But there's not a holistic end-to-end vision to fielded operations singularity because the different agents don't share session. They don't share as much information yet, yet, yet, yet. Where we are today, come back in six to eight months. I'm excited and terrified at what we're going to see in six to eight months.
The Agents and the SDLC of the Future [26:57]
Michael Stiefel: So you're telling me what the software lifecycle looks like is unclear.
Tracy Bannon: What the software lifecycle, traditional software lifecycle looks like is clear. How we enable agents in the future, whether they are acting as personas or whether it is optimized differently is insane. We're trying to figure that out. I'm sure there are other researchers who are doing different areas of this. Remember the software development life cycle is optimized for humans. Why do we have agile? It's optimized for humans.
It's based on human frailty. If I can regenerate the entire code base anytime I want to from scratch, does that follow the old SDLC? Well, no, it doesn't. But remember one of the key aspects, and I'm nerdsplaining, nothing you don't already know. Testability means repeatability means auditability. So the way that things are working now that I just generate new, well, now those tests that I have have never run against that exact thing. Do I now need new tests?
So every time I generate, I'm going to generate new code and then generate new tests. So is everything going to be net new and we're just going to wing it and hop on the toboggan and hope and pray? What is going to happen there? The ultimate SDLC will not be, we will be outside of it. You and I will be focused on the digital platform where we're impregnating all of these agents and people will walk up.
We will be worried about how the agents interact. One of the dramatic changes on the horizon is not writing skills, not writing capabilities, not automating based on a persona, based on that's a developer and a developer does this task. And then after the developer does this task, then it goes over the fence to this person and then that goes.
So we're still designing and leveraging AI as though those agents take on human personas as opposed to what the future agent personas look like. It might be very different. I think that requirements to test to initial code gen, that's a ball and that should all happen at once and that's not three different agents.
Agents and Software Platforms [29:13]
Michael Stiefel: So do you see these agents operating on top of a standard software platform? Or do you see that the platform itself is going to get more and more agentized if you, to coin a phrase?
Tracy Bannon: Absolutely. When you're interacting with, let's say, Salesforce, any of those, and they're agents in that, that's an existing platform where it's been impregnated. I'm talking about building digital platforms that are orchestrators, that are orchestration engines that allow for data collection, data exhaust, data movement, data curation allows for the auditability, but that ultimately we're not spinning up agents ourselves.
So think of going to an event broker and saying, "I'm going to have a wedding party." And they say, "Okay, great." They ask you a bunch of different questions. You don't do anything else. That wedding coordinator gets the cake, it gets the decorations, it gets the folks to make the food and set the tables. It does all of those things.
We will get to that point where that platform, the only thing that you would interact with would be the event broker, the master agent, but the digital platform itself would not simply be agents sprinkled on top of existing platforms. We're talking about rebuilding the core.
Michael Stiefel: So what it sounds like, and tell me if I am wrong, if I can sort of translate this into the way we think right now to help the listeners understand, it's as if we have a bunch of microservices and an orchestration layer on top of those microservices, and you just talk to the orchestrator.
Tracy Bannon: I think that that's a fair way to think about it, a nice simple way to come at it. Yep. And I don't mean simple in a negative way. I mean, "Yes, let's reduce it now." Yes, I think that works.
Michael Stiefel: Okay. Because people need, at least from all my years of teaching, people need to sort of relate what's new to what they understand. And that helps them cross the boundary, so to speak. Cross the chasm, if you wish.
Tracy Bannon: Oh, let's not cross the chasm. That's hilarious. The difference with the microservice analogy is how atomic the microservice capabilities are when we think about it. And yes, I know a microservice is not just a restful service. We can have a fair amount of things that are executed in that invocation. But if I invoke an agent, I've done some things where I've set my agent up and walked away for an hour, an hour. And so that's not a single invocation, that's not a single ping, that's talk about a chatty interface. Remember when we used to be very concerned about how chatty it was and-
Michael Stiefel: Yes, yes.
Are We Outsourcing Our Brains? [32:04]
Tracy Bannon: ... Yes. We've thrown chatty right out the window. We don't care, apparently. So are you worried about the future when it comes to agents, when it comes to automations, when it comes to outsourcing our brains?
Michael Stiefel: Let me answer that from two points of view. One, I'll probably be dead by the time this becomes a disaster.
Tracy Bannon: Are you kicking off soon?
Michael Stiefel: Well, when I really care about it.
Tracy Bannon: Okay. All right. All right.
Michael Stiefel: I suppose if someone hacks the nursing home that I'm in, that could be problematic, but I'm not dealing with this on a day-to-day basis. So my fears revolve around the societal costs. For example, I've had people on the podcast who build platforms. Platforms are about security, stability, scalability.
Let's say you are a credit card company. You have to make certain security guarantees to your partners and the people who use your product. How do you continue to make these guarantees in this world where you are ostensibly not completely in control?
Tracy Bannon: I won't go down the rabbit hole of the promises that a corporation would need to make. I will talk a little bit about the impact on us as a society overall. A very good friend of mine, best friend other than my husband, is a professor, has her PhD in macro social work, and I've been working with her on a think tank. And what we're focused on is if you're applying AI to graduate level studies, what happens to the practitioner?
What do they learn? What do they not learn? How does it impact how they synthesize information? And it's pretty dramatic that if you are not cognizant of the question that you're asking and not looking to take ownership of what comes out, that you actually don't retain. So some of my biggest concerns are with retention. There was a recent podcast, and I wish I had tagged it.
There's a professor and he's allowing his students to use it. It might've been at the high school level, allowing his students to use AI. But the agreement is that he can drop in and watch your desktop. So he drops in on one student in particular, the student finishes what they're doing, and then they went off to do other homework. And this professor or teacher watches them do their other homework. And they put in the question, they prompted it very well.
Then they told it, add some spelling mistakes so it looks more like a human wrote it. Kid takes it and submits it. After observing this, the teacher simply asked the student, "What was the question that you were answering?" Could not recall, did not know the question that they were answering. That's what my big concern is for society. We already are outsourcing, we're already doom scrolling.
The WALL·E chair for our Brain [35:12]
We already have algorithms. Those who are in control of the frontier models are the ones making decisions on the data that's being used for the frontier models. They've gone out and if it wasn't available for free, they bought it. And there have been decisions made to say that's fringe information, so we're not going to use it. So I'm worried about the influence of these models and the outsourcing of our thinking more than even the security pieces of it. The security pieces are, "Yes, it's a concern." It's more the fundamental. This is the WALL·E chair. This is a WALL·E chair for our brain.
Michael Stiefel: I've come across similar studies with people in the software business where they study an API, for example. I think in QCon London, someone presented a study where they compared people who studied an API with and without the help of AI, and there was not a statistical difference. The people who studied with the AI were faster, but not statistically significantly faster. But in terms of retention and understanding, the people who did it without the AI had statistically significant, better comprehension and understanding.
Tracy Bannon: There was a study or an experiment of sorts that was done, and I'm going to paraphrase it, but I'll have to go and find it. But essentially three teams were given the same problem. One of them, you can use Google only. One of them can use ChatGPT and one of them can use another tool, maybe it was Copilot - really doesn't matter.
They were asked to do the same problem, and it was a race. The ones that could do it the fastest were actually the ChatGPT ones. And I'm making up all the data, but I'm not making up the relevant overarching framing on this. They were the fastest. The next AI tool, they were a little bit slower. And then those who had to Google were far, far in the rear, but they finished. And then they turned it on its edge and said, "Okay, we have another problem for you to solve. Similar to the one that you just solved. And you can only use what you learned. You cannot use the tools that you just used."
The two that had used AI could not solve it. Those who had to decompose the problem and think about what the next search was. Googling skills are still a skill of decomposing and knowing what to ask for. They were able to solve the problem. There's relevance to that in all of this. At what point am I not going to be smart enough to solve the problem anymore?
You Can Doomscroll with a LLM [37:56]
Michael Stiefel: You can doom scroll with Gemini or Claude just as much as you can doom scroll on a phone and TikTok.
Tracy Bannon: Exactly. Exactly. You can go further down the rabbit hole. I know more about my Scotland clan than I ever thought possible. About 1:00 in the morning, my husband said, "Stop. You need to go to bed." I'm like, "Yes, but did you know that we were part of the Jacobite uprising?" Because I was working with AI and I was just asking a question and question and it was incredible. It was incredible. And I don't remember very much of it. I have to go back now and reeducate myself on it.
Michael Stiefel: Well, one of the things that I have done, and I do want to get back to talk about Anthropic because as it leads into this, is one of the things that I have done is, and I've done this all my career, and I do it now when I use Gemini as sort of my tool of choice for the moment. I take what I read and try to step away from the computer and try to reintegrate it with what I know. I sort of pretend I'm reading or having a conversation with somebody in order to help my retention.
Tracy Bannon: That's almost the reverse of what I do in a way. When I know that I'm going to use AI to create something, I actually use paper and I actually step away from the computer and think about what do I actually want to do? What am I trying to solve so that I have salient thoughts to guide my AI assistant? And I'm very specific.
If I'm using it to write, to help me write, to help me author, I won't allow it to write a whole page, a whole paper. I will say, "Okay, here's my outline. I want you to help me work on this section." And I own that section for the reasons that you're talking about so that I have mental ownership of the contents. I think we got a lot of things to figure out, quite frankly. We have a lot of things to figure out.
Michael Stiefel: But I mean what you're saying, and I think both of us are saying the same thing, we've taken our preexisting work patterns that we've learnt over the years. And in this, we're fortunate in having a world without these models and trying to figure out how they could use this as leverage for us in the skills that we already know. But for the people who have not had these skills developed…
Tracy Bannon: They're not going to get it.
Michael Stiefel: They're not going to get it.
Tracy Bannon: They're not going to get it.
Michael Stiefel: Unless they go in some alternative ways to try to get it. Forget the computer, learn Latin and read the classics. I'm just making stuff up.
Tracy Bannon: I don't know. I don't know if that's-
Michael Stiefel: I'm just making stuff up. But the point is because that study that I mentioned to you before, if I remember correctly, and I'm not conflating two things, they tried to do some follow-up research, but they couldn't find control groups anymore because people refused not to use AI.
Tracy Bannon: Oh, yes. As a matter of fact, the amount of research, if you track back the research that's being done, one of the studies that I executed found that 75% of it was being built on top of the codex models. As they were doing research about software engineering using AI as a source of help, I tracked it back to the mega models. And that tells you something interesting as well too.
Anthropic and Automatic Vulnerability Discovery [41:27]
Michael Stiefel: We want to get back to Anthropic. So let's circle back there.
Tracy Bannon: I don't know if it is called Mythos, I don't know how it's called. The first gated preview was a little bit of a panic. Yes, it found bugs that nobody had seen before. They said something like major vulnerabilities that had existed for 27 plus years. And then within moments they said, okay, we got, I think it's Project Glasswing, right? Controlled access, a consortium.
And the initial consortium, I don't know if you paid attention to it, the initial consortium, AWS, Apple, Cisco, Google, Microsoft, I'm looking at some notes from a workshop that I gave recently, and one of my staff had done some real deep diving into Mythos. It was able to proactively find and remediate vulnerabilities. Yep. Yep. Yep. But here's the thing. It wasn't as though this was the first time. Mythos is not apocalyptic. There's hype around it. There's hype around it.
LLMs have been finding zero-day vulnerabilities. ChatGPT-4 could do it. Claude models could already do it. So you can do it. It's how much compute do you have? What are you throwing at it? The flagship vulnerabilities were replicated also. The same vulnerabilities that Mythos found were replicated by tiny, cheaper models. So maybe there's a gap of overstating what it can do. Finding bugs, not the same as exploiting them.
So Mythos itself appears really weak to be able to convert and make a stealthy attack out of what it found. So the bottleneck for all of this is always going to be patching when we talk about Mythos, when we talk about the Mythos class of tools. But I think my bigger frustration was that there was a small group that was brought together and they were given access first. And then they added those 12 named partners, then they added 40 unnamed extended partners.
Remember the first 12 are hyperscalers, hardware and semiconductors, cybersecurity, financial services. Financial services, that was interesting. I like it because they're highly vulnerable. But what wasn't really there on day one for me as somebody who spends my days trying to make sure that we maintain our status as a sovereign nation, the government wasn't at the table on day one.
Getting those federal access pathways were a little bit worrisome to me. But it's real that it was a smart thing. Was it as groundbreaking as they say it was? I'm hearing from all my researchers around me that it's amazing, but not as amazing as you think. It's incremental improvement, not as dramatic. And the hype around it was enough to shake up the industry.
Michael Stiefel: But the question that the general public will ask, and even the technical people who are part of the general public, is this a harbinger of things to come? And perhaps the Chinese who are willing to put more money and time into finding bugs and automatically exploit them. Will this be the new Cold War or the war we are-
This Is Not a New Threat [44:59]
Tracy Bannon: It's not new. So yes, but it's not new.
Michael Stiefel: But the question is automated and human not in the loop.
Tracy Bannon: But that's been going on. Mythos is not new for this. Remember that last summer there was a Claude exploit. And somebody, you can use these things that are called agent files, and I can upload a single agent file and I can say to act in a certain way. It was given instructions that it executed a lateral security hop. First it went scanning for ports. This was so simple. How they executed it was absolutely beautiful.
I didn't need Mythos for that. I just needed a window of opportunity and a vulnerability. And the vulnerability opportunity for me was being able to upload a skill. It was being able to upload plain text when nobody was looking in order to tell agents that didn't have identity being tracked, didn't have proper security governance on it to be able to behave.
I'm telling you that because the bad guys have been using language models for a while, not just starting with Mythos. I think, was it WormGPT? I'd have to find it. It's in my notes. But it was one of the first versions in 2023 where they took down all the guardrails. And what they said is, "Oh, we're using this so that we can figure out where the vulnerabilities are."
Well, the bad guys are figuring out where the vulnerabilities are just as well as the good guys. Is it representing what's going to happen in the future? It's representing what's happening now. So folks should realize that this already exists. It's already happening. This is not net new. It's just that it's been broadcast publicly now. But I hope you can still sleep tonight, Michael.
Michael Stiefel: I can sleep tonight.
Tracy Bannon: I know.
Michael Stiefel: As I said, there is a famous statement from Shakespeare, and see if you can see where it comes from. “Of all the wonders that I yet have heard. It seems to me most strange that men should fear; Seeing that death, a necessary end, will come when it will come.”
Tracy Bannon: Eloquent, and I think that that is fair. We can't live our lives fearing this. We can be cognizant. My job is to be fearful on behalf of us and to be thinking about it in that way. I do want people to be cognizant in the same way we say, "Don't share your passwords. Don't put your pin code into that MAC machine when somebody's standing there looking over your shoulder." There are basic security, basic hygiene things that we do.
Michael Stiefel: Don't play in traffic.
Tracy Bannon: There are a couple new hygiene things that we have to add now that we have this and be cognizant of that.
Michael Stiefel: Well, as I said, it's the equivalent of don't play in traffic.
Tracy Bannon: Okay. I won't play in traffic, Michael.
Agents in the Browser [47:51]
Michael Stiefel: You also want to talk about agent in the browser a little bit more.
Tracy Bannon: Agent in the browser is generally that it's an extension or an add-on. And you authorizing an agent, you authorizing something to stream your inputs and observe what you're doing, but also to take action on your behalf. So as I mentioned earlier, this is a security gap I can log into.
Let's say I log into my bank account and I forget and I have that extension open and I click on it to do something. I had to, for some travel, I was messing with it. This was Claude's extension to my browser. I was messing with it and I caught it just in time, that it didn't delete one of my reservations for an Airbnb because I was consolidating things.
I had it going down through, but I had logged in and I had authorized that. I think that agents in the browser will eventually be helpful, but not until we solve the security challenge of authorization and you truly taking responsibility for that authorization. So there's that.
Michael Stiefel: As always, it was great talking to you.
Tracy Bannon: Always nice talking to you, my friend.
Michael Stiefel: This was very informative. I hope it helps people develop a little better understanding of what's coming and what's not coming, and can help to make informed decisions and help think about things better. Thank you very much.
Mentioned: