E062 - AI or Not - David Timm and Pamela Isom

Welcome to "AI or Not," the podcast where we explore the intersection of digital transformation and real-world wisdom, hosted by the accomplished Pamela Isom. With over 25 years of experience guiding leaders in corporate, public, and private sectors, Pamela, the CEO and Founder of IsAdvice & Consulting LLC, is a veteran in successfully navigating the complex realms of artificial intelligence, innovation, cyber issues, governance, data management, and ethical decision-making.

A bid protest can look like a fight over a contract award, but it’s also one of the few tools that keeps federal procurement transparent, competitive, and less vulnerable to bad decisions. We sit down with David Timm, a claims and bid protest partner at Burr & Forman, to unpack how protests work at the agency level, the GAO, and the Court of Federal Claims and why the smartest “win” often happens before proposals are even due.

We get practical about the pre-award phase: using RFIs and solicitation Q&A to resolve ambiguities, challenge overly restrictive requirements, and raise set-aside concerns like the rule of two. David explains why early engagement can save money, reduce contention, and improve your chances of award, especially if you’re a small business navigating the FAR and tight proposal timelines.

Then we tackle the fast-growing collision between government contracting and generative AI. David shares what he’s seeing in procurement tribunals: hallucinated case law, misstated holdings, and filings that trigger warnings or sanctions, including monetary penalties and dismissed cases. We also explore a new frontier for proposal strategy, including solicitations that disclose AI-assisted evaluation, what contractors should ask about which large language model is used, and why AI summaries can create compliance failures through the “lost in the middle” problem.

We close with two tools you can use immediately: submit early and confirm receipt to avoid “late is late” landmines, and build a simple risk-versus-importance matrix to decide when AI is worth it. Where does AI help your proposals most, and where is it too risky? 

For more information, please see this resource:

https://www.burr.com/government-contracting/gen-ai-misuse-in-procurement-litigation

Transcript

[00:00] Pamela Isom: This podcast is for informational purposes only.

[00:26] Personal views and opinions expressed by our podcast guests are their own and not legal advice,

[00:34] neither health, tax, nor professional nor official statements by their organizations.

[00:42] Guest views may not be those of the host.

[00:50] Hello and welcome to AI or not, the podcast where business leaders from around the globe share. Wisdom and insights are needed right now to address issues and guide success in your artificial intelligence and your digital transformation journey at large.

[01:08] I am Pamela Isom and I am your podcast host.

[01:13] So we have a special guest with us today,

[01:17] David Timm.

[01:19] So interestingly,

[01:21] David is a claim and bid protest partner at Burr and Foreman and he will explain more about that. But David, welcome to AI Or Not.

[01:32] David Timm: Thank you so much for having me, Pamela. It's a pleasure to be here.

[01:36] Pamela Isom: Thank you for joining me. Now, we met as we were collaborating on AI and learning pathways for government contractors. So I'm so glad that we had the opportunity to meet.

[01:48] So, to start this conversation,

[01:50] let's have you tell me more about yourself and that journey that you've traveled and how your journey is shaping where you are today.

[01:59] David Timm: Yeah, thanks.

[02:00] So I went to the George Washington University Law School,

[02:04] and you may or may not know, it has a very good public procurement program, a very good government contracts program.

[02:11] And so right from the get go, I remember sitting in the room in orientation and the dean telling the group of us that only about 20% of us would become litigators.

[02:24] And I didn't grow up knowing any lawyers so personally. So I thought to myself at the time, well, I mean, what are the rest of them doing if they're not.

[02:33] If they're not litigating?

[02:35] And I quickly learned that there are lots of other ways that lawyers can help people or do work that's not specifically doing disputes. But I always personally wanted to do sort of the more prototypical litigation.

[02:49] And so naturally at gw, because the procurement program was so good,

[02:54] I got really interested in bid protests and public procurement and claims and navigating the Federal acquisition regulation.

[03:03] And I had the pleasure of working at the Transportation Security Administration.

[03:08] I was an intern there for a year during law school.

[03:11] Shortly after that, my next summer was spent at my first firm as a summer associate where all I did there for the majority of of my career was government contracts litigation.

[03:23] And then about a year ago, myself and a number of others came over to Burr Informan and we just have continued our practice the same as we've been doing this whole time.

[03:36] So I really love in particular bid protests. I think they're extremely interesting and I know there's. They sometimes get a bad rap. You think bid protest, you think vexatious incumbent.

[03:47] But I think really the bid protests are very interesting from an overall systemic perspective because the public procurement situation is that the government is the one who is in charge of evaluating bids.

[04:04] And so it doesn't really make sense for them to judge themselves.

[04:09] Right.

[04:10] And so what you need is there's a case that discusses this, but you need a bunch of different private attorneys general to bring claims on behalf of the procurement system overall to avoid things like corruption, to avoid the government paying really high prices for goods and services.

[04:28] And the way that that works, it's sort of this self interested system where contractors who didn't get the award but see some problem that has real effect on the procurement system,

[04:42] they are allowed to protest, they go to the agency level, they go to the Government Accountability Office, or they go to the Court of Federal Claims and they say, hey, something about this process was wrong, maybe we should have received the award.

[04:55] Or maybe there was something about the way that the solicitation was drafted that unfairly excluded us and reduced competition.

[05:02] And while there certainly are frivolous bid protests and we don't want too many bid protests to slow down the system,

[05:10] it's always good to remember that they play a very important part in, in keeping the overall procurement system clean and functioning well. Because even the threat of protests, the fact that someone could protest even if they don't,

[05:25] is something that keeps our government officials thinking. How do we make sure that this is all performed above board,

[05:34] that we're transparent with the contractors and with industry?

[05:37] So that's a little rant on bid protests. I love thinking about them from,

[05:42] you know, not only my client's perspective, but also from the overall procurement system perspective.

[05:47] Pamela Isom: And where are you headed?

[05:49] David Timm: I think this is what I want to do for the rest of my career. I want to help contractors. You know, this is a very difficult area to be in industry.

[06:00] There are lots of regulations, there are very competitive procurements,

[06:05] and there's lots of considerations that, and choices that you have to make that make a difference to the bottom line of the companies that I represent,

[06:15] many of which are small businesses,

[06:18] women owned, minority owned,

[06:20] service disabled, veteran owned small businesses.

[06:23] And I think that it's really important that they have someone to come alongside them and it doesn't have to be me. There are plenty of very well qualified lawyers to do the same thing as me,

[06:33] to advise them. I've seen this before and I think that you should take this path because of my experience and because of different changes in the law or things that have affected how the courts are resolving things.

[06:47] And so I like to stay on top of the latest cases.

[06:51] And I tend to write a lot about where litigation is at the Court of Federal Claims and at the Government Accountability Office and what is really pressing and impactful on contractors today.

[07:05] Pamela Isom: So I am glad to hear you say what you're saying because we all need to know where those resources are and especially in this particular area, because it's complex.

[07:17] And so I appreciate you sharing that insight.

[07:21] So tell me more about bid protesting and how should contractors work in the pre bid process,

[07:30] especially to make sure that the solicitation requirements are met.

[07:35] David Timm: Yeah,

[07:36] so I think when you think of a bid protest, typically I think your mind goes to there's been an award of the contract and it's gone to someone.

[07:45] And the protester in that scenario, the post award scenario, is saying, hey, I should have gotten the award, not that other guy.

[07:54] And so one of the things I like to tell contractors is you don't want to be in that situation where you're trying to claw your way back to the award.

[08:04] Of course, sometimes it happens and I like those protests. They're interesting and useful and sometimes necessary.

[08:10] But I think there's a lot more room for contractors to take control of the process early, and that's to work through the RFI process, the request for information and the Q and A question and answer process as soon as the solicitation has been drafted, or even earlier, if you can get in the ear of the agency to help the scope of work,

[08:33] if it's going to look something like what your company does to try and get into that process to shape it.

[08:39] Similarly, if you're reading the solicitation and you're seeing things in there that are confusing or that don't make sense,

[08:47] you're considering how do we bid this? This is ambiguous.

[08:50] This requirement is difficult for anyone to meet, let alone us.

[08:55] Or maybe it's set aside for only specific businesses and you don't think it should be.

[09:01] Or maybe it's not set aside and there are multiple small businesses, the rule of two, for instance,

[09:07] that would be able to bid on it.

[09:08] All of these different considerations happen before the contract is ever awarded. And there are a whole a litany of different types of bid protests that you can file at that stage and actions you can take even before filing a bid protest that are much cheaper and in some cases even more effective to try and limit your risk when you make the bid to make sure that you're going to meet the requirements,

[09:35] to make sure that you are going to have a chance at award ultimately. I mean, that's the thing the contractors care about the most. And the great thing about the pre bid protest is that compared to the post award protest,

[09:51] pre bid,

[09:53] there's not a winner announced yet. Right.

[09:55] So in a lot of cases, the post award process is very contentious. And what that means is that there's two people that you need to fight against to get a win in the protest.

[10:08] The winner of the contract, or if it's a multiple award contract, potentially multiple winners,

[10:13] as well as the government who is likely to try and defend their award decision.

[10:18] So in the, in the pre award context,

[10:21] you're just trying to help shape the solicitation to make it fair for everyone.

[10:26] And in fact, other contractors might agree with the questions that you're raising and the government might agree and then they might make the changes without even having to file a protest.

[10:36] So I think it's really important to be very engaged before the solicitation. But definitely as soon as the solicitation is available,

[10:43] you need to early on think about if there are any problems there that you can potentially work on to solve with the government. And it's everyone's sort of coming together shoulder to shoulder, hey, let's make this a little bit clearer.

[10:57] Let's make sure that this is properly set aside and that can make for a happier,

[11:02] more competitive and better procurement overall.

[11:06] Pamela Isom: How do we do that? How would you go about getting involved in bid protesting at that early stage?

[11:13] David Timm: Yeah, so I think it's really important to talk to your proposal folks about hey, what are you seeing in this solicitation that might be a problem for us?

[11:25] And then I hate to make it really general, but I mean, we could talk about specific situations. Like a very simple one is we want to bid on something, but we're not able to for whatever reason the solicitation is set aside or it's not set aside and,

[11:41] or there are requirements in the solicitation that are overly restrictive of competition.

[11:47] And so we think about these things, these obstacles or these ambiguities in the solicitation. We talk to our proposal folks, we talk to the folks who are doing the work,

[11:57] who are going to be doing the work if we win the contract and we say, hey, what would help us get a, a better chance of winning this? And then we take the intelligence that they give us their expertise and we want to craft usually either requests for information or Q and A Questions for the government to answer through the Q and A process.

[12:17] And what you'll find, I mean, I see lots and lots of solicitations.

[12:21] What you'll find is the government often,

[12:24] even though they've issued a solicitation, they're often very willing to make amendments to the solicitation to change things.

[12:31] And so it's sort of an iterative process,

[12:34] but clarifying things at the earliest possible stage is optimal. And then of course, if the government doesn't agree, but it's a deal breaker for you, then you can consider escalating and moving it into the protest arena.

[12:49] Or maybe you say, yeah, we actually do have to talk to a lawyer, or you can protest yourself pro se in some cases and, and figure out where we're going to take it.

[12:59] Are we going to go to the agency level? Are we going to go to the Government Accountability Office or to the Court of Federal Claims? And that's a whole other discussion.

[13:07] But I think hopefully that answers your question there.

[13:10] Pamela Isom: Yeah, that answered the question. I appreciate that.

[13:13] So. And we'll get into why that's important when you answer this next question.

[13:17] So let's talk about your research on generative AI misuse and the procurement tribunals.

[13:25] David Timm: Yeah,

[13:26] so as I mentioned before, I really like to stay on top of the latest cases.

[13:31] And I started noticing,

[13:33] I mean, this has been a couple years,

[13:35] but there are folks out there who are putting together databases about what are called hallucinations or plausible errors in filings at different courts across the world. And there's global hallucination database maintained by a friend of mine named Damien Charlatan.

[13:50] And there's like 1500 or 1600 cases in there at this point.

[13:55] Well, that was very interesting to me. And I started to see it happening in the places where I practice, right at the Court of Federal Claims,

[14:04] at the Boards of Contract Appeals, at the Government Accountability Office,

[14:08] at the Small Business Administration's Office of Hearings and Appeals, and at the Federal Circuit.

[14:14] And it was last year that we saw the first case at one of those procurement tribunals that specifically said, hey, this person made a filing and there were cases in there that were fake.

[14:27] They were either made up or they were misstated or they said that they stood for a holding that wasn't actually part of the case.

[14:36] And all of these are very likely the results of someone relying on artificial intelligence,

[14:43] generative artificial intelligence, like a large language model to draft either their bid, protest or their claim or whatever procurement related filing that they're making.

[14:54] So I got really interested in this And I started tracking it and it seemed like something that the industry was very interested in. And so I kept posting and writing about it.

[15:04] And ultimately at the beginning of this year, I wrote a report about all of the cases that I found that had genai misuse, as I call it,

[15:14] in them from 2025, and I've continued to track in 2026. Unfortunately, we're seeing more this year than we saw last year. I think partly because people are using artificial intelligence more, but partly because the procurement tribunals themselves are,

[15:33] they're more aware of the issue and they're more willing now to,

[15:38] to say we're not just going to warn the filer that there was a problem, but we're going to potentially sanction them. That can be a monetary penalty.

[15:47] They could strike the filing itself or they could completely dismiss the case without even touching the merits. So there are very serious consequences if you have these errors in your filings.

[15:59] So I think it's a very interesting issue and that's why I've gotten really involved in it.

[16:05] Pamela Isom: So they're not saying you can't use AI, but they're telling you to check your work?

[16:10] David Timm: Yeah, I mean, everyone is always responsible for whatever they file.

[16:14] So none of the procurement tribunals have said,

[16:17] at least as a broad rule, that you're not allowed to use artificial intelligence in your filings.

[16:23] For instance, the Armed Services Board of Contract Appeals on their website, they say,

[16:29] basically a warning. They say, hey,

[16:32] large language models are capable of this sort of error where it'll make something up completely or it'll twist the words or it'll say that there's a quote in there that doesn't exist.

[16:45] These sorts of errors are possible from generative artificial intelligence. And while we don't ban the use,

[16:53] we want to make sure that you understand that you're responsible for any of those sorts of errors that end up in your brief.

[17:01] And so you know, the Armed Services Board of Contract Appeals has been particularly aggressive in sanctioning and in warning contractors.

[17:09] Government Accountability Office has as well.

[17:12] All of the procurement tribunals so far, except for the Office of Hearings and Appeals have issued at least some warning and most of them have issued some sanction.

[17:23] So yeah, there's it's really, it's not a ban at all. And listen, I'm the co chair of my firm's Artificial Intelligence Committee and I use artificial intelligence in my own practice to a certain extent for safe use cases,

[17:38] which I don't think research is a very safe use case, at least not if you're going to take the research and put it right into your brief without checking it.

[17:46] The solution is, is very simple. I mean, sometimes people respond to my posts on LinkedIn and they say, well, what if we did this technical thing? What, what if we had the LLM check its own work?

[17:59] What if this. What if we did that?

[18:01] And my point of view on this is that what if we just read the case?

[18:06] Yeah,

[18:07] if we read the case and then we said, okay, either it exists and what the generative AI came up with is accurate or not,

[18:17] this would solve 100% of the issues.

[18:20] By the same token, though, I want to point out it's a lot of pro se filers that have this issue. So they're filing without the help of an attorney. But this year there have been a number of lawyers and law firms at the procurement tribunals that have been cited for the same exact issues.

[18:38] And these are not lazy or stupid people.

[18:41] The types of errors that large language models have are very plausible because remember, what it's doing from a technical point of view is it's creating a likely sequence of words.

[18:54] And that means that when you read it,

[18:57] even if it's a fake case, unless you're really digging into it, unless you're going and reading the case and making sure that what it said is precisely correct,

[19:05] it's very easy for your eyes to skim over it because it looks right. It's not like a typical human error where if a human makes.

[19:14] First of all, humans don't really invent cases completely.

[19:18] Maybe they'll think of the wrong one or they'll have a typo in the name of the case or the, or the case number.

[19:25] But typically they don't just make it up completely. So it's a different sort of error. And I don't really.

[19:31] While I think that no one should have these errors, I want to point out bid, protest, process very fast, lots of deadlines coming at you, flying at you, and people are under time pressure, lawyers and pro se filers, contractors.

[19:47] And it's completely understandable that in some cases they get up against the timeline and they don't have enough time to check the case.

[19:54] But if that this can happen ahead of time and you put in controls to make sure that you're being careful and you're leaving yourself enough time, I think you can really reduce these issues almost completely.

[20:07] Pamela Isom: We are always up against time,

[20:11] especially when you're dealing with government contracts. There's a lot of things that happen that are unexpected.

[20:17] I do like something that you said in that they will listen to feedback and they will make adjustments or they will.

[20:29] I've been in situations where I've seen and I don't know if they use AI to help come up with this or not, but I've been in situations where I have seen government take the feedback and then have a special session to address that feedback that they were getting.

[20:47] So I think one thing that we should be mindful of is if we feel inundated or if we feel overwhelmed or confused by something,

[20:55] please speak up. Because they,

[20:58] my experience has been that they will listen, but you might think that they won't because there's like, oh, it's, it can be annoying sometimes. Right? Because there's so much material to read and preview,

[21:08] which is the temptation to lean on the AI.

[21:12] Right. That's where that temptation comes.

[21:14] So thank you for sharing that research that you have done and some of those outcomes.

[21:20] So tell me, how is government using AI for evaluating bids? Or are they?

[21:26] David Timm: Yes, they are. If you go to sam.gov today,

[21:30] checked yesterday morning, there were three solicitations that were active that I immediately found and I know of others either that have already closed or that are forthcoming or are out there.

[21:43] Otherwise,

[21:44] where the government specifically says in the solicitation we are going to use large language models or generative AI tools in the evaluation process of bids.

[21:56] And so my research has really focused on gen AI misuse,

[22:03] typically by contractors in the litigation process.

[22:06] But I've expanded because I think now contractors really need to know how to deal with the possibility and the likelihood that the government is using generative AI in reviewing proposals.

[22:21] I think there's a lot of reasons to be a bit nervous here as contractors,

[22:26] but I don't want you to be scared. I think that if you understand how these things work,

[22:32] you can actually in some cases use that information to your advantage. So I'll give you an example.

[22:38] Earlier we were talking about pre bid requests for information.

[22:44] I think that every contractor now in every single situation where the government hasn't already disclosed that it's using generative AI in the evaluation process,

[22:55] you should ask at least two questions before you bid on a solicitation.

[23:00] The first one is will the agency use generative artificial intelligence to evaluate bids?

[23:06] And the second one follows from the first and it says, if so,

[23:11] which model will the government be using?

[23:15] This is where I talk about a little bit of a potential competitive advantage.

[23:20] So we know from very interesting research on hiring data that large language models prefer the output of the same model.

[23:30] So if you have ChatGPT creating output that is part of a proposal,

[23:37] and then ChatGPT is the same model that's reviewing that proposal and evaluating it. We know from these interesting studies in a slightly adjacent field that ChatGPT will grade its own output in the proposals higher than others,

[23:56] both or other types of models. So, for instance, if you were using Claude and the government is using ChatGPT,

[24:05] but another competitor is using ChatGPT,

[24:08] you might be at a slight competitive disadvantage. And we don't know precisely how government is using large language models in evaluation. But why would you take the risk if you've got the access to both of these tools,

[24:22] or if you can get access for a short period of time just for drafting the proposal? Now, I will say I do caution contractors in drafting,

[24:32] especially when it comes to compliance with solicitation requirements using large language models. And that's for the same reason that we have issues in the litigation space with hallucinations and other things.

[24:45] But there's another problem as well that I talk about a lot, and it's called the lost in the middle problem. And it's, it's called that because when large language models evaluate their input, so let's say it's a proposal or a solicitation,

[25:02] they privilege and are more accurate on the beginning and end of the document or the beginning and the end of the input.

[25:11] So there's some important stuff there in the middle, but it's going to, it's going to perform worse in terms of accuracy and recall on the middle of the document, or if it's a large set of data, it'll do better on the first few documents and better on the last few,

[25:28] but worse on the ones in the middle. So we know this because of there's extensive academic research on it.

[25:35] So when I tell folks to be cautious with drafting with artificial intelligence, your proposal, we know that you can be eliminated for failing to comply with a material requirement of the solicitation.

[25:50] So you're looking at section M and you ask the artificial intelligence tool to shred section M so that you what all the requirements are and it creates a summary,

[26:01] and then you use that summary to then draft the proposal. Well, if it misses one of those material requirements, then all of a sudden when the evaluator is looking at it,

[26:12] they see that you didn't hit that requirement and they eliminate you without even looking closer.

[26:17] So I do want to caution on drafting using large language models. From that perspective, I would always be verifying myself by reading the document.

[26:27] So to make sure that you're not missing anything that's really, really important.

[26:33] Pamela Isom: So thank you for those insights.

[26:35] And so to follow up on that, since I can see that you have such a passion for bidding and you like to talk about protesters and claims and really how.

[26:47] How we can keep ourselves in a good place,

[26:52] can you tell me more about what we could do to stay compliant when it comes to regulations? And just what are some regulations that we should be mindful of?

[27:06] David Timm: Well, there's a lot of them, unfortunately,

[27:10] and small businesses have trouble, I think,

[27:13] because there's so many regulations for them to stay in touch with. So one thing that I know, everybody is thinking about, the revolutionary FAR overhaul.

[27:23] And so I've written a little bit about the RFO and how that's changed things for contractors. One of the, you know, big general directions for the RFO is to move in a way that gives the contracting officers more discretion.

[27:39] But not every change in the RFO does sort of what the.

[27:45] What the makers of it are hoping, which is to simplify the regulations, make fewer of them so that the contractors have fewer things to worry about.

[27:55] In some cases, it can actually make things much more complicated. And so I wrote this article earlier this year, and it was called the Latest Late Rule and the Hydra of the FAR Overhaul.

[28:11] And so I use this analogy to the mythical hydra from Greek mythology, where if you cut off one head of this beast,

[28:20] multiple will spring up in its place. And so I think a lot of the ways that the RFO has proceeded has followed this track where they think that they're cutting away rules that are unnecessary.

[28:36] And what they're really doing is they're removing rules that have arisen for a reason,

[28:43] rules that have helped clarify ambiguities in statutes or that have come about because the FAR drafters saw that there was a lot of litigation on a particular topic,

[28:56] and they wanted to address it to make it. It's simpler for contractors.

[29:02] So it's sort of this counterintuitive thing, like we've reduced the word count of the Federal acquisition regulations,

[29:09] but we've actually made it more complicated, and that's going to create more uncertainty, which can lead to additional litigation.

[29:17] The big example that I give and that I talk about for that particular article is this rule that's been around for decades. It's called the late is late rule.

[29:27] And basically it means if you've submitted your bid and it's received after the deadline,

[29:35] late is late. You're not getting in. It doesn't matter what the circumstances are. But there are actually a Couple of exceptions to the latest late rule in the Federal Acquisition regulations.

[29:45] And one of those is the government control exception. And so basically, in short,

[29:51] it means if the government has received the proposal and it has it in its control before the deadline,

[30:00] then it's the bid is on time. And this usually takes place in an electronic email filing where the bidder has filed and sent their proposal and the government server has accepted it.

[30:17] But then there's a technical error, but it's on the government side, right? So maybe their spam filter kicked out the email and it never arrived in the Contracting Officer's inbox.

[30:29] Well, if the latest late rule did not have any exceptions, that wouldn't be on time. But because this government control exception exists,

[30:38] it is on time. And there's lots of litigation about this. There's usually a few bid protest cases every year.

[30:45] And so the FAR overhaul, in my view, had a perfect opportunity to resolve what a split that arose between two of the main bid protest forums that hear this issue.

[30:58] The Government Accountability Office says that the government control exception doesn't work.

[31:04] The rule is the bid is late,

[31:06] and the Court of Federal Claims comes to the opposite conclusion, which is more favorable to contractors.

[31:13] So this split exists.

[31:15] They're both interpreting the same rule and the same exception.

[31:19] And the FAR overall could have easily resolved this. They could have said, we agree it should be the way that GAO thinks about it, or we agree with the Court of Federal Claims.

[31:30] But instead, what they did is they created different rules for different types of procurements. They left the government control exception for FAR Part 15 procurements, but for commercial procurements, they said now it's just fully within the discretion of the Contracting Officer.

[31:49] So I say all this because there's constant updates and changes and things to stay abreast of. And I think it's really important.

[31:57] The takeaway here is not really about the latest late rule. It's that you need to be following folks who are very closely and well in tune with the changes of in procurement and in the regulations.

[32:11] And when you're following those people, they're going to highlight for you things that you never thought of and never will think of. And sometimes you'll get information more than what you asked for, just like I've given you more information than you wanted on the latest late rule.

[32:25] But you may learn something that you can apply to your own best practices.

[32:30] For instance, I always talk about best practices for the latest late rule, which is very simple to submit the day before,

[32:37] before 5pm if you do that, there's another exception that will mean that it will always be on time. And if you do that, and it's always your practice to do it that way, you'll never have to do any litigation focused on whether you have to go to the Court of Federal Claims or to the GAO to win your case.

[32:56] Pamela Isom: I don't like waiting until the last minute to do anything. I always feel like the system could go berserk or get overloaded or inundated with requests. So I don't like to wait till the last day.

[33:08] But sometimes you have to.

[33:10] And so this is good insights because if you can set a goal to get it in the day prior by 5pm you said we're in kind of a safe zone.

[33:20] Right? I didn't know that. I just do it because I just don't like to wait till the last minute, and I just know anything can go wrong. But I didn't know that.

[33:29] So that's good insight.

[33:30] David Timm: Right. And I also recommend that you contact,

[33:33] either call or email the contracting officer and say, hey, I just want to make sure you received my bid. If it didn't get to your inbox, we sent it. So we'll figure something else out.

[33:44] Maybe hand delivery or get a courier to make sure that it's on time.

[33:49] But you want to have a backup plan as well, and you want to get some sort of confirmation from the

[33:53] Pamela Isom: contracting officer, and that keeps us compliant with the FAR or the overhaul. Is there any other regulations that you want us to be mindful of?

[34:04] David Timm: I mean, there's so many. Everyone is talking about CMMC right now.

[34:08] Pamela Isom: Oh, that's true.

[34:09] David Timm: I think that's really interesting and important.

[34:12] It's not a huge area of my practice, but I think this is one of those things where you need to start trying to figure out whether the contracts that you're on have anything to do with CUI documents or data.

[34:26] And if they do, then I think you need to don't reach out to me because I'm not an expert in helping you get certified, but there are plenty of those out there who can help you.

[34:36] And I would just encourage you to make sure that you're reaching out to a company that's reputable, that has lots of good recommendations. Maybe you talk to someone else who's gotten certified and you ask them, like, what's the ballpark cost here?

[34:49] Because I know there are a lot of people out there trying to take advantage of scared contractors and charge them way more than what it would actually cost or what they might actually need for their particular situation.

[35:03] In Regard to cmmc.

[35:05] Pamela Isom: That's a good point.

[35:06] That's an area that I can help provide expertise.

[35:10] So. But it is true. There are vulnerabilities everywhere and there are people there to take advantage of that.

[35:17] We are to the place where we are about to wrap up.

[35:21] One last question I have for you before I ask my final question is what's your take on Frontier models?

[35:28] David Timm: Yeah, I use all the Frontier models. I use, I use Claude, I use ChatGPT and I use Gemini. I think that it's easy to get overwhelmed and to focus on the marketing and the benchmarks.

[35:39] And you know, they just released Mythos recently or they didn't release it publicly, but they announced it.

[35:46] And it's easy to get very scared about the capabilities of the models. Are these coming for our jobs? And I want to, I think I want to try and reassure folks about trying to see through the hype and understand how these things work from a technical perspective.

[36:03] And one of the ways that I like to do that is I like to talk about how we're getting these improvements in the models.

[36:10] So there's basically three ways that we've gotten improvements on performance capabilities. And the first one we've seen,

[36:19] it's basically, as far as we can tell,

[36:22] maxed out. And that's basically increasing the size of how much information all of the data on all of the Internet and how that's compressed in the pre training process.

[36:33] That scaling and performance gains really sort of ended in 2024.

[36:40] We are not getting huge gains from making the models much larger. And there really isn't much more data for them to ingest, except for synthetic data which can cause something called model collapse, which the Frontier companies are very anxious to avoid.

[36:56] That's sort of the snake eating its tail problem.

[36:59] So the second way that we've gotten performance gains for frontier models is through reinforcement learning. And that's where you take for instance, in the coding space, we take the answers that ChatGPT or Claude is giving us and we have reviewers who have technical expertise and they say, I like this answer better than that answer.

[37:23] And so we're going to tweak the system so that we get more of this answer and not the other answer. And that's called reinforcement learning. And we have seen significant gains in the performance of the models thanks to this process.

[37:36] But it's important to remember that this is a very, very,

[37:41] very expensive process and they have to do it every single time they train a new model and they have to have humans do the reinforcement learning.

[37:50] Right?

[37:51] So this is very expensive. It can't go on like this forever.

[37:55] And they can only do it for things that are in that common use case area.

[38:01] So for instance, if you are a government contractor, I don't think there are people in OpenAI and in Anthropic who are sitting down and thinking, how do we do optimal reinforcement learning for people on the federal acquisition regulations?

[38:20] It's just not quite a big enough market. And they're more focused at the moment on coding use cases because they tend to be tech people from Silicon Valley and they're very interested in that area of improvement.

[38:33] And so when we look at the benchmarks for the frontier models, putting aside the fact that a lot of these are just marketing fluff, the places where we see the most improvement from those first two things, pre training and reinforcement learning,

[38:47] are with regard to coding things like legal, things like in other domains,

[38:53] there's not nearly as much or similar pace of improvement.

[38:57] I just want to help people understand that,

[39:01] that through this process we're not,

[39:04] the companies aren't really claiming that we're 12 to 18 months away from artificial general intelligence anymore. I think that scaling thesis has sort of petered out and people no longer believe it.

[39:16] I never really bought it myself to begin with, but I do want to just give people a little bit of reassurance. These tools are very good and they're getting better,

[39:25] but when it comes to your specific use cases, real world stuff, not just benchmarks,

[39:30] there's always going to be a gap between what they can tweak in terms of the model's pre training and reinforcement learning and what you have to solve and what you have to figure out on your particular use case.

[39:43] And that's why I always preach that you need to be very careful about what you're relying on from the models.

[39:49] Pamela Isom: I agree,

[39:50] like a thousand percent.

[39:52] Okay, so my last question for you is,

[39:55] do you have last minute words of insight or words of wisdom or call to action?

[40:04] David Timm: I do.

[40:05] So I mentioned before,

[40:07] I use large language models in my practice to a certain extent.

[40:12] And the way that I figure out whether I'm going to use a large language model for a particular use case is I have in my head,

[40:21] and I've written it out as well,

[40:23] a matrix where on one axis is importance and on the other axis is risk.

[40:30] So you have sort of four quadrants. You have in the top right quadrant high risk and high importance, and in the bottom left you have low risk and low importance.

[40:41] And then you need to figure out where your use case falls on the high risk and high importance matrix.

[40:49] So, for instance, I am a lawyer, I represent clients in front of courts and tribunals.

[40:55] If most of my work is very high risk and very high importance,

[41:00] if I mess up, it could be my law license or it could be a loss for a client and that could have huge reputational impact.

[41:08] So that's sort of high risk and high importance.

[41:11] On the converse, I always tell people when a coworker texts me and they say, hey, I want to get a drink after work. Are you free?

[41:20] Sometimes I'll go, but sometimes I don't feel like going and I don't want to make up an excuse. So I'll go to the large language model and say, hey, I don't really want to go to get drinks with this coworker, come up with something plausible that's nice and lets them down easy and tells them,

[41:36] next time we'll do it. And ChatGPT or Claude will come up with something that is very nice and very, you know, well thought out. And that's a low risk and a low importance use case.

[41:47] And there are lots of things in between.

[41:50] Maybe it's a low risk, but it's a high importance task. For instance, in the legal area,

[41:57] you know, doing research at the beginning of a case to just figure out what's out there, like what exists in the world of possibilities. And then you start going and clicking more closely on the source material and you rely on that rather than what the large language model said.

[42:13] I think that's pretty high importance and low risk.

[42:16] So my advice to contractors is to create. Your work is not the same as mine.

[42:21] So you should create your own risk and importance matrix. You should test your use cases and figure out where they fall.

[42:28] And then on top of that,

[42:31] what do you need to make sure that the risk is.

[42:34] Is worth the reward?

[42:36] The reward is worth the risk, right? If you're not going to actually save very much time because you have to spend,

[42:43] let's say it's summarizing the section M from a solicitation and you ask it to shred section M so that you have all the compliance in a summarized file,

[42:54] but then you have to read section M anyway to make sure that the lost in the middle problem or hallucination didn't occur,

[43:00] that's going to cause you problems in your proposal.

[43:04] Well, did you actually save any time by using it? To summarize there,

[43:08] if the reward is not worth that high risk, then maybe don't even use there at all. In other cases, maybe there is an roi,

[43:17] but you need specific safeguards in place. Somebody has to do a final check, they have to read the case before we submit a citation, things like that.

[43:26] So I just encourage contractors to use their own use cases in that risk importance matrix, and I think that will help them when figuring out how to use it.

[43:36] Pamela Isom: That was so helpful.

[43:38] David Timm: I'm glad.

[43:39] Pamela Isom: Like, we, like, do it mentally,

[43:42] but you never really sketch it out, right? And so, and if you sketch it out, what you're saying is, now you have something you can refer back to, a repeatable process that you can use.

[43:51] It makes a whole lot of sense. I really, really appreciate that insight.

[43:55] Those are words of wisdom and a call to action.

[43:59] So thank you very much.

[44:01] All right, so it's been a pleasure having you.

[44:03] I'm so glad that you were able to make time for me. Thank you so much for being on AI or Not, a podcast where business leaders from around the globe share wisdom and insights that are needed right now to address issues and guided success in that artificial intelligence and digital transformation journey.