What was discussed
We sat down with Affaf Ahtisham, Head of Engineering at LaunchGood, to get into what changes when agents write most of your code.
LaunchGood is opening production up to non-engineers, which means more PRs, bigger diffs, and code review that can’t keep up. The conversation goes into what the software lifecycle becomes on the other side of that shift:
- Why production signals end up being your best reviewers
- Where a human still needs to stay in the loop, and where they should step back
- How to sell tech-debt work to a C-suite that just wants to go faster
- Agentic QA, the ROI question every engineering leader is now fielding, and honest predictions on where review and validation go next
If you’re figuring out how to ship fast (without shipping chaos), check it out.
Watch the full session below.
Transcript
Shahram (00:00)
Really excited because after someone we’ve been speaking to for a long time, I’ll let him introduce himself, but just really quickly he adds up engineering at LaunchGood. They’re doing a lot of great work. We’ll hear more about that. But today the discussion is really about just how the SDLC is changing really quickly and how can we be a little bit more proactive in how we detect problems. And honestly, like there’s so much advice being thrown around on how to deal with AI and things like that. I think the biggest mistake is to sort of assume certainty. I love conversations like this because we actually talk about on the ground kind of issues, what we’re seeing, what we’re not. So I’m really excited to have Afa on and I’m looking forward to a great conversation.
Affaf (00:44)
For having me guys, I head up engineering at LaunchGood. Previously, I’ve led the Wolve group at Afghie Corp as well. One of the interesting things that LaunchGood has been that we’ve actually been empowering non-engineers to ship as well. So the problem that we’ve been trying to struggle with is we’re shipping a lot. What do we do to kind of match? And what was interesting last week was that the change verification feature that Clary came out with was quite interesting on like how it’s actually you can call it babysitting, but it’s like actually watching the release coming out, right? And one of the ones that is kind of — take a little look at screen here quickly.
Shahram (01:18)
Just talk us through what is this LG deploy off of? I’m assuming you’ve got some channel up there, all right? What does it do? How do releases work?
Affaf (01:26)
Yeah, so we just have our own release bot that we use to kind of just throw out releases into Slack as just working through the GitHub action. So it’s very much deterministic, going through all the steps of like, okay, build the images, deploy the services, deploy the UI. That was really about it. What was interesting or kind of the gap here was that what is happening after each of these steps? This is very much like Booleans, right? Like I’ve done this, now next step, I’ve done this. If there’s no errors, then just move on to the next in the pipeline for the releases, right? So this is one example.
Actually, one of the non-engineers, someone on the DevOps infrastructure team, kind of picked it up to ship it out. So it was kind of interesting where they got the release ready, shipped it out. And then what was interesting was that no one really needed to go to the actual production site and see that, okay, did it change anything? And this was very interesting. Like we enabled this feature, I believe midweek last week and started to kind of play around with it.
And this is a nice example where it’s kind of like shepherding what’s happened after the release, right? So previously it kind of stopped for us with like, okay, something is in production or development wherever the release was going. And now we’re just purely relying on SLOs, we’re relying on logs coming through, like more of our telemetry data essentially. But that was being more reactive, right? Something happened, then we would go back and really dive deeper into it. But what was missing was like, okay, we’re sure that what has shipped out has actually released or…
So, one of the interesting things is that in engineering, the senior engineer persona, it really looks into like, okay, I’ve done my work, I’ve released it. Now I do smoke tests, right? Smoke test the product environment, right? Make sure that what I shipped out actually shipped out. And that’s a very common feel for anyone that has quote unquote like productive paranoia, just looking at making sure that things are coming out. And what was interesting here was that we didn’t need a human per se to do that anymore. We saw the verification sort of kicking in of like, okay, the release hasn’t come.
Some of our pipelines take quite a bit of time to actually even release 10-15 minutes at times, depending on how many services are involved. And in this particular case, like that’s a lot of wastage of like someone trying to babysit a pipeline. But in this particular case, it was interesting where it went through, looked at the change that was there, looking to identify the bug and make sure that it’s actually seen the decrease of the particular sections that were impacted. So we see some of this come in over here. And then
Affaf (03:52)
Kind of like just keep on repeating itself as well a little bit until it’s all the changes are fully hydrated through. So it kept on going through the process and once it was like, okay, this release is actually serving real traffic now. There’s no hydration errors. And what was interesting here was that it was able to identify the previously there was a five percent error rate, now there’s none, but at the same time looking at the noise as well. So in a sense of like, okay, this release is er there’s still some errors here. So we have some other issues that are regular simmering issues are there, but that’s background noise, not necessarily impact.
So being able to distinguish that what is this particular release responsible for and being able to monitor that into production in the past, like this would be more or less an engineer doing it. But increasingly we’re seeing more and more genetic workflows coming and that can enable engineers to really focus on the actual real problems and not necessarily being like the glorified like looking up logs and and whatnot. Right. So that’s of like one of the interesting things here.
Shahram (04:50)
What makes you skeptical or uncomfortable? Because this is a new thing, right? I think it’s an age old question of as engineers, we like to be in control and we like to know exactly what’s going on. So as agents are taking on more of these tasks, as an engineer, how does that hit you and what does idea look like? I feel free to just completely trash what you saw. We’re all trying to learn and build the best thing, right?
Affaf (05:13)
Yeah, totally. Obviously this is their early days, but what I would love to kind of see more of is linking some of the other related issues that might be in Lineai. This might be our setup. We’re still kind of configuring what are the right permissive tool calls to be made. I know we’ve gone back and forth a little bit on like what about data, right? So a lot of bugs or issues or verification sometimes of attributes, not just the actual code that’s shipped or changed, but for a lot of these services also along the lines of the actual data that can compound those types of problems. So how do we do that in a safe…
Launch good specifically, we are within the fintech space, so we have to be very careful with the data that we’re obviously exposing to other systems and services. So we just can’t nilly-willy just open everything and say, hey, AI go wild. That would be very irresponsible for us. So for us, it’s like we want to trial and make sure that we’re working and building that operational comfort and excellence in terms of these new tools and ways of working, right? So I think one of the biggest aspects is that how do we safely introduce data without exposing it to other third-party services? How do we ensure that we’re compliant with the governance controls that we already have in place? I think those are all problems that we’re trying to figure out and how best to kinda go about it and navigate it. But it’s definitely one where we wanna explore further but have to do it in mind that’s safe essentially.
Shahram (06:27)
Makes sense. I think it would be helpful just to take a step back and go back in time a little bit. I think it would be great if you could maybe introduce yourself in terms of your background with Ashikop before you join LaunchCood. From our very first conversation, I got the sense I was like, this guy is AI pilled. He’s super in AI, he’s constantly learning new stuff. Talk us through that journey. What built up your engineering fundamentals and like what got you interested in this space and what keeps you curious?
Affaf (06:53)
Yeah, so I mean what’s interesting here, so a little bit about myself. Like I’ve been in the industry for at this point, twenty years, not launch kit for well over two years now. Previously I was leading Hashi Corp Vault to both managed community edition and the enterprise edition as well. And that was a very interesting space because Vault had a huge footprint. Software verification, what we typically call kiway, and quality of software was such an important aspect of it. It was also very difficult. So I left Hashi at the end of twenty twenty four.
And the world was just starting to change with ChatGPT 3 coming out, 3.5 the year before that. It was interesting where it’s like things are changing. We don’t know exactly what’s changing, but something feels different. I think for me as an engineer, I’ve also gone through and making peace with it as well. I think I read this revelation last year around summertime of like, what is my value as an engineer? Honestly, it was like, okay, if I write code, the value could be questioned. Problems and the business problems that are interesting and actually necessary for the world.
Willem (07:47)
Solid.
Affaf (07:54)
In different areas, and honestly, this is where it’s like we can almost say maybe it is pretty mature, but it’s almost a golden age of that because now we can really solve on real problems and not the ones that about writing code per se. Code generators have been there for ages now. We’ve had border plate code being generated, we have different languages tackling this problem. So these silver bullets have always been there. But what’s been interesting is that that’s become, can become more accessible as well.
So my approach with the eyes are this is something that is definitely changing the world around us. We don’t exactly know where. And this is even last year, which is like, okay, what am I doing with my life at this point? But even now, looking at it, if we’re solving business problems, that’s really going to enable us to move farther, much faster as well. And LaunchCube was quite interesting in the space that at face value is a crowdfunding platform of commerce. Like, how difficult can it be? But because of the regulatory and zone that we reside in is actually very complex.
We have multiple avenues that we need to be protecting the platform against, whether it’s regulatory, financial, technical, there’s different avenues of it. And AI was one of those things where the entire leadership team and many others that we saw as potentially an opportunity, but many others as well, that hey, this could be challenging. So it’s been more or less about like how can we pragmatically introduce this. So last year we slowly introduced different ways, first with incident management, then slowly picked it up. More and more code generation in this past Ramadan.
So Ramadan for us, we essentially see 10 months worth of activity in a single market. So it gets extremely busy very quickly. In this past Ramadan, well over 90% of the code was being committed by one of our company goals for this year. Is that 100% of the staff will be shipping something to production? That includes bizdev representatives. We’re talking about customer support. And these are not just simple like CSS or HTML copy changes, what we would love to see are actually logical changes, things that happen on that backlog that have been burning a hole into people’s minds like, why can’t we just do this?
Well, now you have tools to do that. But in engineering, for my role for it specifically, the other problem is I don’t want sub ones, I don’t want my engineers waking up at two AM in the morning trying to solve that bug that got shipped accidentally. So I think software verification and the quality controls of that is becoming increasingly important. Everyone’s building a coding agent at this point. Like I think the the bar is
Affaf (10:14)
So low at this. And we’ve talked about this in the past that every other vendor about for the past year and a half that we’ve been working with, everyone has a coding agent. Now whether it makes sense or not, they have a coding agent. Even Linear just recently this year introduced their coding agent as well. So it’s all these like our interesting spaces. So the problem is like okay, everyone’s shipping code. You have so many different agents able to do that. But how do you know that you’ve actually shipped the right thing? And that’s both on the principal side but also on the side as well and that’s a problem that’s quite intriguing right now, at least on a personal level. So
Willem (10:50)
So before where are you going in terms of the you have the everybody in the company is gonna be shipping code at least some level of logical change, not superficial, but there’s kind of getting to that.
Shahram (10:51)
Currently land.
Shahram (11:00)
Stages together and I can infer what you need to put in place. How deep in that journey are you? You probably want to have some parts in
Willem (11:03)
Because some base in place already. For example, maybe some secure sandboxing or places for things to execute. So maybe if you can just like a high level of where you are on the journey.
Shahram (11:15)
You don’t give
Affaf (11:22)
We’re definitely on the earlier start of our journey, essentially. We just started a fiscal year about three months ago at this point, actually. So we’re just wrapping up, we’re getting close to wrapping up the first quarter. The approach for us was that let’s do crawl architect. That’s the scenario that we’ve been talking about. And for us, what is very much low-hanging fruit is within like an EPD triangle for example, or Trio, right? And that functional group that we’ve always worked with in, let’s get the designers and the PMs first shipping code, being able to do that like first because they understand some of the tooling, some of the constraints of the systems, get that going.
So we’re just transitioning through that phase right now. There’s like cursor and Slack, for example, that triggers it. The engineers are watching it as well and being like, okay, how do we tweak the context that we’re giving the agents? How do we work with creating those dev environments that previously you would actually either have these you know QA environments that are maintaining these sandbox environments or you’re doing locally with the testing verification. But now how do you that for individuals that don’t have that setup, essentially?
So a lot of that has been going into setting up those toolings, the right context in place, guardrails. It’s been interesting. I talked to some of my peers across other companies and whatnot. And it’s been quite intriguing to talk about that, hey, you know, all the technical debt that we’ve been like shoving under the rug for years with the end to end testing speeds, all of that. Well, guess what? If your leadership wants being fully AI pilled, those are the guardrails that we desperately need now to actually enable AI to perform in a secure, in an efficient manner as well.
Otherwise you’re just like token vaccine or just burning through your budgets at that point because you have no way to ensure quality or maintain that quality control for your organization. So I think for engineering leaders it’s becoming an interesting lever that a lot of the things that we have sometimes kind of pushed aside, we have a new opportunity to bring that back into conversation because if you want to go fast, we need to pay off some of that debt that we’ve been shying away from.
Shahram (13:17)
Has that changed the allocation of how you put engineers behind problems? Has it gone to forty percent of time now spent on paying down tech debt? If it’s not eighty twenty like what’s that number to you?
Affaf (13:30)
It’s work in progress. We’re definitely trying to get more of that focus in and trying to prioritize our roadmaps to include that work as we right now. We essentially establish like a AI task force within engineering as well that is really looking at picking up those very integral pieces that are the multipliers for the organization, right? So one the key things that we did was how do we build these highly scalable QA environments and cluster pipelines in a manner that is sustainable? We have one legacy code base, one modern microservices oriented code base.
So we have a lot of interesting technical choices over the years that we need to kind of wrangle together and build that through. So for us it’s been going through that journey, okay, can we get everything going in one area for an agent to work with? Then comes the next piece of improving our end-to-end test suites. So we were sending the right signals back to the user and making sure that they’re getting the right signals back from their agent. And then from there there’s a lot of training that goes behind it as well.
I think one thing that’s interesting is that we’ve really found the benefit of the multiplayer aspect of agents, essentially. There was that mindset of having an agent that’s restricted to a certain channel, for example, or one to one DM, whether it’s on slide or the web space as well. What we find really interesting is the multiplayer impact of agents that do support that. We saw the adoption with AI increase tremendously. So I kind of joke around that are possible, right? There’s many developers with years of experience behind, only done things a certain way.
And how do we as leaders now help them understand this new way of doing things? Yeah, you have YouTube videos, you have content coming out, but if you’re not actually able to showcase that real examples, it’s a struggle. We saw that with Clarent originally, it was just a few of the individuals doing it, but then as soon as we saw publicly folks doing at Clarent, for example, and even like at Cursor, whatever, whichever agent that we have going at that time, we saw other people looking like, huh, I can do that with that agent. Okay, cool. Now then we would see more of that happening, right? So that multipler aspect has been one of the key growths essentially in terms of helping people adopt and move faster with that, more than any training center essentially.
Shahram (15:39)
I guess it begs the question on what the end goal is, right? Because we started off with this idea that we’re trying to get everybody involved to write code that goes to production. Why is that the goal? Before, it used to be if I’m a designer, I do design work. If I’m a back end engineer, I just stick to back end. And I’m hearing this a lot, right? Like I as a product manager I’m shipping front end code. I’m still I’m unclear. It’s like, you know, just because something can be done does not mean it should be done. I’m curious on your take on why that is better.
Affaf (16:08)
Yeah, I think it’s an interesting proposition for us, right? Within engineering and within I think generally any organization in twenty twenty six at this point is that previously there’s been hard constraints, whether it’s budgetary, whether it’s skill sets of like items in the backlog. They’re just sitting there collecting dust. Or if you’re on a call with the customer, they have this interesting idea and now you’re stuck with like, okay, do I action this by deprioritizing something? Or does this go into the backlog to collect dust now?
And with the way that this new tooling is coming around, think there’s opportunity here where we’ve seen our experiences as well where someone has a problem. I’ll give you an example a couple of weeks ago there was a conversation of like, hey, should we do this experience with a tool tip or a drawer on the mobile, right? Previously this would be a bunch of UX studies I go through, maybe some user interviews with those design mock-ups and whatnot. And it’s like, well, let’s just draw up two PRs. Well, here you go. Which one do you like better, right? Based on experiences and we had some tweaks and stuff. But I think that’s the opportunity that it provides now.
How does this translate to business value? I think that’s the interesting question. And I think we’re still working through that as well. For us, I think it’s how do we get more capabilities or capacity for our product org, right? So today we’re about 25 or so individuals within our product group, and that’s limited by our budget with various different aspects of it. How do we enable the entire organization to not be the builders, right? The concept of a builder is very much. And I think that’s the part that’s interesting, is that hey, that support person, they get an interesting question, or at any point that keeps happening, like being able to rename your credit cards on our platform.
That’s a feature that will never be able to get very high in the I score, for example, if you’re going to a traditional prioritization framework. But why can’t that CX team member or the customer support member just use AI to help build that small feature out, right? Very small, minimal feature, but has a lot of improvements for the end user as well, potentially.
Willem (18:06)
Yeah, I’m curious so I think we could probably extrapolate like why features need to exist, right? There’s every company has many features in the backlog previous than others. But I like extrapolating where this goes because a human ability to generate new ideas is but eventually these code bases will crack. Do you see like if this continues for another or two years, models get
Shahram (18:11)
Mean every company has many some of them more do I love the idea of
Shahram (18:22)
Almost limitless.
Willem (18:35)
Become more agentic. Maybe every person in the company now has five to ten agents running at the same time. And I guess that’s that the bottlenecks for like every piece of CSS and React change. Or back end change operations, we still do, but I think that’s also gonna change a lot of companies. The better these models get.
Shahram (18:41)
Surely there’s a limit tied to the fact that over time people will they’re currently review, right? And we’re letting them go. I mean UI changes often we don’t scrutinize
Shahram (19:01)
Then you
Willem (19:04)
So I don’t know how think about that infinity.
Affaf (19:07)
I think that’s an area that a lot of folks are trying to wrangle with, right? Like there’s a lot of benefit with yes, now we can ship a lot of code, but then how do we validate and verify those changes I think are becoming increasingly important? And I think the part that is interesting is that I think many of us just have hypotheses, right? So for me right now is that code reviews, the practice of code reviews, even at a human level, there were some fallacies with that approach as well. Like if the PR is too big or even if it’s too small, sometimes folks just glaze over.
It’s just human nature. You have human loop, you will always have those types of challenges, right? So we always kind of afforded that type of margin of error. But the problem now is that AI is amplifying that quite a bit. For example, at LaunchCode, we haven’t been completely oblivious to that either. Like we’ve had SEV ones and sev twos, so IS2 incidents essentially over the past few months, actually, as more and more of the developers are using AI to ship more where more PRs are coming in, that ex more code reviews.
Code reviews are actually getting difficult as well because there are larger sets of changes, just because the agent is able to do that. So now the focus has been starting to shift to is that we really want in June org to be very metric-oriented, essentially beyond just the simple availability and SLOs, we want to be looking deeper into things that really tell us how the code is impacting, how the changes are impacting as well. So, your original question, one of the interesting things we’ve been talking about is that we want our teams to be certain metrics within their business function, essentially.
So now the engineers are becoming like there’s a dilemma of back-end or front-end engineer. Now it’s like, no, you’re a product engineer. So it’s like, okay, the stable stakes for you to know the language, the syntax, you have AI to be able to do both back end and front end, but now we need to essentially have them be thinking about the business impact as well. And now with LaunchGate, for example, specifically as we’re looking at non-engineers to be shipping out work, how do we ensure that what they’re shipping doesn’t impact the integrity of the code base and the quality of the experiences as well.
And what we’re betting on is that if we have enough telemetry coming from the metric side of it, we’re able to create that loop that can be enabled and supported by AI, but also be a signal for the humans, the engineers, that hey, something is regressing here at this particular beyond just the simple things like SLOs, diving deeper into areas where we’re talking about not necessarily like the code churn, for example, but more or less about
Affaf (21:34)
How we’re doing our imports of the technical design and some of the constraints around it as well. Some of it is very deterministic controls, right? That we didn’t lean in too heavily in before, but now we are essentially the telescope signals. One of the interesting things with our designers at LaunchCon has been about leaning in heavily towards the heart metrics, which is a philosophy that came through Google or UX, for example, where we’re looking at the amount of tasks being completed, how they’re completing, as a way to tell that, okay, is this particular area of the application regressing or not, essentially?
So for us it’s like how do we get more of these signals out there and then be able to engage on those signals because the lines of code to be reviewed at this point, it’s a losing value at this point. Like there’s just so much coming out, we just cannot stay on top. We cannot draw our or ten X either to kind of manage that load. But there was always human error to begin with too. So I think this is one of those things we have to of balance out, right? And look at the risk essentially. So
Shahram (22:31)
This is a interesting area to go into because what you’re saying, and correct me if I’m wrong, is that we’re moving from this notion of judging work by the artifact and moving more into the outcome. In this case, metrics to me is basically like what did it do, right? Versus what is it? Like what’s the logic. But you said a couple of interesting things that I want to pull out. Because I think you framed it very much from the sense of like what humans need to consume. Because code review is hard.
So I want us to be more metrics oriented so humans are going to consume. But like I want to flip that around and say, is that the end state, or is it that we want metrics for agents to consume so agents can do a better job? Then eventually does the does it move more towards if we just have these high level metrics like SLOs did back in the day, but the lower level metrics are actually for the agents. How do you think about it?
Affaf (23:23)
It’s an interesting one. I mean we’re we’re trying to work through this ourselves right now, but I think about the change representation feature that
Willem (23:30)
For example,
Affaf (23:32)
There were some very interesting tidbits in there. We use data dollar. So we’re looking at the ROM events that are coming out. And it’s quite interesting that if I imagine as we have more of the non-engineers both, the same type of outcomes the output sections that are being measured or surfacing. Now it would be very interesting for AI to pick those up and actually spin up and react to them as well. I don’t think we’ve seen that quite yet with Clary. Maybe Pierre’s approach is all green, so thumbs up for the team that’s what would be interesting is that as we saw, again, sure there was like this simmering noise and that particular change that hey, these are the ordering background noise, we’re always shipping or metting those errors.
Well, why can’t you have an AI agent identify the execute against it? Probably we can’t, but I imagine more and more as we determine as a judgment, I think more and more what we build from a product sense is going to be lot more important than what happens at the design layer or the engineering layer, or even at the feature layer. It’s really looking at the business out of being able to align with those outcomes and understanding that, okay, is this the right thing for us to be investing in at this particular part of this is like within the broader industry as well, is that everyone’s been building a coding agreement, like I said.
Working with you guys, for example, for quite some time now, I was also intrigued of like, hey, you’re now seeing this bubbling of SRE ages coming in and what that means. So I’m kind of curious on like as we talk about change verification, here’s kind of your take on some of this. What have you folks seen and there’s been a lot of interesting conversations with Manson here some what led you towards the change verification aspect that before you even get to investigations? It sounds like you’re trying to stop it earlier, right? Like, hey, let’s not even get there, let’s stop the bleeding much earlier.
Willem (25:16)
Yeah, I can jump in.
Shahram (25:18)
It was a recognition that you want to solve the root cause. The root cause is not in root cause a torrent of AI code coming through the pipeline. And there are many to catch that earlier.
Willem (25:23)
Fraud, right? The cause manifests there, but it’s really generated code coming pipeline. And there are many opportunities too. I think people just can’t keep up in all of these domains. So if you consider a code, that’s an area that’s very hot right now. Drawing their agents code changes. Got people banned because CI can’t keep up with everything we’re on. So it’s failing.
Shahram (25:38)
Center a review. Everyone’s at the post merge now you’ve got patching their commits together. The roll out also looking very closely at can we run a more complete test suites or even generate test?
Willem (25:52)
So during the process we’re
Willem (25:59)
For features that are being rolled out that are maybe conditional or textual based to really that this change, especially is safe. And all along that pipeline you’re basically features that could be almost like
Shahram (26:02)
Contextual those changes really make sure that change if it’s multiple services that plug together or the the set of changes and so tracking changes almost like a hand grenade rolling into prod. Hopefully tank right making sure
Willem (26:21)
You hopefully it’s like a cake, right? Making sure that it stays a cake and doesn’t explode at some point. Not like a Trojan horse now with the Odyssey in rear view. So we’re looking at like dynamic change.
Shahram (26:32)
Check that or a genetic QA prior to rolling out this very telemetry that hopefully expand to look at reliability.
Willem (26:34)
Then visuification is another one where we produce because you know it’s one thing to but you also want to look at business impact. I think that’s something that is on my mind is actually business because every single business there’s a cognitive load, if not
Shahram (26:46)
A lot is how do we know this thing actually benefiting the business second it’s live in the code base for humans or for agents? And can you somehow remove it eventually if it’s not useful? So we want that loop to be closed. Reliability is of the metrics, and can you deal?
Willem (27:00)
And so the loop is going to closed based on while business met the action take downstream of that could be healing or fixing it, but it could also be
Shahram (27:11)
Expanding coverage if you’ve nailed some user flow, maybe you need
Willem (27:14)
More of that or removing it if you haven’t. I mean if it’s safe, then I think you can have everybody contribute. I don’t think is enough. Even then I feel like give you a sense of quality of the system strong enough to maintain all the systems correct for these agents to make changes safely or are they just like cutting through the code base every single time and everything is like refactoring very dynamically.
Shahram (27:22)
But even that I don’t even that at scale, you need some verify quality system or the structures of code to seem
Shahram (27:40)
Every single
Willem (27:44)
So think there’s a lot of techniques that we’re gonna have to apply that maybe we’ve shelved for a while that we have to revisit in terms of
Shahram (27:46)
Needs for code health.
Affaf (27:52)
Very interesting because I think that’s one of the challenges that we’ve been struggling with as well as that okay, we have certain guidelines of like this much test coverage that is necessary. And it’s easier now to write tests when you just tell the agent like, hey, cover the pass here and not so happy about it. But when it comes down to actual experiences, there’s still the usual concept of setting your test cases and you have assertions in place, so it’s still stuck to that very deterministic, very much like pointing with a flow of checking software essentially.
So what you’re talking about as a genetic QA is quite interesting because let’s say you have a brand new feature that was developed or live coded or whatever, it goes out there. I mean give it enough context to understand what are the right expectations versus the wrong expectations, I should have, or assertions, I should say. So it’d be quite interesting in like how this evolved. And there’s a couple of organizations out there solely focus on the… That’s only a subset of the broader picture of like, hey, shift this thing, is it really doing the right thing or is it causing me harm than until like weeks later?
Shahram (28:56)
And these things only show up at scale, right? I mean, I think the whole vibe coding thing took off because there’s just totally different experience with the coding agent in a Greenfield project versus a brownfield. Where on Brownfield we’ve spent decades trying to figure out how to make things safe. I don’t think we ever really cracked it. It’s still a really hard problem. And somehow that has gotten worse, right?
So I think to your question, I see it as a bet on the future that you just can’t rest on being reactive. Like you talked about coding agents having way too many and you also talked about how there’s way too many SRE agents. Like there’s way too many every agent, right? Like I think really as much as the model labs are trying to be on the frontier, I think from the agent side of things, both you and us spend a lot of time thinking, what’s the future? And how do we go there? Because we’re still in this moment of flux. We’re doing the same thing. And the bet really is that MoCo it’s gonna go to production. It’s gonna break in ways that we didn’t expect, and how do you make — how do you build that seat though? Like how do you make it
Willem (29:56)
I think this is an interesting one to talk about as well because I think it’s really different for
Shahram (30:00)
Think it’s every company. It’s different depending on what you’re shipping. I’m imagining if you’re working on vault, having the marketing person ship code, even if they can. I’m not sure if it launched good it’s
Willem (30:07)
You’re not lying.
Willem (30:13)
Same. I’m assuming there’s a lot of guardrails in place. Where you’re basically done for but there’s companies where the UI we can catch it quickly, you know, we can like fix it.
Shahram (30:16)
There’s certain product you lose trust what? Some companies have quite glitches and it’s like low risk. I don’t know how you balance it.
Willem (30:29)
Yeah.
Affaf (30:29)
Yeah, I’ll first talk to Launch Gear for that. There are elements where Launch Gear has helped raise over a billion dollars in paying currency. We’re catching up to our first billion. But in that area as well, if we have issues with our payments or with our payouts, that has a huge trust impact. The is already so prone real when it comes to confidence intelligent trust assignments.
What are some things that we really need to be extra careful about? So anything on payments, ledgering, pay ads, all that, we have to be extra careful. Front engine is per se your account settings, for example, right? If you get it wrong once or twice, it’s not gonna be a major deal. But if it was your payments, then that’s other problem. So I do agree with you. I think there is an appetite and a tolerance as well, depending on what you’re working with. Vault is a really good one because the footprint that we have is huge. You’re talking about carrying way from mainframes all the way to personal machines, for example.
That was the footprint that we had to go through. And remember we looked at some very interesting things like TSIS, we still need to rebrand them. It’s always a thing to I know it’s hundred percent. But it was very much like reminiscence of like the old school chaos engineering, right? Injecting new types of problems into the space to see how these large scale systems that you really wouldn’t see the problems until you’re like five thousand or ten thousand or 100,000 pods in production, you really wouldn’t see the impact until you were at scale. So how do you get ahead of those problems?
And I think that tolerance would be different. Different organizations will approach this differently as well. But the bottom line is there still is that skepticism that still exists. One is that AI is still shipping a lot of slopes, so I don’t trust AI. I’m trusting it or I can use it. I can leverage it as that cane, so to speak, to kind of support me. But otherwise I’m not really trusting and letting go of my decisions to it.
And I think some folks are still on that journey to go through that. And a lot of it depends on how much they can tolerate that risk and how much they can actually be okay with not leveraging the ad for their use cases as well, right? So certain organizations are not moving fast or aren’t compelled to move fast, whether it’s the competitive nature of their space or the domain that they exist in, then those are the challenges that will come in. For example, government, healthcare, all those spaces will be more careful with how they leverage, consume decentral tooling.
And kind of going back to earlier, that one of the things is that certain subsets of defects that we see in software is dependent on the data that caused that defect. So now how do you ensure that you can work with AI in a space where data looks highly governed as well? And we haven’t cracked that up, so to speak, either. And it’s one that’s quite quite challenging. But I think we will figure out ways to do that. But until then, I think different organizations will have.
Willem (32:58)
We’ve been remote.
Willem (33:23)
Thank you.
Shahram (33:23)
Well then, doesn’t it equalize though? Like if I’m in an industry like banking, then all my competitors are also kind of stuck in the same speed. So it’s fine. And if I’m a consumer, I have to go very fast.
Willem (33:35)
Sometimes there’s a bit of luck with what you have in place. This is almost like fastening.
Shahram (33:41)
Like the furious and you’re putting nitra in the machine and the nitro you’re putting into it in the engine slow the future like it’s gonna be healthy it’ll
Willem (33:46)
And the you’re putting today has like sand and if you put too much it’s just gonna like explode at a certain point. But in the future, like a year it’s just like go much faster. And for some teams basis where they can just expand because of the structures they put in place, maybe it’s you know it could spiral completely out of control. It’s almost like you’re playing chicken between the competitors. So I think if you’re a very
Shahram (33:59)
For some teams that have healthy code.
Shahram (34:04)
For others.
Shahram (34:12)
Lucky you’re alone in a space and if you have complex you almost have no hopefully loaded. Until we get encoded cases that
Willem (34:15)
Choice but to try and stand neck with them. And hopefully the system doesn’t employ them itself until you get AI that can maintain database, that can go completely end to end, that can like garbage collect and all that good stuff. So I think it’s a bit scary for some companies. I think if you’re doing field or if you’re starting a completely new company spot. But yeah, you know it’s right because then you also need to establish yourself, etc.
Shahram (34:32)
For some purely green, I think you’re in a very good trade off, right?
Affaf (34:42)
Yeah, that’s very much an ample goes to your example as well where if don’t have those
Willem (34:47)
Person.
Affaf (34:50)
Been taking care of like devx, for example, agents are gonna have libraries with the same types of experiences to kind of implement and execute. So if you don’t have those basics in place and instead have spaghetti code or architecture that is either repeated or included, yeah, it’s just gonna amplify all that you have on you get ahead of that, or you kind of get consumed by this very essentially as well. So that’s what I think will do this challenge is that how do you kind of
Willem (35:10)
Exactly.
Affaf (35:19)
Private space to be able to go through the basics as well, be able to kind of implement those changes that will not only ironically help the humans, but also enable AI to move faster and in more safety as well in those space.
Shahram (35:33)
That’s why I liked your point at the beginning of this conversation about how you’re using the push to do more AI to do the good like to do the hygiene work as well. So that sounds like good advice to any engineering leader listening where CEO is saying, Hey, we gotta go faster, and you’re also saying, Yes, but you know, here’s all this other stuff that we gotta do. I think it probably came naturally to you that this is the right instinct, but I’m more curious how that conversation goes with business leadership.
Because it’s definitely the case that social media is just full of all these, I did a website in a in a day, I did X in one hour. And you know, you’ve got one sort of idea of how fast AI is, but then from an engineering standpoint, you’re probably thinking, whoa, how does that conversation go with business? And maybe what advice do you have for people listening in to manage the conversation in the best way?
Affaf (36:27)
I think this is where it gets a little bit trickier, right? I think as leaders we have to be able to translate those engineering nuances of like, hey, why do we want to invest in those ETU nuances now to leverage AI? And I think being able to translate those into business outcomes that quite frankly it’s just the age old management of stakeholders, right? Where we need to be able to translate those technical nuances to actual business outcomes. And I feel that AI right now is such a hot topic with leadership. Like every C suite is like, hey, why can’t we use more of this AI?
Earlier this year, everyone was talking about token leaderboards, and now halfway through 2026, it’s like, no, token maxing is actually a really terrible thing. Don’t do that. So we’re seeing that shift as well on like what are the drivers for leadership and how do we evaluate the impact of AI in businesses as well. For engineering leaders or technical leaders, I think that’s the tricky part is that we have to be able to provide for that narration of like, hey, that end-to-end test suite will allow our agents to be able to verify those changes.
So if you want work to be shipped out, even when engineers clock out or end their day and it’s happening before they join the next day, you need that verification. You need to be making sure that what you’re shipping out is actually working, right? Or working as we were expecting it to work. How do you do that? Well, you need these types of harnesses to do that. To be driven, be able to articulate them.
Willem (37:50)
Yeah, you see.
Affaf (37:54)
Business stakeholders will be able to understand. You also get an aspect of his costs as well. Yeah, it has gotten expensive over the past few months. We’ve seen that with the model cost going up. We’ve seen that with different types of provisional billing models coming up as well. Just today, Hilo prices changes with his pricing structure. So it’s like yet again, we have to revisit our budgets and figure out the next steps on what we do. So it’s gonna be interesting time and it’s like you have to be okay with the flux.
As engineering leaders, I think we’re gonna be targeted at multiple layers. We have the technical layer with like we work with AI to you know leverage the best of it and not the worst of it. The human layer of like, how do I motivate my team that I think your jobs are safe? Like AI is not gonna replace you per se. How do you do that? And the last piece of it is just the economics of AI. How are you going to be able to leverage AI to provide for that business value and be able to justify the NOI? Talk about AOI. It’s like, what is the cost of actually leveraging all these tools and how you go about it? And I believe that that’s where the optimizations and the learnings will start to kick in is like, not everyone will need to use everything.
The same advice these days is like, hey, everyone doesn’t need to be using Opus File with Fable File. It’s like, you know, bringing a tank to like a knife fight essentially, or as some people like you know from it. So don’t do that. And I think these are the remnants that we’re gonna continue to
Willem (39:11)
Yeah.
Shahram (39:13)
How do you have that ROI conversation though? Like something I keep wondering about is what you’re talking about is definitely a big thing that AI budgets are going up. There’s companies, I think Ramp just did, OpenRouter, there’s a whole bunch of companies that are raising based on saying we’ll save you money on AI. So it’s clearly like something CFOs are thinking about. But it’s been really hard to draw a clear line between AI spend and revenue, right? Effectively you gotta go to the CFO or the CEO and say, hey, if you put money here, we’re gonna get this. And now we’ve just talked about during this conversation that it’s really hard to paint that line. So how do you justify that spend?
Affaf (39:52)
Yeah, that’s million plus dollar question there. The way that I’ve approached this conversation, even at Launchka, is like honesty is the best way to kinda go about it and recognizing that we’re talking about bleeding edge within our space of like how do you build and maintain and manage products, right? All of that is out the window at this point. None of the traditional sense of costs or any way that we measure costs against the actual deliverables, a lot of those things are being turned upside down right now. And one of those fundamentals is like you need to come up with a function, or at least that’s how I’ve been approaching, is like a function to like pilot and then extrapolate like, okay, this is the benefit or the ROI that we’re able to get.
So for example, when we looked at incident management and investigations, we looked at how much time is this saving the engineers from like being able to close down their issues and the more time we’re saving, the more time that they have towards other actually more high value artifacts. So in a more agile sense, they’re able to deliver the value more readily so than the vacuum that investigations or incidents typically cause injury teams to spend on. And that’s where it gets interesting on like, okay, can you put the dollar value? Some people use like pure like you know hours bill type of situation. Others are able to kind of tie in the knot towards like, hey, okay, this was an outage in production, this is a loss of revenue. And we’re able to get, you know, the incident closed up much faster because now we have that AI supporting us.
The basic example is that 2 a.m. when we’re joining an incident and something is going down, some issue has come up, we already have the investigation primarily completed and we have the next steps available. This is one of the things that we really love about some of the tooling that we use, including is that now engineers are coming in, reviewing everything, and now just actually actioning on it right away. So we’re reducing the time and actually the RY is helping to increase that because during that outage, we would have had X amount of loss in business. So now we’re able to kind of trim that down based on how quickly we’re able to respond. That’s just one example.
Shipping features is this you know one of the more challenging ones where I think it’s more about how quickly you can run your experiments to understand where you can get business, how do you want to invest in the future? If you have a bunch of hypotheses, well, with AI, you’re actually able to run a bunch of those experiments collectively, which previously would have been more about a sequence approach. So what the learning would have been over three or six months, not able to get one month and I provide whatever you want to do, right? It’s a brand springboard that’s available for
Shahram (42:21)
It is a notion of being able to punch way above your weight, right? If you set up for it, right? So, you know, like even for us, right? Like we’re a small team, but I think the amount we’ve been shipping and we the way we’ve been able to move would not have been possible for AI and that’s hard to quantify. Focus a little bit on what you said on being in the frontier. Like it’s a breakthrough kind of technology and that’s kind of the narrative. Man, like I think X and LinkedIn, Linked is probably a lot worse, but it’s just full of all these hot takes on where things are going.
It’s you know, one day it’s software factory, then software factory doesn’t make sense, and it’s this, then it’s that. What’s your no BS view of what are you skeptical about? What do you think is actually gonna happen, let’s say six months from now? Where’s the puck going in your view? I I’d be happy to share our takes as well.
Affaf (43:11)
You’re saying six months. I struggle with three months now. And I’ve seen that since last fall, essentially, right? Like everything kind of turned upside down. We’re already seeing it. The pressure of the finances, the economics of AI are really gonna kick in and they’re already starting to creep into conversations. There’s a lot more nuance to like how quickly people are able to jump into new tool and export things. I think that is gonna be very much real. Verification is going to be the problem. We have all of this coming out. We need to be able to understand how do you validate and verify software, true sense of quality, how do we maintain that?
And that’s gonna be a multi-layer or multi-faceted. So both at the product layer, at the engineering quality layer, and honestly, I mean you guys have probably seen this as well. Defining engineering quality has always been challenging to begin with. You go qualitative, you go quantitative, it’s a space in its own and how do we kind of force with that with a real impact to how do we do our craft essentially? So all of that is going to come to a head.
So a couple of hot takes frontier models are not the be all, or the like we have to be using open rated models are going to be a real consideration for many teams to kind of start leveraging and almost before stage of that conversation, even if they’re not already for it, just because of the economics coming in. And the quality aspect is going to really take front hold and it’s going to be a little bit different than the ones that we saw last year, for example, where it was like Repliter or someone where someone dropped their live code apps like database, for example.
Think we’re gonna see a lot more nuanced take on voice quality and what are we really afraid of with AI committed code, right? We saw this a little bit with AWS earlier this year as well. And I think a lot of just your average organizations are really going to start thinking about seeing that these types of problems are really going to help as more and more the adoption comes in essentially. And my horizon for these things is like three months at this point. Like I did the six month, one year thing last year, this whole plan of like every three months we’re gonna review AI tools, our AI budgets didn’t fly. So we’re down to like quarterly and sometimes even monthly in some areas. So options.
Shahram (45:14)
Were there any hot takes from you? I still really think that code should be a single player activity. I think a lot of
Willem (45:24)
Teams will silo engineers kind of like in a Conway’s law manner. Versus having them over each other’s toes and then get stuck, because that’s how you go from like five X back to one point one X. So I think companies that do that correctly will have massive denial, continue to have this collaboration that slows everyone down. But I think there’s gonna be many of these examples of
Shahram (45:30)
At the very least on components, maybe even repos. Run their agents.
Shahram (45:53)
Ways in which graphics technology
Willem (45:54)
We have to follow this and figure out ways to maximally benefit it. And unfortunately, it’s just there’s no show. Yeah, otherwise your competition just
Shahram (45:58)
See there’s no time to slow down it is
Shahram (46:05)
Lapsie.
Affaf (46:07)
Looking at building out one of those like this typical thing of like, Hey, let me rebuild this entire SaaS application using AI and see what the cost savings are. One month later I still have to review those dirty PIs that are out there. So those are the real impacts of it. Yeah, it generated all the code, but was it the right code? I still gotta review it. So
Shahram (46:25)
Yeah.
Willem (46:25)
Sorry.
Shahram (46:28)
I think it’s a good note to end it on. Yeah. This has been great. I think just on my take a little bit was I’m really interested in the ROI side of this discussion because all of the stuff that, you know, we’ve been talking about, like should we use open source models, should we use Fable or Opus or you know token maxing, that’s all optimizing for the eye, right? It’s a cost. It’s like, okay, how do we make this cheaper? But I’d say that if there’s something that is gonna make you ten X more revenue who cares if you’re gonna use fable all day for that?
But we’re just not able to do that. So that’s why I’m so interested in this verification side of things. Like right now it’s early days. We’re looking at, you know, engineering metrics, looking at logs and errors, eventually you want to go to product metrics. But I think the gold standard is really linking it to business metrics so that we can actually tell you, hey, like this change actually impacted something. I it’s like science fiction right now. It’s a long path to get there. But I think the more you can close this gap, then I think some really interesting discussions can happen.
Afa, pleasure as always, sir. Good to have you. And yeah, until next time.
Affaf (47:35)
Sure, sure.
Willem (47:36)
Thanks.