From rowing to steering: a year of AI-assisted engineering at XEO 360

It’s no overstatement to say that the world of software engineering is going through a profound transformation. This article is a one-year retrospective of how the processes, practices and tools at XEO 360 have evolved from its founding in the Summer of 2025 to September 2026. In a nutshell writing code got dramatically cheaper, and almost everything around it got harder, sometimes in non-obvious ways.

First a bit about myself, my company and my team. I’m a software engineer by trade (yes, I subscribe to the notion that some of what we do is engineering). I was a Principal Engineer at Klarna, and I am currently the hands-on CTO of XEO 360.

XEO 360 is an AI visibility startup. We measure how brands and their products show up when consumers ask LLMs like ChatGPT or Gemini for recommendations. That covers aspects like which brands are mentioned, how often they appear, what's said about them, and where that diverges from the brand's own product data. From there, we help brands fix the gaps, from correcting product information to creating content that LLMs actually cite. Under the hood, that means scraping consumer AI surfaces at scale, maintaining a knowledge graph for each brand and respective products, and running natural language processing and analytics workloads over the results.

We're a small, fully distributed team of eight, six of whom are software engineers with different specialities and seniority levels.

A journey through practices, processes and tools:

Act I: Foundations (Summer 2025 - Nov. 2025)

From the very beginning, the processes and ways of working have been designed as remote-first. This comes with a set of unique challenges namely around onboarding, knowledge sharing and fostering team cohesion. I’m a firm believer that writing is thinking, and therefore we made a deliberate choice to adopt processes like RFDs to record and discuss our most important decisions as a company. This doesn’t preclude moving fast and adapting to changing conditions, however there are some decisions where four hours of thinking and deep work can save weeks of effort.

Beyond that, our setup was fairly standard: a TypeScript monorepo, GitHub, pull requests with the customary code review, Slack for collaboration, Linear for issue tracking, and regular team touchpoints (daily standup, weekly demos and monthly retros). We didn't enforce any standard tooling at this stage, and the team was split between Cursor and VS Code with extensions like Claude Code.

Our deployment process was also standard fare, built on GitHub Actions. A developer opens a PR, which triggers automated tests, and after code review it's merged into main. From there it deploys automatically to staging, and a manual gate tags the release and promotes it to production.

During this period, we were operating as a “classic” software shop, with each person typically working on one task at a time, leveraging AI mostly as a combination of glorified autocomplete with StackOverflow. Situationally very useful but requiring close supervision.

Act II: Pair-programming with Rain Man (Dec. 2025 - Apr. 2026)

The release of Claude Opus 4.5 in late November 2025 was a watershed moment. There was a noticeable step change in the quality of LLM outputs in coding tasks, and the team started to lean more heavily on AI to create code.

The amount of PRs and code being shipped increased significantly, and the team started struggling to keep up with code reviews. To be fair we already struggled with this before, but the moment we had 10+ PRs waiting for review this was the breaking point. This is when we introduced a new social contract, whereby if the author of the PR needs a second opinion on any part of the code he should explicitly ask for revision from particular team members - a pragmatic but definitely imperfect solution. The following table captures the number of merged commits and LoC changes over the last year.

Month	Merged commits1	LoC changed2
2025-09	227	71082
2025-10	271	85212
2025-11	251	85270
2025-12	380	139408
2026-01	414	233343
2026-02	482	181529
2026-03	361	239391
2026-04	252	218379
2026-05	356	493237
2026-06	278	199880
2026-07	537	578227
2026-08	534	799963
2026-09	596	610131

Effectively from this point onwards each developer gained access to a team of talented but not very wise juniors at their beck and call. LLMs can be amazing at narrow tasks, but often blind to the surrounding context (ergo the Rain Man reference) and prone to over-engineering. They operate like “tactical tornados”, pumping out code that ends up working, however that does not necessarily mean that we’re building the right thing, or that we’re setting ourselves up to have a codebase that is maintainable in the long run.

Ultimately we witnessed an amazing compression of the SDLC timings. A single developer was now able to plan and implement a feature spanning infrastructure, backend and frontend in record time, without having to reach out to other team members - this is quite liberating. However, in true complex adaptive system fashion, this shift had a knock-on effect in team dynamics: Each developer now had the power of a small team, and combined with the normal startup delivery pressures, this meant that the tooling and incentives were there for people to ship code rather than coordinate with other team members. This is an all too familiar inter-team dynamic now being played out at the individual contributor scale (fun fact: at some point we had six different implementations of Levenshtein distance in our codebase). Overall we saw a decrease in collaboration, measured by the number of RFDs being produced, and of course our planning process also suffered. Plausible agent-generated plans were executed sacrificing some of the team discussion, leading to sub-optimal solutions and situations where new key domain entities were introduced without the team being fully aware of them. This is a hard balance to strike, as we’re very cognizant that maker time needs to be defended, however there needs to be enough common ground for the team to act as a cohesive unit instead of a set of people that happen to pull work from the same queue.

We started seeing some cracks in our foundations, for example, in one month the number of hotfixes we had to ship grew to 21 per month where the previous monthly average was 3. And while we had a pretty strong technical setup from the get go with good test coverage, linters, and automated deployments, this wasn’t enough to account for the combination of more atomized team work (where we spend less time thinking about core abstractions we want to have in the platform) and all the creative ways in which LLMs generate code. Some areas of our codebase started becoming bloated and over-engineered, with a nightmarish dependency graph. In addition to this, larger PRs became more common, which compounded the review problem.

Overall this was a very intense hands-on learning experience (bad experiences included) on the power but also the limitations of AI generated code and AI-centric processes in our particular context. From a more tactical perspective, there were several notable cases where state-of-the-art models of this period failed consistently:

• Preferring solutions that involve multiple queries and in-memory processing over a slightly more complex query that provides the required data straight from the database, rapidly leading to N+1 query type scenarios that hammer your database hard.
• We use Prisma as our ORM, and we noticed a pattern whereby because we introduced queries relying on hand-written raw SQL using queryRaw (basically the “escape hatch” the ORM provides to run SQL queries written by hand), LLMs picked that idiom up and started using it, even when not needed.
• Complete and utter lack of transactions. It’s likely a training artifact but it is common to see LLM generated code that simply ignores that transactions are a thing. This is probably one of the most insidious patterns we’ve encountered thus far.
• Tests that extensively rely on mocking, and in many cases end up being tautological, asserting the mock and not the actual system under test.
• Spurious tests, for example testing that constants don’t change.

Act III: The Cambrian explosion (May 2026 - Sept. 2026)

The increase in hotfixes, the duplicated code, the bloated dependency graph, the slow but steady erosion of common ground, were all warning signs. We found ourselves facing an interesting conundrum: we scaled individual output without scaling both the shared understanding of the team and the required safeguards to sustainably deliver value in the long run. The challenge was therefore to effect a course correction while continuing to ship new features and onboarding new clients.

The first correction was organizational. Professional software development is a team sport, and quality is an emergent system property, therefore, ensuring the team has a shared understanding of the ecosystem and the affordances (shared capabilities, common patterns, UI/UX interaction, etc.) it provides is key. In practice this led to a renewed effort to resume our use of RFDs to communicate key technical decisions, create more collaborative design sessions and re-organize how we structure our work in Linear to ensure that people are working as much as possible towards one shared, common goal at any given point in time.

The second correction was more technical in nature - and really was all about orienting ourselves in this brave new world. The team leaned in heavily to experiment with a bit of everything from new lint rules, AGENT.md configs, skills, to coding agents/harnesses to ways of interacting and delegating work to LLMs. The goal was to learn as much as possible, as fast as possible, so we could better tailor the processes and tools to our needs. This was an organic process, and in hindsight as a technical leader I should’ve done a better job of channeling all that enthusiasm in a more organized manner - as there was quite a bit of noise and strong opinions in the mix. Some experiments paid off, others didn’t work in our ecosystem and a few were counterproductive. Interestingly a lot of what paid off were traditional engineering practices whose value went up once most of our code was written by an LLM.

A good rule of thumb is that if it can be automatically enforced, it probably should be: this is where we had the biggest wins. In our case architecture guardrails became an essential tool to keep the dependency graph in check and start course correcting some of our previous excesses. We are using dependency-cruiser to ensure our codebase maintains certain properties like “core-infrastructure code cannot import anything from the business-domain code”. This helps address the nightmarish dependency graphs we started seeing earlier.

Linting rules were reinforced and are now used to also capture idioms no reasonable human would introduce (e.g. the excessive use of queryRaw). Be opinionated on how you want your code to look and the standards you apply. The diversity of random idioms that are used in a codebase is super-charged by LLMs and even current frontier models to very weird things.

Tests are non-negotiable, they are life-savers however LLMs are frankly bad at writing them. To counter-act this, we’ve invested significant time in setting up the examples and idioms we want to use, and heavily curating our test suite (it’s okay to delete tests as well). Beyond the traditional testing focusing on unit and integration tests, LLMs made more advanced techniques like property-based testing accessible to people who hadn't used them before.

The same principle applies to LLM-intensive processes, which aren't a good fit for traditional unit and integration tests. They're probabilistic by nature and need a different testing approach altogether. Evals have been widely adopted for this. At this stage we are doing mostly offline evals, but even this drives immense value, allowing us to rapidly test new (often cheaper) models, and better assess the impact of changes to prompts and context. Trust me bro is not an engineering approach.

In operational terms, LLMs are an incredible powertool to sift through telemetry data efficiently. This gives every team member more expressive power to understand what’s really happening without necessarily having to know the right incantations of a particular log aggregation system. This helped reduce cycle time for bug fixes significantly, and currently stands at 3h:30m (median) from the moment the issue is raised to when a fix is deployed in staging during working hours - more on our cycle time in a bit.

Git worktrees became a staple of our workflow, and developers now commonly work on several issues in parallel. The combination of a growing test suite and more frequent PRs put extra pressure on our already struggling code review practice, our CI pipeline and developers' machines. During peak hours, merges to main regularly sat waiting in a queue. That forced us to rethink running the full test suite on every PR. PRs now run only the tests affected by their changes, and the full suite runs nightly and on every release. To show how quickly this problem scaled, we tracked three metrics:

• Pre-merge check time: how long CI takes before a PR can be merged.
• Merge queue wait time: how long a PR waits in the queue before it's merged and the automated deployment starts.
• Merge cycle time: the time from a PR being merged to it actually being deployed to staging.

Pre-merge check execution time (minutes):

Month	Median	P90	Max
2026-01	9.7	12.5	14.5
2026-02	11.5	14.4	18.9
2026-03	15.5	18.8	29.6
2026-04	20.3	23.6	34.9
2026-05	14.2	17.0	23.3
2026-06	16.2	19.1	21.3
2026-07	22.2	27.3	31.9
2026-08	30.9	38.0	52.8
2026-09	34.4	54.0	62.2

Merge queue wait times (minutes):

Month	Median	P90	Max
2026-01	0.1	0.1	17.4
2026-02	0.1	0.1	0.2
2026-03	0.1	0.1	12.9
2026-04	0.1	0.1	0.2
2026-05	0.1	0.2	0.3
2026-06	0.1	0.1	0.2
2026-07	0.1	0.1	18.9
2026-08	0.1	5.2	295.2
2026-09	0.5	87.3	413.5

Merge cycle time (minutes):

Month	Median	P90	Max
2026-01	7.1	7.6	24.3
2026-02	6.3	6.7	7.9
2026-03	6.7	11.6	19.2
2026-04	12.2	13.6	27.6
2026-05	12.7	13.8	18.3
2026-06	13.3	14.6	17.2
2026-07	15.1	17.3	35.9
2026-08	14.6	26.4	314.4
2026-09	67.2	145.0	441.6

In a nutshell we deviated quite a bit from the very worthwhile target of 15 minutes or bust. In particular, tail events tell a quite damning story!

At the intersection of product and engineering we also saw some very positive changes. LLMs are great at rapidly iterating and generating cheap disposable prototypes to test different hypotheses (resist the temptation of making them more than that). This is impressive and is an excellent way to anchor discussions around concrete artifacts, as sometimes an image (or clickable prototype) is worth a thousand words. LLMs are also accessible enough that non-technical stakeholders can meaningfully engage in the exercise, allowing for a broader range of voices to contribute to the discussion.

Our biggest failures came from delegating judgement to the machine, or from prescriptive skills that didn't suit our environment and practices.

We experimented with LLM-generated implementation plans. The output is often plausible but full of assumptions, so it needs close inspection. We tried skills like Wayfinder to mitigate this, without much success. We ended up with entire sets of subpar Linear tickets: the classic combination of LLM verbosity, a baffling disregard for separation of concerns and no crisp deliverables. We had multiple instances of bloated prose describing trivial state machines, with nonsensical requirements baked in. That naturally led to bloated, overly defensive code. In one case, ticket-level over-engineering went as far as tweaking database isolation levels on run-of-the-mill read operations. Curating a large number of these tickets isn't worth the effort, and we now discourage its use.

Another disappointment was AI code review tools like Greptile or CodeRabbit. Code review is a genuine problem, and at first glance letting AI review the code sounds like a really good idea. It turns out that these tools are not really optimized to reach a fixed point. Every commit brings a fresh batch of suggestions, and while there are some genuinely good findings, most were false positives or nitpicks. The PR author ends up getting guilt-tripped into trying to catch them all, often past the point of diminishing returns.

When it comes to skill use, the results were very interesting. Generally, the value of the skill is inversely proportional to the complexity of the skill. Simpler skills that offer a very specific capability (e.g. the excellent show-me skill) are great general purpose tools, and great building blocks for internal skills.

More process-oriented skills, such as the very popular skills from Matt Pocock, resulted in significant friction with our existing processes and practices: on one hand the introduction of the CONTEXT.md as the canonical store of domain entity definitions rapidly started clashing with existing documentation (often incorrectly!), on the other hand, the generated ADRs go against the grain of the RFD process. On the documentation front in particular this had real negative value as it is already hard enough to maintain traditional READMEs in a human-readable state, let alone managing a second LLM-centric artifact.

I’m a firm believer that human-readable documentation is worth having, as it is usually a necessary but not sufficient first step in forming a mental model of the system (the other elements are reading the code, and looking at observability data in whatever form it is available to you). The idea that operators don’t need stable artifacts because they can just ask an LLM is naive at best. Just ask yourself: how do you know you’re asking the right questions? That takes judgement, and judgement is built on understanding.

What’s next:

We now have a much clearer notion of what works and what doesn’t in our context. That lets us approach this space more strategically, and over the next few months we want to address the issues affecting our practices, processes and tools.

The practices aspect will focus on two key challenges that are amplified by the fact we are a remote team. The first is the fact that it is very easy to stay in your lane and ship features, only to end up with a whole that is less than the sum of its parts. Part of the answer to this lies in practices we already have such as leaning on RFDs to clearly communicate important decisions and creating explicit opportunities to collaborate. This can be supplemented by new tools such as weekly automated digests that cover the latest changes to the systems we’re working on, so that everyone has better situational awareness.
The second challenge is improving our decision making process on the adoption of team-wide tools and ways of working. We expect this space to continue evolving very rapidly, and we need to have better governance over how we assess new tools and ways of working. This will reduce friction and will also avoid creating knowledge asymmetries that prevent people from getting the most out of the shiny new toys at their disposal.
One big outstanding point is consolidating our documentation. Today it's split across markdown files in our repos, Linear documents and Google Docs, and each has tradeoffs. Markdown in the repo isn't really accessible outside engineering. Linear is great (and renders Mermaid diagrams!), but its collaborative editing isn't at the level of Google Docs.

On the process side of things, the question on everyone's mind is how to improve our software delivery lifecycle as a whole. Code review is one obvious bottleneck, and we will begin experimenting with a better process for flagging code reviews leveraging lightweight decision models to better classify which parts of a PR, if any, require explicit human reviews with potential escalations to specific team-members. For example core database migrations are things we need to guard carefully, while label changes on the frontend, not so much. In parallel with this, we will give another go at code review tools/services, as they may have improved significantly since we last tried them. We are not a developer tooling company so we need to carefully balance what to build and what to buy.

On the tooling front, there is also a lot to do! Our merge cycle time is growing and therefore we need to reduce it. This is in part accomplished by working with smaller PRs and selectively running only the tests that are affected by the changes at hand. We'll also need to decide whether to standardize on an agent harness. Pi, for example, is very extensible, which could let us capture specific steps of our SDLC via extensions. We've started supplementing the team with an "army of Roombas" that takes care of small but important chores, like flagging where documentation and code have drifted apart. We're currently testing Linear Loops for this. Finally, we want to improve how we create fully provisioned worktrees that can eventually run remotely, including ephemeral infrastructure, so developers and agents can fully test what they're building in a secure, isolated environment before opening a PR.

Clearly we have our work cut out for us!

Coding may be solved, what about software engineering?

The last year was one of the most intense periods of my career, and everything described above happened while we kept onboarding clients (including some of the largest consumer goods brands in the world!) and shipping features. Stepping back, a clearer picture is emerging. Our job as software engineers is to exercise professional judgment, and it always has been. What changed is that the part of the job spent typing the right incantations to turn business needs into executable code has shrunk dramatically. The level of abstraction, however, hasn't risen - and here lies part of the rub. There is no such thing as programming in natural language: a detailed enough spec is very close to the code itself, and performance, abstractions and developer ergonomics still matter a great deal. That's why so much of what kept its value is tried-and-tested engineering practice. You still need to understand and care about the code, even if you don't type every line of it.

We're in a transition where the main activity of a software engineer is shifting from rowing (focusing on one thing at a time and producing good code) to steering (managing multiple workstreams, context switching all day, and exercising sound judgment throughout). That is mentally taxing, and at times at odds with the skills, and sometimes the identities, we've built over the years.

The tax is compounded by a latent epistemological problem. How do we know what we know about a system? How do we know the generated code does what it should? Much of that knowledge used to come from actually typing the code, and from a productive kind of laziness: the mix of frugality and foresight that shaped what we chose to build. LLMs remove a lot of those feedback loops. Generated code easily papers over the early signs of bad abstractions, and it often ships more than is needed. That raises two questions: how much does it matter, and what do we do about it?

I think it matters, at least in places. Interfaces, module dependencies, how we handle failure modes and key data models all matter a great deal. Identifying those leverage points and guarding them closely will only become more important.

As for what to do about it, a few directions look promising. The renewed interest in formal verification, and new languages like Bend2, hint at what may come next. LLMs also open interesting possibilities from a human-computer interaction perspective, particularly in helping developers make sense of large systems through better visuals, interactive explainers and views of how a system has evolved over time.

The fundamentals still matter, and I suspect reports of the death of software engineering have been greatly exaggerated. In the midst of all this messyness, it's worth reminding ourselves that, as software people, we're better equipped than anyone to steer this technology and build the future we want to live in.

---

Footnotes

1. For simplicity we only counted commits on our main mono-repo and only commits that eventually were merged into our main branch. ↩

2. The “LoC changed” column excluded:

   • Data and configuration files: JSON/JSONL, CSV, YAML, TOML, text fixtures, snapshots, SVGs, and other files outside the source-code extension list.
   • Lockfiles: package lockfiles, pnpm-lock.yaml, yarn.lock, etc.
   • Documentation: .md, .mdx, etc.
   • Recognized generated code: files under /generated/, or with .gen. / .generated.
   • Binary changes: Git doesn’t provide numeric line counts for these.
   • Merge commits: excluded to avoid recounting their constituent commits.
   ↩