Lalit Maganti (Tag: Big-Tech)

How I Find Problems to Solve as a Staff Engineer

Sat, 25 Jul 2026 14:55:00 +0100

Note: this post was revised after publishing for increased clarity, based on reader feedback.

“How do you find problems worth working on?” a senior engineer I mentor asked me recently. He’s trying to make the jump to staff engineer and realized that the role isn’t just about doing the work he’s assigned. He also needs to get involved in figuring out what his team and org should be building.

Someone else had suggested blocking out time in his calendar to think about the bigger picture. He’d tried that, but hadn’t found it productive, so he asked if I had any alternatives.

I told him I rarely find good problems by staring at a blank page and trying to “think strategically.” Instead, I act like a sponge. I listen to the stream of day-to-day noise, absorb the problems people are having and let them sit in the back of my mind. Over time, some fade away while connections begin to appear between others that initially seemed unrelated. Eventually, I start to see what’s really slowing people down and what my team or I can do about it.

I’ve worked with many engineers who’ve never really tried this. They wait for managers or leads to identify opportunities, then demonstrate their value by solving the hardest assigned problems. That can absolutely lead to promotion. But the projects that have made the biggest impression in my career were the ones where I found and solved an important problem my leaders did not yet realize existed.

One caveat: my experience comes mainly from working on infrastructure and developer tools at large companies, on teams where engineers have a lot of bottom-up autonomy to influence their roadmaps. In a more top-down environment, there may simply be less room to work this way.

Absorb problems, not requests#

People love talking about the problems they are facing: in meetings, chat threads, presentations and email. They explain why their work is hard, complain about what slows them down and describe what they wish they could do.

When something overlaps with my area, I start pulling on the thread. I might ask, “If X existed, would it solve your problem?” or point them at an existing feature in a product I own and ask how much of their use case it covers.

Users often ask for a particular solution instead of explaining their root issue. Rather than taking the request at face value, I keep digging until I understand what they are trying to accomplish and why existing products do not work for them.

As a natural introvert, this sort of ambient listening works particularly well for me. I don’t need to fill my calendar with speculative meetings just to find ideas; there is already an enormous amount of useful information flowing around me during a normal week.

When a problem seems worth exploring, though, I become more active; I need to see how it affects the team’s day-to-day work. I’ll sit with them as they walk me through their workflows and the bugs they’re investigating. When I can, I’ll try working through some of those bugs myself. Seeing the problem firsthand makes it easier to separate what the team actually needs from the solution they asked for.

I also seek out people who see more of the organization than I do: those who own critical systems, work across several teams or have particularly deep insight into the work downstream of my team. I’ll arrange a 1:1 or coffee chat and ask about interesting problems they’ve come across. They may have already seen the same issue in several places and started connecting the dots, giving me a head start on patterns I might otherwise have taken much longer to notice.

Let problems accumulate#

Several times, I’ve been burned by moving too fast. I became excited by a request from a vocal team, built the feature and watched them barely use it. Their priorities had changed, or the request had come from a one-off investigation that no longer mattered. How eager a team was in that moment wasn’t the same as how important the feature was relative to everything else my product needed to support. By hyperfocusing on their request, I lost sight of the bigger picture.

That taught me to let potential problems pile up. Listening the way I do leaves me with far more of them than I could possibly solve, and not all deserve action. Most don’t need to turn into projects the first time I hear about them; waiting can be a superpower.

Waiting means the same problem might pop up independently in different teams, making it a higher priority to solve. Or problems that look different on the surface might turn out to have the same shape, so I can address several use cases in one shot. Or, as I’ve learned painfully, the requesting team didn’t even care that much in the first place.

Instead, I make a mental note and revisit the problem if it comes up again. Other engineers I know write this sort of thing down more systematically. The mechanism is a personal choice: everyone has to figure out what works for them. What matters is keeping unresolved problems around long enough for more evidence to accumulate.

Find the common shape#

Waiting helps me collect evidence, but that alone doesn’t tell me what to build. I still need to work out whether the problems I’ve retained are genuinely related and what, if anything, could address them together.

Perfetto, the performance debugging tool I work on, is a good example. It displays recordings of system activity on a timeline made up of rows called “tracks.” Over a couple of years, teams kept asking for small, specific additions to the UI. One wanted a command to keep their preferred tracks pinned to the top of the screen; the next team wanted the same, but for a completely different set of tracks. Others wanted Perfetto to open already zoomed in on a particular part of a recording, or to show a custom aggregation tuned to what they cared about. A few had stopped waiting for us and built elaborate workarounds with bookmarklets.¹

By the time enough of these had piled up, my head was the usual tangle: the requests themselves, the constraints on each and a handful of half-formed solutions. I’ve learned not to force a solution by just sitting at a desk and thinking. Instead, my best untangling happens on long, aimless walks around London, where connections come more easily when I’m not trying to force them.

What I eventually realized was that none of these teams really wanted the specific feature they’d asked for. Each wanted to personalize Perfetto for their own workflow without imposing their choices on everyone else. The underlying need wasn’t any one feature but rather the ability to extend the UI. When a connection like that finally clicks, it’s one of the best feelings in the job: several awkward requests collapse into a single idea, and possibilities open up that none of them hinted at on their own.

That feeling, though, is exactly when I have to be careful, because a common shape is only a hypothesis and elegance is not evidence. When it happened with extending the UI it turned out to be real, but I’ve been fooled before.

In another recent case I was convinced that building a transparent caching system for querying Perfetto traces would solve issues with sharing large traces and repeated queries. It was only as I wrote the RFC and built a prototype that I realized the elegance was a lie: the two problems wanted genuinely different solutions. I reluctantly split the design in two, both halves of which have since shipped.²

Pressure-test before building#

You’d think this would be the moment I start building, but it usually isn’t. How far I go depends on how sure I am that the idea works and that people actually want it.

If something is useful and low-risk enough, I act straight away: I send the change and let my manager know. When I’m unsure whether an idea will work or how much effort it will take, I build a throwaway prototype instead; it exposes the failure points and gives me something concrete for others to react to. And when an idea is big but I’m convinced by it, I commit to the full effort: weeks or months of work and the hard yards of building support across other engineers and teams.

Through all of it, I’m not only trying to convince other people; I’m also trying to convince myself. Sometimes the honest answer is to stop: if people don’t see the value I do, or we hit a major technical wall, I’d rather drop the idea now than build something no one uses or that becomes a maintenance nightmare. And sometimes it holds up but the timing is wrong, so I park it, ready to spring into action the day it becomes an org priority.

When an idea does hold up, I don’t necessarily need to be the person who builds it. I might implement it, someone else on my team might, or it might change what the org focuses on. Finding and shaping the right problem can have an impact even when I don’t own the implementation.

The Perfetto extensions idea was worth that full effort. We were already building plugins to modularize the UI, but they weren’t enough: teams had to open source all their plugin code, which wasn’t an option for many internal use cases. So before building anything new, I took the problem and my proposal to my manager, teammates and the client teams. I ended up writing two RFCs, having several 1:1s and giving a couple of talks, refining it as the feedback came in.

In the end, I designed and implemented macros as “lightweight extensions”: a way to automate actions in the UI without writing a plugin. Extension servers took the idea further by letting teams share their macros.

Instead of implementing every requested feature ourselves, we gave teams ways to adapt Perfetto to their own needs. Dozens of teams inside Google now use macros and extension servers, and several other companies use extension servers internally too.

Solving useful problems helps me find the next one#

The more often I go through this process, the easier it becomes. When I show genuine interest in someone’s problem, ask useful questions or help solve it, they remember. They start coming to me earlier and bring me into conversations with other people facing related issues.

That gives me a wider view of what is happening across the organization, making it easier to spot patterns and build things people actually need. Solving one of those problems brings me into more conversations, and the loop continues.

Those successes build the kind of trust that comes from long-term stewardship. Early on, I had to turn many of these ideas into something real myself to prove that my judgment was sound. Over time, my manager and org gave more weight to my assessment of what mattered. That allowed me to influence the roadmap without needing to own every project.

This differs from the idea that becoming a staff engineer means replacing technical work with meetings and coordination. For me, conversations are inputs into what I build, not the end result.

Conclusion#

That is what I wanted my mentee to understand: finding problems worth solving isn’t separate from the rest of the job. It comes from staying engaged with people’s work long enough to see what no single request can show you.

These workarounds used bookmarklets to run JavaScript against Perfetto’s internal UI APIs. ↩︎
The original proposal was to use a transparent cache for repeated queries and faster reopening of large traces. As I worked through it, I realized repeated queries were better served by keeping sessions warm in memory, whereas reopening was better served by explicitly exporting a trace into a format designed to load quickly. A transparent disk cache could also retain multi-gigabyte files without the user realizing and would need a new system to manage their lifetime. The proposal was ultimately replaced by warm sessions and streaming table export. ↩︎

Why Senior Engineers Let Bad Projects Fail

Tue, 13 Jan 2026 10:48:00 +0000

When I was a junior engineer, my manager would occasionally confide his frustrations to me in our weekly 1:1s. He would point out a project another team was working on and say, “I don’t believe that project will go anywhere, they’re solving the wrong problem.” I used to wonder, “But you are very senior, why don’t you just go and speak to them about your concerns?” It felt like a waste of his influence to not say anything.

So it’s quite ironic that I found myself last week explaining to a mentee why I thought a sister team’s project would have to pivot because they’d made a poor early design choice. And he rightfully asked me the same question I had years ago: “why don’t you just tell them your opinion?” It’s been on my mind ever since because I realized I’d changed my stance on it a lot over the years.

The answer is that being right and being effective are different.

In large companies, speaking up about what you see as a “bad project” is a good thing. But only in moderation. Sometimes the mark of seniority is realizing that arguing with people who won’t listen isn’t worth it; it’s better to save your counsel.

Bad projects#

What I mean by a “bad project” is many things:

UX: making product complicated, solving a problem which doesn’t exist, breaking existing workflows
Technical: overcomplicated design, wrong library, poor performing architecture
Political: chasing hype cycles, exists primarily to justify a promotion

It’s important to point out that for much of the lifecycle of a project, whether it’s “bad” is highly subjective. Software engineering is largely a game of tradeoffs and making decisions which are not perfect but the best possible with the information available. There often can be disagreements on whether correct choices are made and it only becomes obvious much later on, potentially years after a project has shipped.

But as you become more senior, you’ll start to have “taste” when it comes to software projects and that will cause you to look at some fraction of the software projects and feel “this doesn’t make sense”. And this gut feeling is the sign to me of a “bad project”, one which you can see in advance of when it’s obvious to everyone.

Drawing on my personal experience, the most memorable example was a few years ago at Google . There was a high-profile announcement internally of a “game changer” project that sat right at the intersection of two extremely large organizations. It was technically amazing and elegant, and full of clever ideas for really hard problems.

But I distinctly remember sitting in the room for the announcement, turning to my lead and whispering, “This project has no chance of succeeding, right?” He turned to me and just said, “Yup.” We both realized the problem immediately. The project was entirely based on a platform team asking a flagship product team to give up control of their core user flow: technically the right move, but no lead or PM would ever cede ownership of something that central to another team. Politically, this project was a total fantasy.

The project kept quietly chugging away in the background for almost two years. Every time it got close to launch, it would get pushed back as “not ready yet.” Over time, we heard less and less about it until, eventually, the inevitable “strategic pivot” email appeared in my inbox. Resources were reallocated and the code was deleted. We were told the company “learned a lot from the effort,” but to me it felt like it was doomed from the beginning. Politics and solving the correct problem matter just as much as technical beauty.

Why you cannot stop them all#

When I started noticing “bad projects” and I felt that I had some expertise to share, the temptation for me was to start calling them out. Reach out to the team doing it, tell them “this doesn’t make sense” and explain to them why. Use facts and logic to persuade.

And I did do this. But only for a very short time before I realized that there are a lot of costs to doing this that I just wasn’t thinking about.

Firstly, software companies have an inherent bias for action. They value speed and shipping highly. Concerns, by definition, slow things down and mean people have to look at things which they hadn’t budgeted for. And so unless your concern is big enough to overcome the “push for landing”, there’s little chance for any meaningful change to come from you saying something. In fact, it’s very likely that you’ll be largely ignored.

Related to this, even if the team does take your concern seriously, you have to be careful not to do it too often. Once or twice, you might be seen as someone who is upholding “quality”. But do it too often and you quickly move to being seen as a “negative person”, someone who is constantly a problem maker, not a problem “fixer”. You rarely get credit for the disasters you prevented. Because nothing happened, people forget about it quickly.

There’s also the problem that every time you push back, you are potentially harming someone’s promotion packet or a VP’s “pet project.” You are at risk of burning bridges and creating “enemies”, at least of a sort. Having a few people who disagree in a big company with you is the cost of doing business, but if you have too many, it starts affecting your main work too.

Finally, there is also the psychological impact. There is one of you and hundreds of engineers working in spaces that your expertise might help with. Your attention is finite, but the capacity for a large company to generate bad ideas is infinite. Speaking from experience, getting too involved in stopping these quickly can make you very cynical about the state of the world. And this is really not a good place to be.

Manage influence like a bank account#

So if you cannot stop all the bad projects, what do you do? You get strategic. Instead of trying to fix everything, view your influence as a bank account. You have a certain amount of “influence” coming in every month as you do your job, help people, ship successful projects, and generally remain low friction.

Then, when it matters, you should be ready to make “withdrawals.” Every time you block something or raise concerns, no matter how small, you are writing a check against your balance. But not all checks are the same size:

The $5 Check: A nitpick on a code review. Cheap, daily expense.
The $500 Check: Challenging an architectural decision or pushing back on a timeline. Requires some savings.
The $50,000 Check: Trying to kill a VP’s pet project. This is a massive spend. You might only afford this once every few years.

The problem comes if you spend $5 on every minor inefficiency you see. If you are constantly saying “no” to small things, your account will be empty when you need to write the big check to stop a true disaster.

If you “go overdrawn,” you enter political bankruptcy. People stop inviting you to meetings, they stop asking for your opinion, they essentially start working around you. Once you are bankrupt, your influence drops to zero and you not only harm your ability to influence things but also start hurting your own ability to get things done.

When to spend influence#

Given that we’ve now accepted that we cannot weigh in on everything, we need to figure out when it does make sense to do so.

The most important thing to do first is to be humble and evaluate whether you actually have the expertise to make a judgment. Seniority often brings opinions, but those are not always informed opinions. For example, while I have some frontend experience, I do not feel qualified to give deep advice on it because my knowledge is “enough to get by” rather than deep expertise that comes from long term ownership. It is easy to lose sight of the fact that high-quality judgments require informed opinions. If you find yourself in this position, see yourself as an opinionated observer and stop there.

You must also internalize the fact that just because you say something does not make it the truth. You are raising awareness of a point of view, not issuing a decree. So if some team doesn’t listen to your concerns and decides to go ahead with what they were doing anyway, then you have to accept that and move on: at the end of the day, you’re an engineer, not a CEO with authority over them!

Given these points, I use three main factors to decide when to speak up:

How close is the project to my team?
If it goes wrong, how much impact will it have on my team?
If it goes wrong, how big will the problem be for the company?

Proximity. If a project is close to you, the “price tag” of saying something is lower. If it is within your own team, the cost is near zero because you have high trust and a quick conversation often solves it. If it is in your broader organization, the price goes up; you have to spend social capital and potentially stake your reputation. If it is outside your org? The cost is often prohibitive. You have zero leverage, different reporting chains, and stopping it would require a massive withdrawal.

Team Impact. Sometimes another org does something that deeply affects your work. For example, because Perfetto (the performance tool I work on) has users throughout Google, sometimes a team will ask us to sign off on a very complex integration. This is a classic risk: if things go right, they get the credit, but if things go wrong, your leadership might expect you to help solve a problem you didn’t create. In these cases, the payoff of speaking up is high because you are protecting your team.

Company Scale. Finally, consider the blast radius. Some projects are self-contained; if they fail, they only take themselves down. Others are so intertwined with core systems that their failure causes widespread damage or creates technical debt that persists for years. These can be deadly to the long-term health of a project.

How to act with bad projects#

It’s also not just about when you put your opinions forward but how you do it. There’s a very wide range of actions you can take depending on what you’re facing.

When you intervene#

The nuclear option is to directly say “we should not do this” and try to shut the project down. This almost always requires escalation to your leads and the leads of the owning team, requiring great conviction in both the fact that you’re right and that this project will be actively harmful. But on some occasions, this is the right thing to do, especially if the cost of not saying something can be existential to your project or team.

A slightly softer but still quite risky variant of this is, instead of doing a direct escalation, you raise concerns in directly with the team. Usually this is done with a meeting with the team or a strongly worded “concern” or “rebuttal” doc. The goal is to speak in strong enough terms that the team themselves conclude that this the project might not be a good idea.

Then there are the smaller interventions, nudging things in the right direction. These are perfect for when a team is about to do something that makes sense from a high level but they are going about this the wrong way. I see this often with Perfetto: a team sends a design doc proposing a complex use of Perfetto that I know will cause them pain later. I sit down with them, understand their actual problem, and guide them to a better solution. It costs an hour but saves them months. If you do it right, you can even be seen as a helper rather than a hindrance, even if you do slow down the team.

When you don’t#

Sometimes you conclude that the ROI just isn’t there to do anything direct: the political momentum is too strong, or the issue is too small to justify spending any influence. At this point, what you do depends on how much your team is involved.

If it overlaps with your team’s work heavily then it might be best to make some subtle contingency plans: reducing your dependency on it or building abstractions to cope if it goes away. There is also a long game trick here. Even a bad project usually has an “essence” of a good idea, a specific problem it was trying to solve or an insight it was based on. If it fits with your job, it’s often a good idea to take that essence and see if you can naturally incorporate a better version of that specific solution into your own project. That way, if the bad project stalls or gets canceled, you can be proactive instead of reactive to the fallout.

Alternatively, if you’re not involved, it’s easy: just stay out of the picture. Vent to friendly colleagues in private, commiserate, but in public, live with the reality.

Managing your team through it#

Finally, you must manage your own team through the process. If you can see the flaws in a project, other senior engineers probably see them too. Don’t try to gaslight them or “walk the company line” by pretending a bad project is actually good. It destroys trust.

Instead, be honest about the facts on the ground without going into unnecessary political details. Tell them that you will do the best you can under these constraints.

Conclusion#

So what did I tell my mentee? “I’ve learned that being right and being effective are different things. I could go tell them my concerns. They probably wouldn’t listen. I’d burn some goodwill. And in six months, nobody will remember that I called it, they’ll just remember I was the guy who tried to block their work”.

When you’re earlier in your career, you want to believe that good ideas win on merit, that if you just explain clearly enough, people will see reason. It took me quite some time to accept that big companies don’t work that way.

But this doesn’t mean you stop caring. It means you get strategic about when to spend your credibility. Pick the battles where you can actually change the outcome, where your team will be hurt if you stay silent, where the cost of being wrong is low but the cost of the project failing is high.

And for everything else? You vent to colleagues, you make quiet contingency plans, and you watch. Sometimes you learn something. Sometimes you’re wrong and the project actually works. And sometimes you get to feel that grim satisfaction of predicting exactly how things would fall apart.

None of this is as satisfying as fixing everything. But it works and keeps me sane.

Why I Ignore The Spotlight as a Staff Engineer

Thu, 04 Dec 2025 00:15:00 +0000

Lately I’ve been reading Sean Goedecke’s essays on being a Staff+ engineer. His work (particularly Software engineering under the spotlight and It’s Not Your Codebase) is razor-sharp and feels painfully familiar to anyone in Big Tech.

On paper, I fit the mold he describes: I’m a Senior Staff engineer at Google. Yet, reading his work left me with a lingering sense of unease. At first, I dismissed this as cynicism. After reflecting, however, I realized the problem wasn’t Sean’s writing but my reading.

Sean isn’t being bleak; he is accurately describing how to deal with a world where engineers are fungible assets and priorities shift quarterly. But my job looks nothing like that and I know deep down that if I tried to operate in that environment or in the way he described I’d burn out within months.

Instead I’ve followed an alternate path, one that optimizes for systems over spotlights, and stewardship over fungibility.

We Live in Different Worlds#

The foundational reason for our diverging paths is that Sean and I operate in entirely different worlds with different laws governing them.

From Sean’s resume, my understanding is that he has primarily worked in product teams ¹ building for external customers. Business goals pivot quarterly, and success is measured by revenue or MAU. Optimizing for the “Spotlight” makes complete sense in this environment. Product development at big tech scale is a crowded room: VPs, PMs and UX designers all have strong opinions. To succeed, you have to be agile and ensure you are working specifically on what executives are currently looking at.

On the other hand, I’ve spent my entire career much more behind the scenes: in developer tools and infra teams.

My team’s customers are thousands of engineers in Android, Chrome, and throughout Google ². End users of Google products don’t even know we exist; our focus is on making sure developers have the tools to collect product and performance metrics and debug issues using detailed traces.

In this environment, our relationship with leadership is very different. We’re never the “hot project everyone wants,” so execs are not fighting to work with us. In fact, my team has historically struggled to hire PMs. The PM career ladder at Google incentivizes splashy external launches so we cannot provide good “promotion material” for them. Also, our feedback comes directly from engineers. Adding a PM in the middle causes a loss in translation, slowing down a tight, high-bandwidth feedback loop.

All of this together means our team operates “bottom-up”: instead of execs telling us “you should do X”, we figure out what we think will have the most impact to our customers and work on building those features and tools. Execs ensure that we’re actually solving these problems by considering our impact on more product facing teams.

Compounding Returns of Stewardship#

In the product environments Sean describes, where goals pivot quarterly and features are often experimental, speed is the ultimate currency. You need to ship, iterate, and often move on before the market shifts. But in Infrastructure and Developer Experience, context is the currency.

Treating engineers as fungible assets destroys context. You might gain fresh eyes, but you lose the implicit knowledge of how systems actually break. Stewardship, staying with a system long-term, unlocks compounding returns that are impossible to achieve on a short rotation.

The first is efficiency via pattern matching. When you stay in one domain for years, new requests are rarely truly “new.” I am not just debugging code; I am debugging the intersection of my tools and hundreds of diverse engineering teams. When a new team comes to me with a “unique” problem, I can often reach back in time: “We tried this approach in 2021 with the Camera team; here is exactly why it failed, and here is the architecture that actually works”.

But the more powerful return is systemic innovation. If you rotate teams every year, you are limited to solving acute bugs that are visible right now. Some problems, however, only reveal their shape over long horizons.

Take Bigtrace, a project I recently led; it was a solution that emerged solely because I stuck around long enough to see the shape of the problem:

Start of 2023 (Observation): I began noticing a pattern. Teams across Google were collecting terabytes or even petabytes of performance traces, but they were struggling to process them. Engineers were writing brittle, custom pipelines to parse data, often complaining about how slow and painful it was to iterate on their analysis.
Most of 2023 (Research): I didn’t jump to build a production system. Instead, I spent the best part of a year prototyping quietly in the background while working on other projects. I gathered feedback from these same engineers who had complained and because I had established long-term relationships, they gave me honest and introspective feedback. I learned what sort of UX, latency and throughput requirements they had and figured out how I could meet them.
End of 2023 to Start of 2024 (Execution): We built and launched Bigtrace, a distributed big data query engine for traces. Today, it processes over 2 billion traces a month and is a critical part of the daily workflow for 100+ engineers.

If I had followed the advice to “optimize for fungibility” (i.e. if I had switched teams in 2023 to chase a new project) Bigtrace would not exist.

Instead, I would have left during the research phase and my successor would have seen the same “noise” of engineers complaining. But without the historical context to recognize a missing puzzle piece, I think they would have struggled to build something like Bigtrace.

The Power of “No”#

One of the most seductive arguments for chasing the “Spotlight” is that it guarantees resources and executive attention. But that attention is a double-edged sword.

High-visibility projects are often volatile. They come with shifting executive whims, political maneuvering, and often end up in situations where long-term quality is sacrificed for short-term survival. For some engineers, navigating this chaos is a thrill. For those of us who care about system stability, it feels like a trap.

The advantage of stewardship is that it generates a different kind of capital: trust. When you have spent years delivering reliable tools, you earn the political capital to say “No” to the spotlight when it threatens the product.

Recently, the spotlight has been on AI. Every team is under pressure to incorporate it. We have been asked repeatedly: “Why don’t you integrate LLMs into Perfetto?” If I were optimizing for visibility, the answer would be obvious: build an LLM wrapper, demo it to leadership, and claim we are “AI-first.” It would be an easy win for my career.

But as a steward of the system, I know that one of Perfetto’s core values is precision. When a kernel developer is debugging a race condition, they need exact timestamps, not a hallucination. Users trust that when we tell them “X is the problem” that it actually is the problem and they’re not going to go chasing their tail for the next week, debugging an issue which doesn’t exist.

But it’s important not to take this too far: skepticism shouldn’t become obstructionism. With AI, it’s not “no forever” but “not until it can be done right” ³.

A spotlight-seeking engineer might view this approach as a missed opportunity; I view it as protecting what makes our product great: user trust.

The Alternate Currency of Impact#

The most common fear engineers have about leaving the “Spotlight” is career stagnation. The logic goes: If I’m not launching flashy features at Google I/O, and my work isn’t on my VP’s top 5 list, how will I ever get promoted to Staff+?

It is true that you lose the currency of “Executive Visibility.” But in infrastructure, you gain two alternate currencies that are just as valuable, and potentially more stable.

Shadow Hierarchy

In a product organization, you often need to impress your manager’s manager. In an infrastructure organization, you need to impress your customers’ managers.

I call this the Shadow Hierarchy. You don’t need your VP to understand the intricacies of your code. You need the Staff+ Engineers in other critical organizations to need your tools.

When a Senior Staff Engineer in Pixel tells their VP, “We literally cannot debug the next Pixel phone without Perfetto”, that statement carries immense weight. It travels up their reporting chain, crosses over at the Director/VP level, and comes back down to your manager.

This kind of advocacy is powerful because it is technical, not political. It is hard to fake. When you are a steward of a critical system, your promotion packet is filled with testimonials from the most respected engineers in the company saying, “This person’s work enabled our success”.

Utility Ledger

While product teams might be poring over daily active users or revenue, we rely on metrics tracking engineering health:

Utility: Every bug fixed using our tools is an engineer finding us useful. It is the purest measure of utility.
Criticality: If the Pixel team uses Perfetto to debug a launch-blocking stutter, or Chrome uses it to fix a memory leak, our impact is implicitly tied to their success.
Ubiquity: Capturing a significant percentage of the engineering population proves you’ve created a technical “lingua franca”. This becomes especially obvious when you see disconnected parts of the company collaborating with each other, using shared Perfetto traces as a “reference everyone understands”.
Scale: Ingesting petabytes of data or processing billions of traces proves architectural resilience better than any design doc.

When you combine Criticality (VIP teams need this) with Utility (bugs are being fixed), you create a promotion case that is immune to executive reorganizations.

Archetypes and Agency#

Staff Archetypes

I am far from the first to notice the idea of “there are multiple ways to be a staff software engineer”. In his book Staff Engineer, Will Larson categorizes Staff-plus engineers into four distinct archetypes.

Sean describes the Solver or the Right Hand: engineers who act as agents of executive will, dropping into fires and moving on once the problem is stabilized. I am describing the Architect or the Tech Lead: roles defined by long-term ownership of a specific domain and deep technical context.

The “Luck” Rebuttal

I can hear the criticism already: “You just got lucky finding your team. Most of us don’t have that luxury.”

There are two caveats to all my advice in this post. First, the strategy I have employed so far requires a company profitable enough to sustain long-term infrastructure. This path generally does not exist in startups or early growth companies; it is optimized for Big Tech.

Second, luck does play a role in landing on a good team. It is very hard to accurately evaluate team and company culture from the outside. But while finding the team might have involved luck, staying there for almost a decade was a choice.

And, at least in my experience, my team is not particularly special: I can name five other teams in Android alone ⁴. Sure, they might have a director change here or a VP change there, but the core mission and the engineering team remained stable.

The reason these teams seem rare is not that they don’t exist, but that they are often ignored. Because they don’t offer the rapid, visible “wins” of a product launch nor are they working on the “shiny cool features”, they attract less competition. If you are motivated by “shipping to billions of users” or seeing your friends and family use something you built, you won’t find that satisfaction here. That is the price of admission.

But if you want to build long-term systems and are willing to trade external validation for deep technical ownership, you just need to look behind the curtain.

Conclusion#

The tech industry loves to tell you to move fast. But there is another path. It is a path where leverage comes from depth, patience, and the quiet satisfaction of building the foundation that others stand on.

You don’t have to chase the spotlight to have a meaningful, high-impact career at a big company. Sometimes, the most ambitious thing you can do is stay put, dig in, and build something that lasts. To sit with a problem space for years until you understand it well enough to build a Bigtrace.

By product team I don’t mean “frontend team”: even as a backend engineer, you are still working on some part of what is being served directly to end users. ↩︎
This is not exhaustive, Perfetto is open source and we do also care about external developers but that’s not why we get paid. From the company perspective, time we spent on open source bugs is “wasted” time but we do it because we believe in the mission of open source. I talked about this more in a recent post, On Perfetto, Open Source, and Company Priorities. ↩︎
For what it’s worth, LLMs might not even be the best solution to “let’s put AI into Perfetto”: in my opinion there is lots of value with “old school” machine learning techniques like neural networks. A lot of trace analysis is just pattern matching. This is something I’m hoping to explore more in the coming year! ↩︎
Android Kernel, Android System Health, Android Runtime, Android Camera HAL, Android Bionic ↩︎