The bug with four faces (and the air-gap hiding behind all of them)

TL;DR: A single broken feature in my Solana Seeker mobile app took four rounds to fix. Each round I found “the” root cause, fixed it, and the bug came back wearing a different mask. The real final blocker turned out to be an invisible trailing space in an environment variable — put there by a triple-click in a text file. The lesson isn’t “check for typos.” It’s that a plausible explanation that fits most of the evidence is more dangerous than no explanation at all.

---

I build a mobile app for the Solana Seeker. One of its core actions is a “quick mint” — you take a photo, it gets stored permanently, and it becomes an NFT. The flow, simplified: mint the token against a placeholder, upload the real image to permanent storage in the background, then swap the token’s metadata to point at the real image. Three steps. The user sees one tap.

It worked. Mostly. And “mostly” is where I lost about three days of my life.

Round one: the mint that succeeded but stayed a placeholder

First real test on mainnet — real money, permanent, no take-backs. I mint a photo. The transaction lands, the fee arrives in my treasury, everything the app tells me says success. I check the NFT in my wallet.

It’s still the placeholder. “Developing…”, forever.

Now, the tempting move here is to trust the app. The app said it worked. The chain agreed the mint happened. So surely it’s just… wallet lag? Give it time?

I don’t give things time. I have a background in AV — live sound, lighting, video systems — and you learn quickly that “it’ll probably sort itself out” is how you get a dead mic in front of 2,000 people. So I went looking.

The database row for that mint was frozen at the first step. The token was minted, paid for, real — but the record of it never advanced to the “upload the real image and swap it in” stage. The background job that was supposed to finish the job had never run.

Round two, three, four: the same bug, different mask

Here’s the part that makes this a story and not a bug report.

I handed the investigation to an AI agent I’ve been building this with. It dug in, found a genuine flaw — a code path that, on certain failures, deleted the record instead of preserving it — fixed it, and verified the fix. Clean work.

Except it didn’t match what I’d seen. My mint succeeded. It wasn’t rejected or deleted. So I pushed back: the fix you made is real and good, but it’s not the bug I’m looking at. Look again.

Round two: the actual cause was a mobile-network race. The app’s own “did the transaction confirm?” check would time out — a totally normal thing on a phone — and that timeout threw an error before the finalize step ever fired. The mint had landed on-chain; the app just gave up waiting and never kicked off the finish. Fixed: drive the finish from a signature captured at signing time, not from the flaky confirmation check. The chain is the source of truth, not the phone’s opinion of the chain.

Round three: fixing that exposed a new hole — the retry button. A “failed” mint that had actually landed could be retried, minting a second paid-for copy of the same photo. A double-charge. Fixed that too: before retrying, check whether the thing already landed and recover it instead of re-minting.

Round four: I test again. Mint defers with “transaction not visible to the RPC yet — retry this.” Correct behaviour now… except nothing ever retried it. The record sat there, stranded, forever. Another facet: a deferred finalize with no mechanism to actually re-drive it, and — the real kicker — the signature it needed to retry was never saved server-side, so nothing could find it to retry it. Fixed: save the signature the moment it exists, add a sweep that re-drives deferred finalizes on a schedule, independent of whether the app is even open.

Four rounds. Four genuine, distinct bugs, each hiding behind the last, all wearing the same costume: “quick mint doesn’t finalize.” I fixed all four.

I ran the sweep to recover my stuck test mints.

completed: 0.

The air-gap

Everything was fixed and it still didn’t work. The recovery would find the stranded mints, attempt them, and report every single one as “transaction not visible to the RPC yet” — including transactions that were nine hours old and absolutely, verifiably on-chain.

So I did the dullest possible thing. I took one of those “invisible” transactions and queried it directly against my RPC endpoint, by hand.

It came back instantly. Fully confirmed. Nine hours old. Right there.

So the transaction was visible to my RPC. But the code using that same RPC couldn’t see it. Which meant the code and I were not, in fact, talking to the same RPC — even though the config said we were.

I opened the environment file where I’d stored the RPC URL. It looked correct. Then I remembered how I’d put it there: I’d triple-clicked the URL in a text document to copy it. Triple-click selects the whole line — including the trailing whitespace. I’d pasted the URL into the secret with an invisible space on the end. Every request the background job made went to a subtly malformed endpoint that couldn’t find anything, and dutifully reported “not visible yet.”

The manual query worked because I’d typed that one clean.

Anyone who’s worked in AV knows this feeling in their bones. You check the console, the routing, the DSP, the network config, the clock sync — every abstract, sophisticated thing that could possibly be wrong — and then you find the XLR is half-unplugged. An air-gap. It’s almost always an air-gap. After all the clever solutions, the root cause is a physical-layer nothing.

I re-set the secret. Deployed. Ran the sweep.

completed: 4.

The stranded mints finalized. The images resolved. Real photos, on-chain, permanent. Done.

Lesson learnt

The four-faced bug taught me the obvious thing — check the simple stuff — but that’s not the real lesson, because I did eventually check the simple stuff and it was the last thing I checked, not the first.

The real lesson is about plausible explanations that fit most of the evidence. Every round, I had a story that explained the symptom. Every story was even true — those were all real bugs. But a story that accounts for 90% of what you’re seeing is more dangerous than admitting you don’t understand it, because it lets you stop looking. The 10% it doesn’t explain is the actual bug.

What broke the loop each time wasn’t a cleverer theory. It was going back to ground truth — the actual database row, the actual on-chain transaction, the actual raw RPC response — instead of trusting what any layer told me had happened. The app said success. The fix said solved. The logs said “not visible yet.” All three were confidently, plausibly wrong.

Trust the chain. Trust the row. Trust the raw response. Everything above that is just an opinion — and opinions, it turns out, will happily agree with each other all the way down to the trailing space.