How I Recovered a UniFi Aggregation Switch That Could Barely Stay Online

Sometimes troubleshooting isn't about knowing the right command. Sometimes it's about figuring out how to move seven megabytes across a network connection that refuses to stay alive long enough to transfer a text file.
This is the story of how I troubleshooted and eventually ended up performing a USW Aggregation firmware recovery. The switch had become increasingly unstable, couldn't successfully update its firmware, and eventually forced me into building a crude but surprisingly effective failure-resistant file transfer process.

The Problem

I had a USW Aggregation switch running UniFi Switch OS 6.6.61.

For quite some time it had been exhibiting two symptoms:

• Firmware updates consistently failed
• Network and management connections dropped randomly

Initially this looked like a simple failed upgrade.

It wasn't.

I disabled automatic updates and began trying every update method available:

• Standard controller-driven updates
• Cached updates
• Manual firmware uploads
• Download-and-install updates
• Factory resets
• SSH-based upgrades

Nothing worked.

In fact, after two complete factory resets, the behavior remained exactly the same.

At this point I strongly suspected the issue existed below the configuration layer.

The Cloud Console Wasn't Helpful

Normally I'd use UniFi's Cloud Debug Console.

Unfortunately, the debug sessions lasted roughly 30 seconds before being terminated.

By the time I authenticated and started investigating, the connection would disappear.

That made remote troubleshooting nearly impossible.

So I enabled SSH and connected directly.

Attempting Command-Line Updates

Ubiquiti documents several methods for manually updating switches through SSH.

A typical upgrade process looks something like:

curl -o /tmp/fwupdate.bin https://...

syswrapper.sh upgrade2 &

Simple enough.

Except every time I started downloading firmware, my SSH session immediately died.

This introduced a second problem.

Process Lifetime Was Tied to the Session

If the SSH connection terminates, the associated process often dies with it.

So even if the download had started successfully, it wasn't surviving long enough to complete.

My first thought was:

Easy. I'll use nohup.

Something like:

nohup curl -o /tmp/fwupdate.bin https://...

That would keep the process running after disconnect.

At least in theory.

Unfortunately the download still failed.

The Misleading TLS Error

During testing I kept seeing:

OpenSSL SSL_read: OpenSSL/1.1.1t: error:1408F119:lib(20):func(143):reason(281), errno 0

This sent me down the wrong path.

I assumed TLS negotiation was failing.

To eliminate encryption from the equation, I downloaded the firmware image locally and built a simple HTTP server that hosted the firmware over plain HTTP using node.js and express.js' serve static functionality.

No TLS. No certificates. No encryption.

The downloads still failed. At that point it became clear:

TLS wasn't the problem.

It was simply the first component noticing that the connection had disappeared.

The real issue was that the switch could not maintain a stable connection long enough to transfer the firmware image.

Working Inside BusyBox

One challenge that complicated recovery was the operating system itself.

The switch runs a very minimal environment.

There was:

• No Bash
• No zip
• No unzip
• No advanced utilities
• Limited process management tools

What I did have:

• BusyBox
• curl
• ssh
• sha256sum
• basic shell commands

Enough to survive, but not enough to be comfortable.

Things Continued to Get Worse

While troubleshooting I noticed something unusual.

The more often I got disconnected, the shorter my future sessions became.

After a reboot I might get a minute or two. After several failed attempts I might only get a few seconds.

Eventually I would have less than a second before the connection was terminated.

That observation became important later.

Because it suggested the system was degrading over time rather than failing at a fixed threshold.

Looking for Alternative Recovery Paths

At one point I opened the switch.

I hoped there might be:

• USB storage
• A recovery header
• An SD card
• Some other offline loading mechanism

No luck.

The flash storage was soldered directly to the board.

For anyone considering opening theirs:

I do not recommend it.

There wasn't an obvious recovery path available internally.

What If I Made the Transfer Failure-Tolerant?

At this point I stopped asking:

How do I transfer a 7 MB firmware file?

And started asking:

How do I transfer a 7 MB firmware file over a connection that constantly dies?

Those are very different problems.

My first attempt was writing a retrying download script.

The plan:

1. Start a download
2. Verify hash
3. Retry if validation failed

Unfortunately that was still dependent on maintaining long-lived transfers.

It failed too.

Thinking Like a Deep Space Engineer

Around this point I was reminded of how NASA communicates with distant spacecraft.

Those systems assume communication failures will occur.

They don't expect a perfect connection.

They assume an imperfect one and build around it.

I obviously didn't have access to deep-space communication protocols.

But I could steal the underlying idea:

Break big things into small things.

Splitting the Firmware Into Chunks

The firmware image was roughly 7 MB.

I wrote a PowerShell script that:

• Split the firmware into chunks
• Generated a SHA-256 hash for each piece
• Generated a SHA-256 hash for the final file

The process started with tiny chunks:

Test 1: 1KB

Success.

Test 2: 2KB

Success.

Test 3: 10 KB

Success.

Test 4: 100KB

Success.

Test 5: 200KB

Success.

Test 6: 500KB

Success.

Test 7: 1024KB

Failure.

Interesting.

I had finally found a boundary.

After additional testing I settled on:

512KB chunks

Large enough to be practical, small enough to transfer reliably.

Here is the final script I used to split the firmware file into multiple parts:

$File = "C:\example\path\here\fwupdate.bin"
$ChunkSize = 512KB
$OutDir = "$File.parts"

New-Item -ItemType Directory -Force -Path $OutDir | Out-Null

$stream = [System.IO.File]::OpenRead($File)
$buffer = New-Object byte[] $ChunkSize

$i = 0
while (($read = $stream.Read($buffer,0,$buffer.Length)) -gt 0) {
    $part = Join-Path $OutDir ("{0:D6}.part" -f $i)

    [System.IO.File]::WriteAllBytes(
        $part,
        $buffer[0..($read-1)]
    )

    $hash = (Get-FileHash $part -Algorithm SHA256).Hash.ToLower()
    "$hash  $(Split-Path $part -Leaf)" | Out-File -Append -Encoding ascii "$OutDir\parts.sha256"

    $i++
}

$stream.Dispose()

Verifying Every Transfer

Each chunk was transferred individually using WinSCP.

After every transfer I connected over SSH and verified integrity:

sha256sum 000000.part

If the hash matched, I moved to the next file.

Slow? Absolutely.

Reliable? Yes, finally!

For the first time during the entire recovery effort I was successfully getting data onto the device without corruption.

That was a huge breakthrough.

Reassembling the Firmware

After transferring all sixteen chunks, I reconstructed the firmware image directly on the switch.

cat 000000.part \
000001.part \
000002.part \
000003.part \
000004.part \
000005.part \
000006.part \
000007.part \
000008.part \
000009.part \
000010.part \
000011.part \
000012.part \
000013.part \
000014.part \
000015.part \
> fwupdate.bin

Then I verified the resulting file:

sha256sum fwupdate.bin

The resulting SHA-256 hash matched the firmware image provided by Ubiquiti.

After countless failed attempts, I finally had a valid firmware image on the switch.

The Moment of Truth

With the firmware in place, recovery was surprisingly simple.

syswrapper.sh upgrade2 &

The update process took approximately fifteen minutes.

Then the switch rebooted.

And everything changed.

Suddenly:

• SSH was stable
• Cloud Debug Console worked
• Management traffic stayed connected
• Firmware updates functioned normally

The switch was fully operational again.

Root Cause

Although I can't prove it conclusively without hardware-level forensic analysis, the evidence strongly points to firmware or operating system corruption.

Supporting evidence:

• Multiple factory resets did not resolve the issue
• Failures persisted across configuration resets
• The upgraded firmware immediately resolved all symptoms
• Connectivity issues disappeared after reflashing
• Update mechanisms themselves were unstable until the firmware was replaced

Lessons Learned

Don't Trust The First Error

The OpenSSL error looked significant. It wasn't.

TLS was simply where the failure became visible.

The actual problem was a collapsing connection.

Factory Reset Doesn't Fix Everything

A factory reset resets configuration.

It doesn't necessarily repair corrupted operating system components.

Verify Everything

Without SHA-256 verification I would have had no confidence in any transferred file.

Hash validation turned guesswork into certainty.

Reliability Beats Speed

When transferring a file over an unreliable connection, making the transfer smaller and verifiable can be more effective than trying to optimize throughput.

Sometimes the Solution Is Surprisingly Primitive

Modern networking protocols are incredibly sophisticated.

My final recovery method was essentially:

1. Split file
2. Copy file
3. Verify file
4. Repeat

It was not elegant, but it worked.

Final Thoughts

What started as a routine firmware update became an exercise in fault-tolerant communications, troubleshooting under severe constraints, and working around the limitations of an embedded operating system.

In the end, the fix was simple:

Overwrite the corrupted firmware with a newer version.

Getting the firmware onto the device was the difficult part.

Sometimes engineering isn't about finding the perfect solution.

It's about finding the solution that still works when everything else is broken.

---

Hardware: UniFi USW Aggregation
Original Version: 6.6.61
Recovered Version: 7.5.11
Recovery Method: Manual chunked file transfer + local firmware flash
Outcome: Fully operational system with stable management and update functionality.