Uberspace Operations @ops@uberspace.social

While debugging another problem we observed some loss on our network at FRA3. After some debugging we found an IPv6 address from China Telecom that heavily scanned our network causing lots of ND traffic. We blocked them at 16:00 CEST and wow... It really decreased the loss and the spikes in the latency.

Kundin auf den Philippinen fragt warum IPv6 30% mehr Latenz hat als IPv4 und schickt einen Trace.

Stellt sich raus: IPv4 geht links rum, IPv6 recht rum um den Globus 😂

IPv4: Philippinen -> Hongkong -> Singapur -> Marseille -> Frankfurt

IPv6: Philippinen -> Minneapolis -> Chicago -> New York -> London -> Amsterdam -> Frankfurt

We wish our colleagues at OVH all luck for the hard work of recovery, thats one of the worst case scenarios. 🤗 #hugops

What a great feeling to resolve a ticket with our datacenter provider after over a year of work and a lot of unexpected problems. 🎉

It is totally okay to have some fun with the graphs while rebooting some VMs 🌈

Updated all switches and routers at all locations. Downtime less than 2 Minutes (zero will be possible in the future)! Hopefully the bug is now fixed.

We've planed everything for an update fixing an annoying bug today. And then we got hit by it in the night before. That's bad luck 😂

Our U7 monitoring watch dog 😍 🐶

The mitigation for the sudo vulnerability wasn't that great from a performance perspective. Thankfully there was a update available this morning ;)

A new cluster is born! 🎉

Someone tried to attack all WordPress installations with their xmlrpc.php on all our hosts at once 😂 Happily we were able to block the attack on the routers.

Looks like the hard reboot fixed it

The time on the host since epoch... right this is how time behaves...🧐

You really really don't want your log file to look like this while you are trying to debug an incident with your storage on a specific host.

So we jumped back nearly 30 minutes. Then we were in a time loop for 1 hour (image 2) and then jumped forward to the normal time again.

Did we fly our spaceship too close to a black hole? 🚀

Wenn ihr euer großes 25 Minuten Weihnachtsvideo hosten wollt, macht das doch lieber bei den Freundinnen von Peertube, da macht das mehr Spaß und unser Monitoring schreit auch nicht ;)

https://joinpeertube.org/instances

Tja dank altem Kernel werden aus geplanten 20-25 Minuten Downtime halt schnell mal 86 Minuten. Mist. Aber jetzt läuft wieder alles.

Wir haben mal wieder ein paar SSDs nachgeschoben. Das sieht wie immer sehr hübsch in den Graphen aus 😍

Still not migrated your old U6 account to the new and shiny U7 platform? Make us happy and do it this weekend! EOL is coming in 10 days.