Uberspace Operations @ops@uberspace.social

Und jetzt sind beide Cluster auf Ceph Pacific, das lief wirklich ohne Probleme, sehr schön ;)

Damit sind alle Upgrades durch 🥳

Und C2FRA4 ist auch durch, das ging heute flüssig und ohne Probleme. Nächste Woche dann noch das Ceph auf beiden Clustern ;)

System und Proxmox Upgrades sind beim ersten Cluster in FRA3 fertig. Morgen gehts dann mit C2FRA4 weiter.

Aktuell upgraden wir alle Systeme auf Debian 11/Proxmox 7 und Ceph Pacific. Manche von euch haben dabei vielleicht einige Wackler bemerkt, das tut uns leid. Wir sind leider mehrfach in unerwartete Probleme mit ifupdown2 gelaufen, hoffen aber, dass das jetzt nicht mehr passiert.

mcnaught hat leider gerade am längsten Probleme gemacht. Dafür wandert er jetzt direkt von HDD auf den SSD Pool 🚀

While debugging another problem we observed some loss on our network at FRA3. After some debugging we found an IPv6 address from China Telecom that heavily scanned our network causing lots of ND traffic. We blocked them at 16:00 CEST and wow... It really decreased the loss and the spikes in the latency.

Kundin auf den Philippinen fragt warum IPv6 30% mehr Latenz hat als IPv4 und schickt einen Trace.

Stellt sich raus: IPv4 geht links rum, IPv6 recht rum um den Globus 😂

IPv4: Philippinen -> Hongkong -> Singapur -> Marseille -> Frankfurt

IPv6: Philippinen -> Minneapolis -> Chicago -> New York -> London -> Amsterdam -> Frankfurt

We wish our colleagues at OVH all luck for the hard work of recovery, thats one of the worst case scenarios. 🤗 #hugops

What a great feeling to resolve a ticket with our datacenter provider after over a year of work and a lot of unexpected problems. 🎉

It is totally okay to have some fun with the graphs while rebooting some VMs 🌈

Updated all switches and routers at all locations. Downtime less than 2 Minutes (zero will be possible in the future)! Hopefully the bug is now fixed.

We've planed everything for an update fixing an annoying bug today. And then we got hit by it in the night before. That's bad luck 😂

Our U7 monitoring watch dog 😍 🐶

The mitigation for the sudo vulnerability wasn't that great from a performance perspective. Thankfully there was a update available this morning ;)

A new cluster is born! 🎉

Someone tried to attack all WordPress installations with their xmlrpc.php on all our hosts at once 😂 Happily we were able to block the attack on the routers.

Looks like the hard reboot fixed it

The time on the host since epoch... right this is how time behaves...🧐

You really really don't want your log file to look like this while you are trying to debug an incident with your storage on a specific host.

So we jumped back nearly 30 minutes. Then we were in a time loop for 1 hour (image 2) and then jumped forward to the normal time again.

Did we fly our spaceship too close to a black hole? 🚀