TIL: there is an "sql_exporter". You give it a list of SQL queries, it exports the results as a prometheus compatible endpoint.

github.com/free/sql_exporter

Works quite nicely! Some basic support metrics are now done: number of tickets, number of open tickets, and average age. Tomorrow we'll point something at the endpoint to actually collect them somewhere :)

Show thread

aand finally moving MySQL on wirtanen.uberspace.de onto SSDs. That's the big 71GB one I was scared of earlier.

This will also mark 62 of 87 hosts migrated 馃帀

Show thread

Our 130 role ansible setup is of course not going anywhere soon. While we have numerous problems with ansible, the cost of rebuilding that by far outweighs the dirtiness of our workarounds. Maybe some day.

bundlewrap looks interesting to handle our in-host configuration management: getting user domains into the webserver config and so on. There we're also using ansible, but are _much_ less happy about it. I'll probably take a closer look next week.

Show thread

(Correction: bundlewrap is not young at all, but quite mature. I apparently cannot read github stats properly 馃榿)

Show thread

> bundlewrap.org is a decentralized configuration management system that is designed to be powerful, easy to extend and extremely versatile.

even though it seems to be a rather young project, it looks promising. Also, it's python so easy for us to contribute and fix.

Does anyone have any experiences you'd like to share?

Last week we migrated a couple of hosts onto the new cluster. Today they'll get their shiny MariaDB SSD device to speed things up. This causes a downtime of 10-40 minutes, depending on how much changes in the DB.

Watch is.uberspace.online for updates.

There are now >200 of you, beloved followers 馃帀 welcome to the party! 馃榿 do you have any wishes on what you'd like to see here?

We never collected hard numbers about our performance: time to initial reply, next reply, number of followups, number of open tickets, 鈥 We always knew that it would probably be smart, but it never seemed necessary. Some colleagues worked at big companies before, which judged their performance purely on numbers. We don't want that, but it complicates the topic a bit. But now is the time to define and collect (responsible!) metrics :)

So join us for grafana (鉂わ笍) and RT ran-- content next week! 馃榿

Show thread

next mini project: support metrics!

We're a bit slow at answering support emails right now, because everyone seems to have _a lot_ of time on their hands, which means many additional tickets.

Normally our support load changes quite slowly. If it doesn't (e.g. because of holidays, bad weather, ...), the change is temporary, so we can catch up quickly. This means that planning when to find new colleagues was rather easy. We just trusted our instincts.

Well, right now our instincts don't cut it!

aaand blogpost done! join on our adventure around storage performance, spinning rust, inefficient code and the future 馃弾锔 馃帀 (spoiler: it's now less inefficient and the future is bright.) blog.uberspace.de/u7-storage-p (German only, sorry :/)

鈽-Headsup: in the next release, we'll update from java 1.8 to java 14. It will be updated to 15 and so on in the future, tracking the newest version available in EPEL.

One of us has the rest of the week off and the other dev is busy remote-attending the PostgreSQL conference. Don't expect a lot of a fedi-traffic this week :) pgcon.org/2020/

I'm working on a blog post about the recent storage performance problems, the yum/rpm/mysql SSD migration, the qmail script fail, and future actions. It has many pretty graphs!

Anything in particular you'd like to know or would like me to expand on?

Which language(s) can you read comfortably? (don't worry, this account is very likely to keep English as its primary language.)

Fediverse tooling for our support team: 鉁 馃槑

(I would show you a screenshot, but checking all the pictured toots for their privacy status is a hassle. so you have to take my word for it.)

aand our tests concluded that the change improved the situation by exactly 0% 馃帀 I don't understand why, though 馃槓 the playbook adds RANDOM_DELAY=3 and reloads crond (just to be sure). But the peaks did not change at all.

Oh well, back to the drawing board!

Show thread

It's experiment time!

All the */5, */15 and */20 cronjobs lead to the ugly I/O access pattern shown below. Tonight we're adding a RANDOM_DELAY to each crontab to figure out, if that would help. This means your jobs might be delayed by a few minutes compared to their normal run times. Please bear with us! 馃惢

Every time I tell you, beloved followers, that we goofed something, you give it much more love, boosts and favs compared to something that did great on. So either you like the drama or you want us to goof more things!

Please explain 馃

Show more
Uberspace Mastodon

This server is for internal use only at the moment