hachyderm.io is one of the many independent Mastodon servers you can use to participate in the fediverse.
Hachyderm is a safe space, LGBTQIA+ and BLM, primarily comprised of tech industry professionals world wide. Note that many non-user account types have restrictions - please see our About page.

Administered by:

Server stats:

10K
active users

One of the fall-outs of sorts from the maintenance this morning is it's identified a fairly significant storage issue across our servers. Right now there isn't a huge amount I can realistically do other than invest a lot of money in more servers (And that's not entirely financially viable with the current state of donations from here and Universeodon (And it impacts both)).

I'm looking to see if there are any cost effective solutions that don't introduce issues where the sites are more likely to run slow or otherwise crash but it's easier said than done.

Right now our database server host is running at 99.75% disk usage, and it's having some fairly significant operational impact behind the scene (As well as me loosing a lot of sleep over the fact) and while the container running the database has some headroom, it's not enough.

Our other database host for Universeodon is sitting at a bit over 80% disk usage, and we have one of our more general purpose application servers running at 80% and another at over 90% as well.

This means I'm starting to have to look at if the current scale we are running the site at is sustainable (For both here and Universeodon) and if reducing the amount of servers we have is going to be the only real option forward so we can invest in a larger capacity server for our DB's.

I'm fortunate enough to get a solid amount per month in Donations, with a relatively small chunk being monthly subscriptions (A lot of our monthly income is you kind folks giving ad-hoc contributions) / payments, however we're just starting to out grow our original architecture and design and I am locked in with some of our infrastructure to contractual obligations / purchase agreements which helped to save us a lot of money in the longer term.

MastodonApp.UK and Universeodon won't be going anywhere I want to make that very clear. I just need to figure out if there is a cost effective and scalable way to grow our storage capacity without causing more technical headaches or making things a lot worse. If I can't I will have a look into options to scale back some of our servers and reduce our footprint so we can make investments into new servers with the higher capacity we urgently need.

Emelia 👸🏻

@wild1145 it'd be very interesting to see where database storage is being consumed (e.g., is it all non-interacted with / non-addressed statuses?

Perhaps we need a mechanism in Mastodon to reduce these to just a reference to the remote server. Like you visit that post, it gets refetched & the page shows a "we're refetching this post, if you'd like to view it immediately, click here"

And this would only apply for non-local-mentioned posts & posts without interactions (likes, reblogs, replies)

@thisismissem We managed to shave around 80GB in our statuses table today through a vacume full and ultimately exporting and re-importing the database to fix the encoding issue. One of the bigger issues is needing to re-export the DB, re-build the database VM entirely and re-creating it as the disk has ballooned a lot larger than it actually needs, but at the same time we need 2.5x the size of the DB to be free on disk at any given time for effective / full backups of the DB.

@thisismissem And with a 150GB current size (ish) DB that's hit that critical tipping point where we're starting to see issues with backups reliably working, especially if one tries to run while another is already backing up.

In terms of posts / content, I suspect a lot of the content we store is never seen in terms of remote servers, but that was originally a comfortable trade-off I was willing to make for better content visibility within the server given fetching remote posts isn't a thing.

@wild1145 I am not sure why you need this space for backups. Are you doing SQL backups?
@thisismissem

@renchap @thisismissem Yes, currently it'll dump the database to disk, gzip it and upload it to a bucket. I'm just testing now that I think I have free'd up just enough space that the current VM in it's current config might just be able to perform a single backup at a time (Which was an issue with my old setup as it was backing up different backups but during overlapping times)

@renchap @thisismissem Actually I've just spotted that my number and wording was incorrect, I had meant to say we need more like 1.5x the db size to be free on the disk. That's me confusing total space needed and free space.

@wild1145 @renchap I think others are using pgbackrest for backups?

@wild1145 I will repeat what other said, but you really do not want to use dumps for backups at this scale. Use pgbackrest or another similar solution that relies on block backups, and do not require a copy of the data locally when backing up.
@thisismissem