hachyderm.io is one of the many independent Mastodon servers you can use to participate in the fediverse.
Hachyderm is a safe space, LGBTQIA+ and BLM, primarily comprised of tech industry professionals world wide. Note that many non-user account types have restrictions - please see our About page.

Administered by:

Server stats:

10K
active users

So last week (on Sunday 1 December at 00:00), our server host canceled its service without warning.

TL;DR: we do not recommend using @Hetzner_Online 's service

Everyone else: a short 🧵

Murphy's law states that if things can go wrong, they will. Ideally in the worst possible way.

For us, that meant having our servers disconnected at 00:00 on a Sunday 1st (so likely a scheduled deprecation on their end).

Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong.

Turns out that Hetzner has decided to cancel our account and terminate all servers. There was no warning (yes, we checked our spam folder), and nobody could be reached before Monday morning.

When reached, they could not explain the reason for the cancellation:
Them: - We sent you an email.
Us : -We did not receive it, can you please resend?
Them: - We can't
Us: ಠ_ಠ

In the meantime, all servers had been wiped already so no way to retrieve our data.

If you are looking for a bad case of the Mondays, well, that was one.

@kiwix The poor communication is bad, but this is the worst. If you're terminating service, data should be held for a reasonable time, at least a month, unless it was manually inspected and deemed illegal even to possess (i.e. CSAM), to allow customer to retrieve/migrate it. Immediate deletion is a huge red flag.

@kiwix Even if you don't care about customers terminated fir violation of ToS, immediate deletion for them means same could happen to any customer by technical glitch or employee error. That should not be possible in decent professional hosting.

@dalias @kiwix Hetzner argued they sent a message you don't know when that happened, I have no reason not to believe them.

Someone will have filed an abuse notice due to copyright violations, hetzner will have sent an email and then terminated the account after not receiving a response, that's quite normal, isn't it?

Cassandrich

@juliank @kiwix No, there needs to be a large window between disabling of account and deletion of data.

@dalias @kiwix The Digital Services Act, article 6 requires a hoster in the EU (who like social networks are classified as a information society service):
"upon obtaining such knowledge or awareness, acts expeditiously to remove or to disable access to the illegal content."

In the case of a server where they don't control individual content, removing or disabling access to that content exclusively is not possible, so removing all data is the only thing they can do.

@dalias @kiwix Basically upon receiving a notice of copyright violation on any server of web hosting space, you need to delete the data immediately, or you will be held responsible for it.

You can't go into the server and find the infringing data and only delete that, that'd be prohibitively expensive and infringe on customer privacy rights potentially.

And then it's highly possible you also can't provide details to your customer because that would violate the privacy of the claimant.

@dalias @kiwix In fact, you can not only request the deletion of data but also the deletion of the request to delete the data.

In short, don't host content you don't control, and you don't get into trouble.

Re hosting other people's websites, or running a fediverse servers are clear cut violations. The former may be legal but put you at high risk.

The latter isn't really possible at all since fediverse servers copy content due to their nature and hence are in violation by their nature.

@juliank

> [...] upon receiving a notice of copyright violation [...], you need to delete the data immediately

That's not correct. There are a lot of ways to deny access besides "set it all on fire ASAP". Off the top of my head:

* VM: pause the machine.
* Separate iron: blackhole network traffic to it, or turn it off
* Behind some sort of reverse proxy: block URL in proxy config
* Shared hosting (maintained by the provider): update webserver config not to serve URLs with offending content

@farhaven Sure you can do all that but what's the point? The regulation here applies between the provider and the server operator, as in;

the server operator has uploaded illegal content to the server.

The content on the server hence must be deleted or rendered inaccessible to the server operator themselves.

@juliank Yes, and "render inaccessible" is about the same effort as "nuke entirely".

The problem here is 2-fold, I think:

- The report "there is illegal content on that server" may be spurious. I don't know if that is the case here, but it has definitely been in the past (even recently, see itch.io getting taken offline by a fraudulent DMCA claim). Setting everything on fire basically instantly completely removes the "there was a mistake here, let's quickly undo what we did".

1/2

@juliank The other aspect being that the folks at kiwix weren't even _aware_ that there was an issue they could've rectified themselves. Whether that is by lost email, email in a spam folder, someone overlooking the email, or any other reasons, just completely destroying everything after a warning that was seemingly not reacted to is an extreme measure.

Oh and: do we even know this was because of _actual_ "illegal content" (which, besides CSAM, depends a lot on the local jurisdiction)?

2/2

@farhaven We don't know if it was, but I have found trivial violations in 5 mins (they'd be considered fair use in the US, but no such thing here), anyone contributing to those mirrored sites could use them to torpedo the project.

@juliank @kiwix Um, no. They can disable access to the hosted server without wiping it.

See your own quoted text: "remove or to disable access".

@dalias @kiwix Yes but there's little point in denying kiwix access to their server and keeping it using resources and continue to bill them?

@juliank @kiwix Sure there is. It admits fixing a mistake without a huge PR disaster.

@dalias @kiwix Hetzner locked me out of my account and denies me access to my data while continuing to bill me for it isn't much better than just deleting it tbh.

Like once the action is taken either way, there's no way to get the data back because if you did get your data back it would not be inaccessible

@juliank @kiwix Sure it is. You contact them, figure out what's up and if you can remedy it, get customer service to let you in to take your things and move out, or get back up and running if it was in error.

Same principle as evicting landlord having to let you get your stuff, not being entitled to throw it away.

@dalias @kiwix you can't let them in and move data out because you are required by law to not let them have access to the data, again, as stated, it must be deleted or rendered inaccessible.

If you let them in and move it out you have just made it accessible again.

You don't upload a pirated movie to a hosting site, and then get the right to download it again after it gets flagged.

Here Hetzner is the hosting site, and the server disk contains the pirated content.

@juliank @kiwix "Block access" is about third party/public access to the data, not tenant's access to their own data.

@dalias @kiwix It's funny but it is what it is. Go complain to Brussels.

But if you think about it you create a file sharing association and host a server, it's never public, you are always sharing the data amongst yourselves.

The DSA still applies. Not to you as the file sharing service provider but to the hoster hosting you.

I don't see how to make sure you can get your own content back without introducing a whole bunch of loopholes like that tbh

@juliank @kiwix No hosting provider is going to get in legal trouble for preserving the disk pending contact with a representative of the tenant and allowing nothing but a disk image transfer via control panel or similar if the site isn't reinstated.

We're not talking about running a private warez server behind login on a live server with routed IP. Just single "moving out data".

But the more important part is possibility to reinstate in the event of human error. This doesn't require any exfil

@dalias @kiwix I think this is where it gets silly because the regulation isn't written for three parties.

To give an example, f you host a social network and have the issue, you delete the users content.

But now if you don't host the social network yourself, copyright owners can just complain to your hoster, and then ask them to delete your social network.

The provider has the option of forwarding this to you as their customer but oh well if you don't respond they need too act themselves

@juliank @kiwix They can act by turning off the server or disabling the routes to it. Not deleting it.