Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
hachyderm.io is one of the many independent Mastodon servers you can use to participate in the fediverse.
Hachyderm is a safe space, LGBTQIA+ and BLM, primarily comprised of tech industry professionals world wide. Note that many non-user account types have restrictions - please see our About page.

Administered by:

Server stats:

9.4K
active users

hachyderm.io: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.7

Rudel Saldivar

For those who are content creators or publishers (blog content), some ways to protect your content from AI scraping (training data):

- Use the well known option to control bot site indexing, using robots.txt. There are reports that some AI bots don't respect the robots.txt, which is too bad.

- If you use CDN, you can interrogate the request header and block it at the edge before even reaching your site.

#ai#aibot#cloudflare
Jul 04, 2024, 08:29 AM·Public·Web
0boosts·0favorites
Public
Rudel Saldivar

Option 1:
You can block AI crawlers by adding them to your site’s robots.txt file as disallowed user agents (according to each AI company’s instructions.)
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: CCbot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: PiplBot
Disallow: /

Option 2:
Popular CDN providers like Cloudflare make this easy to do, see lnkd.in/gVdU9y7U

lnkd.inLinkedInThis link will take you to a page that’s not on LinkedIn
Public
Rudel Saldivar

You can also combine both 1 & 2 option.

ExploreLive feeds
About