Shakedown Social

Kevin Karhan :verified:<a href="https://daedal.io/@thomzane" class="u-url mention" rel="nofollow noopener" target="_blank">@thomzane</a> the problem with <a href="https://robotstxt.org" rel="nofollow noopener" translate="no" target="_blank">https://robotstxt.org</a> / <a href="https://infosec.space/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a> is that <a href="https://social.tchncs.de/@cgudrian/114980303354716730" rel="nofollow noopener" target="_blank">it's a mere ask</a> and not something enforced.<ul><li>And with "enforced" I mean <code>.htaccess</code> or even better <a href="https://infosec.space/tags/iptables" class="mention hashtag" rel="nofollow noopener" target="_blank">#iptables</a> at the hoster's site.</li></ul>Personally I tend to go full "asshole mode" on <a href="https://infosec.space/tags/hostile" class="mention hashtag" rel="nofollow noopener" target="_blank">#hostile</a> actors like <a href="https://infosec.space/tags/NSAbook" class="mention hashtag" rel="nofollow noopener" target="_blank">#NSAbook</a>!

Kevin Karhan :verified:<a href="https://ohai.social/@lina" class="u-url mention" rel="nofollow noopener" target="_blank">@lina</a> np. I think it's important to highlight such projects like <a href="https://cuiiliste.de" rel="nofollow noopener" translate="no" target="_blank">https://cuiiliste.de</a> and actually point people at it in the hopes that <a href="https://mastodon.social/@ooni" class="u-url mention" rel="nofollow noopener" target="_blank">@ooni</a> and <a href="https://mastodon.social/@citizenlab" class="u-url mention" rel="nofollow noopener" target="_blank">@citizenlab</a> add it to their <a href="https://github.com/citizenlab/test-lists/" rel="nofollow noopener" target="_blank">test lists...</a><ul><li>Personally I did add it to my <a href="https://github.com/greyhat-academy/lists.d/blob/cacf5b9fbd80affd34d760c50b0b7333def000bf/blocklists.list.tsv#L28" rel="nofollow noopener" target="_blank">blocklist lists</a> not as an endorsement [far from it!] but as a point of reference for <a href="https://infosec.space/tags/CUII" class="mention hashtag" rel="nofollow noopener" target="_blank">#CUII</a>... </li></ul>Maybe someday I (or someone else) can get around and make a <a href="https://infosec.space/tags/VirusTotal" class="mention hashtag" rel="nofollow noopener" target="_blank">#VirusTotal</a> - style <a href="https://infosec.space/tags/URL" class="mention hashtag" rel="nofollow noopener" target="_blank">#URL</a> / <a href="https://infosec.space/tags/Domain" class="mention hashtag" rel="nofollow noopener" target="_blank">#Domain</a> checker that provides more than a mere "is clean" / "is sus" / "is dangerous" assessment but a more nuanced answers like:<ul><li>"This domain is blocked by CUII for alleged copyright infringement" </li></ul>or<ul><li>"This domain belongs to <a href="https://infosec.space/tags/NSAbook" class="mention hashtag" rel="nofollow noopener" target="_blank">#NSAbook</a>" </li></ul>and allow for granular, informed decisions (i.e. allow accessing it as a client, but blocking the entire <a href="https://infosec.space/tags/ASN" class="mention hashtag" rel="nofollow noopener" target="_blank">#ASN</a> from accessing one's Servers and thus block non-consensual <a href="https://infosec.space/tags/Scrapers" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scrapers</a> that violate <a href="https://infosec.space/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener" target="_blank">#RobotsTXT</a>.) <ul><li>I'm just shit at programming anything with a <a href="https://infosec.space/tags/GUI" class="mention hashtag" rel="nofollow noopener" target="_blank">#GUI</a> but hey, I'm just a <a href="https://infosec.space/tags/Sysadmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#Sysadmin</a> not a <a href="https://infosec.space/tags/Frontend" class="mention hashtag" rel="nofollow noopener" target="_blank">#Frontend</a> person...</li></ul>

Thom ZaneHow do we get Crawl-delay into the RFC 9309 robots.txt standard? Big tech hides behind RFC 9309 as an excuse to ignore Crawl-delay because they want to crawl as fast as they want.<a href="https://daedal.io/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a> <a href="https://daedal.io/tags/sysadmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#sysadmin</a>

teufelswerkSemrush ist eines der bekanntesten SEO-Analyse-Tools auf dem Markt. Es durchsucht Websites regelmäßig mit seinem Bot (SemrushBot), um Daten wie Keywords, Backlinks, Rankings und vieles mehr von deiner Website zu erfassen und zu analysieren. Hier sind 5 effektive, schnell umzusetzende Methoden, wie du Semrush von deiner Website aussperren kannst. 👇 <a href="https://teufelswerk.net/semrushbot-blockieren-so-schuetzt-du-jede-website-egal-ob-wordpress-joomla-typo3-oder-statisch/" rel="nofollow noopener" translate="no" target="_blank">https://teufelswerk.net/semrushbot-blockieren-so-schuetzt-du-jede-website-egal-ob-wordpress-joomla-typo3-oder-statisch/</a><a href="https://social.tchncs.de/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#SEO</a> <a href="https://social.tchncs.de/tags/semrush" class="mention hashtag" rel="nofollow noopener" target="_blank">#semrush</a> <a href="https://social.tchncs.de/tags/botblocker" class="mention hashtag" rel="nofollow noopener" target="_blank">#botblocker</a> <a href="https://social.tchncs.de/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#bots</a> <a href="https://social.tchncs.de/tags/website" class="mention hashtag" rel="nofollow noopener" target="_blank">#website</a> <a href="https://social.tchncs.de/tags/websecurity" class="mention hashtag" rel="nofollow noopener" target="_blank">#websecurity</a> <a href="https://social.tchncs.de/tags/cybersecurity" class="mention hashtag" rel="nofollow noopener" target="_blank">#cybersecurity</a> <a href="https://social.tchncs.de/tags/wordpress" class="mention hashtag" rel="nofollow noopener" target="_blank">#wordpress</a> <a href="https://social.tchncs.de/tags/joomla" class="mention hashtag" rel="nofollow noopener" target="_blank">#joomla</a> <a href="https://social.tchncs.de/tags/typo3" class="mention hashtag" rel="nofollow noopener" target="_blank">#typo3</a> <a href="https://social.tchncs.de/tags/nginx" class="mention hashtag" rel="nofollow noopener" target="_blank">#nginx</a> <a href="https://social.tchncs.de/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a> <a href="https://social.tchncs.de/tags/htaccess" class="mention hashtag" rel="nofollow noopener" target="_blank">#htaccess</a>

Christian Gudrian“robots.txt” is like leaving your front door open with a “please don’t steal here” sign next to it. <a href="https://social.tchncs.de/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebDev</a> <a href="https://social.tchncs.de/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a>

Jonathan BaileyI asked ChatGPT about the recent copyright news. It rehashed my latest column and misconstrued the facts. But why was it on my site at all?<a href="https://www.plagiarismtoday.com/2025/07/23/chatgpt-ignores-robots-txt-rehashes-my-column/" rel="nofollow noopener" translate="no" target="_blank">https://www.plagiarismtoday.com/2025/07/23/chatgpt-ignores-robots-txt-rehashes-my-column/</a><a href="https://mastodon.world/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a> <a href="https://mastodon.world/tags/ChatGPT" class="mention hashtag" rel="nofollow noopener" target="_blank">#ChatGPT</a> <a href="https://mastodon.world/tags/OpenAI" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenAI</a> <a href="https://mastodon.world/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#RobotsTxt</a>

George E. 🇺🇸♥🇺🇦🇵🇸🏳️‍🌈🏳️‍⚧️Here's <a href="https://bofh.social/tags/Cloudflare" rel="nofollow noopener" target="_blank">#Cloudflare</a>'s <a href="https://bofh.social/tags/robots-txt" rel="nofollow noopener" target="_blank">#robots-txt</a> file: <pre><code># Cloudflare Managed Robots.txt to block AI related bots. User-agent: AI2Bot Disallow: / User-agent: Amazonbot Disallow: / User-agent: amazon-kendra Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: bigsur.ai Disallow: / User-agent: Brightbot Disallow: / User-agent: Bytespider Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Diffbot Disallow: / User-agent: DigitalOceanGenAICrawler Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: FacebookBot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: iaskspider/2.0 Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: img2dataset Disallow: / User-agent: Kangaroo Bot Disallow: / User-agent: LinerBot Disallow: / User-agent: MachineLearningForPeaceBot Disallow: / User-agent: Meltwater Disallow: / User-agent: meta-externalagent Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: Nicecrawler Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: PanguBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: PetalBot Disallow: / User-agent: PiplBot Disallow: / User-agent: QualifiedBot Disallow: / User-agent: Scoop.it Disallow: / User-agent: Seekr Disallow: / User-agent: SemrushBot-OCOB Disallow: / User-agent: Sidetrade indexer bot Disallow: / User-agent: Timpibot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: YouBot Disallow: /</code></pre>

smegI've had the robots.txt to block ChatGPT from touching my site in place for months. Yet it's a referrer?<a href="https://assortedflotsam.com/tags/chatgpt" class="mention hashtag" rel="nofollow noopener" target="_blank">#chatgpt</a> <a href="https://assortedflotsam.com/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#llm</a> <a href="https://assortedflotsam.com/tags/privacy" class="mention hashtag" rel="nofollow noopener" target="_blank">#privacy</a> <a href="https://assortedflotsam.com/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a>

zeyus :gopnik_implant_b:‎Hey does anyone know if there's still a working zip bomb style exploit that can be deployed on a static site/JS (or as a asset/resource)? Specifically to target web scrapers and AI bullshit? The second any server goes online now it's immediately bombarded by stupid numbers of requests.<a href="https://corteximplant.com/tags/hacking" class="mention hashtag" rel="nofollow noopener" target="_blank">#hacking</a> <a href="https://corteximplant.com/tags/aislop" class="mention hashtag" rel="nofollow noopener" target="_blank">#aislop</a> <a href="https://corteximplant.com/tags/crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawlers</a> <a href="https://corteximplant.com/tags/webscraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#webscraping</a> <a href="https://corteximplant.com/tags/webcrawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#webcrawler</a> <a href="https://corteximplant.com/tags/robots" class="mention hashtag" rel="nofollow noopener" target="_blank">#robots</a> <a href="https://corteximplant.com/tags/zipbomb" class="mention hashtag" rel="nofollow noopener" target="_blank">#zipbomb</a> <a href="https://corteximplant.com/tags/zipbombing" class="mention hashtag" rel="nofollow noopener" target="_blank">#zipbombing</a> <a href="https://corteximplant.com/tags/exploit" class="mention hashtag" rel="nofollow noopener" target="_blank">#exploit</a> <a href="https://corteximplant.com/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a> <a href="https://corteximplant.com/tags/server" class="mention hashtag" rel="nofollow noopener" target="_blank">#server</a> <a href="https://corteximplant.com/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a>

Dawn Tåke 🌙 :sparkletrans:Hi, got a question.Is there a standard for Anti-AI/Anti-SEO etc robots.txt file? Or a trustworthy site that explains how to build one if prefab isn't available? Is there anything else I should consider? Thanks.<a href="https://tech.lgbt/tags/AskFedi" class="mention hashtag" rel="nofollow noopener" target="_blank">#AskFedi</a> <a href="https://tech.lgbt/tags/TechHelp" class="mention hashtag" rel="nofollow noopener" target="_blank">#TechHelp</a> <a href="https://tech.lgbt/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener" target="_blank">#RobotsTXT</a> <a href="https://tech.lgbt/tags/RobotsDotTXT" class="mention hashtag" rel="nofollow noopener" target="_blank">#RobotsDotTXT</a>

Preston Maness ☭<a href="https://www.tiktok.com/@alberta.nyc/video/7465916806939659563?lang=en" rel="nofollow noopener" translate="no" target="_blank">https://www.tiktok.com/@alberta.nyc/video/7465916806939659563?lang=en</a><a href="https://tenforward.social/tags/DeepSeek" class="mention hashtag" rel="nofollow noopener" target="_blank">#DeepSeek</a> <a href="https://tenforward.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://tenforward.social/tags/OpenAI" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenAI</a> <a href="https://tenforward.social/tags/China" class="mention hashtag" rel="nofollow noopener" target="_blank">#China</a> <a href="https://tenforward.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a>

Kevin Karhan :verified:<a href="https://infosec.space/@fennix" class="u-url mention" rel="nofollow noopener" target="_blank">@fennix</a> the fact that neither <a href="https://social.bund.de/@bsi" class="u-url mention" rel="nofollow noopener" target="_blank">@bsi</a> nor <a href="https://ec.social-network.europa.eu/@EUCommission" class="u-url mention" rel="nofollow noopener" target="_blank">@EUCommission</a> make honoring <a href="https://infosec.space/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener" target="_blank">#RobotsTXT</a> legally mandatory under penalty of fines and forced disconnects is a problem.<a href="https://infosec.space/tags/WhatYouAllowIsWhatWillContinue" class="mention hashtag" rel="nofollow noopener" target="_blank">#WhatYouAllowIsWhatWillContinue</a> applies here and I kniw some folks intent to literally ban entire ASNs for hosting crawlers because those literally <a href="https://infosec.space/tags/DDoS" class="mention hashtag" rel="nofollow noopener" target="_blank">#DDoS</a> sites offline and criminally incompetent, value-removing middlemen like <a href="https://infosec.space/tags/ClownFlare" class="mention hashtag" rel="nofollow noopener" target="_blank">#ClownFlare</a> <a href="https://www.youtube.com/watch?v=Hi5sd3WEh0c" rel="nofollow noopener" target="_blank">do jack shit about even when tasked to do so.</a><a href="https://infosec.space/tags/sarcasm" class="mention hashtag" rel="nofollow noopener" target="_blank">#sarcasm</a> <a href="https://infosec.space/tags/vent" class="mention hashtag" rel="nofollow noopener" target="_blank">#vent</a> <a href="https://infosec.space/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a> <a href="https://infosec.space/tags/LLM" class="mention hashtag" rel="nofollow noopener" target="_blank">#LLM</a> <a href="https://infosec.space/tags/Enshittification" class="mention hashtag" rel="nofollow noopener" target="_blank">#Enshittification</a>

Kevin Karhan :verified:<a href="https://mastodon.neilzone.co.uk/@neil" class="u-url mention" rel="nofollow noopener" target="_blank">@neil</a> <a href="https://kind.social/@ThreeGerbilsInACoat" class="u-url mention" rel="nofollow noopener" target="_blank">@ThreeGerbilsInACoat</a> also note that <a href="https://infosec.space/tags/InternetArchive" class="mention hashtag" rel="nofollow noopener" target="_blank">#InternetArchive</a> disregards the <a href="https://infosec.space/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener" target="_blank">#RobotsTXT</a> file...

Ecologia Digital<a href="https://mato.social/tags/Robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#Robotstxt</a> <a href="https://mato.social/tags/CrawlerBacklash" class="mention hashtag" rel="nofollow noopener" target="_blank">#CrawlerBacklash</a> Trickle-down effects: "people start blocking all crawlers, and some crawlers are very important, for search indexing, internet archiving, some are used for academic research, and so the bad behaviours of all these <a href="https://mato.social/tags/AIcompanies" class="mention hashtag" rel="nofollow noopener" target="_blank">#AIcompanies</a>, and the backlash to it, is kind of fundamentally changing how the Internet works, how it is remembered and indexed..." <a href="https://pca.st/yto6v3il?t=11m34s" rel="nofollow noopener" translate="no" target="_blank">https://pca.st/yto6v3il?t=11m34s</a>

🧿🪬🍄🌈🎮💻🚲🥓🎃💀🏴🛻🇺🇸Y'all really putting a file on your webserver that says "don't look *here* if you're a bot!" and expecting people not to look there first 🤡 <a href="https://mastodon.social/tags/webDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#webDev</a> <a href="https://mastodon.social/tags/robotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotsTxt</a> <a href="https://mastodon.social/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#bots</a> <a href="https://mastodon.social/tags/search" class="mention hashtag" rel="nofollow noopener" target="_blank">#search</a> <a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mastodon.social/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#llm</a> <a href="https://mastodon.social/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://mastodon.social/tags/privacy" class="mention hashtag" rel="nofollow noopener" target="_blank">#privacy</a> <a href="https://mastodon.social/tags/web" class="mention hashtag" rel="nofollow noopener" target="_blank">#web</a>

Ecologia Digital"…the <a href="https://mato.social/tags/backlash" class="mention hashtag" rel="nofollow noopener" target="_blank">#backlash</a> to AI tools from content creators and website owners who do not want their work to be used for AI training purposes without permission or compensation is not only real but is becoming increasingly widespread. The analysis also highlights the limitations of robots.txt—while many companies respect robots.txt instructions, some do not. Perplexity have been caught circumventing & ignoring <a href="https://mato.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a>."<a href="https://www.404media.co/the-backlash-against-ai-scraping-is-real-and-measurable/" rel="nofollow noopener" translate="no" target="_blank">https://www.404media.co/the-backlash-against-ai-scraping-is-real-and-measurable/</a>

Ecologia Digital"…researchers estimate that in the 3 data sets—called C4, RefinedWeb and Dolma—5% of all data, and 25% of data from the highest-quality sources, has been restricted…set up through the <a href="https://mato.social/tags/RobotsExclusionProtocol" class="mention hashtag" rel="nofollow noopener" target="_blank">#RobotsExclusionProtocol</a>, a method for website owners to prevent automated bots from crawling their pages using a file called <a href="https://mato.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a>."<a href="https://www.nytimes.com/2024/07/19/technology/ai-data-restrictions.html?unlocked_article_code=1.8k0.8eMA.cGAaZ0i10aZE&smid=nytcore-ios-share&referringSource=articleShare" rel="nofollow noopener" translate="no" target="_blank">https://www.nytimes.com/2024/07/19/technology/ai-data-restrictions.html?unlocked_article_code=1.8k0.8eMA.cGAaZ0i10aZE&smid=nytcore-ios-share&referringSource=articleShare</a>

🧿🪬🍄🌈🎮💻🚲🥓🎃💀🏴🛻🇺🇸Web scrapers work by finding URLs in a page and then visiting those URLs to find more URLs recursively.What's stopping us from serving them infinite trees of URLs, filled with random garbage?<a href="https://mastodon.social/tags/webDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#webDev</a> <a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a> <a href="https://mastodon.social/tags/scrapers" class="mention hashtag" rel="nofollow noopener" target="_blank">#scrapers</a> <a href="https://mastodon.social/tags/scraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraping</a>

🧿🪬🍄🌈🎮💻🚲🥓🎃💀🏴🛻🇺🇸Blocking robots.txt is not very cyberpunk<a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mastodon.social/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#llm</a> <a href="https://mastodon.social/tags/cyberpunk" class="mention hashtag" rel="nofollow noopener" target="_blank">#cyberpunk</a> <a href="https://mastodon.social/tags/openAI" class="mention hashtag" rel="nofollow noopener" target="_blank">#openAI</a> <a href="https://mastodon.social/tags/perplexity" class="mention hashtag" rel="nofollow noopener" target="_blank">#perplexity</a> <a href="https://mastodon.social/tags/anthropic" class="mention hashtag" rel="nofollow noopener" target="_blank">#anthropic</a> <a href="https://mastodon.social/tags/gemini" class="mention hashtag" rel="nofollow noopener" target="_blank">#gemini</a> <a href="https://mastodon.social/tags/claude" class="mention hashtag" rel="nofollow noopener" target="_blank">#claude</a> <a href="https://mastodon.social/tags/stableDiffusion" class="mention hashtag" rel="nofollow noopener" target="_blank">#stableDiffusion</a> <a href="https://mastodon.social/tags/aiArt" class="mention hashtag" rel="nofollow noopener" target="_blank">#aiArt</a> <a href="https://mastodon.social/tags/llms" class="mention hashtag" rel="nofollow noopener" target="_blank">#llms</a> <a href="https://mastodon.social/tags/chatGPT" class="mention hashtag" rel="nofollow noopener" target="_blank">#chatGPT</a> <a href="https://mastodon.social/tags/copilot" class="mention hashtag" rel="nofollow noopener" target="_blank">#copilot</a> <a href="https://mastodon.social/tags/robots" class="mention hashtag" rel="nofollow noopener" target="_blank">#robots</a> <a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a> <a href="https://mastodon.social/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://mastodon.social/tags/search" class="mention hashtag" rel="nofollow noopener" target="_blank">#search</a> <a href="https://mastodon.social/tags/google" class="mention hashtag" rel="nofollow noopener" target="_blank">#google</a> <a href="https://mastodon.social/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a>

🧿🪬🍄🌈🎮💻🚲🥓🎃💀🏴🛻🇺🇸AI scraped all your photos so that I could look up the names of flowers which completely justifies the scraping in my mind.<a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#robotstxt</a> <a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mastodon.social/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#llm</a> <a href="https://mastodon.social/tags/stableDiffusion" class="mention hashtag" rel="nofollow noopener" target="_blank">#stableDiffusion</a> <a href="https://mastodon.social/tags/openai" class="mention hashtag" rel="nofollow noopener" target="_blank">#openai</a> <a href="https://mastodon.social/tags/copilot" class="mention hashtag" rel="nofollow noopener" target="_blank">#copilot</a> <a href="https://mastodon.social/tags/gemini" class="mention hashtag" rel="nofollow noopener" target="_blank">#gemini</a> <a href="https://mastodon.social/tags/ip" class="mention hashtag" rel="nofollow noopener" target="_blank">#ip</a> <a href="https://mastodon.social/tags/copyright" class="mention hashtag" rel="nofollow noopener" target="_blank">#copyright</a>

Recent searches

Search options

Administered by:

Server stats:

#robotstxt