Ecologia Digital<p>"…researchers estimate that in the 3 data sets—called C4, RefinedWeb and Dolma—5% of all data, and 25% of data from the highest-quality sources, has been restricted…set up through the <a href="https://mato.social/tags/RobotsExclusionProtocol" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsExclusionProtocol</span></a>, a method for website owners to prevent automated bots from crawling their pages using a file called <a href="https://mato.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a>."</p><p><a href="https://www.nytimes.com/2024/07/19/technology/ai-data-restrictions.html?unlocked_article_code=1.8k0.8eMA.cGAaZ0i10aZE&smid=nytcore-ios-share&referringSource=articleShare" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">nytimes.com/2024/07/19/technol</span><span class="invisible">ogy/ai-data-restrictions.html?unlocked_article_code=1.8k0.8eMA.cGAaZ0i10aZE&smid=nytcore-ios-share&referringSource=articleShare</span></a></p>
shakedown.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A community for live music fans with roots in the jam scene. Shakedown Social is run by a team of volunteers (led by @clifff and @sethadam1) and funded by donations.
Administered by:
Server stats:
271active users
shakedown.social: About · Status · Profiles directory · Privacy policy
Mastodon: About · Get the app · Keyboard shortcuts · View source code · v4.4.2
#robotsexclusionprotocol
0 posts · 0 participants · 0 posts today
Flipboard Tech Desk<p>Reddit announced on Tuesday that it’s updating its Robots Exclusion Protocol, Tech Crunch reports:<br><a href="https://techcrunch.com/2024/06/25/reddits-upcoming-changes-attempt-to-safeguard-the-platform-against-ai-crawlers/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">techcrunch.com/2024/06/25/redd</span><span class="invisible">its-upcoming-changes-attempt-to-safeguard-the-platform-against-ai-crawlers/</span></a></p><p><a href="https://flipboard.social/tags/Reddit" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Reddit</span></a> <a href="https://flipboard.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://flipboard.social/tags/Robotsexclusionprotocol" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Robotsexclusionprotocol</span></a></p>
Cory DoctorowLong thread/7
TrendingLive feeds
Mastodon is the best way to keep up with what's happening.
Follow anyone across the fediverse and see it all in chronological order. No algorithms, ads, or clickbait in sight.
Create accountLoginDrag & drop to upload