#HeadsUp, my entire website was scraped by a new #AI #LLM scraper #Ai2Bot . #UserAgent is:
Mozilla/5.0 (compatible) Ai2Bot-Dolma (+https://www.allenai.org/crawler)
Some of the IPs:
100.27.119.55
100.28.58.124
100.28.84.26
174.174.51.252
18.205.170.247
3.214.92.12
34.198.246.31
44.194.249.41
44.196.112.226
44.196.49.185
50.19.235.114
54.144.167.95
54.162.34.15
54.236.199.27
54.237.131.196
54.80.81.52
Would have blocked earlier if I had noticed. Reads robots.txt, no idea if respected.