News publishers are finally finding out the worth of their human-written content. They don’t want to give it for free to AI Giants. Now, Getting the latest information for the LLMs will not be easy as Apple’s AI crawlers are getting blocked.<\/p>\n\n\n\n
Highlights:<\/strong><\/p>\n\n\n\n So, this is what happened. Apple launched a tool that allows websites to opt out of their content from being accessed, especially for training their AI models. This is a great thing for website owners but maybe not for the iPhone manufacturer. At least, that’s what WIRED’s new report<\/a> says.<\/p>\n\n\n\n As we know, AI models like GPT-4o (technically you can say ChatGPT) need information to train their systems. They get this by getting content from websites on the internet. But they also need the latest up-to-date information about the current events to provide as much accurate information as they can. <\/p>\n\n\n\n So, they also want to get the information from news publishers. But that doesn’t mean they get it for free. They need permission from these sites, otherwise, it would be an infringement of Intellectual Property!<\/p>\n\n\n\n Big news publishers like The New York Times, The Financial Times, Vox Media, and The Atlantic have already chosen to block Apple’s AI crawlers from using their content.<\/strong><\/p>\n\n\n\n The new crawler called Applebot-Extended<\/a> gives control to publishers over how their content can be used to train Apple\u2019s foundation models that will later be used to power generative AI features across their products. <\/p>\n\n\n\n The company said, “Allowing Applebot-Extended will help improve the capabilities and quality of Apple\u2019s generative AI models over time<\/em>.” But why should news publishers give their hard-worked content for free?<\/p>\n\n\n\n News Publishers have updated their robots.txt files to block Applebot-Extended user agents. They\u2019re doing this to make sure their content isn\u2019t used without their permission. Vox Media\u2019s Lauren Starke said:<\/p>\n\n\n\n “We\u2019re blocking Applebot-Extended across all of Vox Media\u2019s properties, as we have done with many other AI scraping tools when we don\u2019t have a commercial agreement with the other party.”<\/p>\nLauren Starke, Vox Media<\/cite><\/blockquote>\n\n\n\n This could lead to changes in how AI models are trained and might also result in new deals where companies pay to access content. OpenAI has partnered with many news publishers<\/a> recently and they have made a deal with Stack Overflow<\/a> as well.<\/p>\n\n\n\n So, if the AI giants want content, they need to pay the price for it. Nothing comes free for them. The decisions being made now could have a big impact on how AI and online content work together in the future.<\/p>\n\n\n\n Note that the new user agent is an update to their original web crawler, Applebot. While the latter helps power search features like Siri, this new extension lets websites choose if their data can be used to train their AI models. News publishers don’t have a problem with Applebot but they are not ready for Applebot-Extended.<\/p>\n\n\n\n In a similar study, Ben Welsh has found that about 25% of the news websites out of 1,167 (mainly from the US) are blocking the AI crawlers as well.<\/p>\n\n\n\n Conclusion: Find out why many major news publishers are not allowing Apple’s Applebot crawlers to access their content for AI training.<\/p>\n","protected":false},"author":8,"featured_media":6362,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jnews-multi-image_gallery":[],"jnews_single_post":null,"jnews_primary_category":{"id":"","hide":""},"footnotes":""},"categories":[57],"tags":[56,97,332],"class_list":["post-6361","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","tag-ai","tag-apple","tag-applebot"],"_links":{"self":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/6361","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/comments?post=6361"}],"version-history":[{"count":2,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/6361\/revisions"}],"predecessor-version":[{"id":6365,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/6361\/revisions\/6365"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/media\/6362"}],"wp:attachment":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/media?parent=6361"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/categories?post=6361"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/tags?post=6361"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
Why Are Websites Blocking Apple\u2019s AI?<\/strong><\/h2>\n\n\n\n
Here’s why News Publishers are doing it.<\/strong><\/h2>\n\n\n\n
\n
<\/strong>The fight over AI data scraping is heating up, with more websites blocking their content from being used to train LLMs. This could change how AI is developed and how online content is protected. The choices being made today will likely shape the future of AI and digital content.<\/p>\n","protected":false},"excerpt":{"rendered":"