Australian Kitchen is akin to a web search engine. We have a web crawler, or ‘bot’, which crawls the internet and finds web pages with recipes on them. These are added to the Australian Kitchen index so we can refer our users to web sites with recipes that match their preferences.
For most sites, australian-kitchen-bot shouldn’t access your site more than once every few seconds on average. However, due to network delays, it’s possible that the rate will appear to be slightly higher over short periods. In general, australian-kitchen-bot should download only one copy of each page at a time. If you see that australian-kitchen-bot is downloading a page multiple times, it’s probably because the crawler was stopped and restarted.
australian-kitchen-bot was designed to be distributed on several machines to improve performance and scale as the web grows. Also, to cut down on bandwidth usage, we run many crawlers on machines located near the sites they’re indexing in the network. Therefore, your logs may show visits from several machines at Australian Kitchen, all with the user-agent
australian-kitchen-bot/1.0 (Australian Kitchen https://australian.kitchen/bot; email@example.com). Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server’s bandwidth.
If you want to stop australian-kitchen-bot from crawling recipes on your site, then you’ll be losing the opportunity to welcome hordes of referred visitors from Australian Kitchen. But if you feel you have to, then it’s pretty easy to block access to files and directories on your server using a robots.txt file. The robots.txt file will need to be in the top directory of your server to work.
Although we have never been asked to, we will respond to take down requests. If you would like us to remove content that you own from the Australian Kitchen index, please contact us. You will be expected to demonstrate that you own the content.