|
Yahoo Bot
I import my logs into a SQL Dbase for later reporting in crystal & i've noticed one thing when i only look at major bot visits... Yahoo seems to hit my robots.txt file all the time, like there will be 4 hits to that inbetween a hit to an actual page. Anyone noticed this? And it seems like they seem to hit the same pages over & over again and not hitting a lot of the interior pages. I have a sitemap and i have tons of links within the site, not sure why it can't seem to find othe pages to hit.
|