New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Looking through logs, dawgy.pw has been crawled 23 times since Feb 1
A lot, especially after a new update.
It's different for everyone. Generally Google loves finding new content. You'll be assigned a crawl budget based on how important they think your site is and how often your pages change.
Google Chrome is a google bot
Almost every day. The time varies though. I'm running an unimportant forum with about 10 replies a day and something like three new topics a week. When i google its name using the Tor browser it ends up at page 3 or 4.
Website is there since 2010, and used to be lots more active in the beginning.
My sites exist to serve content, generally ad-free, so I don't really care about tracking any "metrics". That said, it's a pretty easy search to do:
Aaaaaaand Cloudflare has decided you don't get to see my
zgrephandiwork. LET really ought to shut that off. Summary is ~350 hits on an average day.I can't be bothered to look any of that up. It's combined logging for a handful of domains that I've had for over a decade with content that changes infrequently.
I think that some times
On a popular site here at Mexico, googlebot went crazy for weeks, having like... 5000 hits per minute or so from google addresses ¬_¬
You sure it was Google addresses and not Google user agent? We've seen a botnet attacking port 80 from numerous IPs, not belonging to Google, but what was distinctive, all requests tried to mimic Google bot user agent.
Two things
My mistake, was every ten minutes. Still too high.
And yes, verified with rdns and whois (most 66.x or 68.x something like that).
And magically got fixed. The client didn't wanted to modify the configs at the webmaster tools panel because they were afraid of losing positioning.
https://support.google.com/webmasters/answer/48620?hl=en
Yeah that Cloudflare stuff is super annoying...there's been a lot of times where I have typed up some long post and tried to include commands to help someone out only to have Cloudflare kick me at the last second and then I don't even feel like bothering with the post anymore.
Massive. Multiple times a day.
The CloudFlare WAF is completely customizable and they could fix it by checking the firewall event log, identifying the rule that's being triggered, and modifying or disabling it. I'm not sure why they haven't done that here.
Well on my site every week and it checks every url which I mentioned in sitemap.xml. There is frequency and I had it weekly.
Thank You!
I know u.u
But as I said, the clients didn't wanted to touch that
this..same with me..you can set it up with sitemap.xml
for me everytime there is new post/update only..