We’ve been noticing a few odd things lately with Google:

• New sites aren’t getting spidered – or not as quickly as earlier this year. Webmaster tools gives a generic message about the website not being listed in Google’s index, along with a link to a video that seems to mostly be about websites that get themselves banned for violating Google’s terms of service. Also existing websites that are growing aren’t always having the new content added as quickly as before – or rather it happens inconsistantly lately.
• Google PageRank tools don’t seem to be working any more. I’ve tried a number of them lately.
• While we’re on the topic of PageRank, it seems to be even less relevant than before. In one controlled scenario where we have many listings showing for a specific search term, a PR5 page is showing on the third page, while much lower PR websites are showing on the first page. Sorry, I can’t be more specific, but it is a fairly controlled scenario. All of the pages involved are similar in size with similar numbers of occurrences of this keyword.
• Searches are often slow. As far as I can tell, this isn’t just my internet connection. Its been years since I’ve had the Google homepage time out on me.
• Search listings sometimes change dramatically in short time periods.

All of the above seems to indicate that Google is gearing their entire system up for something big. Speculation among my staff says they’re going to try to make everything realtime (or close to it) in order to compete with Twitter. That means that they’re going to try and reorder all of their indexes very quickly (rather than weekly or possibly daily) in order to try and provide something closer to the immediate zeitgeist that one can obtain through Twitter.

Having some idea of the size of Google’s indices, and a vague notion that the number of servers in their demesne is in the low millions, the scale of this boggles my mind.