Order of Allow & Disallow can make different search engines see your site differently. Google’s logic is quite different from that of Bing.

We are running an experiment to document the behavior & suggested approach. This post will be updated with the results.

Care need to be taken when using special characters such as space and !$&’()*+,-.:;<=>@[\]^_`{|}~

Search engines, browsers & web servers have differences in handling these characters.

Presently we are running an experiment to understand behavior of search engines. This blog will be elaborated with the result.