quickobot – QuickoLabs Robot

If you’re reading this, chances are you’ve seen our robot visiting your site while looking through your server logs. When we crawl to populate our index, we advertise the “User-agent” string “quickobot”. If you see the agent “quickobot”, that’s probably our system is crawling your website to build index.

We are building on-demand organic search marketing products, trying to build something useful for all the website owners. In order to provide accurate and reliable information, we regularly crawl the web and build our index.

We have recently initiated our crawlers and it would take few months before our first index is available. We intend to provide free service similar to Yahoo site explorer. In 2012, Yahoo Site explorer will discontinue their service, Google, Bing and other search engines provide limited access to backlink information, making it difficult for webmasters to analyze link data. QuickoLabs want to fill this gap by building exhaustive link index and providing service to webmasters across the globe.

Our bot does retrieve and parse robots.txt files, and it looks for robots META tags in HTML. These are the standard mechanisms for webmasters to tell web robots which portions of a site a robot is welcome to access. If you notice our bot misbehaving and not respecting rules, please drop us a line at Robots at QuickoLabs and we will investigate the problem.

Sysadmins/robots.txt

Our bot follows the robots.txt exclusion standard, which is described at http://www.robotstxt.org/wc/exclusion.html#robotstxt. Depending on the configuration, our robot may obey different rules. To make it simple to send our bot away, we’ll always obey rules for “qucikobot”. Here are the different cases.

When we’re running to populate our index, we’ll advertise the agent “quickobot”, and obey rules for “quickobot” if they exist.

To ban all bots from your site, place the following in your robots.txt file:
User-agent: *
Disallow: /

To ban only quickobot from your site, place the following in your robots.txt file:

User-agent: quickobot
Disallow: /

Webmasters/Robots META

If you do not have permission to edit the /robots.txt file on your server, you can still tell robots not to index your pages or follow your links. The standard mechanism for this is the robots META tag, as described at http://www.robotstxt.org/wc/meta-user.html.

To tell quickobot, and other robots, not to index your page or follow your links, insert this META tag into the HEAD section of your HTML document:

Of course, you can control the “index” and “follow” directives independently. The keywords “all” or “none” are also allowed, meaning “index,follow” or “noindex,nofollow”, respectively. Some examples are:


name="robots" content="index,follow">
content="index,nofollow">
content="noindex,follow">
content="none">

If there are no robots META tags, or if an action is not specifically prohibited (ie. neither “nofollow” or “none” appears), quickobot will assume it is allowed to index or follow links.