Net-Teams, Inc.
HOME | Membership Websites | SMM Solutions | CRM Solutions | Online Training Systems | Publishing | Clients | Guarantee | Log In

Harnessing the Power of Robots.txt

4
Submitted by Writer's Cramp Syndications | RSS Feed | Add Comment | Bookmark Me!

Once we have a website up and running, we need to make sure that all visiting search engines can access all the pages we want them to look at.

Sometimes, we may want search engines to not index certain parts of the site, or even ban other SE from the site all together.

This is where a simple, little 2 line text file called robots.txt comes in.

Robots.txt resides in your websites main directory (on LINUX systems this is your /public_html/ directory), and looks something like the following:

User-agent: *
Disallow:

The first line controls the “bot” that will be visiting your site, the second line controls if they are allowed in, or which parts of the site they are not allowed to visit…

If you want to handle multiple “bots”, then simple repeat the above lines.
So an example:

User-agent: googlebot
Disallow:

User-agent: askjeeves
Disallow: /

This will allow Goggle (user-agent name GoogleBot) to visit every page and directory, while at the same time banning Ask Jeeves from the site completely.
To find a “reasonably” up to date list of robot user names this visit http://www.robotstxt.org/wc/active/html/index.html

Even if you want to allow every robot to index every page of your site, it’s still very advisable to put a robots.txt file on your site. It will stop your error logs filling up with entries from search engines trying to access your robots.txt file that doesn’t exist.

For more information on robots.txt see, the full list of resources about robots.txt at http://www.websitesecrets101.com/robotstxt-further-reading-resources

Click for Details --> Customer Generating Machine <--


Contact Us
Support and Sales
Contact Us

LinkedIn Recommendation: Jonah William - Owner at Tessier Marketing - I thought I was well versed in social media before I found Teo. He has helped me round my skill set with great information only he seems to teach. I went on the web and looked for it later, but it's just not there. The ShareThis.com counters for instance, which definitely don't work right... can't find that anywhere, but it's easy to prove once you know what to test for! Teo really has a way of getting to what works and what works best! - March 17, 2012, Jonah was Teo's client

Welcome!

Search Articles On Net-Teams

Featured [Domains] Articles:
Net-Teams - Helping Businesses Prosper With Custom CRM, SMM and Online Training - Net-Teams, Inc. (NTI) is a technology and marketing firm and offers access to a core set of system t...
The Benefits Of A Membership Program For Your Website - Building membership through your website allows you to automate the acquisition of prospects and cus...
eWorkshop Hosting - The More Effective Way to Build Your Business with Online Ed - More and more companies are using eWorkshops to reach out to customers, prospects and employees. An ...
eWorkshop Publishing From Net-Teams - As many people are discovering, self-publishing is a time consuming venture, which takes time away f...
What is Social Media Management And Why Is It So Critical? - Whether or not you have a customer relationship management (CRM) system in place, there is one key r...

Related Tags (related articles): Domains (44), search engines (186), web permissions (1), search engine optimization (96), seo (244)