PassMark Logo
Home » Forum

Announcement

Collapse
No announcement yet.

zoom v5 and robots.txt

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • zoom v5 and robots.txt

    Hi I am trying to come up with a good robots.txt. This is my last try yet some URL,s were blocked that should not. Here is my robots.txt please for some help as to what I mite be doing wrong.

    User-agent: *
    Disallow:/

    User-agent: BadBot
    Disallow: /

    User-agent: *
    Allow:
    User-agent: *
    Allow:/support_center/
    User-agent: *
    Allow:/music/
    User-agent: *
    Allow:/searchworld/
    User-agent: *
    Allow:/forums/
    User-agent: *
    Allow:/band world/
    User-agent: *
    Allow:/band world/younggalaxy/

  • #2
    I would suggest you refer to The Web Robots FAQ here, which contains a detailed explanation and many examples:
    http://www.robotstxt.org/wc/faq.html

    Not sure what you are trying to achieve with the above robots.txt, but it is in many ways flawed. For example, the very first two lines blocks off the entire site, so all other "Allow:"s that you apply afterwards are irrelevant. In most cases, you only need to use "Disallow:"s as all other paths that are found or requested by the robot is assumed to be allowed. It is also unnecessary to specify the same User-agent repeatedly.

    Please see the FAQ above for more information.
    --Ray
    Wrensoft Web Software
    Sydney, Australia
    Zoom Search Engine

    Comment


    • #3
      thank you I found this one in the Google search. Liked it as the URL,s I want
      to be indexed are the lesser of all URl,s.It seemed to be less work as there are
      a lot of core URL,s to be denied!!!! this robots.txt has already found the trash bin!
      Last edited by Bruce; Jul-20-2007, 02:07 AM. Reason: smiles did not work

      Comment

      Working...
      X