(from http://www.robotstxt.org/wc/norobots.html#examples)

The following example “/robots.txt” file specifies that no robots should visit any URL starting with “/cyberworld/map/” or “/tmp/”, or /foo.html:

# robots.txt for http://www.example.com/User-agent: *Disallow: /cyberworld/map/ # This is an infinite virtual URL spaceDisallow: /tmp/ # these will soon disappearDisallow: /foo.html

This example “/robots.txt” file specifies that no robots should visit any URL starting with “/cyberworld/map/”, except the robot called “cybermapper”:

# robots.txt for http://www.example.com/User-agent: *Disallow: /cyberworld/map/ # This is an infinite virtual URL space# Cybermapper knows where to go.User-agent: cybermapperDisallow:

This example indicates that no robots should visit this site further:

# go awayUser-agent: *Disallow: /[/code:1sf7nkiy]