[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[dvd-discuss] Searchable list archives.
- To: <dvd-discuss(at)cyber.law.harvard.edu>
- Subject: [dvd-discuss] Searchable list archives.
- From: Scott A Crosby <crosby(at)qwes.math.cmu.edu>
- Date: Sat, 27 Apr 2002 22:14:22 -0400 (EDT)
- Cc: <wendy(at)cyber.law.harvard.edu>
- Reply-to: dvd-discuss(at)cyber.law.harvard.edu
- Sender: owner-dvd-discuss(at)cyber.law.harvard.edu
One of the things that sorta surprised me was that the openlaw DVD list
wasn't coming up on any of my searches of 'Vivian's Vow'. I would have
expected google to have indexed the archives by now and searches for
'Sara Glover' or 'Vivian's Vow' to be returning our messages.
But its not.
I checked out http://cyber.law.harvard.edu/robots.txt
---
#keep crawlers out of archive
User-agent: *
Disallow: /archive/
Disallow: /lists/
Disallow: /msdoj/discuss/
#allow cyber-search to get there
User-agent: htdig/3.1.5 (wendy@eon.law.harvard.edu)
Disallow:
Disallow: /msdoj/discuss/
---
And I think that this policy should be changed. Google-searchable list
archives are a valuabale resource for ourselves, for others, and can serve
as an advertisement of our cause.
So.. Whats the verdict about changing robots.txt to allow search engines
(especially google) to index the list archives?
Scott