Re: [dvd-discuss] Searchable list archives.

[microlenz(at)earthlink.net]

I don't know the reason but would be interested in hearing it. 

OTOH...the thought of Jack "Boots" Valenti and his Internet Jackboots descening 
upon my humble abode in less than 40minutes after making their decision to 
search my home for DVDs (I don't even own a PLAYER!) sends shivers up my 
spine...(Hey do you think they'd do anything during RUSH HOUR :-)..Hey at least 
Jim Tyre is up in Culver City!

Attempts at mirth aside...the Internet is more than just a broadcast medium. It 
can also be selective...

On 27 Apr 2002 at 22:14, Scott A Crosby wrote:

Date sent:      	Sat, 27 Apr 2002 22:14:22 -0400 (EDT)
From:           	Scott A Crosby <crosby@qwes.math.cmu.edu>
To:             	<dvd-discuss@eon.law.harvard.edu>
Copies to:      	<wendy@eon.law.harvard.edu>
Subject:        	[dvd-discuss] Searchable list archives.
Send reply to:  	dvd-discuss@eon.law.harvard.edu

> One of the things that sorta surprised me was that the openlaw DVD list
> wasn't coming up on any of my searches of 'Vivian's Vow'. I would have
> expected google to have indexed the archives by now and searches for
> 'Sara Glover' or 'Vivian's Vow' to be returning our messages.
> 
> But its not.
> 
> I checked out  http://cyber.law.harvard.edu/robots.txt
> 
> ---
> 
> #keep crawlers out of archive
> User-agent: *
> Disallow: /archive/
> Disallow: /lists/
> Disallow: /msdoj/discuss/
> 
> #allow cyber-search to get there
> User-agent: htdig/3.1.5 (wendy@eon.law.harvard.edu)
> Disallow:
> Disallow: /msdoj/discuss/
> 
> ---
> 
> And I think that this policy should be changed. Google-searchable list
> archives are a valuabale resource for ourselves, for others, and can serve
> as an advertisement of our cause.
> 
> So.. Whats the verdict about changing robots.txt to allow search engines
> (especially google) to index the list archives?
> 
> Scott
>