UdmSearch v2.0  by UdmSearch Developers Team.
Search engine over the web based on MySQL database.
Please see the file COPYING for license information
===================================================


General
-------

UdmSearch is a MySQL based search engine over the web.
UdmSearch consists of two parts. The first part is indexing mechanism (indexer).
Indexer walks over html hypertext references and stores found words and
new references into mysql database. The second part is cgi program
(search.cgi or search.php3 if you prefer php) to find words using 
data created by indexer.


Main features are:
* HTTP support
* proxy support
* FTP via proxy support
* text/html with <a href="">,<area href=""> and <frame src="">
and text/plain support
* basic authorization support. One could easy index password protected
intranet HTTP servers.
* it's reenterant. One could use several indexing processes at the same time.
* stop-list support
* flexible configurability of indexed url
* <MATA NAME="robots" content="..."> and robots.txt support
* both C cgi and PHP3 support
* boolean query language support
* best documents are found first
* it's very fast (thanks Monty for greatest database)
* russian koi8-r and windows-1251 character set support


Where to get it?
----------------

Check for latest version of UdmSearch at:
ftp://ftp.izhcom.ru/pub/unix/UdmSearch/
http://search.udm.net/about/

Mailing list
------------
There is now mailing list related to UdmSearch:

udmsearch@search.udm.net is list for discussion of UdmSearch and
related announcements.

To subscribe to list: 
send latter with line "subscribe udmsearch" in the body 
to majordomo@search.udm.net


How do I use it?
----------------

Please read run/indexer.conf and other *.conf samples first 
and take a look at create/create.txt.

Note that UdmSearch does not index common words like  "i","am","she","he", etc...
See stop.*.txt in 'create' directory of UdmSearch sources.

1. How to index:

  Just run indexer for example once a week (a day, a hour ...) to find 
the latest modifications in your web sites. Indexer will reindex expired
documents. If you want to reindex not expired documents also, you can
use 'indexer -a'. Indexer have -t,-u,-s command line arguments to allow 
reindexing only the part of database. 

  If you want to clear whole database use 'indexer -C'. You can also delete
only the part of database by using -t,-u,-s filters.

  If you run 'indexer -S', it will show database statistics,
count of total and expired documents of each status. -t,-u,-s filters
are taken in affect in this mode too.

The meaning of status is:
0 - new (not indexed yet) url
HTTP response code also.
Some of HTTP codes are here: 
200 - "OK" (url is successfully indexed)
301 - "Moved Permanently" (redirect to another URL)
302 - "Moved Temporarily" (redirect to another URL)
303 - "See Other" (redirect to another URL)
304 - "Not modified" (url is not modified since last indexing)
401 - "Authorization required" (use login/password for given URL)
403 - "Forbidden" (you have not access to this URLs)
404 - "Not found" (there was references to URLs that do not exist)
500 - "Internal Server Error" (error in cgi, etc)
503 - "Service Unavailable" (host is down, connect timeout)
504 - "Gateway Timeout" (read timeout when retrieving document)

If UdmSearch founds url with HTTP 301,302,303 code it will index url
given in "Location: " field of HTTP-header instead.

HTTP 401 means that this url is password protected. One can
use AuthBasic command in indexer.conf to set login:password for
this URL(s).

HTTP 404 means that you have incorrect reference (or reference to
recource that does not exist) in you document. Check 'referer' field
in url table. You can also check such referers by 'indexer -I -s 404'.

If you have bad connection with HTTP server,
you can run several indexer simultaniously with the same indexer.conf file.
I have successfully tested 30 simultanious indexers.

Notes for several indexers at the same time:
    * One can run several indexer with the different config files on
    different mysql database.

    * It is not recommended to use the same MySQL database with different
    indexer.conf files! First indexer could add something but second could 
    delete it. This process could never stop.

You can also insert indexer into your crontab job.


2. How to find:

First open search.cgi in your web server location:
http://your.web.server/path/to/search.cgi

If you preffer PHP:
http://your.web.server/path/to/search.php3 if you have handler for php3
documents in your HTTP server configuration or if you have not
http://your.web.server/cgi-bin/php.cgi/path/to/search.php3


To find something just type words you want to find and press SUBMIT button.
For example, "mysql odbc". You should not use quot " in query,
it's written here only to devide query from other text. UdmSearch
will find any urls, that contain word "mysql" or word "odbc". Best
documents will be found first.

If you want more advanced results and use PHP you can use query language.
search.cgi does not have advanced search yet. It is on TODO.
search.php3 understands next commands:

& - logical AND.
For example, "mysql & odbc". UdmSeach will find any URLs that 
contain both "mysql" and "odbc".

| - logical OR. For example "mysql|odbc". It's just the same that "mysql odbc".
Space " " is equal for "|". UdmSearch will find any URLs, that contain
word "mysql" or word "odbc".

~ - logical NOT. For example "mysql & ~odbc". 
UdmSearch will find urls that contain word "mysql" and do not contain
word "odbc" at the same time. Note that ~ just excludes word from result.
Query "~odbc" will find nothing! UdmSearch compose WHERE condition in
mysql query using all of the words in search query (to make search very quick):
.... WHERE word in ('all','of','the','words','have','been','typed')

() - group command to compose more complex queries.
For example "(mysql | msql) & ~postgres".
Query language is very simple (but very powerful!!!). 
Just consider query as common logical expression.


Disclaimer (see COPYING for details)
------------------------------------

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
