	Parameters of fulltext database locus are specified in a text file
locus.opt, divided into sections. The three basic sections are locus (used
by all programs), cruncher and grazer; user-defined objects (document parts
and operators - see file search) have their own sections.
	Section starts by the name of the section in square brackets on
a separate line. Every line in the section specifies value of one parameter
(if it doesn't start with '#' - in that case it's a comment). Every parameter
has a name and a type: int, double, string or string list. Values of type int
can be written not only as numbers, but also as words 'yes' (1) and 'no' (0).
Strings are delimited by double quotes and cannot contain character 0.
Characters double quote, newline, tab and backslash must be quoted (like in 
C strings). String list is a sequence of strings separated by spaces.
	Most parameters have implicit values, which are used when the 
parameter is not specified in options file. Parameters without implicit
values must be specified by user. Selected parameters are:


Basic
=====
	name				type			implicit value
-------------------------------------------------------------------------------
section locus:
	base_dir			string			none
	- directory where cruncher creates index files and grazer looks for
them
section cruncher:
	source_dir_list			string list		none
	- list of directories where cruncher looks for files to index; their
subdirectories are searched as well (depth-first); no shell expansion is performed
on them - use absolute paths
	source_mask_list		string list		none
	- masks of document files


Search parameters
=================
	name				type			implicit value
-------------------------------------------------------------------------------
section grazer:
	focus_weight			double		      	0.2
	correlation_weight 		double     		0.2
	prominence_weight		double			0.2
	locality_weight			double			0.2
	order_weight			double			0.2
	relevance_threshold		double			0.5
	- implicit operator parameters (see file search for details)

	soft_operators			string list		empty 
	- names of soft operators corresponding to sections in
named_objects_file
	named_objects_file		string			"locus.opt"
	- file with definitions of named operators; it has the same format as
locus.opt
	query_expression_file		string			""
	- if not empty, names the file with query expression (see file search
for its syntax)

	interval_start_file		string			""
	- if not empty, interpreted as name of indexed file from which searching
should begin
	interval_start_offset		int			0
	- if interval_start_file isn't empty, interval_start_offset must be
offset of some document in that file (0 is always a valid offset)
	start_is_in_interval		int			yes
	- flag determining whether low bound belongs to the searched interval
	interval_stop_file		string			""
	- if not empty, interpreted as name of indexed file at which searching
should end
	interval_stop_offset		int			0
	- if interval_stop_file isn't empty, interval_stop_offset must be
offset of some document in that file (0 is always a valid offset)
	stop_is_in_interval		int			yes
	- flag determining whether high bound belongs to the searched interval


Stoplist
=========
Is used to make index files smaller and indexation faster. It's based on the
observation that words which are in every document do not distinguish these 
documents, so it's not useful to search for them, and so it's not necessary to 
index them (this is not quite true, but if you see that, you don't need my
explanation). Stoplist is specified by a text file; every line of that file
contains one word (from the first column; the rest of line after a first
whitespace character is ignored). Words in documents are compared (case
insensitively) with words in stoplist and matching ones are dropped. An
example english stoplist is included, but it's better to generate your own.
	To generate a stoplist, index part of your database (let's say 10%).
From that, generate wordlist and use its first few lines as a stoplist (when
you add a word into the stoplist, its occurences are removed from the database).

	name				type			implicit value
-------------------------------------------------------------------------------
section cruncher:
	stop_list_file			string			""
	- non-empty value is a name of text file from which cruncher adds
words to the stop list (before starting indexation)
section grazer:
	word_list_file			string			""
	- if not empty, interpreted as a name of file into which grazer writes
(before it starts searching) all indexed words, sorted by number of documents
in which they occur, number of occurences and alphabetically


Document interpretation
=======================
I don't recommend changing these options for an existing database.

	name				type			implicit value
-------------------------------------------------------------------------------
section locus:
	doc_start_rx			string			""
	- regular expression matching documents starting in the middle of 
a file; one input file can contain multiple documents, considered logically
isolated - grazer searches them separately and identifies them by file name and
position where document begins; when empty, the whole file is treated as one
document
	basic_separators		string			" \n\r\t"
	- word delimiters; changing this option is not recommended
	additional_separators		string			""
	- word delimiters to add; for example when indexing HTML, it's useful
to add "<>", so that tags are recognized as separate words (and can be dropped)
	basic_punctuation		string		"\r\t.,;:\'\"()!?\0x1a"
	- basic string of characters stripped off beginning and end of every
word before processing it further; changing this option is not recommended
	additional_punctuation		string		"[]{}@#$%^&*-_=|\/"
	- additional string of characters stripped off beginning and end of
every word before processing it; this one can be modified
	early_case_conversion		int		2
	- case conversions performed by cruncher and grazer; possible values:
	0 - no conversions in cruncher, grazer looks for variants of every
word depending on late_case_conversion
	2 (for forward compatibility - all non-zero values behave the
same) - both cruncher and grazer convert words to lowercase


Document structure
==================
If your documents (e-mail messages, HTML etc.) have a common structure, you can
instruct cruncher to store it so that grazer can use it when searching. You do
it by defining document parts, delimited by regular expressions. At the
beginning of a document begins "plain" part. Other parts begin when document
text matches some push_rx expression and end at pop_rx expression of active
document part. When one part ends, the previous one becomes active.

	name				type			implicit value
-------------------------------------------------------------------------------
section locus:
	doc_parts			string			empty
	- names of explicit document parts corresponding to sections in
named_objects_file; the word "plain" is reserved for default part and cannot
be in this list
document part section:
	push_rx				string			""
	- regular expression matching beginning of document part (empty string
never matches)
	pop_rx				string			""
	- regular expression matching end of document part (or empty string which
never matches)

Example: mailing list
With these options:
doc_parts = "from" "subject"
[from]
push_rx = "^From:"
pop_rx = "\n"
[subject]
push_rx = "^Subject:"
pop_rx = "\n"

A message like this one:
=== cut ===
From:         preedy@NSWC-WO.ARPA
Subject:      digest format
To:           VIRUS-L@LEHIIBM1.BITNET
Keywords: edited

Ken,
     It sounds like a good idea to me to start getting the "digests".
I am tired of getting duplicates and ascii trash.  Hope this isn't too
much of a bother for you and you can continue to do it.  Thanks for
the effort.  It is appreciated.  You are saving a lot of people
precious time.


         Pat Reedy
         preedy@NSWC-WO.ARPA


[Ed. Thanks for the input!  The effort involved is actually quite
minimal thanks to a set of GNU EMACS digestifying routines by David
Steiner.  It takes me about as much time to create a digest as it
would to read my own mail.]
=== end cut ===

would be partitioned into:
from part:
=== cut ===
From:         preedy@NSWC-WO.ARPA
=== end cut ===
subject part:
=== cut ===
Subject:      digest format
=== end cut ===
and plain part:
=== cut ===
To:           VIRUS-L@LEHIIBM1.BITNET
Keywords: edited

Ken,
     It sounds like a good idea to me to start getting the "digests".
I am tired of getting duplicates and ascii trash.  Hope this isn't too
much of a bother for you and you can continue to do it.  Thanks for
the effort.  It is appreciated.  You are saving a lot of people
precious time.


         Pat Reedy
         preedy@NSWC-WO.ARPA


[Ed. Thanks for the input!  The effort involved is actually quite
minimal thanks to a set of GNU EMACS digestifying routines by David
Steiner.  It takes me about as much time to create a digest as it
would to read my own mail.]
=== end cut ===

Of course you can be fancier than that - for example, it would be posible to 
define a part for editor's comments. If you devise some examples you're proud 
of, let me know.


Compressed documents
====================
locus can access data in compressed files - with some important restrictions:
	1) you can use at most one extension of such files and one
decompressing program.
	2) in every archive must be exactly one file having the same name
(with any extension).
	If you want to index compressed data, you must set options 
compressed_ext, decompress_dest_dir and decompressor_command_line.  N.B: locus
does not expect unpacking program to destroy the original archive, like gunzip
does. If you use gunzip, set decompressor_command_line to some script wrapping
gunzip.

	name				type			implicit value
-------------------------------------------------------------------------------
section locus:
	compressed_ext			string			""
	- compressed file extension (i. e. ".gz")
	decompress_dest_dir		string			""
	- directory in which locus expects to find unpacked file after running
decompressor_command_line
	decompressor_command_line	string			""
	- name and parameters of external unpacking program; before running it,
locus substitutes its substrings source_file_subst and dest_dir_subst (if they
exist)
	source_file_subst		string			"%src"
     - substring of decompressor_command_line to be replaced by name of
compressed file
	dest_dir_subst			string   	   	"%dest"
	- substring of decompressor_command_line to be replaced by target
directory


Output
======
	grazer outputs warnings, progress and query result. Warnings and
progress always go to standard output, result can be redirected to a file.
Result is prolog, list of documents and epilog. Document is document id (file
name and optionally offset in the file), document relevance and list of
"clots", i. e. clusters of words found near to each other. Clot contains
highlighted passages (found words) and default passages (their neighbors).

	name				type			implicit value
-------------------------------------------------------------------------------
section grazer:
	output_file			string			""
	- if not empty, names the file into which result is printed

	spot_distance			int         		8
	- maximal distance (in words) of found words which are grouped into one
clot; must be between 2 and 1000
	doc_limit			int			0
	- maximum number of displayed documents; 0 means no limit
	max_highlights			int			0
	- maximal number of highlighted passages (basically found words)
displayed in one clot; 0 means no limit
	show_relevance			int			no
     - if yes, grazer shows metrics (focus, correlation, prominence, locality  
and order - see file search) of every displayed document, so that you see why
it was found and refine your search

	Default output formatting uses ANSI colors. Beware: due to a bug, clots
can display twice at the same screen position, obscuring part of the output. In
option values, colors are coded by a three-digit integer: hunderts means
attribute (0 normal, 1 bold/bright, 2 underscore, 3 blink, 4 reverse, 5
concealed), tens foreground and ones background (0 black, 1 red, 2 green,
3 yellow, 4 blue, 5 magenta, 6 cyan, 7 white).
	name				type			implicit value
-------------------------------------------------------------------------------
section grazer:
	doc_name_color			int         	bright yellow on black
	text_color			int		white on black
	occ_color			int		bright green on black
	- color of found word
	head_tail_color			int		bright yellow on black


	You can change output formatting with templates, which are similar to
format strings for printf(), but they use numbered arguments (every template
has its own list of them). Every argument can be used multiple times or not at
all. For example doc_file_template can be "%0\n" (just a file name),
"%1File %0\n" (string in the appropriate color) etc.
	name				type			implicit value
-------------------------------------------------------------------------------
section grazer:
	not_found_template		string			"\t%0\n"
	- unindexed word warning; note that this is part of warnings, not result
		%0 - the word
	prolog_template	string		string			""
	- no parameters
	epilog_template			string			"%0"
		%0 - ansi sequence to set output to default_color
	doc_file_template		string			"%1File %0:\n"
	- document id for documents starting at offset 0
		%0 - file name
		%1 - ansi sequence to set output to doc_name_color
	doc_section_template		string		"%2File %0 (from %1)\n"
	- document id for documents starting in the middle of a file
		%0 - file name
		%1 - offset in file
		%2 - ansi sequence to set output to doc_name_color
	doc_end_template		string			""
	- no parameters
	metric_template			string			"%0 = %1\n"
	- value of a normalized metric for this document (see file search); not
used unless show_relevance is set to yes
		%0 - metric name
		%1 - metric value
	relevance_template		string			"result = %0\n"
	- relevance of this document; not used unless show_relevance is set to
yes
		%0 - the value
	clot_head_template		string			"%0*"
		%0 - ansi sequence to set output to head_tail_color
	clot_break_template		string			"%0\n...\n"
		%0 - ansi sequence to set output to head_tail_color
	clot_tail_template		string			"%0\n"
		%0 - ansi sequence to set output to head_tail_color
	passage_template		string			"%3%0%1%2"
		%0 - punctuation at the beginning of text
		%1 - the text
		%2 - punctuation at the end of text
		%3 - ansi sequence to set output to text_color
	highlighted_passage_template	string			"%3%0%1%2"
		%0 - punctuation at the beginning of text
		%1 - the text
		%2 - punctuation at the end of text
		%3 - ansi sequence to set output to text_color

Example: settings for HTML output
output_file = "/home/vb/locus/ufidx/r.html"
prolog_template = "<html>\n<menu>\n"
epilog_template = "</menu>\n</html>\n"
doc_file_template = "<ol> <a href=%0>%0</a>\n\t<menu>\n"
doc_end_template = "</ol>\n"
metric_template = "\t<ol>%0 = %1</ol>\n"
relevance_template = "\t<ol><b>relevance = %0</b></ol>\n\t</menu>\n"
clot_head_template = ""
clot_break_template = ""
clot_tail_template = "\n"
passage_template = "%0%1%2"
highlighted_passage_template = "%0<b>%1</b>%2"
show_relevance = yes


Miscellaneous
=============
	name				type			implicit value
-------------------------------------------------------------------------------
section cruncher:
	find_counter			int			10
	read_counter			int			1000
	index_counter			int			400
	- these numbers control frequency of cruncher messages (like
"reading 1. file (0%)") - the message is printed every counter iterations;
if you have a much faster machine than mine (which is not that improbable),
you may want to increase them

section grazer:
	temp_dir			string			""
	- explicit temp directory; if empty, /tmp is used
	late_case_conversion		int			2
	- used when early_case_conversion is 0; specifies case conversions 
grazer performs on every word from the query to get variants it actually
searches for
	0: as specified, no other variations
	1: as specified and all uppercase
	2: as specified, all lowercase, capitalized (first uppercase, the
	rest lowercase), all uppercase


Parameters not documented here are internal and/or experimental. Use The Source.
