===========================================================================
	           NOTES ON PIMP (Perfect Internet Mail Popper)
===========================================================================

Introduction:

The perfect part is meant as a joke.

This is the very first release of my pop mail retrieval client.
Please realize when using this client that it is in a very rough
state.  Currently this release is a working release without
any neat features.  Bugs may be abound.  If you do encounter a bug,
please "make debug" if using linux, if not "make otherdebug".  If
you are not using linux, you must remove malloc_dump() from pop3.c.
Then, re-run and try to recreate the error.  Dump the stdout and
stderr output to a file and mail to me, please.  Specify the architecture,
version of relevant packages (kernel, compiler, libraries), and what
you did to create the error.

The Makefile is not very well written (because of time constraints)
and if you're not compiling on a system with gcc/glibc, you may
have to do a little hacking.  Also, the debugging sections of the
Makefile compile in a FreeBSD malloc that provides some cool debugging
functions.  The malloc source, however, generally does not compile on anything
aside from linux (and FreeBSD).  See the how to install note below.

The original code base was popmail 1.6.  Since this client is a 
total rewrite I'm releasing this as a seperate package.  
popmail 1.6 was written by Guido Trentalancia.
I don't believe there is even a single line remaining from the original
code base.  However, I feel obligated to thank Mr. Trentalancia for
giving me an idea to work from (even if the idea was based on what
I believed to be failings).

Command line args are handled very very poorly.  I didn't diverge much
from popmail1.6 in the handling of command line args.  I promise I'll
rewrite this when I can.

I have sucessfully compiled this on Linux 1.3.x using glib and gcc,
Solaris 2.4 using cc, IRIX using gcc&glibc (IRIX's libc does not have alloca),
and BSDi using gcc/glibc.  

Note that the debugging lines in Makefile will not work on Solaris or IRIX
'cause it tries to compile in a BSD malloc that provides cool malloc debugging.
You must compile with "otherdebug" if you're compiling w/o gcc/glibc.
However, you will have to create a function called malloc_dump();
that does nothing.

IRIX's native c libraries does not have alloca.  You must use glibc.

If you can not get it to compile on any other systems, please let me know.
I do not have access to every architecture, so I couldn't test it anything
not mentioned above.  I would be interested to know if it will compile on
AIX with the native compiler, but, unfortunately, our AIX box here has been
unplugged for a few months and the systems group has had trouble finding
someone to plug it back in.  (note: it's plugged back in, but, it's
been restricted to only systems personnel use, so I can't use it anyways)

pimp v1.0 was checked for out of range memory use and leaked memory
using Purify(tm) and was optimized using Quantify(tm).  Excellent
software.  Check http://www.pure.com for more details.  Tres elite.
I owe Bob for showing me the light.

pimp v1.0 is copyrighted in 1996 under the GNU Public Licence, v2.
For a copy of the GPL, please contact the GNU project.

I was considering naming this bpk, but, I think pimp is a more catchy
name.  Plus, it describes what the author is. (note: actually, I was
just too lazy to change all the references to pimp)

Oh, and, if you find this document long and rambling, tough luck.

Author: rmw@io.com (Robert Mark Waugh)

===========================================================================

How To Install:

Look at config.h and pimp.h.
These files are kinda wacky right now but should be better next release.
You also might want to look at the Makefile.  You may also want to put
a "-m486" in the CFLAGS if compiling on a 486 or better.

Then, type "make".  It will create a file called pimp.  
"make install" will install it for you, based on the BINDIR and MANDIR
makefile definitions.
If you want to install it to be sgid mail so that the group ownership of
the mail spool file will be correct, use "make sgidinstall" instead.

Other Systems:
For solaris, type "make solaris"
For debugging on a gcc/glibc compiler, "make debug"
For non-gcc/glibc, "make otherdebug"
If you encounter problems, try "make other"

If you still encounter problems, mail me a bug report including the output
from make and your libc/compiler/architecture/kernel versions.

Note, if you do a "make debug" then a "make clean" it will _not_ remove
malloc.o (created only in a "make debug").

Also, look at the pop.conf.example if you want to install a system wide
pop.conf.  It's location is specified in config.h.  You can also specify
a non-system pop.conf on the command line using the flag -f.


===========================================================================

Speed Comparisions:

Note that these comparisions were run over a heavily laden local thin ethernet
to a 4 80mhz processor 256mb sparc 1000e running qpop.

The client is a 486 dx4 100 w/ 32 mb RAM, pci bus IDE running X and a
load of <0.1.

Also, note that popclient's speed is dramatically decreased because it
invokes by default a system MDA and waits for the child to end before
continuing.  While I will support configurable MDA's in future releases,
it will not be the default setting.

Worst case
----------
All \r\n's, 16 mails,  500k each

Without running time(2):
popmail version 1.6: 33 seconds
popclient 3.0b6: 1 minute 26 seconds
pimp version 1.0: 48 seconds

With running time(2), output of time(2):
pomail version 1.6:
11.27u 11.820s 0:39.43 58.5% 0+0k 0+0io 58pf+0w
popclient 3.0b6:
32.150u 38.500s 1:39.88 70.7% 0+0k 0+0io 1309pf+0w
pimp version 1.0:
25.190u 11.770s 1:15.24 49.1% 0+0k 74pf+0w

Long line case
--------------
80 characters/line, 16 mails, 500k each

Without running time(2):
popmail version 1.6: 36 seconds
popclient 3.0b6: 1 minute 38 seconds
pimp version 1.0: 32 seconds

With running time(2), output of time(2):
popmail version 1.6:
10.610u 12.780s 0:46.88 49.8% 0+0k 0+0io 47pf+0w
popclient version 3.0b6:
25.560u 26.000s 1:50.05 45.0% 0+0k 0+0io 1271pf+0w
pimp version 1.0:
19.88u 11.900s 0:42.90 74.0% 0+0k 0+0io 52pf+0w

Average Mail Case
-----------------
60 characters/line, 2000 mails, 10k each

Without running time(2):
popmail version 1.6: 3 minutes 12 seconds
popclient version 3.0b6: 16 minutes 2 seconds
pimp version 1.0: 2 minutes 19 seconds

With running time(2), output of time(2):
popmail vrsion 1.6:
30.600u 50.080s 3:38.96 36.8% 0+0k 0+0io 54pf+0w
popclient version 3.0b6:
102.380u 402.850s 16:19.93 51.5% 0+0k 0+0io 154116pf+0w
pimp version 1.0:
28.240u 26.840s 2:44.43 33.4% 0+0k 0+0io 50pf+0w


===========================================================================

Conclusions, explanations, etc
------------------------------

My pop mail client is still in it's very early stages.  I spent most of
my time optimizing it for the average mail case.  Obvious, the standard
pop clients are slightly better for extreme case mails, but, the fact
is extreme cases are not average cases.  My client is aproximately twice
as efficient in average case mail transfers.  The reasons for this are
many.
The above comparisions are kinda bogus, but it gives an idea of how it
will run on your system.
Also, please note that I have not been able to try this on a machine
running over SLIP/PPP.  I imagine that my speed gains may very well
go down because of the necessity to poll the socket.  Maybe not.
Please let me know about your experiences.

1. I slurp the entire mail into a string, probing the socket every 100th
   of a second or until it's finished.  Then, I remove all the \r's by
   pointer indexing and offseting.  I have found that in cases where
   the mail doesn't get so big that it hits swap my pop mailer is faster.
   If you have very little free memory, do not use this client at this
   time.  At some point I will modify it so that it can be configured
   to only slurp in a definable amount of bytes before parsing and writing.

2. I do not open and close the mailbox for every single mail.

3. Most mailers do a localtime() for every mail, but, when the difference
   in time stamps is in terms of seconds, and in most cases, sub-second,
   this causes too much overhead use.  I've found that localtime() consumes
   around 27% of the consumed cpu time in the mail formatting loops (not
   overall... overall about 10%).  A little note, I will, in the future
   redo the timestamp for each entry in the conf file to provide a more
   accurate reflection of the time.  Because of the 2 second delay (see
   below), depending on the number of entries, it can be very off.
   The reason for not doing this in this release is I thought of this
   the day before I've slated my release date.

4. I did, however, lose a lot of performance because I flushed the buffers
   rather frequently to ensure little, if no, loss of data.

5. I'm also using a FreeBSD malloc that was compiled with lots of error
   checking elements.  This probably made the biggest difference.
   I'm guessing this made a 10% increase in performance lossage.
   I will remove this from the distribution, so, please assume all
   performance will be 10% in favor of mine.

5. Also, I attempted to extract a correct From_ line (which no other
   pop client does) from the mail.  This caused a significant decrease
   in performance, but, I would rather pay for the decrease in exchange
   for compliance.  The From_ line extraction works like this:
   a. look for Return-Path: header line.  Extract correct From_ address.
   b. failing this, look for From:.
   c. If there's an address within a <,> pair, extract it for From_.
      This is bogus, but, some BSD systems do not put a Return-Path:
      header line.  In my opinion, this is less bogus than some stupid
      popclient From_ line.  You can bite me, mrg... :)
   d. Failing even this, insert "Poorly_formed_header" for the From_
      field.  Definite bogosity.

6. I encountered a problem with opening another socket to the same
   host immediately after disconnecting.  I resolved this with a
   hack.  I sleep() for 2 seconds after closing the socket.
   Definite bogosity, need to find a resolution.  So, 2 seconds
   of every test above is consumed doing nothing.  This is only
   used if you are reading from a config file.  In my tests, I
   did use a config file.


===========================================================================

NOTE: Memory use
----------------
My pop client will consume, at most, 3x(+3) bytes/mail in terms of the linked
list used to store mail length and mail number, and 2x bytes in terms
of length of the mail because of the necessity to create a same sized
buffer for removing all the \r's.  If you plan on receiving a lot of
mail that is 1/2 the size of your average available memory (ie, if you
are running w/ 8mb and have 4mb free, you will see severe performance
drops if you get a 2mb+ mail).  This will be resolved in a later
version.

Note: Protocols
---------------
Currently pimp only uses pop3.  This will change.
Later releases will cover apop authentication, pop2, and imap.
I do not intend to implement rpop.  rpop is a bad idea.

Note: Protocol adherence
------------------------
pimp follows the pop3 protocol closer than other pop clients.
This will become even more true in the next release.

Note: Lame POP servers
----------------------
If you encounter a problem with pimp client consistently on a particular
pop server, please uncomment the lines listed as debugging in
the Makefile.  Send the output from the stderr to me via email.
Some servers do not follow very strictly the standards for pop servers,
and I will make provisions for the differing types of lameness as they
come up.... as opposed to throwing in a whole bunch of worst case
stupidity.  If it fails, don't worry, even if you have deletion on,
you will not lose your mail.  It will probably fail during user authorization.

BTW, I do know that clients should be liberal in what they accept and
servers should be conservative in what they send, but, the approach I
took to reading data from the socket requires me to find some sort of
standard way of knowing when the server is finished.  I used the .\r\n
at the end of a reply to mean end of transmission.  I still probe for
a little to see if it is done, but, it could fail.  I'm working on this.
However, I found a server once (though I'm not sure which one) that doesn't
put a .\r\n at the end of it's reply's.  I haven't found one like it since the
server I was testing that was using that version went down.  If you find
one, please let me know what version it is.

===========================================================================

Oh, and please give me a new job.

rmw@io.com
