Toward a Forth-like Programming Language Specifically For Unix User-Space

The Forth programming language(1) has been around since 1971. Unix(2) has
been around a while longer, depending on what you call unix. Around 1973
UNIX was ported from assembly to C. C seems to have been written for the
purpose of portable-izing UNIX. History indicates that re-writing UNIX in
C was indeed a very wise move, and unheard of at the time. However, as a
Forth aficionado, and even as someone who has begrudgingly become
competent in C, I can't help but be chagrined by Thompson, Ritchie et al's
choice of a portability vehicle. 

C is designed for performance, as is necessary even today for a system
language. It has other features which seem to have been geared for the
purpose of writing UNIX, such as allowing recursion, which is helpful with
recursive data structures like hierarchical filesystem directory
abstractions. However, what features it does have seem to have been very
carefully chosen to have minimal adverse effect on performance. The
machine C models is a typical generic CPU. This allows the compiler to
produce code implementing the C virtual machine in almost purely native
code on most unix-capable CPUs.  There are a few typical CPU features
missing from the C virtual machine, such as a "carry bit" or the concept
of "carry" that it represents, but for the most part one can write C that
translates to very few machine instructions in most situations. 

The performance discussed so far is program performance. Forth, on the
other hand, was written to enhance programmer performance. Programmer
performance is impossible to quantify, unfortunately, but anecdotally,
Forth is widely believed to achieve it's programmer performance goals as
well as C achieves it's program performance and systems programming goals. 

Probably the most remarkable thing about Forth as of 2000 is that it
hasn't vanished. Forth's best-known advantage is that a complete Forth
system, that is, a Forth system that is usable as a development
environment and rudimentry operating system, can be strikingly compact. A
Forth with a terminal interface, a primitive editor, some mass-storage
control and other rudiments, for a small-word microcontroller, is often on
the order of 8 kilobytes in total size. This is compelling for embedded
work, where the crucial aspects of a Forth system can often fit on the
target machine. Forth therefor thrives in the embedded world. 

Compactness is less of a factor on the desktop, and in networking
applications. Now that commodity desktop machines are capable of running
truly sophisicated operating systems, and now that truly sophisticated
operating systems are freely available, it would seem that the languages
and utilities best known on those operating systems would suffice.
However, Forth has quite a bit more than campactness going for it, and as
these other advantages become better known it should have wider impact.
That is not to say that Forth's influence has been narrow. Postscript is
built on a two-stack machine. Postscript seems to have been influenced by
Forth's interactive flexibility.

What is Forth? The author of the language isn't sure anymore. The usual
knock on Forth in this regard is "If you've seen one Forth, you've seen
one Forth." Let me attempt a definition to serve for the duration of this
essay... 

	A Forth is a set of instructions for a computer with two stacks, a
	return stack and a data stack, and with routines available in 
	terms of those instructions to interactively create new routines 
	in terms of those instructions.

So, Forth is defined in terms of, and defines, a 2-stack CPU. It's
interactive. It's self-sufficient. It's not necessarily an interpreter in
the usual sense. The 2-stack machine may not be virtual, since hardware
Forth engines exist. Radiation hardened Forth engines are well known in
space applications, and I'm told that the 6809, for example, has 2
hardware stack pointers. 

The other way in which Forth is not an interpreter in the usual sense is
that most Forth implementations on non-Forth hardware use threaded code. 
The Forth interpreter is divided into an inner and an outer interpreter. 
The outer interpreter provides the usual user interface, but the inner
interpreter provides a nesting of new routines ("word definitions" in
Forth parlance)  such that a new extension is only outer-interpreted as
far as it's name, which is the minimum amount of text interpreting
possible to use a keyword given interactively. The words comprising a new
definition are only inner-interpreted when the new name is invoked. The
inner interpreter is usually some form of address-interpreter, but
implementations vary widey.  "Subroutine-threaded" Forths are almost pure
compilers. 

Threaded code is not "byte-code" or "p-code". Moore was well aware of such
schemes when he wrote Forth. Bytecodes don't nest like threaded code. 
This difference between Forth and other schemes bears some elaboration.  A
new bytecode routine is reduced to bytecodes, and a string of bytecodes is
kept to represent the routine. A text string is kept to represent that
string of bytecodes. No bytecode for the new routine is created, so the
text interpretation of the new routine's name is never avoided. I am
generalizing wildly here, but that I believe is the general picture. 

Threaded code nests, such that text names are only ever parsed by the
outer interpreter. 

This is an example Forth definition... 

	: newguy oldguy 4 otheroldguy ;

That creates a new keyword, "newguy", which when used interactively will
call "oldguy" and "otheroldguy" by the threading mechanism, the fast
simple inner interpreter, without thier names being present in the code of
"newguy", or anywhere else other than in thier own original definitions. 
Furthermore, newguy is now available to be threaded into further
definitions, such that it's code also will be invoked quickly by the inner
interpreter. This is why Forth is best thought of as being distinct from
an interpreter in the usual sense. 

Forths typically have performance closer to pure compiled code than
other `interpreters' do. This is even with the disadvantage of modelling a
fundamentally different CPU than the one the Forth may be running on.  As
such, Forth may or may not be suitable for writing an OS on a one-stack
machine, but my position is that Forth is clearly under-used in
user-space.

I have been writing a Forth-like language with 3 stacks. I don't claim
that my 3-stack scheme is of interest, but abandoning standard Forth
practice has caused me to consider some ways in which a Forth-like
language would be more comfortable to unix adepts. The prevailing
standard, the ANSI standard for Forth(4), is geared mostly for what Forth
has been most used for, small systems. Even is cases Like Open
Firmware(3), Forth is still running a small machine, although that small
machine can boot Solaris or MacOS. There are several Forths available for
unix, including the first Forth-in-C, but a unix-specific Forth might well
be more than that.

CASE
 Let's start with the obvious, lowercase. ANSI Standard Forth is
uppercase. I wouldn't dream of presenting such code to a unix programmer. 
Such individuals know how to use tr, dd or some dedicated util to change
case to thier preference.

RPN
 Reverse-Polish Notation. In Forth...
	3 4 +    7 ok

 Even more off-putting to normal humans than all-uppercase. However, RPN
is part of the deep wisdom of Forth. The failure of the Forth community
has not been to fail to come up with a non-RPN Forth. The failure has been
to not make sufficiently clear what a huge win RPN is for over-all
simplicity. 

I'm old enough to have had heated arguments in a land surveying truck
between myself, taking the Casio/non-RPN side, and the more affluent party
chief, my boss, who could afford to take the Hewlett-Packard/RPN side. RPN
is not how I want to think about surveying calculations. HP handhelds
stayed with RPN long after Casio et al came out with calculators that did
normal 
	2 + 3 = 
	5
 It is WELL worth thinking about computer computations in RPN, however.
Forth has no parser.  It just has a tokenizer. RPN allows this, and the
simplicity win keeps coming back.

On unix, uppercase can go. RPN, however, can stay. 

SYSCALLS
 Forth means total control of the machine it runs on. If it's on a host
OS, that means, among other things, access to that OS's API. Forth
struggles to reconcile it's pre-existing self-sufficient I.O. facilities
with those found on the host. My approach to that struggle is to provide
all the host syscalls by thier host names or similar, such as open, fork,
chroot and so on. I provide syscalls as Forth fundamental keywords. Within
Forth, Forth "dup" takes precedence over unix "dup", which becomes
"dupFD". The name conflicts are few. Syscalls that take string arguments
need high-level wrappers, but simple calls like sync can exist just as raw
calls. 

I consider syscalls to be part of the core VM of the Forth for a
particular host. Syscalls ARE the machine in question, as much as OR, AND
and +.  Analogously, on x86 for example, a direct Forth keyword interface
to inb/outb type I.O. instructions is called for. 

TBC....

