How POSTPONE should work

[ruv]

Preface

The well known words lit, and compile, are required for illustrations.

In a Forth system with direct/indirect threaded code and united code space and data space (so compile, is equivalent to , Comma), these words can be implemented as follows:

: compile, ( xt -- ) , ;
: lit ( -- x ) ( R: addr1 -- addr2 ) r> dup cell+ >r @ ;
: lit, ( x -- ) ['] lit compile, , ;

In any standard Forth system lit, can be defined as:

: lit, ( x -- ) postpone literal ;

For better understanding the terms "compilation semantics", "interpretation semantics", "execution semantics", "current definition", please refer to the section 2.1 Definitions of terms of Forth-2012.

The term "word" is used in the notion of a named Forth definition.

Overview

The behavior of postpone word, when it's encountered by the Forth text interpreter in compilation state, is to determine compilation semantics for its parsed argument (immediate argument) and append these semantics to the current definition.

Compilation semantics for a word can be either special or ordinary (see the section 3.4.3.3 Compilation semantics of Forth-2012). To work correctly, postpone should distinguish these cases and generate code according to the different patterns.

Ordinary case

Ordinary compilation semantics for a word are to append the execution semantics of the word to the current definition. The word compile, does exactly that.

For example, let foo is an ordinary word. Then, the quotation [: ['] foo compile, ;] performs the compilation semantics for foo. Hence, the phrase postpone foo (in a definition body) is equivalent to the phrase [ [: ['] foo compile, ;] compile, ] (which appends this quotation behavior to the current definition), which is equivalent to just ['] foo compile,. I.e., postpone foo (in the definition body) can be replaced by ['] foo compile,.

It means that postpone foo should generate the latter phrase immediately during compilation, i.e. the behavior of postpone foo, during compilation, should be equivalent to execution of the phrase ['] foo lit, ['] compile, compile, , which appends ['] foo compile, to the current definition.

So the following definitions are equivalent (for an ordinary word foo):

: postpone-foo ( -- ) postpone foo ; immediate
: postpone-foo ( -- ) ['] foo lit, ['] compile, compile, ; immediate

Special case

Special compilation semantics (if any) are specified individually per a word. And they should not be equivalent to appending execution semantics (the ordinary case).

On the system level they may be implemented in various ways.

Immediate words

In the classic single-xt+immediate-flag Forth systems, special compilation semantics are only implemented via the immediacy mechanism.

The compilation semantics for an immediate word are to perform the execution semantics of this word in compilation state. NB: if they are performed in interpretation state, then the interpretation semantics for this word are performed, that can produce different effects. An example of such a standard word is s" (from FILE) in the classic Forth systems.

For example, let bar is an immediate word. Then, executing bar (e.g. via ['] bar execute) in compilation state performs the compilation semantics for bar, and in interpretation state it performs the interpretation semantics for bar. To perform the compilation semantics for bar no matter what, we need a helper word execute-compiling, which enters compilation state if it's required.

In a classic Forth system the word execute-compiling can be implemented as follows:

: enter-compilation  ( -- )  1 state ! ; \ it's system-dependent
: leave-compilation ( -- )  0 state ! ;
: compilation ( -- flag ) state @ 0<> ;

: execute-compiling ( i*x xt --j*x )
  compilation if  execute  exit  then
  enter-compilation  execute  leave-compilation
;

Now the phrase ['] bar execute-compiling performs the compilation semantics for bar regardless of the initial value of state. Hence, the phrase postpone bar is equivalent to the phrase ['] bar execute-compiling (i.e., the former may be replaced by the latter in a definition body). It means that postpone bar should generate the latter phrase immediately during compilation, i.e. the behavior of postpone during compilation in this case should be equivalent to execution of the phrase ['] bar lit, ['] execute-compiling compile, (which appends ['] foo execute-compiling to the current definition).

So the following definitions are equivalent (for an immediate word bar):

: postpone-bar ( -- ) postpone bar ; immediate
: postpone-bar ( -- ) ['] bar lit, ['] execute-compiling compile, ; immediate

See also About POSTPONE semantics in edge cases to find a proof of this approach correctness in any edge case.

Full implementation

According to the above cases, in a classic single-xt+immediate-flag Forth system, postpone can be implemented as follows:

: postpone ( "name" -- )
  compilation 0= -14 and throw \ ensure that `find` is executed in compilation state
  bl word find dup 0= -13 and throw 1 = ( xt flag-special )
  swap lit, if ['] execute-compiling else ['] compile, then compile,
; immediate

Actually, this implementation is valid in any Forth system that provides a compatible implementation for the word find.

Limited implementation #

Formally, the word postpone provides a way to perform the compilation semantics for a word regardless of the initial value of state

But before this word was introduced, a standard program might perform special compilation semantics only in compilation state. The Forth-79 and Forth-83 standards explicitly disallowed performing compilation semantics in interpretation state (they use other terminology, certainly).

Taking into account this restriction on programs, postpone can be implemented slightly simpler. In a classic single-xt+immediate-flag Forth system it can be implemented as follows:

: postpone ( "name" -- )
  state @ 0= -14 and throw \ ensure that `find` is executed in compilation state
  bl word find dup 0= -13 and throw 1 = ( xt flag-special )
  0= if lit, ['] compile, then compile,
; immediate

So when postpone is applied to an immediate word, it just appends the execution semantics of this word to the current definition — exactly like [compile] in Forth-83.

With this postpone, the following definitions are equivalent (for an immediate word bar):

: postpone-bar ( -- ) postpone bar ; immediate
: postpone-bar ( -- ) ['] bar lit, ['] execute compile, ; immediate
: postpone-bar ( -- ) ['] bar compile, ; immediate

Having the mentioned restriction on programs by the tradition, and having incorrect behavior only in edge cases, this simpler implementation became popular, unfortunately.

To formally allow such implementation for postpone, and hold the letter of the standard in the same time, and avoid incorrect behavior in the edge cases, the Forth-94 technical committee decided to explicitly restrict standard programs and disallowed them to perform compilation semantics in interpretation state — see RFI Q99-027 (copy1,copy2).

So, the limited implementation for postpone may be provided by a standard Forth system, with the corresponding restriction on programs. But to avoid unexpected effects in edge cases, it's better to provide a complete implementation for postpone.

[SirWumpus]

Why should the edge cases matter? How often are they likely to occur?

[ruv]

to avoid unexpected effects in edge cases, it's better to provide a complete implementation for postpone.

Why should the edge cases matter?

To fall into an edge case concerning postpone, the following conditions should be met:

1. For some immediate word bar the behavior in interpretation state is not equivalent to the behavior in compilation state.
2. In the definition of another word foo, postpone is applied to bar (i.e. : foo .... postpone bar ... ;).
3. The word foo is executed in interpretation state (e.g. : baz ... [ foo ] ... ; or just foo outside of a definition, if it's allowed).
4. The code generated by postpone is executed in interpretation state.

A minimalistic example:

: bar state @ 0<> . ; immediate  : foo postpone bar ;  foo

In such a case, as well as in any other, all standard Forth systems should show the same behavior as far as none ambiguous conditions are met.

But if a system provides a limited implementation for postpone, it shows a different behavior and so makes a problem for a user. For the above example it prints 0, while a standard system (without environmental restrictions) prints -1.

A user can stumble upon this problem when he debug his code interactively by means of [ ... in a definition body, or chooses to generate code in interpretation state, or defines interpretation semantics for a word using the compilation semantics for another word (standard or user defined) , and this word in the Forth system meets the condition 1 above, etc.

For example, let the system defines the interpretation semantics for if to show an error in the following way:

: if  state @ 0= abort" only for compilation" postpone if ; immediate

(thus, this word meets the condition 1, but it's a hidden implementation detail and the user should not be bothered by that)

And let the user implements an ordinary word if, that performs the compilation semantics for if as:

: if, ( c: -- orig ) postpone if ;

Then he tries to test this word:

: test [ if, ]  1 . then 2 . ;  1 test 0 test

And it just does not work, what is quite unexpected.

How often are they likely to occur?

Сompared to what? In any case, an unexpected behavior in edge cases is very irritating for a user, since in non edge cases a behavior usually meets expectations.

If you are a standard Forth system implementer, it does not matter how often an edge case likely to occur — your system should be standard compliant in all edge cases.

The ambiguous conditions defined in RFI Q99-027 were included into none further revisions of the standard. So, to be standard compliant in the edge cases about postpone, you have two options:

• either declare the following ambiguous condition as an environmental restriction on programs (you might also refer RFI Q99-027/A99-027):
  • an ambiguous condition exists if a program tries to perform some compilation semantics in interpretation state;
• or provide a correct complete implementation for postpone.

Many systems don't provide a correct postpone and imply the corresponding ambiguous condition, but don't explicitly declare it. But they should do it to be formally standard compliant.

[AntonErtl]

A minimalistic example:

: bar state @ 0<> . ; immediate  : foo postpone bar ;  foo

In such a case, as well as in any other, all standard Forth systems should show the same behavior as far as none ambiguous conditions are met.

But if a system provides a limited implementation for postpone, it shows a different behavior and so makes a problem for a user. For the above example it prints 0, while a standard system (without environmental restrictions) prints -1.

I just tried a number of standard systems (Gforth, iforth, lxf, sf, VFX), and they all behave the same way: They all print 0. And the same is true if I replace postpone with [compile] and that's consistent with traditional implementation techniques (e.g. fig-Forth). Given that postpone is intended as a replacement for [compile], this is as it should be.

And that's certainly consistent with how I interpret the standard. I know that you have done some heavy language lawyering to devise an interpretation of the standard that supports your position above, but your position is inconsistent with common practice. Moreover, a more straightforward interpretation of the standard actually predicts that the minimalistic example above must output 0.

For example, let the system defines the interpretation semantics for if to show an error in the following way:

: if  state @ 0= abort" only for compilation" postpone if ; immediate

(thus, this word meets the condition 1, but it's a hidden implementation detail and the user should not be bothered by that)

And let the user implements an ordinary word if, that performs the compilation semantics for if as:

: if, ( c: -- orig ) postpone if ;

Then he tries to test this word:

: test [ if, ]  1 . then 2 . ;  1 test 0 test

And it just does not work, what is quite unexpected.

I just tried

: if, postpone if ; : test [ if, ] 1 . then 2 . ; 1 test 0 test

on gforth, iforth, lxf, sf, and VFX64, and the code above works on all of them and prints "1 2 2", as I expected. My position is that the state-smart implementation of if above is just a non-standard implementation of if, but you might also argue that you are referring to Forth-94 as modified by RFI A99-027 rather than Forth-2012, and in that case the test program is not a standard program.

Concerning how often corner cases occur: There have been cases where I did not expect a particular corner case to ever play a role, and then it turned up in some way or other. As a system implementor, you may be tempted to cut corners, but that means you also cut the corner cases, and users of your system will get unpleasant surprises.

[ruv]

Anton, hi! I know you position, we discussed this topic a lot 😃 But thank you for your response here! Maybe we will find a consensus eventually.

You ignored my reasoning concerning postpone and ]. Could you state your objections there?

I just tried a number of standard systems (Gforth, iforth, lxf, sf, VFX), and they all behave the same way: They all print 0.

All these systems imply the corresponding ambiguous condition and environmental restriction on programs.
It can be proofed by a test.

Given that postpone is intended as a replacement for [compile], this is as it should be.

Given that [compile] was in Forth-83 that disallowed to perform compilation semantics in interpretation state, the same restriction was intended in Forth-94, but was not expressed explicitly due to an omission. Having this environmental restriction in the systems, your result is allowed, but is not required.

And that's certainly consistent with how I interpret the standard.

I think, your interpretation is inconsistent :) Let's continue on that in the frame of #103

on gforth, iforth, lxf, sf, and VFX64, and the code above works on all of them and prints "1 2 2", as I expected.

It just means that each of these systems defines if in another way, or provide a correct postpone. So you have missed the initial assumptions of my example.

but you might also argue that you are referring to Forth-94 as modified by RFI A99-027

No. "Forth-94" refers the well known latest full version of the standard ANSI X3.215-1994 issued by the Technical Committee X3J14.

My position is that the state-smart implementation of if above is just a non-standard implementation

My position is that this implementation for if is a standard compliant as far as postpone is correct. Forth-94 was intended to allow the classic single-xt+immediate-flag implementations, but you seem to believe, without enough ground, that it rejects them.