HISTORY.TXT
                       (for DosLynx v0.35b, August 2007)
                               by Fred C. Macall

Introduction

In this document, I'll try to take up where I left off in the last
history.txt and not rehash the history of the previous versions.
However, there is a cumulative list of unresolved bugs, issues, and To Do(s)
at the end.  If you want to know everything else there is to know about my
previous releases of DosLynx, you can get much of that history from:
http://users.ohiohills.com/fmacall/dlx2xdoc.zip .
That archive collects the info.htm and history.txt documents for all eight
of my DosLynx v0.2xb releases.  For the rest, the documents for
DosLynx v0.30b through v0.34b are available at:
http://users.ohiohills.com/fmacall/dlx30/info.htm
http://users.ohiohills.com/fmacall/dlx30/history.txt
http://users.ohiohills.com/fmacall/dlx31/info.htm
http://users.ohiohills.com/fmacall/dlx31/history.txt
http://users.ohiohills.com/fmacall/dlx32/info.htm
http://users.ohiohills.com/fmacall/dlx32/history.txt
http://users.ohiohills.com/fmacall/dlx33/info.htm
http://users.ohiohills.com/fmacall/dlx33/history.txt
http://users.ohiohills.com/fmacall/dlx34/info.htm
and
http://users.ohiohills.com/fmacall/dlx34/history.txt .

In July, I extended or enhanced DosLynx:  To support nine new Unicode
character entities.  To provide limited support for UTF-8 document encoding.
And, to support (anti)caching directives, in HTTP Headers and in
HTML <META HTTP-EQUIV= . . . tags.  With these things accomplished, the next
enhancement most appropriate for DosLynx seemed to become the addition of
https/SSL support.

I started looking into this and found that my first choice seems to be:
Include the Open SSL package in DosLynx or reinvent that whole wheel.
The first step of including the Open SSL package in DosLynx is to "install"
it into my development system.  To do this, I have to decide on my runtime
and development environments, for DosLynx with SSL support.
And, relate these to the environments that Open SSL might support.
Well, the ready made Open SSL installations for PCs all seem to be limited
to 32 or 64 bit runtime environments.  If the present 16 bit DosLynx runtime
environment(s) aren't possible or aren't practical with Open SSL, a big
change or addition will be needed in my DosLynx development environment and
tools.

Did you notice that change of tense in the preceding paragraph?  My seat of
pants reckoning started telling me that the addition of https/SSL support
may be an extra long project.  It seems too bad to make those July
enhancements wait for it.  So, I've decided to make a smaller than usual
DosLynx release first and then start on adding https/SSL support.

Approximating a Few Unicode Characters

The work on full ISO-8859-1 character set support, for DosLynx v0.33b,
included provision of consistent handling for unrecognized or out-of-range
character entity sequences.  This was disclosed in the history.txt document
for DosLynx v0.33b.  The new handling makes sure that troublesome character
entity sequences get passed through unchanged into presented documents.
As a result, character entity sequences such as &#8217; started showing up
in some of the documents I viewed with DosLynx.  This sequence's value,
being greater than 255, was deemed out-of-range.  When given as a named
character entity, this character appeared as &rsquo; which stands for
"right single quote mark".  This sequence went unrecognized, too.
Both examples represent a Unicode character that isn't included in the
ISO-8859-1 character set.  Most of the documents I view are the works of
english speaking authors.

I started keeping a list of the troublesome character entity sequences that
were showing up.  And, soon realized that almost all were from a relatively
small set of characters!  These are the nine characters listed in the
info.htm document for the present release of DosLynx.  Presently, I consider
full support of Unicode in DosLynx to be a subject for further study.
However, I wondered if there might be some better way to handle the nine
Unicode characters appearing in my list.  I've settled on giving them ASCII
or ISO-8859-1 approximations and handling their numeric entities as
recognized exceptions.  Even if the new approximations are imperfect, their
replacement of those previously unprocessed character entity sequences
greatly improves the appearance of the presented documents.

My first step was to add the nine additional entity names to the
initialization of static CONST char entities[ ][8], in HTMLDTD.C.
At the same time, corresponding entries had to be added to the
initialization of static unsigned char ISO8bit[ ], in HTML.C.
(This also meant having to adjust the comments for most of the entries in
both of these initialization lists.)  At this point, eight bit character
codes had to be assigned for the additional characters.
I've assigned direct ASCII approximations for the four additional quote mark
characters.  ASCII accent grave and apostrophe characters approximate the
left and right single quote marks.  The ASCII quote character approximates
both the left and right double quote marks.  This seems preferable to using
doubled accent grave and apostrophe characters for these.  Because, with the
latter approximations, the ordering of adjacent single and double quote
marks would become indiscernible.

I've assigned or ratified character codes in the ISO-8859-1 "undefined"
128 to 159 range, for the remaining five additional characters.
In doing this, I've followed the informal assignments given by
Chuck Musciano and Bill Kennedy, in Appendix E of their book:
HTML, The Definitive Guide, Third Edition, August 1998.
These (re)assignments are:  Horizontal ellipsis (133), Bullet (149),
En dash (150), Em dash (151), and Trade mark sign (153).  In the end,
Horizontal ellipsis and Trade mark sign get multi-character ASCII
approximations.  The dashes and Bullet, remaining, get translations which
are appropriate for both PC Code Page 437 and PC Code Page 850.

These (re)assignments, approximations, and translations have been handled by
extending (static) const char ISO2pccp[ ], in GRIDTEXT.CPP, to cover the
whole range of 128 to 255.  And, by some (more) revision of
HText_appendCharacter( ), also in GRIDTEXT.CPP.  entities[ ][8], ISO8bit[ ],
ISO2pccp[ ], and HText_appendCharacter( )'s role in translation from
ISO-8859-1 character coding to PC character coding were all introduced in
the history.txt document for DosLynx v0.33b.  At this point, the nine 
additional named character entities became recognized.  They began being
replaced by their approximations.  And, quit appearing as unrecognized named
character entity sequences in presented documents.

Handling for the nine additional numeric character entities starts in
UnEscapeEntities( ), in HTML.C, and in SGML_character( )'s case S_cro, in
SGML.C.  Both of these have been extended to accumulate a numeric character
entity's value in a long int.  This insures against overflow in the
accumulation of numeric character entity values up to nine digits long.
Finding a numeric character entity value greater than 255, these routines
provide that value in a call to new function
extern "C" int trunicod(unsigned int ucval), in new module TRUNICOD.CPP.

In addition to trunicod( ), TRUNICOD.CPP contains two static int arrays of
nine entries each.  The first contains the numeric character entity values
which are to become recognized exceptions.  While the second contains the
corresponding character codes assigned as described above.
trunicod( ) searches for the value it has been given in the first array.
Finding a match, it returns the corresponding character code from the second
array.  Otherwise, it returns zero.

Finding a non-zero return from trunicod( ), its callers assign that back to
their numeric character entity value and continue normally.
Otherwise, they follow their exception path and pass their out-of-range
numeric character entity sequence through to the presentation.
With these arrangements, the nine troublesome character entity sequences all
became replaced by their approximations and quit appearing in presented
documents.

Limited Support for UTF-8 Document Encoding

After completing the previous section's work, I found only DosLynx's
handling of UTF-8 encoded documents remaining lacking, in the area of
"special character handling".  And, UTF-8 seems to have become a popular
choice for HTML documents.  (These assessments are based on my previously
stated interest in the works of english speaking authors.)  So, I wanted to
try to improve the handling, of UTF-8 encoded documents in DosLynx, if
possible.

UTF-8 is thoroughly described, though perhaps not "defined", in RFC 3629.
It is an octet based variable length encoding for Unicode character numbers,
up to 21 bits long.  It is fairly compatible with ISO-8859-1 because the
first 256 Unicode character numbers correspond to the ISO-8859-1 character
set.  And, because UTF-8 assigns its shorter encodings to the lower numbered
Unicode characters.  In particular, the UTF-8 encodings for the 128 ASCII
characters exactly match the eight bit versions of those characters included
in ISO-8859-1.  This is why DosLynx has come to nearly its present point
without having to recognize UTF-8 encoding.  As long as a document contains
only ASCII characters, it doesn't matter whether its character set is called
ISO-8859-1 or UTF-8.  On the other hand, now that DosLynx pretty well
supports all of ISO-8859-1, and then some, we expect it to do as much with
UTF-8.  I resolved to see about having DosLynx support the UTF-8 encoded
Unicode characters corresponding to ISO-8859-1, together with the nine
characters discussed in the previous section.

The UTF-8 encodings for all Unicode character numbers greater than 127 are
sequences of two or more octets.  All octets of all multi-octet UTF-8
sequences begin with a one bit.  So, multi-octet UTF-8 sequences are easily
recognized as non-ASCII.  However, they certainly haven't escaped redefining
the characters in the upper half of the ISO-8859-1 character set.
(Pun intended.)  The objective stated above implys handling two octet and a
few three octet UTF-8 sequences.  But, we can't just pluck them out of what
may be an ISO-8859-1 stream.  So, we first have to look for charset=UTF-8,
in Content-Type: HTTP Headers and in the corresponding
HTML <META HTTP-EQUIV= . . . tags.  Is the work needed for this kind of
thing familiar, yet?

First, comes a declaration for extern int utf8stat, in GLOBALS.H.
utf8stat gets instantiated in GLOBALS.CPP.  And, it gets cleared to zero at
the beginning of TURLView::TURLView( ), in TURLVIE2.CPP.  Next, comes an
extension or revision of HTMIME_put_character( ), in HTMIME.C.
After recognizing Content-Type: and a field ending white space character or
semi-colon, case CONTENT_TYPE in case GET_VALUE in HTMIME_put_character( ),
now sets new state CHARSET and goes to getrest.  Which is a new label for
the case SET_COOKIE routine for accumulating the remainder of a Header line.
This routine has been extended to assign any remainder of a
Content-Type: Header line to new global PRIVATE char * cpchars, in HTMIME.C.
HTMIME_put_character( )'s SET_COOKIE state and case were introduced in the
history.txt document for DosLynx v0.34b.  CHARSET has now been added to
typedef enum _MIME_state { . . . } MIME_state, in HTMIME.C, as well.

HTMIME_put_character( )'s case NEWLINE includes a new if (cpchars) clause.
Here, any white space character(s) included in cpchars' string get removed.
Then, a case insensitive check is made for ;charset=utf-8 , at the beginning
of any string remaining.  Given a match while utf8stat is (still) zero,
utf8stat gets set to one.  Then, cpchars' string is free(d) and cpchars gets
cleared.  That last part's clean up and clearing also gets done in
HTMIMEConvert( ), if necessary.

A big new if ( . . . ) clause, in case HTML_META in HTML_start_element( ) in
HTML.C, provides the same kind of recognition and utf8stat updating, for
 . . . charset=UTF-8 , appearing in an
HTML <META HTTP-EQUIV="Content-Type" CONTENT= . . . tag.

A new if (utf8stat) enabled clause, at the beginning of SGML_character( ) in
SGML.C, looks for two and three octet UTF-8 sequences.  And, decodes the
Unicode character number each sequence encodes.  Non-zero utf8stat values,
other than one, are used to remember the state of this process while within
a UTF-8 sequence.  As in the extended numeric character entity processing,
trunicod( ) in TRUNICOD.CPP, gets called whenever a Unicode character number
greater than 255 appears.

In contrast with character entity processing, what should be done with
exceptions here isn't so clear.  Because, many exceptions may simply be due
to the occurrence of a Unicode character that we (still) aren't handling.
Though, invalid UTF-8 sequences are also possible and ought to be recognized
when they present themselves.  In either case, the UTF-8 sequence source
octets for those exceptions, presented as ISO-8859-1 characters, won't be as
unambiguous or readable as character entity sequences are.
Generally, we tend to pass the source sequence through unchanged when we
meet with an exception here.  But, we aren't being very faithful about doing
that where there is an extra effort required.

This support has at least two more limitations, presently.  First, a local
file copy of a UTF-8 encoded HTML document may not get displayed correctly.
If it doesn't contain an HTML <META HTTP-EQUIV="Content-Type" CONTENT= . . .
tag specifying charset=UTF-8.  This is because the HTTP Header(s) saved in
a local file copy of an HTML document (still) don't get processed in the
course of such a copy's presentation.  And second, though a received
document may be accompanied by an HTTP Header line such as
Content-Type: text/plain; charset=UTF-8
the charset=UTF-8 part of that won't get honored.  This is because the
DosLynx processing stack for text/plain doesn't include SGML_character( ).

Support for (Anti)Caching Directives

Since DosLynx started supporting Forms, it's lack of recognition for the
various (anti)caching directives has sometimes been seen to be an issue.
With its support for Cookies, in version 0.34b, this lack has been seen more
frequently.  This topic was discussed under the heading:
Cookies Support To Do, in the history.txt document for DosLynx v0.34b.
It had become clear that DosLynx needed to support (anti)caching directives.
I am referring to "(anti)caching directives" because I have come to think of
these as provisions for overriding HTTP's default document caching strategy.

RFC 2616, Hypertext Transfer Protocol -- HTTP/1.1, section 13.13,
acknowledges and/or describes browser "history mechanism(s)" that seem to
trump most or all (anti)caching directives.  The RFC seems to take these
very seriously.  It seems to me that the main value of this kind of function
is that it may do away with document refetches that otherwise would result
from honoring (anti)caching directive(s).  (If and when memory availability
allows that.)  When a number of an index type document's links are being
followed.  In July, I thought the DosLynx "history mechanism" wouldn't be
hard to give up.  If I had high speed Internet service, this expectation
might have been fulfilled.  But, as a user of a dial up Internet service, I
eventually found the extra refetches resulting from the new support to be
very annoying.  So, toward the end of this development cycle, I added
histmode= configuration for the new support.  This is described in an
updated sample DOSLYNX.CFG file, of course.

(Anti)caching directives come from three sources.  These are:
HTTP status code responses and response Headers and
HTML <META HTTP-EQUIV= . . . tags.  The (anti)caching directives I've
identified are:  Status code responses other than 200, 203, 300, 301, or
410.  Response Header(s) containing:  Pragma: no-cache ,
Expires: <date/time in past or unrecognizable> , and
Cache-Control: no-cache (but not no-cache= . . . ) or
Cache-Control: max-age=0 .  And, HTML <META HTTP-EQUIV= . . . tag forms of
the Pragma: no-cache and Expires: <date/time in past or unrecognizable>
Headers.
  
Recognizing the (anti)caching directives starts out much as in recognizing
the charset=UTF-8 designation.  First, a declaration for extern int nocache
has been added to GLOBALS.H.  nocache gets instantiated in GLOBALS.CPP.
And, initialized to zero early in TURLView::TURLView( ), in TURLVIE2.CPP.
Early in HTLoadHTTP( ), in HTTP.C, nocache gets set to one.  If a status
code of 200, 203, 300, 301, or 410 gets decoded later, nocache gets cleared
back to zero.  As results of this arrangement, code 300 and 301 redirects
(from HTTP) to, say, FTP will end with nocache clear.  Any other redirect(s)
to FTP will end with nocache set.  While redirects that stay with HTTP will
end with nocache set in accord with the last status code obtained.
We'll watch to try to see if these results need any modification.

Next, HTMIME_put_character( ) in HTMIME.C, has been extended to recognize
Cache-Control: . . . , Expires: . . . , and Pragma: . . . response Header
lines.  States CEE, CACHE_CONTROL, EXPIRES, and PRAGMA have been added to
typedef enum _MIME_state { . . . } MIME_state for this.  Pragma: Header
lines get handled by a new case PRAGMA, in case GET_VALUE in
HTMIME_put_character( ).  Essentially, this is a case insensitive check for
the string no-cache .  A match gets noted in the nocache global variable.
Cache-Control: and Expires: Header lines get handled mostly in common with
Set-Cookie: Header lines.  Once Cache-Control: or Expires: has been
recognized, the remainder of one of these Header lines gets accumulated in
a string located by a new PRIVATE char * cpcctrl or PRIVATE char * cpexpire
variable.

A new if (cpcctrl) clause, in case NEWLINE in HTMIME_put_character( )
begins by eliminating all of the string's white space character(s).
Then, on the basis of any comma separator(s) possibly present, it uses case
insensitive compares to check each of the string's field(s) for no-cache or
max-age=0 .  Any match gets noted in the nocache global variable.
Afterwards, cpcctrl's string gets free(d) and cpcctrl gets cleared.

Expires: response Header line handling is much like
Set-Cookie: Expires= . . . response Header attribute field handling.
So, I've extracted the code for that from readcky( ), in CKIESRCV.CPP.
And, placed it in new function extern "C" int chkdtim(const char * cpdtim),
in new module CHKDTIM.CPP.  readcky( ) and CKIESRCV.CPP were introduced in
the history.txt document for DosLynx v0.34b.
chkdtim( ) returns:  A negative value for a date/time string value in the
past.  Zero for an unrecognizable date/time string.  And, a positive value
for a date/time string value in the future.  The extracted code has been
replaced with an if (chkdtim( . . . clause, in readcky( ).

The following new clause, in case NEWLINE in HTMIME_put_character( ),
serves Expires: response Header lines:

/*  If have an Expires: Header line parsed, see about setting the no caching
/*  indicator.
*/
        if (cpexpire)
            {
/*  If the Expires: Date/TIMe field value is unrecognizable or past,
/*  don't cache the document in process.
*/
            if (chkdtim(cpexpire) <= 0)
                nocache |= 4 ;

            free(cpexpire) ;
            cpexpire = 0 ;
            }
        
As with cpchars strings, cpcctrl and cpexpire strings also get cleaned up
and cleared in HTMIMEConvert( ), when necessary.

Two more big new if ( . . . ) clauses, in case HTML_META in
HTML_start_element( ) in HTML.C, provide the same kind of recognition and
nocache updating for no-cache appearing in an
HTML <META HTTP-EQUIV="Pragma" CONTENT= . . . tag.  And, for a date/time
string value unrecognizable or in the past, in an
HTML <META HTTP-EQUIV="Expires" CONTENT= . . . tag.

The added configuration arrangement begins with a declaration for
extern int histmode, in GLOBALS.H.  histmode gets instantiated with an
initial value of zero, in GLOBALS.CPP.  A new clause in
TDosLynx::EvalConfigFile( ), in TDOSLY14.CPP, recognizes and encodes any
configured histmode= value into histmode.

Once a document's presentation has been accomplished, TURLView::TURLView( )
checks to see if nocache has been set in the course of this preparation.
If so, and if histmode is zero or if histmode and ckymode coincide in values
for AUTO and (Enabled and) ON, 'TURLView( ) makes an
HText_inval(HTp_HyperDoc) call to get the presentation marked stale.
That will insure that it won't get reused without being reloaded.
And, that it gets free(d) as soon as it goes unviewed.
HText_inval( ), in GRIDTEXT.CPP, was introduced in the history.txt document
for DosLynx v0.27b.  ckymode was introduced in the history.txt document for
DosLynx v0.34b.

Updated Score

As explained under the heading:  Keeping Score, in the history.txt for
DosLynx v0.20b, I am maintaining an informal written list of DosLynx issues
and their resolutions.  In addition to the issues it's always had, this list
now includes issues that have come along in connection with the DosLynx
Protected Mode version.  At the time of this writing, for the release of
DosLynx v0.35b (in late August 2007), my list has about 537 issues.
Of these, about 384 (over 70 percent) have been resolved or ameliorated.
This leaves a total of about 153 issues pending.

If you will look ahead to the next section for a moment, you may note that
several of the pending issues include the parenthetical observation:
"This is hardly an issue with the DosLynx Protected Mode version."
As far as users of the DosLynx Protected Mode version are concerned, the
issue(s) to which this observation applies have been resolved.  On the other
hand, the DosLynx Protected Mode version does have a few issues, of its own,
that don't apply to the Real Mode version.  A study identifying issues that
don't apply to both versions finds about 147 issues pending for the
Real Mode version.  And, about  142 issues pending for the Protected Mode
version.

In the next section, I'll list a few of the 153 pending issues that haven't
been mentioned above.

To Do

As explained in the Introduction, I am now hoping to provide https/SSL
support in the next release of DosLynx.  If I have to develop a 32 bit
version of DosLynx to accomplish this, there may be a longer than usual
interval before that next release.

Here is a summary of some of the pending issues not discussed anywhere
above:

  -  Provide https/SSL/TLS support.  (This is expected by many Web servers
     that support data entry.)

  -  Provide Login (original AUTHINFO) support for the news client?

  -  Support the HTML Form <INPUT TYPE=file . . . variation.  (This is part
     of the so called "file upload" capability.)

  -  Find some way to guard against making duplicate Post requests.
     (Bypass:  Move the anchor cursor off the submit button after using it,
     if/when you resume, or first return to, a window containing a submitted
     Form!)

  -  Provide some way for retrying a temporarily failed e-mail send attempt?

  -  Support the HTML Form <BUTTON . . . tag
     (like <INPUT TYPE=submit . . . )?

  -  Automatically replace space(s) with %20 , in links being followed or
     other URLs requested.  (Bypass:  Use the Navigate|Show Destination URL
     (Alt-U) menu entry or command to paste a troublesome link's URL into
     the File|Open URL... (F3) dialog's history.  Then, edit it for use.)

  -  Provide a convenient Cookie(s) file viewing dialog or tool.

  -  How about changing the File|Open URL... (F3) and local file dialogs to
     start with the cursor at the bottom of the history list?  How about a
     fast way to get to the directories, at the end of a long files list, in
     a local file dialog?  (Bypass:  After entering a history or other list,
     use Ctrl-Pg Dn to quickly move to its end.  Use Ctrl-Pg Up to quickly
     move back to its start.)

  -  Find a way for the Protected Mode version to successfully support the
     stock Microsoft mouse driver in Windows 2000 windows.  (This isn't an
     issue with the DosLynx Real Mode version.  Bypass:  Use mouse=
     configuration to disable support of the mouse, where necessary.)

  -  Find a graceful way for the FTP client to deal with directory reports
     that are too large to be memorized.  (This is hardly an issue with the
     DosLynx Protected Mode version.)

  -  Find a way for the File|Save Source and Navigate|Download Selection
     menu entries to handle FTP directory reports that are too large to be
     memorized.  (This is hardly an issue with the DosLynx Protected Mode
     version.)

  -  When your display is set in a VESA mode upon running DosLynx, its
     arrangement(s) for saving and restoring the screen mode may break down.
     (Bypass:  Set your display into a "legacy mode" before running
     DosLynx.)

  -  Provide for displaying directory entry details, beyond filename, from
     the WAR-FTP Daemon's DOS style directory reports.

  -  Provide a configurable "download directory".  (Bypass:  use the command
     line /P option (as demonstrated in DOSLYNX.BA_) and set your DOS
     default drive and path to specify the directory you want to download
     into.)

  -  Provide a "disk view" number at the bottom of the screen that isn't
     limited to the 2.1 GB maximum size of a DOS FAT16 partition.  So, the
     free space available for a DosLynx temporary directory in a large DOS
     FAT32 partition won't be under-reported.  (Presently, the disk view
     number stays at or about 2147418112 as long as the large partition has
     at least that much space free.)

I hope you will let me know what you think about the way the DosLynx
Protected Mode version has turned out.  Or, if you have any comment(s) or
question(s) in any connection with DosLynx v0.35b!  As I don't have a
'286 based PC, presently, I am especially interested in hearing from anyone
who is able to try the DosLynx Protected Mode version on such a system.

fcm