HISTORY.TXT
                         (for DosLynx v0.39b, May 2009)
                               by Fred C. Macall

Introduction

In this document, I'll try to take up where I left off in the last
history.txt and not rehash the history of the previous versions.
However, there is a cumulative list of unresolved bugs, issues, and To Do(s)
at the end.  If you want to know everything else there is to know about my
previous releases of DosLynx, you can get that history from:

http://users.ohiohills.com/fmacall/dlx2xdoc.zip
and
http://users.ohiohills.com/fmacall/dlx3xdoc.zip .

dlx2xdoc.zip collects the info.htm and history.txt documents for all eight
of my DosLynx v0.2xb releases.  dlx3xdoc.zip does likewise for all nine of
my DosLynx v0.3xb releases.

Cookies Support for Short FQHNs

Of course, we won't share cookies we've received from sourceforge.net with
(any of the many) other hosts in the net top level domain.  But, we may want
to support sourceforge.net's cookies when we contact sourceforge.net.
And, we may want to do that, in general, for any host.  Even those, like
sourceforge.net, with a fully qualified host name (FQHN) that is too short
to include a related domain (name) part.  DosLynx wasn't allowing that.
(Though it wasn't issuing a reject message for sourceforge.net's cookies,
either.)

This limitation came from assumptions backed up by checks for short FQHNs,
made in ckypname( ), getckies( ), and savecky( ) in CKIESRCV.CPP.
These routines and this module were introduced in the history.txt document
for DosLynx v0.34b.  Removing the limitation was mostly a matter of
adjusting the checks to allow host names one qualification level shorter
than before.  And, adjusting the code to eliminate a couple of assumptions
about the minimum length of the host names being processed in cookies.
The code that makes "domain match" checks has not been so adjusted.
So, if a sourceforge.net cookie should ever include a domain=.net attribute,
DosLynx will (still) reject it.

While in there, it was easy to add HTTPOnly to the list of recognized but
unused cookie attribute names.  Thinking this through also prompted a small
extension in readcky( ), also in CKIESRCV.CPP.  Now, while processing saved
cookies, it will check for a match with the (present) list of recognized
cookie attribute names.  Any saved "cookie" found matching an attribute name
will be ignored.

DHCP Regression

Roberto, who uses a spanish ISP with domain ono.es, wrote to tell me that
the DHCP support in DosLynx quit working for him in version 0.38b.
Tests he ran for me, with the released version of TCPINFO and an
experimental version, pretty much established that the trouble was due to
adding DHCP option(s) to the DosLynx DHCP discover message.

Meanwhile, no one wrote to say that DosLynx v0.38b had fixed a DHCP issue
for them.  So, I decided to grandfather the DosLynx v0.36b flavor DHCP
support, for my_ip=DHCP configuration.  And, add a new my_ip=EDHCP
configuration for specifying the extended DHCP discover message.
If you found the DosLynx v0.38b style DHCP support to your liking, you will
need to change your DosLynx configuration to specify my_ip=EDHCP, with
version 0.39b.

The changes for this fix came in _dobootp( ), in PCBOOTP.C.
set_values( ), in PCCONFIG.C.  main( ), in TCPINFO.C.  And, in DOSLYNX.CFG.

Form Submission And Recognition

Walter Scott wrote to tell me of trouble that came upon submitting (search)
Form(s) found in document(s) from:
http://news.google.com .
(Walter's comments and reports have provided a number of new items for my
informal DosLynx issues list.)  The trouble started with the Form tag's
action="/news?pz=1&ned=us&hl=en" and method="get"
attributes.  The Form's first three "controls" are:
<input type="hidden" name="pz" value="1">
<input type="hidden" name="ned" value="us">
and
<input type="hidden" name="hl" value="en"> .

So, the URL resulting from this Form's submission looked like:
http://news.google.com/news?pz=1&ned=us&hl=en?pz=1&ned=us&hl=en&q= . . .
The Navigate|Show Destination URL (Alt-U) menu entry or command applied
to the Form's Submit Button demonstrated that result.  In turn, Google acted
as though it wasn't prepared to process a request containing two query
parts.  (The parts beginning with question marks.)  I can only guess about
how or why this situation has developed.

I have seen one other Web site with a similar problem.  The troublesome Form
from that site has an action= attribute ending in a question mark.
So, that Form's submission results in a doubled question mark.
And, that's something that site's server doesn't handle.

In both cases, the first query part is completely matched by the beginning
of the second query part.  So, a bypass seems pretty easy and safe:
When a URL resulting for Form submission with method=get contains two query
parts, delete the first query part if it is matched by the beginning of the
second query part.  I have implemented this bypass in TURLView::getChild( ),
in TURLVI18.CPP.

The Navigate|Show Destination URL (Alt-U) menu entry or command gets its
information from 'getChild( ).  So, once you have DosLynx v0.39b installed,
you won't be aware of this issue in any case where it gets bypassed
successfully.  Unless your use of some other browser or viewer discloses
a Form with an action= attribute containing a question mark.
 
Documents carrying Form tag(s) lacking an action= attribute pose a somewhat
related issue.  What default should be provided for a missing action=
attribute?  Well, action="" seems to represent a relative URL that is empty.
We take it to mean the containing document's URL.  By extension, a missing
action= attribute might be taken to mean the same thing.

DosLynx had been ignoring such Forms.  Sometimes robbing their containing
document of its central essence.  So, I decided to correct that.
The needed change has been made in HTML_start_element( )'s case HTML_FORM,
in HTML.C.

Secure Connections for the E-mail Sending Client

Last fall, my ISP "transitioned" its e-mail service to make it a
full-fledged "Google App", based on the Gmail service.  Since then, I
haven't been able to use my old DOS e-mail tools nor the DosLynx e-mail
client.  Because, Gmail's SMTP and post office server(s) require that their
clients use secure (i.e.:  TLS/SSL) connections.  That is something the old
DOS e-mail clients, including the DosLynx e-mail client, have never done
before.

I didn't want to tackle this issue in DosLynx v0.38.  But, in the end, it
seems as though I probably could have.  The main complication is that there
seem to be two protocol variations being used for secure SMTP sessions:
The SSL Protocol defined by Netscape, beginning in 1995.
And, Secure SMTP Over TLS defined in RFCs 2487 and 3207, beginning in 1999.

Netscape's protocol usually is used on an SMTP host's or server's port 465.
In essence, the client simply makes a TCP/IP connection with the server.
And, then uses OpenSSL, via an SSL_connect( ) call, to negotiate a secure
TLS/SSL session.

The RFCs define some additional negotiation to be performed in the clear
before making the same SSL_connect( ) call.  The principal component of this
negotiation is an ESMTP STARTTLS command.  Apparently, SMTP servers may opt
to perform this protocol on either port 25 or port 587.  This negotiation is
like the ESMTP AUTH LOGIN negotiation:  The ESMTP server advertises STARTTLS
in response to the client's EHLO command.  Seeing that advertisement, the
secure SMTP client issues a STARTTLS command.  If all is well, the server
gives a 220 go ahead response.  Then, the client makes its SSL_connect( )
call.

Once an SSL connection has been established, the session continues with a
220 go ahead from the server, in the Netscape protocol case.
Then, for all cases, the session continues with the client's usual EHLO.
And, likely, with an AUTH LOGIN negotiation.  Though, of course, these are
encrypted once an SSL connection has been established.

The smtphost= configuration extension made in DosLynx v0.34b provides for
the port number configuration needed.  I thought of trying to overload that
port number configuration for selecting a security protocol.
But, decided against that.  Instead, a new smtpsec= configuration item
provides that.  Its settings may be None (the default), LegacySSL, or
STARTTLS.  Any one of those possibilities may be used together with any
smtphost= port number specification or its default (port 25).
These things are repeated in comments, in a new section for smtpsec=
configuration, in the updated example DOSLYNX.CFG file.

As usual, the new smtpsec= configuration item gets recognized in
TDosLynx::EvalConfigFile( ), in TDOSLY14.CPP.  'EvalConfigFile( ) leaves its
result, for smtpsec= configuration, in a new smtpsec global.
The smtpsec global is declared in GLOBALS.H and instantiated in GLOBALS.CPP.
All the rest of this enhancement is implemented, in #ifdef USE_SSL clauses,
in TDOSLY15.CPP.  The global declarations, there, consist of:

#include <ssl.h>
extern SSL * SSLphand ;
extern SSL_CTX * ssl_ctx ;

extern "C"
{
SSL * HTGetSSLHandle( ) ;
void HTSSLInitPRNG( ) ;
}

These refer to objects instantiated in the OpenSSL routines and in HTTP.C.
The objects in HTTP.C were introduced in the history.txt document for
DosLynx v0.38b.

Next, as in HTTP.C, there are #defines for HTTP_NETREAD( ),
HTTP_NETWRITE( ), and HTTP_NETCLOSE( ).  Calls to these replace all the
existing s_read( ), s_write( ), and s_close( ) calls.  They allow all the
code following the SSL_connect( ) call to do without explicit tests for
whether or not an SSL connection is in process.

Declarations and initialization added to TDosLynx::mailDeveloper( ), for
secure connections, consist of:

int status ;
BOOL try_tls = TRUE ;

SSLphand = 0 ;

Next, after a TCP/IP connection has been made, comes the possible STARTTLS
negotiation.  Then, as in HTTP.C, a possible sequence of HTGetSSLHandle( ),
SSL_set_fd( ), HTSSLInitPRNG( ), and SSL_connect( ) calls.
The SSL_connect( ) call's return is handled just as in HTTP.C.
So, an initial SSL_connect( ) call tries to set up a TLS connection.
If that fails, another SSL_connect( ) call will get made, to try to set up
an SSL connection.

I now have sent DosLynx e-mail via Gmail, with each of the following two
configuration arrangements:

smtphost=smtp.gmail.com:465 
smtpsec=LegacySSL

smtphost=smtp.gmail.com:587
smtpsec=STARTTLS

Updated Score

As explained under the heading:  Keeping Score, in the history.txt for
DosLynx v0.20b, I am maintaining an informal written list of DosLynx issues
and their resolutions.  In addition to the issues it's always had, this list
now includes issues that have come along in connection with the DosLynx
Protected Mode versions.  At the time of this writing, for the release of
DosLynx v0.39b (late in April 2009), my list has about 631 issues.
Of these, about 443 (over two thirds) have been resolved or ameliorated.
This leaves a total of about 188 issues pending.

If you will look ahead to the next section for a moment, you may note that
several of the pending issues include the parenthetical observation:
"This is hardly an issue with the DosLynx Protected Mode versions."
For the most part, these are memory management issues that may be important
with the DosLynx Real Mode version.  However, since the DosLynx
Protected Mode versions have memory to spare, these aren't significant
issues with them.  As far as users of a DosLynx Protected Mode version are
concerned, the issues to which these observations apply have been resolved.
On the other hand, the DosLynx Protected Mode versions do have a few issues,
of their own, that don't apply to the Real Mode version.

A study identifying issues that don't apply to all three versions finds
about 169 issues pending for the Real Mode version.  About 164 issues
pending for the 16-bit Protected Mode version.  And, about 168 issues
pending for the 32-bit Protected Mode version.  (These figures tend to be a
bit "noisy".  They depend on how we count issues that pertain to the
OpenSSL support that only the 32-bit Protected Mode version has.)
In the next section, I'll list a few of the 188 pending issues that haven't
been mentioned above.

To Do

With both https/SSL support and secure e-mail support working in the
DosLynx 32-bit Protected Mode version, I feel that I have reached all of
my higher priority goals for DosLynxS.  Whether either of the 16-bit
versions will ever be able to incorporate OpenSSL support still remains to
be seen.  As the previous section indicates, my informal DosLynx issues
list still has a lot of pending issues.  But, few of those seem to me to be
"burning" issues.  If DosLynx has a burning issue, I think it is to find
some way to give it some support for Java Scripts!  However, right now, I
can't plan on having that ready in the next release.

So, for the immediate future, I am going to spend less time on DosLynx.
And focus on developing a PMSMTP version for working with SMTP hosts that
require secure connections.  After that is done, I hope to develop a PMPOP
replacement that will work with POP hosts that require secure connections.
If you use DosLynx at least occasionally, I hope you will write and tell me
what feature(s) or fix(es) would make it more useful for you.

Here is a summary of some of the pending DosLynx issues not discussed
anywhere above:

  -  Provide https/SSL/TLS support for the DosLynx 16-bit version(s).
     (This is expected by many Web servers that support data entry.)

  -  Provide Login (original AUTHINFO) support for the news client?

  -  Support the HTML Form <INPUT TYPE=file . . . variation.  (This is part
     of the so called "file upload" capability.)

  -  Find some way to guard against making duplicate Post requests.
     (Bypass:  Move the anchor cursor off the submit button after using it,
     if/when you resume, or first return to, a window containing a submitted
     Form!)

  -  Provide some way for retrying a temporarily failed e-mail send attempt?

  -  Support the HTML Form <BUTTON . . . tag
     (like <INPUT TYPE=submit . . . )?

  -  Automatically replace space(s) with %20 , in links being followed or
     other URLs requested.  (Bypass:  Use the Navigate|Show Destination URL
     (Alt-U) menu entry or command to paste a troublesome link's URL into
     the File|Open URL... (F3) dialog's history.  Then, edit it for use.)

  -  Provide a convenient Cookie(s) file viewing dialog or tool.

  -  How about a fast way to get to the directories, at the end of a long
     files list, in a local file dialog?  (Bypass:  After focusing on a
     list of local files, use Ctrl-PgDn to move to its end, quickly.
     Use Ctrl-PgUp to move back to its start, quickly.)

  -  Find a way for the 16-bit Protected Mode version to support the stock
     Microsoft mouse driver in Windows 2000 DOS windows, successfully.
     Find a way for the 32-bit Protected Mode version to support the stock
     Microsoft mouse driver in Windows NT DOS windows, successfully.
     (This isn't an issue with the DosLynx Real Mode version.
     Bypass:  Use mouse= configuration to disable support of the mouse,
     where necessary.)

  -  Find a graceful way for the FTP client to deal with directory reports
     that are too large to be memorized.  (This is hardly an issue with the
     DosLynx Protected Mode versions.)

  -  Find a way for the File|Save Source and Navigate|Download Selection
     menu entries to handle FTP directory reports that are too large to be
     memorized.  (This is hardly an issue with the DosLynx Protected Mode
     versions.)

  -  When your display is set in a VESA mode upon running DosLynx, its
     arrangement(s) for saving and restoring the screen mode may break down.
     (Bypass:  Set your display into a "legacy mode" before running
     DosLynx.  Or, (re)do the display set up you want afterward.)

  -  Provide for displaying directory entry details, beyond filename, from
     the WAR-FTP Daemon's DOS style directory reports.

  -  Provide a configurable "download directory".  (Bypass:  use the command
     line /P option (as demonstrated in DOSLYNX.BA_) and set your DOS
     default drive and path to specify the directory you want to download
     into.)

  -  Provide a "disk view" number at the bottom of the screen that isn't
     limited to the 2.1 GB maximum size of a DOS FAT16 partition.  So, the
     free space available for a DosLynx temporary directory in a large DOS
     FAT32 partition won't be under-reported.  (Presently, the disk view
     number stays at or about 2147418112 as long as the large partition has
     at least that much space free.)

I hope you will let me know what you think about the way the DosLynx
Protected Mode versions have turned out.  Or, if you have any comment(s) or
question(s) in any connection with DosLynx v0.39b!  As I don't have a
'286 based PC, presently, I am especially interested in hearing from anyone
who is able to try the DosLynx 16-bit Protected Mode version on such a
system.

fcm