In order to use HTTPS to communicate between the Viewer and the Relay Server, the following are needed.
The private secret key file and server certification file locations are specified in the configuration file. (SKEY_PEM_File and CERT_PEM_File)
The certificate of CA is added to CA.pem (Certificate of CA for Second Life Server) of the Viewer with a text editor. CA.pem is in C:\Program Files\SecondLife\app_settings for Windows XP. When starting the Viewer, use https instead of http following the --loginuri option.
You also need to start the Relay Server with the -as option.
Communication between the Relay Server and Second Life servers uses HTTPS communication by default. However, the Relay Server is not verified from the stand point of the Second Life server.
If verification of the Relay server is needed the CA.pem mentioned above is copied to an appropriate to Relay Server directory and the location is specified with CA_PEM_File in the configuration file.
Relay Server (sl_relay) also needs to be started with the -aca option.
Note that if verification fails the connection is not made.
You can make a server certificate for yourself, usually by setting CA. Please read the OpenSSL documentation for details on how this is done.