https://wiki.preterhuman.net/index.php?title=Compressed_TCP/IP-Sessions_using_SSH-like_tools&feed=atom&action=historyCompressed TCP/IP-Sessions using SSH-like tools - Revision history2024-03-29T15:08:31ZRevision history for this page on the wikiMediaWiki 1.35.0https://wiki.preterhuman.net/index.php?title=Compressed_TCP/IP-Sessions_using_SSH-like_tools&diff=14574&oldid=prevNetfreak: Created page with "<pre> Compressed TCP/IP-Sessions using SSH-like tools Sebastian Schreiber <Schreib@SySS.de> 2.2.2000 1. Introduction In the past, we used to compress files in or..."2020-07-29T22:44:55Z<p>Created page with "<pre> Compressed TCP/IP-Sessions using SSH-like tools Sebastian Schreiber <Schreib@SySS.de> 2.2.2000 1. Introduction In the past, we used to compress files in or..."</p>
<p><b>New page</b></p><div><pre><br />
Compressed TCP/IP-Sessions using SSH-like tools<br />
Sebastian Schreiber <Schreib@SySS.de><br />
<br />
2.2.2000<br />
<br />
1. Introduction<br />
<br />
In the past, we used to compress files in order to save disk space.<br />
Today, disk space is cheap - but bandwidth is limited. By compressing<br />
data streams, you achieve two goals:<br />
<br />
1) You save bandwidth/transfered volume (that is important if you have<br />
to pay for traffic or if your network is loaded.).<br />
<br />
2) Speeding up low-bandwidth connections (Modem, GSM, ISDN).<br />
<br />
This HowTo explains how to save both bandwith and connection time by<br />
using tools like SSH1, SSH2, OpenSSH or LSH.<br />
<br />
2. Compressing HTTP/FTP,...<br />
<br />
My office is connected with a 64KBit ISDN line to the internet, so the<br />
maximum transfer rate is about 7K/s. You can speed up the connection<br />
by compressing it: when I download files, Netscape shows up a transfer<br />
rate of up to 40K/s (Logfiles are compressable by factor 15). SSH is a<br />
tool that is mainly designed to build up secure connections over<br />
unsecured networks. Further more, SSH is able to compress connections<br />
and to do port forwarding (like rinetd or redir). So it is the<br />
appropriate tool to compress any simple TCP/IP connection. "Simple"<br />
means, that only one TCP-connection is opened. An FTP-connections or<br />
the connection between M$-Outlook and MS-Exchange are not simple as<br />
several connections are established. SSH uses the LempleZiv (LZ77)<br />
compression algorithm - so you will achieve the same high compression<br />
rate as winzip/pkzip. In order to compress all HTTP-connections from<br />
my intranet to the internet, I just have to execute one command on my<br />
dial-in machine:<br />
<br />
ssh -l <login ID> <hostname> -C -L8080:<proxy_at_ISP>:80 -f sleep<br />
10000<br />
<br />
<hostname> = host that is located at my ISP. SSH-access is required.<br />
<br />
<login ID> = my login-ID on <hostname><br />
<br />
<proxy_at_ISP> =the web proxy of my ISP<br />
<br />
My browser is configured to use localhost:8080 as proxy. My laptop<br />
connects to the same socket. The connection is compressed and<br />
forwarded to the real proxy by SSH. The infrastructure looks like:<br />
<br />
<br />
<br />
64KBit ISDN<br />
My PC--------------------------------A PC (Unix/Linux/Win-NT) at my ISP<br />
SSH-Client compressed SSH-Server, Port 22<br />
Port 8080 |<br />
| |<br />
| |<br />
| |<br />
|10MBit Ethernet |100MBit<br />
|not compressed |not compressed<br />
| |<br />
| |<br />
My second PC ISP's WWW-proxy<br />
with Netscape,... Port 80<br />
(Laptop)<br />
<br />
<br />
<br />
3. Compressing Email<br />
<br />
3.1. Incoming Emails (POP3, IMAP4)<br />
<br />
Most people fetch their email from the mailserver via POP3. POP3 is a<br />
protocol with many disadvantages:<br />
<br />
<br />
1. POP3 transfers password in clear text. (There are SSL-<br />
implementations of POP/IMAP and a challenge/response<br />
authentication, defined in RFC-2095/2195).<br />
<br />
2. POP3 causes much protocol overhead: first the client requests a<br />
message than the server sends the message. After that the client<br />
requests the transferred article to be deleted. The server confirms<br />
the deletion. After that the server is ready for the next<br />
transaction. So 4 transactions are needed for each email.<br />
<br />
3. POP3 transfers the mails without compression although email is<br />
highly compressible (factor=3.5).<br />
<br />
You could compress POP3 by forwarding localhost:110 through a<br />
compressed connection to your ISP's POP3-socket. After that you have<br />
to tell your mail client to connect to localhost:110 in order to<br />
download mail. That secures and speeds up the connection -- but the<br />
download time still suffers from the POP3-inherent protocol overhead.<br />
<br />
<br />
<br />
It makes sense to substitute POP3 by a more efficient protocol. The<br />
idea is to download the entire mailbox at once without generating<br />
protocol overhead. Furthermore it makes sense to compress the<br />
connections. The appropriate tool which offers both features is SCP.<br />
You can download your mail-file like this:<br />
<br />
<br />
<br />
scp -C -l loginId:/var/spool/mail/loginid /tmp/newmail<br />
<br />
<br />
<br />
But there is a problem: what happens if a new email arrives at the<br />
server during the download of your mailbox? The new mail would be<br />
lost. Therefore it makes more sense to use the following commands:<br />
<br />
ssh -l loginid mailserver -f mv /var/spool/mail/loginid<br />
/tmp/loginid_fetchme<br />
scp -C -l loginid:/tmp/my_new_mail /tmp/loginid_fetchme<br />
<br />
A move (mv) is a elementary operation, so you won't get into truble if<br />
you receive new mail during the execution of the comands. But if the<br />
mail server directories /tmp/ and /var/spool/mail are not on the same<br />
disc you might get problems. A solution is to create a lockfile on the<br />
server before you execute the mv: touch /var/spool/mail/loginid.lock.<br />
You should remove it, after that. A better solution is to move the<br />
file loginid in the same directory:<br />
<br />
ssh -l loginid mailserver -f mv /var/spool/mail/loginid<br />
/var/spool/mail/loginid_fetchme<br />
<br />
After that you can use formail instead of procmail in order to filter<br />
/tmp/newmail into the right folder(s): formail -s procmail <<br />
/tmp/newmail<br />
<br />
3.2. Outgoing Email (SMTP)<br />
<br />
You send email over compresses and encrypted SSH-connections, in order<br />
to:<br />
<br />
<br />
· Save network traffic<br />
<br />
· Secure the connection (This does not make sense, if the mail is<br />
transported over untrusted networks, later.)<br />
<br />
· Authenticate the sender. Many mail servers deny mail relaying in<br />
order to prevent abuse. If you send an email over an SSH-<br />
connection, the remote mail server (i.e. sendmail or MS-exchange)<br />
thinks to be connected, locally.<br />
<br />
If you have SSH-access on the mail server, you need the following<br />
command:<br />
<br />
ssh -C -l loginid mailserver -L2525:mailserver:25<br />
<br />
If you don't have SSH-access on the mail server but to a server that<br />
is allowed to use your mail server as relay, the command is:<br />
<br />
ssh -C -l loginid other_server -L2525:mailserver:25<br />
<br />
After that you can configure your mail client (or mail server: see<br />
"smarthost") to send out mails to localhost port 2525.<br />
<br />
4. Thoughts about performance.<br />
<br />
Of course compression/encryption takes CPU time. It turned out that an<br />
old Pentium-133 is able to encrypt and compress about 1GB/hour --<br />
that's quite a lot. If you compile SSH with the option "--with-none"<br />
you can tell SSH to use no encryption. That saves a little<br />
performance. Here is a comprise between several download methods<br />
(during the test, a noncompressed 6MB-file was transfered from a<br />
133MHz-Pentium-1 to a 233MHz Pentium2 laptop over a 10MBit ethernet<br />
without other load).<br />
<br />
<br />
<br />
+-------------------+--------+----------+-----------+----------------------+<br />
| | FTP |encrypted |compressed |compressed & encrypted|<br />
+-------------------+--------+----------+-----------+----------------------+<br />
+-------------------+--------+----------+-----------+----------------------+<br />
| Elapsed Time | |7.6s | 26s | 9s | 23s |<br />
+-------------------+--------+----------+-----------+----------------------+<br />
| Throughput | 790K/s | 232K/s | 320K/s | 264K/s |<br />
+-------------------+--------+----------+-----------+----------------------+<br />
|Compression Factor | 1 | 1 | 3.8 | 3.8 |<br />
+-------------------+--------+----------+-----------+----------------------+<br />
<br />
<br />
<br />
5. Greetings<br />
<br />
Thanks to Harald König <koenig@tat.physik.uni-tuebingen.de>, who used<br />
rcp in order to download complete mailboxes. The latest version of<br />
this howto is available on http://www.syss.de/howto.<br />
</pre><br />
<br />
[[Category:How-To]]</div>Netfreak