Behind the scenes: accessing legacy FTP servers

A few weeks ago when we officially unveiled the Cobalt RaQ 3 server being online on The Real Internet, we knew security would be a real concern.  After all, what made the Cobalt servers so useful is that they integrated a full stack of open source software together with a primitive but friendly web UI.  Anyone who has been working in IT can tell you that a legacy server with a bunch of legacy software is a security problem waiting to happen.

So, we deployed the RaQ on a private LAN and at first only exposed tcp/80 to the world through NAT.  But we wanted to give our community (you) a chance to actually use the RaQ in its mostly-original form, so we also needed a way to also let users have FTP access or some other similar way to upload files.  

After looking at various options to reverse proxy FTP, or provide a web interface frontend for FTP, we deployed a solution which I thought worked out very nicely:

ProFTPD with the mod_proxy module.  ProFTPD has been one of the top open source *nix FTP daemons for the last 20+ years, the original Cobalt RaQ even uses ProFTPD as its FTP server.  

TJ Saunders, the ProFTPD maintainer, started development of the mod_proxy module in 2015 and it has become quite feature rich.  It works in both forward and reverse proxy modes, in our case we want to use it as a _reverse_ proxy where our FTP users will connect to it, and then it connects to the legacy backend FTP server which is walled off from public access.  In addition to providing a protection layer to not directly expose the RaQ FTP server to the Internet, it also leverages TLS to encrypt our users' data across the Internet.  The mod_proxy documentation mentions use of this with IIS servers, which sounds like a neat project someone worked on, I initially forgot IIS included an FTP server.  

If you have a hosting account on our RaQ, when you connect to the FTP server info we provided, you're actually connecting to ProFTPD version 1.3.8 custom compiled on AlmaLinux 8.  It is then proxying your FTP connection to the RaQ which runs ProFTPD 1.2.0.

This solution was great except we ran into one bug with mod_proxy:  It couldn't handle the wall of "helpful" text that the RaQ FTP server displays after a successful login:

This multi-line text wasn't handled properly in mod_proxy, causing the FTP client to not get a response code indicating a successful login.

Fortunately, this was an easy workaround by commenting out the DisplayLogin option in the RaQ's /etc/proftpd.conf file.  I have been emailing with TJ Saunders over the last few weeks, and he is now working on a fix in the code:

Right now, when mod_proxy reads an FTP response from the backend/origin server, even for a multi-line response, it reads that _entire response_ into an internal "response" object.  That single response object is then fed into the "send this response to the frontend client" machinery.  But the multi-line nature of the backend response, in this case of a DisplayLogin file, is being lost -- such that the frontend client sees an FTP response with embedded CRLF ("\r\n") sequences, which then breaks most FTP clients.

The fix will be to preserve that multi-line nature all the way through the mod_proxy machinery.  That's what I'm working through now.

So, we're looking forward to testing out that fix soon, but wanted to share some behind the scenes details.

-Mark @ The Serial Port