It seems to me that a case can be made that the different
models of trust and key management underlying the two
protocols are actually one of the most significant and
useful differences between them. And that these differences
can actually be leveraged to build a complex system with
some interesting properties. I know the models of trust and authentication were discussed above, but I will repeate some of that information here because that is the focus of my comments and I want to highlight some things about them. Most of the discussions I have seen about the differences between FTPS and SFTP focus on administration and firewall navigation. Classic FTP is not very firewall-friendly, as it were, because of the need to open two ports per session and the need in one of the more efficient permutations, to open one of the ports inbound. These are naturally also concerns for FTPS because it is an extension of FTP. SSH, and consequently SFTP, only needs one outbound port as I understand it.

WHat FTPS brings to FTP, however, is a hierarchical model
of trust and public key management based on X.509 certificates
and certificate authorities. SSH has a direct model of trust
based on direct comparisons of key copies in local repositories and key management based on out-of-band direct distribution of public keys. Note that self-signed certificates can be used in FTPS. This is common in actual business practice although In my opinion it technically violates standard up to TLS 1.0. There may be an acknowledgement of it in newer versions of TLS.
I haven’t checked recently and it is not necessary for these
comments. The point about self-signed certificates that I want to make here is that you they are essentially functionally the same in terms of trust model as SSH keys. They have to be exchanged out-of-band for obvious reasons and authentication of the information in them performed by direct comparison with a local copy, again for obvious reasons.

I think it can be argued that direct trust is in some ways
unwieldy, especially on a large scale, because of the need to
keep up to date local copies of public keys and, in the case
of SSH keys, to manually associate them with remote systems.

These were some of the problems that CA trust was invented
to address. With CA trust, you do not need to maintain local
copies of server certificates. You just need to maintain
secure local copies of issuing certificates. If you trust
the issuing certificates, you can trust the information in
server certificates. You can rely on the CA to maintain
information about revocation of server certificates instead
of having to modify a local database if a key is compromised.

Even more interestingly, in my opinion, you can use CA trust
and client authentication to build closed trading partner
networks where access can can be administred centrally and
independently of the databases on the server systems if you
want to maintain a private CA. You simply issue client
certificates to those you want to allow on your network and
revoke them when you want to shut off access. The CA and
its signing certificates can be located deep within a secure
enterprise infrastructure protected by physical and procedural
controls and administered by separate individuals from network
servers used for communication. You just have to be able to
publish CRLs or run an OCSP responder somewhere the communication servers can get to it.

The idea of a hybrid model is that if you are using the system to transmit business documents or messages that contain all the information needed by business systems for processing (for instance EDI) you could consider putting FTPS servers in a DMZ area and using CA trust to control access to them by trading partners, perhaps in conjunction with firewall access rules. If the business documents have all the information needed for further processing, you do not need to propagate any meta-data from the communication servers further. If that is the case, you could find a way to aggregate inbound business documents into one or a few directories. Then you could consider putting SFTP servers that can access the machines with those directories
inside the DMZ and only opening outbound firewall ports to them. Because you would be administering potentially only a few SFTP servers and clients, you would not have issues with managing large numbers of direct trust relationships.

These are just some thoughts about these prococols based on my experience with them. I did not do any detailed specific research for these comments. They are presented here merely as ideas for discussion. They may be entirely misguided. Use them at your own risk and at your own discression. Any security architecture should have a detailed evaluation by competent, preferably outside, experts before being implemented by an organization.

The firewall concern is a big deal, especially for a busy server, with lots of concurrent connections. It’s not uncommon that an FTP server, which supports passive mode connections, needs open inbound port ranges in the 100′s, sometimes even the 1000′s. These are ports that an organization has to open to the internet and monitor for abuse. But, most network engineers already know to cringe about this aspect of passive mode FTP.

However, there’s another side to the firewall story: the clients. I’ve been finding that an increasing amount of organizations are blocking outbound as well, on the firewall. Passive mode ports (which lack an allocation standard) must be defined as permissible destination ports. Another big network headache.

In terms of encryption and authentication, SFTP supports key-based authentication and fingerprinting; whereas, FTPS does not. And, in my honest opinion, I’ve yet to see any significant, practical advantages a CA-signed cert has over self-signed, in the area of FTP. For instance, FTPS clients can store host SSL certs as approved key material.