SFTP 文件传输

1. SFTP 简介

SFTP(SSH File Transfer Protocol，也称 Secret File Transfer Protocol)，是一种基于SSH（安全外壳）的安全的文件传输协议。使用SFTP协议可以在文件传输过程中提供一种安全的加密算法，从而保证数据的安全传输，所以SFTP是非常安全的。

但是，由于这种传输方式使用了加密/解密技术，所以传输效率比普通的FTP要低。SFTP是SSH的一部分，SFTP没有单独的守护进程，它必须使用SSHD守护进程（默认端口22）来完成相应的连接操作，sftp服务作为SSH的一个子服务，是通过/etc/ssh/sshd_config配置文件中的Subsystem实现的，如果没有配置Subsystem参数，则系统是不能进行sftp访问的。所以，要分离SSH和sftp服务的画，基本思路就是创建两个SSHD进程，分别监听不同的端口，一个作为SSH服务的deamon，另一个作为sftp服务的deamon.

1.1 Spring Integration核心组件

1. SftpSessionFactory： sftp客户端与服务端的会话工厂，客户端每次访问服务器时都会创建一个session对象，且可以通过SftpSessionCaching 将session对象缓存起来，支持session共享，即可以在一个会话上进行多个channel的操作。如果session被充值，则在最后一次channel关闭之后，将断开连接。isSharedSession为true时session将共享。

2. SftpSessionCaching: sftp会话缓存工厂。通过poolSize和sessionWaiteTimeout来设置缓缓池的大小和会话等待超市时间。缓存池默认是无限大，超市事件默认是Integer.MAX_VALUE。

3. SftpRemoteFileTemplate: 基于SftpSessionFactory创建的sftp文件操作模板类，其父类时RemoteFileTemplate。支持上传、下载、追加、删除、重命名、列表、是否存在等。基于输入输出流实现。

4. SftpInboundChannelAdapter: sftp 入站通道适配器。可同步远程目录到本地，且可监听远程文件的操作，可实现下载

5. SftpOutboundChannelAdapter:sftp 出站通道适配器。实际是一个 sftp 消息处理器，将在服务器与客户端之间创建一个消息传输通道。此处的消息指的是 Message 的 payload，其支持 File、byte[]、String。其支持 ls、nlst、get、rm、mget、mv、put、mput 操作

6. Channel Adapter: 通道适配器，实际上是适配消息在客户端和服务器之间的传输。inbound adapter 是接收其它系统的消息，outbound adapter 是发送消息到其它系统

7. @ServiceActivator: 将注解作用的方法注册为处理消息的站点，inputChannel 表示接收消息的通道

2. 环境搭建

docker run -p 22:22 -d atmoz/sftp foo:pass:::upload

3. 代码工程

实现文件上传和下载

1. pom.xml

   <?xml version="1.0" encoding="UTF-8"?>
   <project xmlns="http://maven.apache.org/POM/4.0.0"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <parent>
           <artifactId>springboot-demo</artifactId>
           <groupId>com.ygb</groupId>
           <version>1.0-SNAPSHOT</version>
       </parent>
       <modelVersion>4.0.0</modelVersion>
   
       <artifactId>sftp</artifactId>
   
       <properties>
           <maven.compiler.source>8</maven.compiler.source>
           <maven.compiler.target>8</maven.compiler.target>
       </properties>
       <dependencies>
           <dependency>
               <groupId>org.springframework.boot</groupId>
               <artifactId>spring-boot-starter-web</artifactId>
           </dependency>
   
           <dependency>
               <groupId>org.springframework.boot</groupId>
               <artifactId>spring-boot-autoconfigure</artifactId>
           </dependency>
           <dependency>
               <groupId>org.springframework.boot</groupId>
               <artifactId>spring-boot-starter-test</artifactId>
               <scope>test</scope>
           </dependency>
           <dependency>
               <groupId>org.springframework.integration</groupId>
               <artifactId>spring-integration-sftp</artifactId>
               <!--            <version>5.4.1</version>-->
               <version>5.2.8.RELEASE</version>
           </dependency>
           <dependency>
               <groupId>org.springframework.boot</groupId>
               <artifactId>spring-boot-configuration-processor</artifactId>
               <optional>true</optional>
           </dependency>
           <dependency>
               <groupId>org.projectlombok</groupId>
               <artifactId>lombok</artifactId>
               <optional>true</optional>
           </dependency>
       </dependencies>
   </project>

2. service

   import com.et.sftp.config.SftpConfiguration;
   import com.et.sftp.service.SftpService;
   import lombok.extern.slf4j.Slf4j;
   import org.springframework.integration.file.remote.FileInfo;
   import org.springframework.integration.sftp.session.SftpRemoteFileTemplate;
   import org.springframework.stereotype.Component;
   import org.springframework.web.multipart.MultipartFile;
   
   import javax.annotation.Resource;
   import java.io.*;
   import java.util.List;
   
   
   @Slf4j
   @Component
   public class SftpServiceImpl implements SftpService {
   
       @Resource
       private SftpRemoteFileTemplate remoteFileTemplate;
   
       @Resource
       private SftpConfiguration.SftpGateway gateway;
   
       /**
        * single file upload
        *
        * @param file File
        */
       @Override
       public void uploadFile(File file) {
           gateway.upload(file);
       }
   
       /**
        * single file upload by byte[]
        *
        * @param bytes bytes
        */
       @Override
       public void uploadFile(byte[] bytes, String name) {
           try {
               gateway.upload(bytes, name);
           } catch (Exception e) {
               log.error("error:", e);
           }
       }
   
       /**
        * uopload by path
        *
        * @param bytes
        * @param filename
        * @param path
        */
       @Override
       public void upload(byte[] bytes, String filename, String path) {
           try {
               gateway.upload(bytes, filename, path);
           } catch (Exception e) {
               log.error("error:", e);
           }
       }
   
       /**
        * list files by path
        *
        * @param path
        * @return List<String>
        */
       @Override
       public String[] listFile(String path) {
           try {
               return remoteFileTemplate.execute(session -> {
                   return session.listNames(path);
               });
           } catch (Exception e) {
               log.error("error:", e);
           }
           return null;
       }
   
   
       /**
        * list file and directory by path
        *
        * @param path
        * @return List<String>
        */
       @Override
       public List<FileInfo> listALLFile(String path) {
           return gateway.listFile(path);
       }
   
       /**
        * download
        *
        * @param fileName 
        * @param savePath 
        * @return File
        */
       @Override
       public File downloadFile(String fileName, String savePath) {
           try {
               return remoteFileTemplate.execute(session -> {
                   remoteFileTemplate.setAutoCreateDirectory(true);
                   boolean existFile = session.exists(fileName);
                   if (existFile) {
                       InputStream is = session.readRaw(fileName);
                       return convertInputStreamToFile(is, savePath);
                   } else {
                       return null;
                   }
               });
           } catch (Exception e) {
               log.error("error:", e);
           }
           return null;
       }
   
       /**
        * read file
        *
        * @param fileName
        * @return InputStream
        */
   
       @Override
       public InputStream readFile(String fileName) {
           return remoteFileTemplate.execute(session -> {
               return session.readRaw(fileName);
           });
       }
   
       /**
        * files is exists
        *
        * @param filePath 
        * @return boolean
        */
       @Override
       public boolean existFile(String filePath) {
           try {
               return remoteFileTemplate.execute(session ->
                       session.exists(filePath));
           } catch (Exception e) {
               log.error("error:", e);
           }
           return false;
       }
   
       public void renameFile(String file1, String file2) {
           try {
               remoteFileTemplate.execute(session -> {
                   session.rename(file1, file2);
                   return true;
               });
           } catch (Exception e) {
               log.error("error:", e);
           }
       }
   
       /**
        * create directory
        *
        * @param dirName
        * @return
        */
   
       @Override
       public boolean mkdir(String dirName) {
           return remoteFileTemplate.execute(session -> {
               if (!existFile(dirName)) {
   
                   return session.mkdir(dirName);
               } else {
                   return false;
               }
           });
       }
   
       /**
        * delete file
        *
        * @param fileName 
        * @return boolean
        */
       @Override
       public boolean deleteFile(String fileName) {
           return remoteFileTemplate.execute(session -> {
               boolean existFile = session.exists(fileName);
               if (existFile) {
                   return session.remove(fileName);
               } else {
                   log.info("file : {} not exist", fileName);
                   return false;
               }
           });
       }
   
       /**
        * batch upload (MultipartFile)
        *
        * @param files List<MultipartFile>
        * @throws IOException
        */
       @Override
       public void uploadFiles(List<MultipartFile> files, boolean deleteSource) throws IOException {
           try {
               for (MultipartFile multipartFile : files) {
                   if (multipartFile.isEmpty()) {
                       continue;
                   }
                   File file = convert(multipartFile);
                   gateway.upload(file);
                   if (deleteSource) {
                       file.delete();
                   }
               }
           } catch (Exception e) {
               log.error("error:", e);
           }
       }
   
       /**
        * batch upload (MultipartFile)
        *
        * @param files List<MultipartFile>
        * @throws IOException
        */
       @Override
       public void uploadFiles(List<MultipartFile> files) throws IOException {
           uploadFiles(files, true);
       }
   
       /**
        * single file upload (MultipartFile)
        *
        * @param multipartFile MultipartFile
        * @throws IOException
        */
       @Override
       public void uploadFile(MultipartFile multipartFile) throws IOException {
           gateway.upload(convert(multipartFile));
       }
   
       @Override
       public String listFileNames(String dir) {
           return gateway.nlstFile(dir);
       }
   
       @Override
       public File getFile(String dir) {
           return null;
       }
   
       @Override
       public List<File> mgetFile(String dir) {
           return null;
       }
   
       @Override
       public boolean rmFile(String file) {
           return false;
       }
   
       @Override
       public boolean mv(String sourceFile, String targetFile) {
           return false;
       }
   
       @Override
       public File putFile(String dir) {
           return null;
       }
   
       @Override
       public List<File> mputFile(String dir) {
           return null;
       }
   
       @Override
       public String nlstFile(String dir) {
           return gateway.nlstFile(dir);
       }
   
       private static File convertInputStreamToFile(InputStream inputStream, String savePath) {
           OutputStream outputStream = null;
           File file = new File(savePath);
           try {
               outputStream = new FileOutputStream(file);
               int read;
               byte[] bytes = new byte[1024];
               while ((read = inputStream.read(bytes)) != -1) {
                   outputStream.write(bytes, 0, read);
               }
               log.info("convert InputStream to file done, savePath is : {}", savePath);
           } catch (IOException e) {
               log.error("error:", e);
           } finally {
               if (inputStream != null) {
                   try {
                       inputStream.close();
                   } catch (IOException e) {
                       e.printStackTrace();
                   }
               }
               if (outputStream != null) {
                   try {
                       outputStream.close();
                   } catch (IOException e) {
                       log.error("error:", e);
                   }
               }
           }
           return file;
       }
   
       private static File convert(MultipartFile file) throws IOException {
           File convertFile = new File(file.getOriginalFilename());
           convertFile.createNewFile();
           FileOutputStream fos = new FileOutputStream(convertFile);
           fos.write(file.getBytes());
           fos.close();
           return convertFile;
       }
   }

3. SftpService接口

   import org.springframework.integration.file.remote.FileInfo;
   import org.springframework.web.multipart.MultipartFile;
   
   import java.io.File;
   import java.io.IOException;
   import java.io.InputStream;
   import java.util.List;
   
   
   public interface SftpService {
   
   
       void uploadFile(File file);
   
       void uploadFile(byte[] bytes, String name);
   
       
       void upload(byte[] bytes, String filename, String path);
   
       
       String[] listFile(String path);
   
       List<FileInfo> listALLFile(String path);
   
       
       File downloadFile(String fileName, String savePath);
   
   
       InputStream readFile(String fileName);
   
       
       boolean existFile(String filePath);
   
   
       boolean mkdir(String dirName);
   
       boolean deleteFile(String fileName);
   
       
       void uploadFiles(List<MultipartFile> files, boolean deleteSource) throws IOException;
   
       
       void uploadFiles(List<MultipartFile> files) throws IOException;
   
       
       void uploadFile(MultipartFile multipartFile) throws IOException;
   
   
      
       String listFileNames(String dir);
   
   
       File getFile(String dir);
   
      
       List<File> mgetFile(String dir);
   
     
       boolean rmFile(String file);
   
       
       boolean mv(String sourceFile, String targetFile);
   
      
       File putFile(String dir);
   
       
       List<File> mputFile(String dir);
   
   
       //void upload(File file);
   
       //void upload(byte[] inputStream, String name);
   
       //List<File> downloadFiles(String dir);
   
   
       String nlstFile(String dir);
   }

4. 配置config

   import com.jcraft.jsch.ChannelSftp;
   import org.springframework.beans.factory.BeanFactory;
   import org.springframework.boot.context.properties.EnableConfigurationProperties;
   import org.springframework.context.annotation.Bean;
   import org.springframework.context.annotation.Configuration;
   import org.springframework.expression.common.LiteralExpression;
   import org.springframework.expression.spel.standard.SpelExpressionParser;
   import org.springframework.integration.annotation.Gateway;
   import org.springframework.integration.annotation.MessagingGateway;
   import org.springframework.integration.annotation.ServiceActivator;
   import org.springframework.integration.core.MessagingTemplate;
   import org.springframework.integration.file.FileNameGenerator;
   import org.springframework.integration.file.remote.FileInfo;
   import org.springframework.integration.file.remote.session.CachingSessionFactory;
   import org.springframework.integration.file.remote.session.SessionFactory;
   import org.springframework.integration.file.support.FileExistsMode;
   import org.springframework.integration.sftp.gateway.SftpOutboundGateway;
   import org.springframework.integration.sftp.outbound.SftpMessageHandler;
   import org.springframework.integration.sftp.session.DefaultSftpSessionFactory;
   import org.springframework.integration.sftp.session.SftpRemoteFileTemplate;
   import org.springframework.messaging.Message;
   import org.springframework.messaging.MessageHandler;
   import org.springframework.messaging.handler.annotation.Header;
   import org.springframework.messaging.handler.annotation.Payload;
   
   import javax.annotation.Resource;
   import java.io.File;
   import java.util.List;
   
   
   
   @Configuration
   @EnableConfigurationProperties(SftpProperties.class)
   public class SftpConfiguration {
   
       @Resource
       private SftpProperties properties;
   
   
       @Bean
       public MessagingTemplate messagingTemplate(BeanFactory beanFactory) {
           MessagingTemplate messagingTemplate = new MessagingTemplate();
           messagingTemplate.setBeanFactory(beanFactory);
           return messagingTemplate;
       }
   
       @Bean
       public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
           DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
           factory.setHost(properties.getHost());
           factory.setPort(properties.getPort());
           factory.setUser(properties.getUsername());
           factory.setPassword(properties.getPassword());
           factory.setAllowUnknownKeys(true);
   //        factory.setTestSession(true);
   //        return factory;
           return new CachingSessionFactory<ChannelSftp.LsEntry>(factory);
       }
   
       @Bean
       public SftpRemoteFileTemplate remoteFileTemplate(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
           return new SftpRemoteFileTemplate(sftpSessionFactory);
       }
   
   
       @Bean
       @ServiceActivator(inputChannel = "downloadChannel")
       public MessageHandler downloadHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
           SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "mget", "payload");
           sftpOutboundGateway.setOptions("-R");
           sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
           sftpOutboundGateway.setLocalDirectory(new File(properties.getLocalDir()));
           sftpOutboundGateway.setAutoCreateLocalDirectory(true);
           return sftpOutboundGateway;
       }
   
   
       @Bean
       @ServiceActivator(inputChannel = "uploadChannel", outputChannel = "testChannel")
       public MessageHandler uploadHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
           SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);
           handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
   //        handler.setChmod();
           handler.setFileNameGenerator(message -> {
               if (message.getPayload() instanceof File) {
                   return ((File) message.getPayload()).getName();
               } else {
                   throw new IllegalArgumentException("File expected as payload.");
               }
           });
           return handler;
       }
   
       @Bean
       @ServiceActivator(inputChannel = "uploadByteChannel")
       public MessageHandler multiTypeHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
           SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);
           handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
           handler.setFileNameGenerator(message -> {
               if (message.getPayload() instanceof byte[]) {
                   return (String) message.getHeaders().get("name");
               } else {
                   throw new IllegalArgumentException("byte[] expected as payload.");
               }
           });
           return handler;
       }
   
   
       @Bean
       @ServiceActivator(inputChannel = "lsChannel")
       public MessageHandler lsHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
           SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "ls", "payload");
           sftpOutboundGateway.setOptions("-R"); 
           return sftpOutboundGateway;
       }
   
   
       @Bean
       @ServiceActivator(inputChannel = "nlstChannel")
       public MessageHandler listFileNamesHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
           SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "nlst", "payload");
           return sftpOutboundGateway;
       }
   
       @Bean
       @ServiceActivator(inputChannel = "getChannel")
       public MessageHandler getFileHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
           SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "get", "payload");
           sftpOutboundGateway.setOptions("-R");
           sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
           sftpOutboundGateway.setLocalDirectory(new File(properties.getLocalDir()));
           sftpOutboundGateway.setAutoCreateLocalDirectory(true);
           return sftpOutboundGateway;
       }
   
       /**
        * create by: qiushicai
        * create time: 2020/11/20
        *
        * @return
        */
       @Bean
       @ServiceActivator(inputChannel = "abc")
       public MessageHandler abcHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
           SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);
           handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
           handler.setFileNameGenerator(message -> {
               if (message.getPayload() instanceof byte[]) {
                   System.out.println("receive message:" + new String((byte[]) message.getPayload()));
                   message.getHeaders().forEach((k, v) -> System.out.println("\t\t|---" + k + "=" + v));
                   return "ok";
               } else {
                   throw new IllegalArgumentException("byte[] expected as payload.");
               }
           });
           return handler;
       }
   
       /**
        *
        *  the #root object is the Message, which has two properties (headers and payload) that allow such expressions as payload, payload.thing, headers['my.header'], and so on
        *
        *  link{ https://stackoverflow.com/questions/46650004/spring-integration-ftp-create-dynamic-directory-with-remote-directory-expressi}
        *  link{ https://docs.spring.io/spring-integration/reference/html/spel.html}
        * @return
        */
       @Bean
       @ServiceActivator(inputChannel = "toPathChannel")
       public MessageHandler pathHandler() {
           SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
           // automatically create the remote directory
           handler.setAutoCreateDirectory(true);
           handler.setRemoteDirectoryExpression(new SpelExpressionParser().parseExpression("headers[path]"));
           handler.setFileNameGenerator(new FileNameGenerator() {
               @Override
               public String generateFileName(Message<?> message) {
                   return (String) message.getHeaders().get("filename");
               }
           });
           return handler;
       }
   
   
       /**
        * <ul>
        * <li>ls (list files)
        * <li> nlst (list file names)
        * <li> get (retrieve a file)
        * <li> mget (retrieve multiple files)
        * <li> rm (remove file(s))
        * <li> mv (move and rename file)
        * <li> put (send a file)
        * <li> mput (send multiple files)
        * </ul>
        *
        * @author ：qiushicai
        * @date ：Created in 2020/11/20
        * @description：outbound gateway API
        * @version:
        */
       @MessagingGateway
       public interface SftpGateway {
   
           //ls (list files)
           @Gateway(requestChannel = "lsChannel")
           List<FileInfo> listFile(String dir);
   
   
           @Gateway(requestChannel = "nlstChannel")
           String nlstFile(String dir);
   
   
   
           @Gateway(requestChannel = "getChannel")
           File getFile(String dir);
   
           @Gateway(requestChannel = "mgetChannel")
           List<File> mgetFile(String dir);
   
           @Gateway(replyChannel = "rmChannel")
           boolean rmFile(String file);
   
           @Gateway(replyChannel = "mvChannel")
           boolean mv(String sourceFile, String targetFile);
   
           @Gateway(requestChannel = "putChannel")
           File putFile(String dir);
   
           @Gateway(requestChannel = "mputChannel")
           List<File> mputFile(String dir);
   
           @Gateway(requestChannel = "uploadChannel")
           void upload(File file);
   
           @Gateway(requestChannel = "uploadByteChannel")
           void upload(byte[] inputStream, String name);
   
           @Gateway(requestChannel = "toPathChannel")
           void upload(@Payload byte[] file, @Header("filename") String filename, @Header("path") String path);
   
           @Gateway(requestChannel = "downloadChannel")
           List<File> downloadFiles(String dir);
   
       }
   }

5. SftpProperties

   import lombok.Data;
   import org.springframework.boot.context.properties.ConfigurationProperties;
   
   @Data
   @ConfigurationProperties(prefix = "sftp")
   public class SftpProperties {
       private String host;
       private Integer port;
       private String password;
       private String username;
       private String remoteDir;
       private String localDir;
   }

6. 配置文件

   ##sftp properties
   sftp.host=127.0.0.1
   sftp.port=22
   sftp.username=foo
   sftp.password=pass
   sftp.remoteDir=/upload
   sftp.localDir=D:\\tmp\\sync-files

4. 测试

1. 文件是否存在

   @Test
   void testExistFile() {
       boolean existFile = sftpService.existFile("/upload/home222.js");
       System.out.println(existFile);
   }

2. 列出目录下的文件

   @Test
   void listFileTest() {
       sftpService.listALLFile("/upload").stream().forEach(System.out::println);
   }

3. 下载文件

   @Test
   void testDownLoad() throws Exception {
       sftpService.downloadFile("/upload/home.js", "D:\\tmp\\c222c.js");
   }

4. 上传

   @Test
   void uploadFile() {
       sftpService.uploadFile(new File("D:\\tmp\\cc.js"));
   }