# File Transfers, Synchronisation & Shared Storage Unison, NFS, rSync, SCP # Unison ==================================================================================== #### What is Unison? **-----------------------------------------------------------------------------------------------------------------------------------------------** #### Install Unison Unison will need to be installed on both servers that are sharing files. RHEL: ``` yum install unison ``` Debian: ``` apt install unison ``` **-----------------------------------------------------------------------------------------------------------------------------------------------** #### Configuration of unison Before configuring unison itself, you need to ensure that the hosts have shared keys (since this connection is made via SSH). ``` ssh-keygen -t rsa ssh-copy-id root@otherserver ``` Once that's sorted, the unison service itself can be configured. default unison config file: /root/.unison/default.prf ``` root = / root = ssh://b4sed-01// path = var/spool/cron/ path = etc/passwd path = etc/shadow path = etc/group path = etc/motd path = etc/drbd.conf path = etc/cluster/cluster.conf path = etc/php.ini path = etc/nginx/ ignore = Name access.log* ``` To have unison run automatically, you'll need to configure a cron: ``` vi /usr/local/bin/sync.sh ``` Contents of the file (the SISTER= value needs to be updated). ``` #!/bin/bash SISTER=ABC-WEBDB-01 [ -f /var/run/file_sync.pid ] && exit 1; trap "{ rm /var/run/file_sync.pid; exit; }" EXIT; trap "{ echo 'Bailing out!' 1>&2 ssh -T -p2020 root@$SISTER <<<'killall unison; exit' &>/dev/null }" KILL ABRT INT TERM HUP SEGV touch /var/run/file_sync.pid /usr/bin/unison -sshargs "-p 2020" -batch -terse -silent -owner -group -numericids -prefer / ``` Once added, then the cron needs setting up ``` crontab -e * * * * * /usr/local/bin/sync.sh > /root/.unison/unison.log ``` You would then need to add a cron on the 2nd server, you'll need to add this with the '-prefer' omitted, as below: ``` #!/bin/bash SISTER=ABC-WEBDB-01 [ -f /var/run/file_sync.pid ] && exit 1; trap "{ rm /var/run/file_sync.pid; exit; }" EXIT; trap "{ echo 'Bailing out!' 1>&2 ssh -T -p2020 root@$SISTER <<<'killall unison; exit' &>/dev/null }" KILL ABRT INT TERM HUP SEGV touch /var/run/file_sync.pid /usr/bin/unison -sshargs "-p 2020" -batch -terse -silent -owner -group -numericids / ``` ==================================================================================== #### Multi-server (2+) Unison setup To preface this; Advanced Unison topologies such as the one I detail on this page aren't ideal, one reason for this is that any change requires that unison be run bidirectionally for each node - meaning that synchronisation is definitely not instant. Depending on the content type, a better option might be to have an NFS Share configured. There are different methods that can be used to setup Unison when more than 2 servers are installed, for example, ring and star topology. In this example, I've focussed on the star topology: [![image.png](https://bookstack.b4sed.xyz/uploads/images/gallery/2024-06/scaled-1680-/SJcimage.png)](https://bookstack.b4sed.xyz/uploads/images/gallery/2024-06/SJcimage.png) To set the context for this example and give some explanation; This setup consists of 6 servers, with 1 of those servers acting as the 'master' - being the middle of the star. Unison should be configured on only the master node, from here, you'll need to configure the .prf files and alter the Unison cronjob script accordingly. ------------------------------------------------------------------------------------------------------------------------------------------------ ### Configuration 1\. Install Unison on the master node Decide which node is going to be the master, and install Unison on there. 2\. ==================================================================================== # NFS ## NFS (Guidance based on RHEL/CentOS 7, packages and commands may differ depending on OS) ==================================================================================== ### NFSSHARE SERVER CONFIGURATION #### 1. Install NFS packages on NFS Server ``` yum install nfs-utils rpcbind ``` #### 2. Start on-boot If this is **not** a cluster then start these at boot time: ``` systemctl enable --now rpcbind nfs-server nfs-lock nfs-idmap ``` #### 3. exports Once installed, we can look to configure the nfs share within the /etc/exports file: ``` vim /etc/exports ``` Add the below config (updated with IP of the client server that need to be able to access the nfsshare. ``` /nfsshare IP_IP_IP_IP(rw,sync,no_root_squash) ``` To specify multiple client servers, add separate lines, as below: ``` /nfsshare IP_IP_IP_IP(rw,sync,no_root_squash) /nfsshare IP_IP_IP_IP(rw,sync,no_root_squash) /nfsshare IP_IP_IP_IP(rw,sync,no_root_squash) ``` Once added, you can publish the exports changes using the below command: ``` exportfs -a ``` **=================================================================================** ### NFS CLIENT CONFIGURATION #### 1. Install the required NFS packages: ``` yum install nfs-utils ``` #### 2. Mount the NFSSHARE (won't persist reboot until next step): Firstly, create the file path that you wish for the NFSSHARE to be mounted to: ``` mkdir /nfsshare ``` The below command mounts the NFSSHARE. Ensure to replace the IP\_IP\_IP\_IP:nfsshare with the NFS server IP, and location of the NFSSHARE on the NFS server. Finally, also ensure to update the second /nfsshare on the command to the path you wish for the share to be mounted to on the client server. ``` mount -t nfs IP_IP_IP_IP:/nfsshare /nfsshare ``` #### 3. Test the configuration Test that mounting the nfsshare has worked using the df -h command. You should now see an entry dedicated to the NFSSHARE, as below: #### 4. Permanently configure the NFSSHARE Once you've confirmed that the NFSSHARE is working, you can then look to add the fstab entry that will allow this configuration to persist time & reboots. ``` vim /etc/fstab ``` Add the below link, ensuring the update the syntax as mentioned [above](https://bookstack.b4sed.xyz/link/57#bkmrk-the-below-command-mo) ``` IP_IP_IP_IP:/nfsshare /nfsshare nfs rw,relatime,vers=4,_netdev,timeo=100,retrans=4 0 0 ``` Once you've added this, ensure that this is now working by forcing all fstab-configured mounts to mount: ``` mount -a ``` # rSync ==================================================================================== #### What is rSync? rSync is a file transfer command that can be used for both local and remote transfers.

Remember to be careful with rSync - it's a synchronisation command and can overwrite files. example: I have 2 directories: /: directory1: |\_>file1 directory2: |\_> file1 |\_> file2 If I run `rsync directory1 directory2`, then file2 would be removed, and file1 would be synced.

**-----------------------------------------------------------------------------------------------------------------------------------------------** #### rSync command syntax rsync \[option\] \[[user@host](mailto:user@host)\] **Basic local file transfer** ``` rsync /sourcefile /destinationfile ``` **Basic remote file transfer:** ``` rsync /sourcefile user@IP_IP_IP_IP:destination/location/on/remote/server ``` **rSync Tunelling (secure rsync over SSH)** ``` rsync -e ssh /sourcefile user@IP_IP_IP_IP:destination/location/on/remote/server ``` ##### **flags:**

-a	archive
-z	gzip compression
-u	skip existing files at destination
-r	recursive
-P	show progress of transfer
--exclude=	specify files to ignore (ie "\*.log") (needs to be paired with --include). Appended to command prior to source file specification.
--include=	(only needed if using --exclude) If you're excluding files from an rsync transfer, you'll need to add --include="\*". Appended to command prior to source file specification.
--dry-run	Test run of rSync to display the changes that would be made. No file synchronisation is performed with this option enabled. Appended to end of command.

==================================================================================== # FTP Troubleshooting ==================================================================================== #### FTP Passive Mode Passive ports are used to allow multiple FTP connections by moving open FTP connections from port 21 to a port specified in the passive port range (40,000-40,100 is the standard ANS passive port range ). If your FTP client is using passive mode, you'll usually see an output similar to the below. We can use this to calculate the port being used for passive mode (which we can then check for any restrictions on). `80,244,185,220,156,149 ` First 4 numbers are the server IP, Multiply 5th number by 256 = x X+6th number = passive port which is being used 156x256=39936+149=40085 ------------------------------------------------------------------------------------------------------------------------------------------------ #### SFTP SFTP, which stands for Secure File Transfer Protocol, is a network protocol that provides file access, file transfer, and file management functionalities over any reliable data stream. Unlike FTP (File Transfer Protocol), which is often used with FTP over SSL/TLS (FTPS) for security, SFTP inherently provides secure file transfer through the SSH (Secure Shell) protocol. The primary cPanel/Plesk user can use SFTP if enabled. ==================================================================================== ### FTP Troubleshooting ------------------------------------------------------------------------------------------------------------------------------------------------ ##### cPanel Identify which FTP server is running ``` lsof -i:21 ``` For Pure-FTPd: ``` /var/cpanel/conf/pureftpd/local ``` For ProFTPD: ``` /var/cpanel/conf/proftpd/local ``` Add this line to set which ports your server should use. ``` PassivePortRange: 40000 40100 ``` If your server is behind a firewall and you are seeing unroutable address errors, add the following line, replacing `123.123.123.123` with your server’s public IP: ``` ForcePassiveIP: IP_IP_IP_IP ``` Restart `Pure-FTPd` by running: ``` /usr/local/cpanel/scripts/setupftpserver pure-ftpd --force ``` Allow inbound connections on the passive port range. ------------------------------------------------------------------------------------------------------------------------------------------------ ##### Plesk Plesk also uses the ProFTPD server, but the configuration is slightly different. **Plesk Onyx:** Edit/create the file `/etc/proftpd.d/55-passive-ports.conf` Add the following configuration this file: ``` PassivePorts 40000 40100 ``` Restart the FTP service to pick up the changes: ``` systemctl restart xinetd ``` On your firewall, allow inbound connections on the passive port range. If your server is behind a firewall and you are seeing unroutable address errors, add the following line, replacing `123.123.123.123` with your server’s public IP: ``` MasqueradeAddress IP_IP_IP_IP ``` ------------------------------------------------------------------------------------------------------------------------------------------------ # vSFTPd `vsftpd` (Very Secure FTP Daemon) is a popular FTP server for Linux systems. To use vSFTPd, you'll need to install the `vsftpd `package. \------------------------------------------------------------------------------------------------------------------------------------------------ #### vSFTPd Config Options

pasv\_enable=YES pasv\_min\_port=40000 pasv\_max\_port=40100	Enable passive mode and set port range
pasv\_address=your.external.ip.address	Specify FTP listening IP.
anonymous\_enable=NO	Disable anonymous access
write\_enable=YES	Enable file uploads
local\_enable=YES	Enable local users
chroot\_local\_user=YES	Enable user chroot
chroot\_list\_enable=YES chroot\_list\_file=/etc/vsftpd.chroot\_list	Configure chroot bypass for users.
ssl\_enable=YES	Enable SSL
rsa\_cert\_file=/etc/ssl/certs/vsftpd.pem rsa\_private\_key\_file=/etc/ssl/private/vsftpd.pem	Specify SSL certificate files for FTPS/ES
force\_local\_data\_ssl=YES force\_local\_logins\_ssl=YES	Force SSL usage
ssl\_tlsv1=YES ssl\_sslv2=NO ssl\_sslv3=NO	SSL Protocol options
ssl\_ciphers=HIGH	SSL cipher
force\_local\_data\_ssl=NO force\_local\_logins\_ssl=NO	Enable FTPES

\------------------------------------------------------------------------------------------------------------------------------------------------ #### Adding users for FTP usage 1\. In order to create an account that can use VSFTPd, you will first need to set up a user on the server that you want to transfer files to and from. ``` useradd guest ``` 2\. Once created, you'll want to set a password for that user ``` passwd guest ``` 3\. Also disable shell access for the user ``` usermod -s /sbin/nologin guest ``` \------------------------------------------------------------------------------------------------------------------------------------------------ #### Chrooting users Chrooting a user in `vsftpd` ensures that the user is restricted to their home directory and cannot navigate to other parts of the file system. 1. Add the user Either alter the existing user's home directory, or add a new user to be used for FTP ``` sudo adduser --home /var/ftp/ftpuser ftpuser ``` 2\. Set a password ``` passwd ftpuser ``` 3\. Set home directory permissions ``` sudo chown ftpuser:ftpuser /var/ftp/ftpuser sudo chmod 755 /var/ftp/ftpuser ``` 4\. Configure vsftpd Ensure that the following is present within /etc/vsftpd.conf ``` chroot_local_user=YES ``` \------------------------------------------------------------------------------------------------------------------------------------------------ #### FTPS & FTPES ``` # Enable SSL ssl_enable=YES # Paths to the SSL certificate and key rsa_cert_file=/etc/ssl/certs/vsftpd.pem rsa_private_key_file=/etc/ssl/private/vsftpd.pem # Require SSL for both data and login force_local_data_ssl=YES force_local_logins_ssl=YES # Allow anonymous users to use SSL allow_anon_ssl=YES # SSL protocol options ssl_tlsv1=YES ssl_sslv2=NO ssl_sslv3=NO # Strong ciphers ssl_ciphers=HIGH # Optional: Require SSL reuse for data connections require_ssl_reuse=NO # Enable Explicit SSL (FTPES) # By default, vsftpd will use implicit FTPS (default port 990) # If you prefer explicit FTPS (FTPES), enable the following: force_local_data_ssl=NO force_local_logins_ssl=NO # Explicitly request SSL for login ssl_request_cert=YES ``` ====================================================================================