Note: You can specify a directory other than /. ie if you wanted to review the contents of /var, you would format the command as follows:
``` ncdu /var ``` ##### [Find](https://linux.die.net/man/1/find) We can also use the find command to locate files over a specified size. ``` find / -type f -size +300M > filelist.txt ``` The above command would search / (and all subdirectories) for any files over 300M in size. There are various options available for size specification: - G-> for gibibytes - M-> for megabytes - K-> for kibibytes - b-> for bytes ------------------------------------------------------------------------------------------------------------------------------------------ #### Quick wins /var/log/journal - this can be vacuumed to reduce the size: ``` journalctl --vacuum-size=500M ``` ##### View largest directories in / ``` du -h --max-depth=1 / 2> /dev/null ``` ------------------------------------------------------------------------------------------------------------------------------------------ ##### Disk Reserve Most linux filesystems (depending on FS type) will have a segment reserved for special processes. The default percentage is 5. We can use the [tune2fs](https://linux.die.net/man/8/tune2fs) command for ext2/ext3/ext4 filesystems. Check disk reserve: ``` tune2fs -l /dev/partition ``` Changing disk reserve: ``` tune2fs -m2 /dev/partition ``` ------------------------------------------------------------------------------------------------------------------------------------------ #### Inodes Check for inode usage: ``` du --inodes -d 1 / 2>/dev/null ``` # Quick Guide: Expanding Disks ==================================================================================== ### Virtual Disk ExpansionTLDR; MBR can partition a drive into 4 partitions. Not ideal for large drives. Old standard that's being phased out in favour of GPT.
##### `GPT `- GUID Partition Table The GUID Partition Table (GPT) is a modern standard for the layout of partition tables on a physical storage device. GPT is part of the Unified Extensible Firmware Interface (UEFI) standard, which is designed to replace the older BIOS firmware interface used by PCs. Benefits of GPT include: **Larger Disk and Partition Support** - GPT allows for a virtually unlimited number of partitions. GPT can support disks larger than 2 terabytes (TB), up to 9.4 zettabytes (ZB). **Redundancy** - GPT stores a primary partition table at the beginning of the disk and a backup partition table at the end of the disk.TLDR; GPT is more modern, handles larger disks & partitions, and also has redundancy features.
------------------------------------------------------------------------------------------------------------------------------------------------ ### Configuration & management of partitions ------------------------------------------------------------------------------------------------------------------------------------------------ #### fdisk fdisk (format disk) is a command-line utility that can be used for making changes to MBR disk partitions.fdisk is primarily designed for use with MBR partitioned disks. Check the disk partitioning your disk is using before making changes (fdisk -l /dev/devicename). If your disk is GPT, see [here](https://bookstack.b4sed.xyz/link/145#bkmrk-gdisk).
View current partitions and partitioning standard: ``` fdisk -l /dev/devicename ``` To enter into the fdisk utility to manage partitions: ``` fdisk /dev/devicename ``` We then have the following options available:| `n` | Create a new partition |
| `d` | Delete a partition |
| `i` | print current boot record |
**Example;** Let's say we have a single drive, and want to create a 10GB partition. This is an MBR drive. Firstly, enter into the fdisk utility: `fdisk /dev/diskname` From there, the following options can be used to add a new partition with a size of 10GB. Command (m for help): n Partition type: p primary e extended Select (default p): p Partition number (4-128, default 4): #left at default value First sector (34-16777182, default 16775168): #left at default value (typically) Last sector, +/-sectors or +/-size{K,M,G,T,P} (16775168-16777182, default 16777182): +10G NOTE: fdisk shouldn't really be used for partitioning GPT drives. NOTE: Leaving the last sector blank will automatically cause the remainder of space on the drive to be allocated.
------------------------------------------------------------------------------------------------------------------------------------------------ #### gdisk gdisk is the GPT equivalent of fdisk, meaning that it's designed specifically for partitioning disks using GPT formatting.gdisk is primarily designed for use with GPT partitioned disks. Check the disk partitioning your disk is using before making changes (fdisk -l /dev/devicename). If your disk is MBR, see [here](https://bookstack.b4sed.xyz/link/145#bkmrk-fdisk-%28format-disk%29-).
View current partitions and partitioning standard: ``` gdisk -l /dev/devicename ``` To enter into the gdisk utility to manage partitions: ``` gdisk /dev/devicename ``` We then have the following options available:| `n` | Create a new partition |
| `d` | Delete a partition |
| `i` | print current boot record |
**Example;** Let's say we have a single drive, and want to create a 10GB partition. This is a GPT drive. Firstly, enter into the fdisk utility: `gdisk /dev/diskname` From there, the following options can be used to add a new partition with a size of 10GB. Command (? for help): n Partition number (5-128, default 5): First sector (34-2047, default = 34) or {+-}size{KMGTP}: Last sector (34-2047, default = 2047) or {+-}size{KMGTP}: +5G Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): NOTE: fdisk shouldn't really be used for partitioning GPT drives. NOTE: Leaving the last sector blank will automatically cause the remainder of space on the drive to be allocated.
------------------------------------------------------------------------------------------------------------------------------------------------ #### Parted Whilst gdisk and fdisk are designed for use with specific partition formatting (GPT or MBR), the parted command can be used to manage partitions on both. Parted is also a mini-CLI tool.Parted is a more versatile tool, but is also much ore complex to use. Also may not be installed by default on some systems. I would personally advise sticking with fdisk and gdisk.
Enter the parted CLI: ``` parted /dev/devicename ``` We then have the following options:| `print` | print current partition configuration |
| `mkpart ` | create new partition |
BTRFS whilst being much more advanced in it's capabilities than the alternatives, also has it's own shortcomings. For example, additional ability means additional complexity, BTRFS is also known to be temperamental.
BTRFS offers features such as: cross server partitions ##### SWAP SWAP is technically a partition type, but it isn't focused on traditional data storage. Rather SWAP is allocated space on a storage device used by a Linux System that allows for the storage device to be used as an alternative to RAM. [See this page for more info.](https://bookstack.b4sed.xyz/books/linux/page/swap) ------------------------------------------------------------------------------------------------------------------------------------------------ #### Configuring filesystems Once a disk has been partitioned, we can then look to create a filesystem on that disk. You can also create a filesystem on an unpartitioned disk, this will cause the entire disk to be formatted into the specified filesystem type. ##### Check existing filesystem types: ``` df -T ``` or ``` lsblk -f ``` ##### Create a filesystem on a device/partition There are lots of different filesystem types available in Linux. To simplify the process for configuring filesystems, there are symlinks to binaries added in most systems that can be used to specify filesystem types: ``` root@test:~# ls -l /usr/sbin/mk* lrwxrwxrwx 1 root root 8 Mar 23 2022 /usr/sbin/mkdosfs -> mkfs.fat -rwxr-xr-x 1 root root 133752 Jun 1 2022 /usr/sbin/mke2fs -rwxr-xr-x 1 root root 14720 Apr 9 15:32 /usr/sbin/mkfs -rwxr-xr-x 1 root root 22912 Apr 9 15:32 /usr/sbin/mkfs.bfs -rwxr-xr-x 1 root root 482560 Feb 24 2022 /usr/sbin/mkfs.btrfs -rwxr-xr-x 1 root root 35144 Apr 9 15:32 /usr/sbin/mkfs.cramfs lrwxrwxrwx 1 root root 6 Jun 1 2022 /usr/sbin/mkfs.ext2 -> mke2fs lrwxrwxrwx 1 root root 6 Jun 1 2022 /usr/sbin/mkfs.ext3 -> mke2fs lrwxrwxrwx 1 root root 6 Jun 1 2022 /usr/sbin/mkfs.ext4 -> mke2fs -rwxr-xr-x 1 root root 52048 Mar 23 2022 /usr/sbin/mkfs.fat -rwxr-xr-x 1 root root 43408 Apr 9 15:32 /usr/sbin/mkfs.minix lrwxrwxrwx 1 root root 8 Mar 23 2022 /usr/sbin/mkfs.msdos -> mkfs.fat lrwxrwxrwx 1 root root 6 Nov 1 2022 /usr/sbin/mkfs.ntfs -> mkntfs lrwxrwxrwx 1 root root 8 Mar 23 2022 /usr/sbin/mkfs.vfat -> mkfs.fat -rwxr-xr-x 1 root root 391952 Feb 9 2022 /usr/sbin/mkfs.xfs -rwxr-xr-x 1 root root 22704 Jan 10 13:54 /usr/sbin/mkhomedir_helper -rwxr-xr-x 1 root root 12453 Jun 14 2023 /usr/sbin/mkinitramfs -rwxr-xr-x 1 root root 14648 Jun 1 2022 /usr/sbin/mklost+found -rwxr-xr-x 1 root root 72072 Nov 1 2022 /usr/sbin/mkntfs -rwxr-xr-x 1 root root 47496 Apr 9 15:32 /usr/sbin/mkswap ``` You can then create a filesystem using the format shown above.Example; I have just added an additional disk to my server (/dev/sdb). I've then split that disk into 2 separate partitions (/dev/sdb1 & /dev/sdb2). I'd like to format /dev/sdb2 as an EXT4 filesystem: `mkfs.ext4 /dev/sdb2`
Alternatively, you can use the full command syntax to specify a desired filesystem type: ``` mkfs -t type /dev/devicename ``` ------------------------------------------------------------------------------------------------------------------------------------------------ #### Mounting filesystems & fstab ##### Syntax to mount a filesystem (not persistent): ``` mount /dev/devicename /mountpoint ```Using the above command will force the system to try and identify the filesystem type of the filesystem you're mounting. The preferred and safer method is to specify the type manually, as shown below: `mount -t ext4 /dev/devicename /mountpoint`
##### Syntax to mount a filesystem (persistent) -fstab To persistently mount a filesystem, we need to add an entry to the `/etc/fstab` file instructing the system to do this. ``` vim /etc/fstab ``` Once editing, you can then add a line, following the format below: /dev/devicename /mountpoint fstype defaultsExample; I've added a new additional disk to my machine, I've partitioned the disk into 2 (/dev/sdb1 & /dev/sdb2) and I've created filesystems on both (EXT4 on /dev/sdb1, XFS on /dev/sdb2). The below lines could be added to /etc/fstab to persistently mount these filesystems: `/dev/sdb1 /mnt/mount1 ext4 defaults 0 0` `/dev/sdb2 /mnt/mount2 xfs defaults 0 0` Once the entries have been added, the `mount -a` command can be used to mount all entries in /etc/fstab.
You'll notice on the above fstab entries, there is 'defaults'. This specifies that the system default options should be used, however, there are lots of options available for us to set, some common examples I've listed below:| ro | read only |
| rw | read & write |
| async | write operations may be cached and written to disk at a later time. This can improve performance but may risk data integrity in case of unexpected shutdowns. |
| sync | writes have to be completed before the next write starts. Ensures better data integrity at the cost of performance. |
| First number (Dump Field) (potential value 0 or 1) | determines whether disks will be backed up when the dump command is run. The dump command is deprecated so this value can safely be set to 0. |
| Second number (Pass Field) (potential value 0 or 1) | This field determines the order in which `fsck` checks filesystems during the boot process. - A value of `0` means that the filesystem should not be checked by `fsck` during system boot. - A value of `1` or higher indicates the order in which `fsck` should check the filesystem. |
The swappiness value is actually a kernel tunable - [see here for more information](https://bookstack.b4sed.xyz/link/41#bkmrk-kernel-tunables)
#### How does a system decide how often to use SWAP over physical memory? In Linux, the `/proc/sys/vm/swappiness` file is a kernel parameter that controls the swappiness behavior of the virtual memory system. Swappiness is a setting that determines how aggressively the kernel will use swap space. #### Understanding Swappiness **Swappiness Value**: The value of swappiness can range from 0 to 100. 0: The kernel will avoid swapping out processes as much as possible, preferring to keep data in RAM. **100**: The kernel will aggressively swap processes out of physical memory and move them to swap space, keeping more RAM free. **Swappiness at 0**: The kernel will prioritize keeping processes in RAM, only swapping out data when absolutely necessary. This setting is useful for systems where you want to minimize latency and keep performance high for active applications, such as on a desktop or server where response time is critical. **Swappiness at 100**: The kernel will swap out data more readily, even if there is still available RAM. This setting can be useful in certain scenarios where you want to maximize the use of RAM for disk caches and reduce the amount of cached data being cleared to make room for process memory. ------------------------------------------------------------------------------------------------------------------------------------------------ ### Managing swappiness configuration ##### Check swappiness value ``` cat /proc/sys/vm/swappiness ``` #### Changing swappiness Since swappiness is a kernel tunable, the only method to make a swappiness value change persist a reboot is to [create a custom configuration in /etc/sysctl.d](https://bookstack.b4sed.xyz/link/41#bkmrk-kernel-tunables). I'll also detail the steps below: ##### Change swappiness value (persistent) Create a new custom configuration file in /etc/sysctl.d ``` vim 00-custom-swap.conf ``` Within this file, place the following string (with your desired value added) ``` vm.swappiness=10 ``` Once created, apply the settings using the below command: ``` sysctl -p ``` ##### Change swappiness value (not persistent) To change the swappiness value on a machine for the current session (doesn't persist reboot), we need to manually update the swappiness file: ``` vim /proc/sys/vm/swappiness ``` The file should contain only a number - this being the swappiness value. Change this number to your desired value. For example, to set a swap value of 60: ``` root@test:~# cat /proc/sys/vm/swappiness 60 ``` Once the swappiness value has been changed, we then need to switch off/switch on swap to pull in the new value: ``` swapoff -a swapon -a ``` ------------------------------------------------------------------------------------------------------------------------------------------------ ### SWAP storage types For SWAP to work in Linux, there has to be sections of a disk partitioned specifically for this purpose. There are 2 types of disk allocation that we can use to define swap storage: 1. swap file 2. swap partition A swap file and a swap partition both serve the same fundamental purpose in Linux: to provide additional virtual memory by swapping out inactive pages from the RAM to disk. However, they differ in their implementation, flexibility, and some performance aspects. Here's a detailed comparison between the two: ##### Swap Partition - A swap partition is a dedicated section of the disk specifically set aside for swap space. - It does not reside within any filesystem and is recognized by its partition type (typically "Linux swap"). - Swap partitions can offer slightly better performance because they avoid the overhead of filesystem management. - The dedicated nature of swap partitions can make disk I/O more predictable and efficient for swapping purposes. ##### Swap File - A swap file is a regular file within an existing filesystem, such as ext4, XFS, etc. - It is easier to resize a swap file compared to a swap partition. You can simply create a larger or smaller file and configure it without repartitioning the disk. - Swap files can be created, resized, and managed without needing to repartition the disk, making them more convenient for adding or adjusting swap space on the fly. - Swap files may have slightly lower performance compared to swap partitions due to filesystem overhead. ------------------------------------------------------------------------------------------------------------------------------------------------ ### Configuring SWAP storage Check the current swap partition configuration ``` swapon --show ``` #### SWAP Partition ##### Create and enable a swap partition: ``` mkswap /dev/devicename swapon /dev/devicename ``` The above will create a SWAP partition and enable it until reboot. If we want to create a persistently enabled swap partition, then we'll need to add an entry in /etc/fstab to specify this: ``` vim /etc/fstab ``` Add an entry like the following, ensuring to change the /dev/devicename path: ``` /dev/devicename none swap sw 0 0 ``` #### SWAP File ##### Creating a SWAP file: ``` fallocate -l 1G /swapfile ``` The above creates an empty 1GB file. Set the correct permissions: ``` chmod 600 /swapfile ``` Initialise the SWAP file: ``` mkswap /swapfile ``` The above formats the file into SWAP format. Enable the SWAP file (not persistent): ``` swapon /swapfile ``` Persistently enable the SWAP file: ``` vim /etc/fstab ``` Add the following contents: ``` /swapfile none swap sw 0 0 ``` ==================================================================================== # Archiving and Compression ==================================================================================== #### gzip ==================================================================================== #### tar (.tar) - Tape Archive Take multiple files and compress them into a single file. -c -create archive -f -filename -x -extract -z -compress/decompress -f -filename ##### Creating an archive (uncompressed) ``` tar -cf archivename.tar filestoarchive ``` ##### Creating an archive (compressed) ``` tar -czf archivename.tar.gz filestoarchive ``` ##### Extracting a tar archive (compressed) ``` tar -xzf archivename.tar.gz ``` ##### Extracting a tar archive (uncompressed) ``` tar -xf archivename.tar ``` ==================================================================================== #### DD - Convert and copy ##### Full disk backup ``` dd if=/dev/sda of=/pathtobackupfile.img ``` By default, dd will take the backup 1 block at a time - meaning it can take a while. We can add the 'bs' option to specify how much data dd should copy at a time. ``` dd if=/dev/sda of=/pathtobackupfile.img bs=1M ``` For sensitive copies, we can add the conv=sync flag to have dd compare the original data and the copy for 100% accuracy - this will take a lot longer. ``` dd if=/dev/sda of=/pathtobackupfile.img bs=1M conv=sync ``` ##### Compression We can also pipe the dd output into gzip for compression. ``` dd if=/dev/sda bs=1M | gzip -o > pathtobackupfile.gz ``` To uncompress: ``` gunzip pathtobackupfile.gz | dd of=/dev/sdc bs=1M ``` ==================================================================================== xz (.xz) - cpio (.io) - # fstab ==================================================================================== ### fstab (File Systems Table) The `fstab` (file systems table) is a system configuration file (`/etc/fstab`) used to define how disk partitions, filesystems, and other storage devices should be mounted and integrated into the filesystem at boot time. \------------------------------------------------------------------------------------------------------------------------------------------------ ### Purpose of `fstab` - **Mounting Filesystems:** `fstab` tells the operating system which filesystems to mount and where to mount them in the directory structure. - **Automating Mounting:** It allows for the automatic mounting of filesystems at boot time without user intervention. - **Specifying Options:** It provides options for mounting, such as read/write permissions, mount points, and special parameters for specific filesystems. \------------------------------------------------------------------------------------------------------------------------------------------------ ### File Format The `fstab` file consists of lines, each of which describes a filesystem. Each line contains six fields, separated by spaces or tabs: 1. **Filesystem**: The block device or remote filesystem to be mounted (e.g., `/dev/sda1`, `UUID=xxxxx`, `LABEL=xxxxx`, `/server/share`). 2. **Mount Point**: The directory where the filesystem will be mounted (e.g., `/`, `/home`, `/mnt/data`). 3. **Type**: The type of filesystem (e.g., `ext4`, `ntfs`, `nfs`, `tmpfs`). 4. **Options**: Mount options (e.g., `defaults`, `noatime`, `ro`, `rw`). Multiple options are comma-separated. 5. **Dump**: A number indicating whether the filesystem should be backed up by the `dump` utility (`0` for no, `1` for yes). 6. **Pass**: The order in which filesystems should be checked at boot time by the `fsck` utility (`0` for no check, `1` for the root filesystem, `2` for other filesystems).You never need to enable dump - it's an old outdated command.
*example:* ``` #| -a | aggregate disk IO over time (duration of command) |
| -c | specify a number of IO requests to make |
| -i | interval between requests |
| -s | request size (default 4kb) |
| -q | suppress regular output, only show statistics |