1. Setup VM with:
CPU - 4
Memory - 8GB
Hard disk - 100GB (for software installation) SATA
Three additional hard drives - 8GB (for zfs test) SATA or NVME
Have to use DEBUG build, where we have debug macro
names of other pools
Do not create pools with names that can be used in the tests, i.e. testpool*. Otherwise tests can work incorrectly.
Setup process is described in this instruction.
Note : If you already created dilos_BASE BE according the instruction, and there is no any new tested stable version, just execute the following command:
beadm activate dilos_BASE
In other case if you have already the dilos_ISO_orig BE - make the command:
beadm activate dilos_ISO_orig
Then reboot and start the instruction from the section Step 6. Prepare VM for Tests Running:
reboot
If you have no one of these 2 BEs then start the instruction from the beginning.
Note : Create ztest user if you forgot to do it according the instruction:
sudo useradd -m -d /var/ztest -g staff -s /bin/bash ztest
sudo passwd ztest
echo "ztest ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers.d/ztest
2. Install packages for zfs tests:
Note : If you already did it on the previous step when preparing VM to run, then you can skip this step and go to the next one.
apt update apt install system-test-zfstest testrunner system-file-system-zfs-tests python3 screen
These packets contain kernel drivers, so reboot is required after packets installation.
reboot
3. Update /etc/sudoers:
sudo -E /usr/bin/sed -i.bak '/secure_path/a\Defaults exempt_group+=staff' /etc/sudoers sudo -E /usr/bin/sed -i.bak 's/ requiretty/ !requiretty/' /etc/sudoers
4. Create auxiliary scripts required for cli_user tests:
Create a file /usr/share/zfs/zpool.d/upath with this content or execute the following command:
su - mkdir /usr/share/zfs/zpool.d cat > /usr/share/zfs/zpool.d/upath <<EOT #!/bin/sh if [ "\$1" = "-h" ] ; then echo "Show the underlying path for a device." exit fi # shellcheck disable=SC2154 echo upath="\$VDEV_UPATH" EOT sed -i '/./!d' /usr/share/zfs/zpool.d/upath
Then create a file /usr/share/zfs/zpool.d/iostat with the following content or execute the following commands:
cat > /usr/share/zfs/zpool.d/iostat <<EOT
#!/bin/sh
#
# Display most relevant iostat bandwidth/latency numbers. The output is
# dependent on the name of the script/symlink used to call it.
#
helpstr="
iostat: Show iostat values since boot (summary page).
iostat-1s: Do a single 1-second iostat sample and show values.
iostat-10s: Do a single 10-second iostat sample and show values."
script="\${0##*/}"
if [ "\$1" = "-h" ] ; then
echo "\$helpstr" | grep "\$script:" | tr -s '\\t' | cut -f 2-
exit
fi
if [ "\$script" = "iostat-1s" ] ; then
# Do a single one-second sample
interval=1
# Don't show summary stats
brief="yes"
elif [ "\$script" = "iostat-10s" ] ; then
# Do a single ten-second sample
interval=10
# Don't show summary stats
brief="yes"
fi
if [ -f "\$VDEV_UPATH" ] ; then
# We're a file-based vdev, iostat doesn't work on us. Do nothing.
exit
fi
out=\$(iostat -x "\${VDEV_UPATH##*/}" \
\${interval:+"\$interval"} \
\${interval:+"1"} | tail -n 2)
# Sample output (we want the last two lines):
#
# Linux 2.6.32-642.13.1.el6.x86_64 (centos68) 03/09/2017 _x86_64_ (6 CPU)
#
# avg-cpu: %user %nice %system %iowait %steal %idle
# 0.00 0.00 0.00 0.00 0.00 100.00
#
# Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
# sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
#
# Get the column names
cols=\$(echo "\$out" | head -n 1)
# Get the values and tab separate them to make them cut-able.
vals=\$(echo "\$out" | tail -n 1 | tr -s '[:space:]' '\\t')
i=0
for col in \$cols ; do
i=\$((i+1))
# Skip the first column since it's just the device name
if [ \$i -eq 1 ]; then
continue
fi
# Get i'th value
val=\$(echo "\$vals" | cut -f "\$i")
echo "\$col=\$val"
done
EOT
sed -i '/./!d' /usr/share/zfs/zpool.d/iostat
After that do the following steps:
cd /usr/share/zfs/zpool.d cp iostat iostat-10s cp iostat iostat-1s chmod +x upath iostat iostat-10s iostat-1s
5. Login by user ztest and run tests:
You can run tests with the special test script. The script can be created with any text editor or by executing the following command:
cat > z.sh <<EOT
#!/bin/bash
LOG="/var/tmp/zfstest.\$(date +%F-%T).txt"
ipa=\$(sudo ifconfig | grep -A 1 vmxnet3s0 | sed '1d; s/^[ \\t]*inet[ \\t]*//; s/[ \\t]*netmask.*\$//')
sudo /sbin/zpool destroy testpool
sudo -E /usr/sbin/svcadm disable -s svc:/system/fmd:default
sudo -E find /var/fm/fmd -type f -exec rm {} \\;
sudo -E /usr/sbin/svcadm enable svc:/system/fmd:default
# we need 3 x 8G drives with 512b sector size
export DISKS="\$1 \$2 \$3"
export KEEP="rpool data"
export ZFS_USE_ONLY_DISKS=yes
#remove partition on disks
for D in \$DISKS
do
sudo -E /usr/bin/dd if=/dev/null of=/dev/dsk/\${D}p0 bs=512 count=512K
done
# cleanup drives before tests:
sudo -E /sbin/zpool create -f test123321 \$DISKS
sudo -E /sbin/zpool destroy test123321
sudo -E rm -rf /tmp/mnt* /tmp/ufs.* /tmp/log.* /tmp/tmp.* /tmp/rst* /tmp/tmpfs*
test -d /var/tmp/test_results && sudo rm -rf /var/tmp/test_results
sudo -E /usr/sbin/devfsadm -C
uname -a > \${LOG}
echo "IP Address : \${ipa}" >> \${LOG}
echo "" >> \${LOG}
echo "Disk IDs : \$DISKS" 2>&1 | /usr/bin/tee -a \${LOG}
# run tests
/bin/ksh /opt/zfs-tests/bin/zfstest \$* 2>&1 | /usr/bin/tee -a /var/tmp/z.tmp
echo "Results Summary" >> \${LOG}
echo "" >> \${LOG}
skip=\$(sed -n '/^SKIP[ \\t]*/p' /var/tmp/z.tmp | sed 's/^SKIP[ \\t]*//; s/[ \\t]*\$//')
fail=\$(sed -n '/^FAIL[ \\t]*/p' /var/tmp/z.tmp | sed 's/^FAIL[ \\t]*//; s/[ \\t]*\$//')
pass=\$(sed -n '/^PASS[ \\t]*/p' /var/tmp/z.tmp | sed 's/^PASS[ \\t]*//; s/[ \\t]*\$//')
total=0
if [ "\${pass}" != "" ]; then
total=\$((\${total} + \${pass}))
fi
if [ "\${fail}" != "" ]; then
total=\$((\${total} + \${fail}))
fi
if [ "\${skip}" != "" ]; then
total=\$((\${total} + \${skip}))
fi
echo "TOTAL TESTS: \$total" >> \${LOG}
sed '1,/Results Summary/d' /var/tmp/z.tmp >> \${LOG}
sudo rm -f /var/tmp/z.tmp
EOT
chmod 777 z.sh
After running of this script you will find the log in the /var/tmp/zfstest.<ISO TIME>.txt file. It will contain information in the format that is used in Test Results.
Before running of this script you have to get IDs of tree 8-GB disks from this VM. You can do it with the diskinfo command and run the test script with these IDs in parameters (you can find IDs in the DISK column of the diskinfo output), for example:
ztest@zone:~# sudo diskinfo -p TYPE DISK VID PID SIZE RMV SSD SATA c2t0d0 VMware Virtual SATA Ha> 107374182400 no no SATA c2t1d0 VMware Virtual SATA Ha> 8589934592 no no SATA c2t2d0 VMware Virtual SATA Ha> 8589934592 no no SATA c2t3d0 VMware Virtual SATA Ha> 8589934592 no no ztest@zone:~# ./z.sh c2t1d0 c2t2d0 c2t3d0
Wait about 5h-6h or more for full tests and found full logs at:
/var/tmp/test_results/<ISO TIME>
If you use the SCREEN utility, your command will look like this:
ztest@zone:~# screen ./z.sh <disk1> <disk2> <disk3>
Then you can disconnect in any time by pressing Ctrl-A d and connect back in the new SSH session by the command:
ztest@zone:~# screen -r
6. Run in stability mode
Tests (that are prepared according instructions in the previous section) could be run in the stability mode. To do this just copy this script to the, let say, stab_zfs.h file and make it executable or do these commands:
cat > stab_zfs.sh <<EOT
#!/bin/bash
get_duration() {
IFS=':' read -r -a tm <<< "\$1"
num=\${#tm[@]}
delta=\$((\${tm[0]} * 3600))
if [[ \$num -gt 1 ]]; then
delta1=\$((\${tm[1]} * 60))
(( delta += \$delta1 ))
if [[ \$num -gt 2 ]]; then
(( delta += \${tm[2]} ))
fi
fi
echo \$delta
}
get_time_diff() {
local cur_time=\$(date +%s)
echo \$((\$cur_time - \$1))
}
print_time_diff() {
local delta=\$(get_time_diff \$1)
local sec=\$((\$delta % 60))
let "amin = \$delta / 60"
let "hour = \$amin / 60"
local min=\$((\$amin%60))
if [ -n "\$2" ]; then
printf " Running time: %02d:%02d:%02d\\n" \$hour \$min \$sec >> \$2
else
printf " Running time: %02d:%02d:%02d\\n" \$hour \$min \$sec
fi
}
log_extra() {
echo "---------------- ZPOOL LIST ------------------" >> \$1
zpool list >> \$1
echo "---------------- ZFS LIST ------------------" >> \$1
zfs list >> \$1
echo "---------------- PS AX ------------------" >> \$1
ps ax >> \$1
echo "---------------- TOP -n 10 ------------------" >> \$1
top -n 10 >> \$1
}
i=1
mode=0
TOTALLOG="/var/tmp/stabzfs.\$(date +%F-%T).txt"
dsk1=\$1
shift
dsk2=\$1
shift
dsk3=\$1
shift
echo "============ ZFS Stability test started at \$(date +%F-%T) ==============" > \${TOTALLOG}
if [[ \$# < 2 ]]; then
echo "Missing parameters in the command" | /usr/bin/tee -a \${TOTALLOG}
exit 1
fi
case "\$1" in
-i) mode=1
# number of iterations is provided
maxiter=\$2
echo "Stability Test : launched for \$maxiter iterations at \$(date)." | /usr/bin/tee -a \${TOTALLOG}
;;
-d) mode=2
# duration of tests is provided
delta=\$(get_duration \$2)
echo "Stability Test : launched for \$2 interval (\$delta seconds) at \$(date)." | /usr/bin/tee -a \${TOTALLOG}
;;
-t) mode=3
# stop date/time is provided
stopdt=\$2
;;
*) echo "Stability Test : \$1 is a wrong option" | /usr/bin/tee -a \${TOTALLOG}
echo "Usage : stab_zfs.sh <disk1> <disk2> <disk3> (-i <iterations> | -t <HH:MM[:SS]> | -d <HH:MM[:SS] [DAY-MONTH-YEAR]>]) [-s | <test name>]"
exit 1
;;
esac
shift
shift
if [[ \$mode -eq 3 ]]; then
# stop time/date mode
if [[ \$# > 0 ]]; then
date -d \$1 +%s
stopat=\$?
if [[ \$stopat -eq 0 ]]; then
# stop date is entered after the time
# if it is ommited then date is treated as current one
stopdt="\$stopdt \$1"
shift
fi
fi
echo "Stability Test : launched until \$stopdt at \$(date)." | /usr/bin/tee -a \${TOTALLOG}
stopat=\$(date -d "\$stopdt" "+%s")
fi
GLOB_START_TIME=\$(date +%s)
while [[ \$mode -gt 0 ]]
do
case "\$mode" in
1) # number of iterations is provided
if [[ \$i -gt \$maxiter ]]; then
break
fi
;;
2) # duration of tests is provided
difftm=\$(get_time_diff \$GLOB_START_TIME)
if [[ \$difftm -gt \$delta ]]; then
break
fi
;;
3) # stop date/time is provided
tmpat=\$(date +%s)
if [[ \$stopat -le \$tmpat ]]; then
break
fi
;;
*) break
;;
esac
echo "----------------------- Pass \${i} : started \$(date)" | /usr/bin/tee -a \${TOTALLOG}
START_TIME=\$(date +%s)
./z.sh \$dsk1 \$dsk2 \$dsk3
print_time_diff \${START_TIME} \${TOTALLOG}
echo "----------------------- Pass \${i} : finished \$(date)" | /usr/bin/tee -a \${TOTALLOG}
(( i++ ))
done
echo "Tests are finished successfully at \$(date) with \${i} passes." >> \${TOTALLOG}
print_time_diff \${GLOB_START_TIME} | /usr/bin/tee -a \${TOTALLOG}
EOT
sed -i '/./!d' stab_zfs.sh && chmod 777 stab_zfs.sh
The test script could be run in 3 modes:
for a specific number of iterations
In this case you have to put the -i option and enter the number of iterations, for example, to set 10 iterations:
screen ./stab_zfs.sh <disk1> <disk2> <disk3> -i 10
to work for the specific period of time
In this case you have to put the -d option and enter the time duration in the HH:MM[:SS] format, for example, to work for 10 hours and 15 minutes:
screen ./stab_zfs.sh <disk1> <disk2> <disk3> -d 10:15
or, for example to work for 20 seconds:
screen ./stab_zfs.sh <disk1> <disk2> <disk3> -d 00:00:20
and finally to stop at the specified date and time
In this case you have to put the -t and enter the date and time of stop in the HH:MM[:SS] [DAY-MONTH_YEAR] format, for example, to work from the start to 17:00 of 8th of August 2023 year you will enter:
screen ./stab_zfs.sh <disk1> <disk2> <disk3> -t 17:00 8-AUG-2023
If date is omitted, and only time is entered, it’s supposed that date is the current one. I.e. to stop today at 19:00 you will enter:
screen ./stab_zfs.sh <disk1> <disk2> <disk3> -t 19:00
After the script is finished common log will be put to the /var/tmp/stabzfs.<ISO TIME>.txt file. Logs for every separate pass will be created according the previous section.
If test was crashed or hanged on any pass you can get additional information by executing the following commands:
cat > getinfo.sh <<EOT #!/bin/bash uname -a diskinfo > /dev/null diskinfo > /dev/null echo "-------------------------- DISKINFO ---------------------" diskinfo echo "------------------------- BEADM LIST --------------------" beadm list echo "------------------------- ZPOOL LIST --------------------" zpool list echo "-------------------- LAST SUCCESSFULL TEST --------------" tail -1 /var/tmp/z.tmp echo "------------------- LAST SUCCESSFULL COMMAND -------------" tail -1 /var/tmp/test_results/current/log EOT chmod 777 ./getinfo.sh ./getinfo.sh