Run ZFS tests in stability mode

Tests (that are prepared according instructions in the previous section) could be run in the stability mode. To do this just copy this script to the, let say, stab_zfs.h file and make it executable or do these commands:

cat > stab_zfs.sh <<EOT
#!/bin/bash
get_duration() {
  IFS=':' read -r -a tm <<< "\$1"
  num=\${#tm[@]}
  delta=\$((\${tm[0]} * 3600))
  if [[ \$num -gt 1 ]]; then
    delta1=\$((\${tm[1]} * 60))
    (( delta += \$delta1 ))
    if [[ \$num -gt 2 ]]; then
      (( delta += \${tm[2]} ))
    fi
  fi
  echo \$delta
}
get_time_diff() {
  local cur_time=\$(date +%s)
  echo \$((\$cur_time - \$1))
}
print_time_diff() {
  local delta=\$(get_time_diff \$1)
  local sec=\$((\$delta % 60))
  let "amin = \$delta / 60"
  let "hour = \$amin / 60"
  local min=\$((\$amin%60))
  if [ -n "\$2" ]; then
    printf " Running time: %02d:%02d:%02d\\n" \$hour \$min \$sec >> \$2
  else
    printf " Running time: %02d:%02d:%02d\\n" \$hour \$min \$sec
  fi
}
log_extra() {
  echo "---------------- ZPOOL LIST ------------------" >> \$1
  zpool list >> \$1
  echo "----------------  ZFS LIST  ------------------" >> \$1
  zfs list >> \$1
  echo "----------------    PS AX   ------------------" >> \$1
  ps ax >> \$1
  echo "---------------- TOP  -n 10 ------------------" >> \$1
  top -n 10 >> \$1
}
i=1
mode=0
TOTALLOG="/var/tmp/stabzfs.\$(date +%F-%T).txt"
dsk1=\$1
shift
dsk2=\$1
shift
dsk3=\$1
shift
echo "============ ZFS Stability test started at \$(date +%F-%T) ==============" > \${TOTALLOG}
if [[ \$# < 2 ]]; then
  echo "Missing parameters in the command" | /usr/bin/tee -a \${TOTALLOG}
  exit 1
fi
case "\$1" in
  -i) mode=1
      # number of iterations is provided
      maxiter=\$2
      echo "Stability Test : launched for \$maxiter iterations at \$(date)." | /usr/bin/tee -a \${TOTALLOG}
      ;;
  -d) mode=2
      # duration of tests is provided
      delta=\$(get_duration \$2)
      echo "Stability Test : launched for \$2 interval (\$delta seconds) at \$(date)." | /usr/bin/tee -a \${TOTALLOG}
      ;;
  -t) mode=3
      # stop date/time is provided
      stopdt=\$2
      ;;
  *)  echo "Stability Test : \$1 is a wrong option" | /usr/bin/tee -a \${TOTALLOG}
      echo "Usage : stab_zfs.sh <disk1> <disk2> <disk3> (-i <iterations> | -t <HH:MM[:SS]> | -d <HH:MM[:SS] [DAY-MONTH-YEAR]>]) [-s | <test name>]"
      exit 1
      ;;
esac
shift
shift
if [[ \$mode -eq 3 ]]; then
  # stop time/date mode
  if [[ \$# > 0 ]]; then
    date -d \$1 +%s
    stopat=\$?
    if [[ \$stopat -eq 0 ]]; then
      # stop date is entered after the time
      # if it is ommited then date is treated as current one
      stopdt="\$stopdt \$1"
      shift
    fi
  fi
  echo "Stability Test : launched until \$stopdt at \$(date)." | /usr/bin/tee -a \${TOTALLOG}
  stopat=\$(date -d "\$stopdt" "+%s")
fi
GLOB_START_TIME=\$(date +%s)
while [[ \$mode -gt 0 ]]
do
  case "\$mode" in
  1) # number of iterations is provided
     if [[ \$i -gt \$maxiter ]]; then
       break
     fi
     ;;
  2) # duration of tests is provided
     difftm=\$(get_time_diff \$GLOB_START_TIME)
     if [[ \$difftm -gt \$delta ]]; then
       break
     fi
     ;;
  3) # stop date/time is provided
     tmpat=\$(date +%s)
     if [[ \$stopat -le \$tmpat ]]; then
       break
     fi
     ;;
  *) break
     ;;
  esac
  echo "----------------------- Pass \${i} : started \$(date)" | /usr/bin/tee -a \${TOTALLOG}
  START_TIME=\$(date +%s)
  ./z.sh \$dsk1 \$dsk2 \$dsk3
  print_time_diff \${START_TIME} \${TOTALLOG}
  echo "----------------------- Pass \${i} : finished \$(date)" | /usr/bin/tee -a \${TOTALLOG}
  (( i++ ))
done
echo "Tests are finished successfully at \$(date) with \${i} passes." >> \${TOTALLOG}
print_time_diff \${GLOB_START_TIME} | /usr/bin/tee -a \${TOTALLOG}
EOT
sed -i '/./!d' stab_zfs.sh && chmod 777 stab_zfs.sh

The test script could be run in 3 modes:

for a specific number of iterations

In this case you have to put the -i option and enter the number of iterations, for example, to set 10 iterations:

screen ./stab_zfs.sh <disk1> <disk2> <disk3> -i 10

to work for the specific period of time

In this case you have to put the -d option and enter the time duration in the HH:MM[:SS] format, for example, to work for 10 hours and 15 minutes:

screen ./stab_zfs.sh <disk1> <disk2> <disk3> -d 10:15

or, for example to work for 20 seconds:

and finally to stop at the specified date and time

In this case you have to put the -t and enter the date and time of stop in the HH:MM[:SS] [DAY-MONTH_YEAR] format, for example, to work from the start to 17:00 of 8th of August 2023 year you will enter:

If date is omitted, and only time is entered, it’s supposed that date is the current one. I.e. to stop today at 19:00 you will enter:

After the script is finished common log will be put to the /var/tmp/stabzfs.<ISO TIME>.txt file. Logs for every separate pass will be created according the previous section.

If test was crashed or hanged on any pass you can get additional information by executing the following commands: