You can do less if you feel up to it. The rest gives your muscles a bit of time to recover so you can do more sets with good form in total.
If you consistently feel the rest isn't necessary you can probably do higher level workouts
And during the rest time just do nothing, sometimes I might collect some laundry or small chores like that.