The gain could be too low even if I used a voltmeter to set it to the
subs' recommended RMS ratings?

Voltmeter to set the gains? Why?

Setting gains is simple. You crank it all the way up for the sub amp (at
least in your case), and then adjust the other gains relative to that one.
Use your ears, not meters.