Hard to tell without listening to it. One possibility is your
recording device has some sort of automatic level control, which when
hit with speech is bringing up the background noise level between
words. Once this has been recorded there's no easy way to get it out.