A government minister describes the number as "unacceptable" but says Labour is "making progress".