Currently taking the week off bike riding (or any serious exercise) - I do 3 weeks "on" and 1 week "off" - to recover.
What are the alternatives?
1) Just train all the time - sounds more productive but in reality you can't push yourself all the time, you end up ill or injured.
2) Don't push so hard - yes, you can keep going week after week but you aren't really stretching yourself and things won't improve like they could/should.
3) You ride infrequently, never building up fatigue - a safe option, and you may make some progress, but it's not really a training programme that builds significant capacity, more just riding about and "keeping" fit.
Taking a week off is justified if you have trained hard, you'll need the time off if you have done the 3 weeks right.