There's enjoying things, and being a servant of pleasure.
Plato puts it well in Gorgias; where he makes an analogy of two men tasked with keeping jars filled with wine, honey and milk.
The first man has jars of good quality, and is able to fill them up and they stay full and then he can go on about his life and be happy, the task of keeping them full causes him no grief. The second man has cracks and holes in his jars, so the pleasure always leaks out and he needs to spend every day topping them up in order to avoid misery. He's completely enslaved by the task and it's causing him daily grief to keep them full.
I think the moral of the story is that if you find yourself in that situation, you should repair or replace the jar instead of spending every day in a futile struggle to keep it full.
It's more useful to view resolve, moral continence, self-control as things that you do rather than things you have been granted.
But on the other hand, even the most minor enjoyment (especially the kind that will make you guilty later) is still somehow a form of enjoyment.
Perhaps the bigger lessons is that small pleasures can be expensive in the long term. More than we think.