Custom order requests are covered elsewhere as something worth handling thoughtfully when you say yes, but knowing how to decline gracefully matters just as much — a badly handled "no" can cost you a customer relationship (and a review) that a well-handled one wouldn't.
The key move is responding quickly, even if the answer is no, and being specific about why rather than giving a vague brush-off. "That's a bit outside what I do" leaves a customer unclear whether to ask again differently or just feel rejected. "I mostly work in illustrated, hand-drawn style pieces, and this request is looking for something more photorealistic, which isn't really my specialty" gives a clear, specific, non-personal reason that doesn't leave room for the customer to wonder if they did something wrong in asking.
Offering an alternative, even a small one, softens a decline considerably. This might be redirecting them to an existing design that's close to what they wanted, offering a genuinely different but related option you can actually deliver well, or in some cases, being willing to point them toward a general direction (without naming a specific competitor, which can feel like an odd endorsement) where they might find what they're looking for.
Timing and tone matter as much as content — declining warmly and promptly ("Thanks so much for thinking of me for this! It's a bit outside my usual style, but I really appreciate you asking") tends to leave the relationship intact and the customer still favorably disposed toward your existing catalog, versus a slow, cold, or overly apologetic decline that can feel awkward or discouraging on the receiving end.
It's worth remembering that declining a request that's genuinely outside your skill set or style protects your reputation more than accepting and delivering something mediocre would — a poorly executed custom piece outside your actual strengths risks a negative review far more than a polite, prompt decline ever would. Saying no clearly and kindly is a legitimate, professional response, not a failure of customer service.