Muffins are often misunderstood. For a long time, I misunderstood them, but I'd now like to think I understand them properly. Mainly, I know that they are not cupcakes. They don't have to be as sweet as cupcakes, and they shouldn't be as tender as cupcakes. They don't have to be as uniform in texture as cupcakes. They don't have to be healthy, but it's not bad for them to have some nutrition.
A muffin should have a hearty crumb. Ideally, it would have a crispy-crackly top or outer layer with a softer, but still chewy, interior. You should be able to taste what kind of flour you use. Butter or oil should make a difference.
Yes, I know, it sounds like some weirdo health freak baker talking, but it's true. When you go for a muffin, you at least want to think you're eating something semi-nutritious, whether you realize it or not. Peaches are nice, but they're even better when you throw a little ginger in there. Rye flour does something nice to pumpkin and banana. I put rosemary and fig butter in my oatmeal the other day, and those flavors would make a nice muffin.
Everything you've been doing with your muffins is great. Make muffins you like. But just know that if I'm your muffin man, you'll be getting something faintly sweet, hearty, unassuming. All great qualities, in muffins and in people.