"Lightly damp" is the key. You're not going to hurt anything with a slightly damp rag and a light wipe-down. What you don't want to do is saturate the glove with water. If it's just dust, you might not even need to dampen the rag. Just wiping off the dust with a clean towel might be all you need to do.
Heavily saturating a glove with water is a glove killer! Have you ever seen a glove that's been left out in the rain? It ain't purdy.
We all know that oil and water don't mix. When a glove gets saturated, two bad things can happen. First, the water that soaks through the leather will displace the natural oils (oil and water don't mix). The oils will be driven to the surface and evaporated away. I've seen a few gloves that got left in the rain wind up being as hard as a rock.
The other bad thing is that when water reaches the interior leather or padding they tend to stay damp longer than the outer shell, because no air can circulate to it to let it dry. That sets up perfect conditions for mold and mildew to form.
I usually clean my glove a few times a season, just when it needs it. I've used a damp rag before. What I really like to use is my magic conditioner rag!
Since I condition a lot of gloves, I have some rags (actually, square sections cut from an old cotton t-shirt) that get used for nothing but conditioner. One of them I've used to apply everything from lanolin to Lexol to vaseline, to the point that the rag is kind of permanently saturated a bit from the residue. You can grab this rag and wipe down a glove real fast, getting off any loose dust or dirt and giving the leather a very light conditioning, all at the same time.