My opponent is not playing the challenge I've issued them!
When you send an opponent a challenge they will have a 7 day window to complete this challenge. If this is not done within that time, gems will be refunded back to their respective accounts.
You can also remind an opponent by nudging them by going into your messages and selecting the relevant challenge the clicking/pressing the nudge button. This will send a pop-up to the opponent reminding them of the challenges. This can also be done by going to the challenges tab under your profile page and selecting the challenge from there.