It appears on a list if their in friends and rivals mode too. Then they have the choice to join the game or keep searching for a different one.
Your best bet is to figure out where the people on your list live in the world. So you can plan a good time to battle.
I live in Australia, so try playing on weekends between 4-6(if your from the USA).
p.s- I don't think your on my friend list, but I'll add you if you want.
Mines: 1246-2544-3720
Hunter: HYPA
Black
Name: Chris
FC: 3138-8446-5805