TBH, Friends that don't play also get data transmitted.
So yes delete unnecessary friends, try using Locale search settings, not Locale Only or Default. If you do use Locale settings already. Try using Default, other players in your area might have bad or good connections, it's worth a shot.
Depending on your ISP, turn off other devices in your house to get better internet connection. Say if you have AT&T, don't worry about it they are dedicated to each device, Time Warner Cable on the other hand splits itself between active devices in your range.
Sir Alex, the Ruby Knight of Everlasting Life