To unlock houses, first you must have an adult cat or invent a Ghost Zapper with the inventing table, found in "Build/Buy" mode. You will keep inventing random objects until you finally create the Ghost Zapper. Note: the Zapper is different than the Ghost Positioning System, just keep inventing.
Once you have either the cat or Ghost Zapper, wait until night-time after 9pm, then check your Town Map for the ghost icon on various locations throughout town. Some nights, there will be 3, others only 1. Ghosts may not appear on the lot you wish to unlock, just wait for the next night and try again. Go to that location with either your cat or Sim with a Zapper, locate the ghost in that lot, select the ghost with the option "Capture Ghost" and Viola, you have a brand-new ghost in your inventory. You may sell the ghost, release it, or use it to charge the various inventions you create with your inventing table.
After you have captured enough ghosts from the locked lot, it will eventually unlock and you may move into it yourself or simply wait for new Townies to take over.
Hopefully I gave you enough info, but feel free to message me asking for help.