Retrieving the Harbinger ( sword in the stone )

So I powered thru the game w/o upgrading strength whatsoever and finally got to pull on the sword so i can build up my strength. After pulling it I had to stop playing for a while so I hero saved and started playing again later. After I built up my strength i was short xp for lvl 5 physic so i grinded thru Albion to get enough to get the 5-2-3 you need, but when i tried to pull it nothing happened. So im thinking I messed up by trying to pull it on the different hero saves before I upgraded my strength. Can anybody verify this? Im pretty bummed i gotta do it over again to get the sword and I dont want to screw it up again. And it sucks having no sword wielding abilities or health bar. Thanks guys