Finish the game four times (all four modes) and get a large amount of stamina, Travel to the ends of the world and locate the patch of ivy on the northeast side of the shrine. Climb up the ivy and follow trail all the way up and around the ledge to the right. Jump across at the other end of the ledge to reach the next set of ivy. Continue on until you get to another ledge above that you must climb around. Once at the other side of this ledge, climb to the right then shimmy across another ledge that leads behind a column. Jump to the ivy-covered column, then continue climbing. The ivy will lead to a walkway. Climb the stairs and walkway to the left of the garden. REMEMBER: Eating fruit from the trees causes damage. You must defeat a seventeenth Colossus to continue to the end of the path to the top of the temple.
It is possible to reach the Secret Garden early. Find the patch of ivy behind the Temple you start at (almost directly under the huge bridge you came in on). If your grip meter is high enough, you can just jump straight up the side until you reach the first ledge, which has a small patch of leaves on it. Position yourself directly under that patch of leaves. Continue to hold Grip/Duck and jump straight up. Your character will immediately duck in those leaves, allowing you to refill any lost grip. Then, continue around and up the side until you reach the path leading to the garden.