If you can't hold on to your toe with your knee straight in utthita hasta padangusthasana (one hand big toe pose) then bend your knee and hold on to it instead.
If the problem is lack of grip then grab on to the side of the foot instead. It's no longer "big toe pose" but then again, neither is it if you are using a towel!
Another option is to wrap a towel around your foot and use both hands to hold on to the towel.
This pose could be thought of as a standing hamstring stretching yoga pose for the lifted leg.
Names aside, if your hamstrings lack flexibility, try not to compensate by rounding your spine.
To work on your standing leg stability, practice first with the foot turned out slightly and the standing knee slightly bent. Try squeezing the outer thigh and hip and the inner thigh. Also try squeezing the hamstring.
Practice this first with your lifted knee bent, and then with it straight (or only slightly bent). Hold for only a breath or two then switch legs.
Next, as your leg stability improves, work towards doing it with a straight knee and your foot facing forwards.
One way to work towards the flexibility required in this pose is to rest your foot on a table, book shelf, low wall or even a tree limb.
Keep your standing leg and spine stable and focus on relaxing your hamstring so that you can then position your leg on something higher.
If your problem is balance, work on keeping your balance by using your standing foot to feel where your center of gravity is.
Work at keeping your center of gravity centered between the outside and inside edges of your standing foot. Also position your center of gravity forwards enough that your toes naturally press down. You can then use your toes to help keep your center stable. If your sensitivity is good, you can practice balancing without the use of your toes. In this case try shifting your weight onto your heel and keep it there. Relax your toes.
To practice stabilizing your body, start by activating the feet and ankle of your standing leg. Make both strong and rigid. Then activate the muscles of your lower leg. Hug the bones with the muscles housed in the lower leg.
Do the same with your knee joint.
Stabilize the hip of your standing leg by sucking the thigh bone of that leg into its hip socket. From there stabilize your mid section, spine, ribcage and shoulder of the arm supporting your leg.
If using the towel option and both arms then stabilze both shoulders. To stabilize your shoulders pull your shoulders towards their respective elbows. (Thanks Tim.)
With respect to the Utthita Hasta Padangusthasana lifted leg, you can focus on relaxing the back of the thigh so that you can lift the leg higher.
Or you can try activating the back of the thigh by pulling your leg down while at the same time using your hand to pull up.
Another option is to focus on reaching the leg out of the hip. Make your leg feel long. And at the same time pull up with the arm or pull up and back.
Experiment with these variations whether the leg is to the front or the side.
In the Ashtanga Yoga Standing Sequence, the first two variations of this pose are with the hand grabbing the big toe first with your leg to the front and then open your lifted leg to the side.
When moving the leg to the side, try externally rotating the leg so that the knee points slightly rearwards.
Then move your leg to the front again, bend forward once and kiss your knee and then lift your torso, release your hand and hold your leg out to the front.
Hold each position for 5 breaths.
Do the right leg first (stand on your left foot) and then the left leg.
Ashtanga Yoga Poses: Utthita Hasta Padangusthasana 1, 2 and 3
Especially when holding your leg out to the front without using your hand, you may find this a lot easier to do if you bend your standing leg.
This gives you more room to stabilize your pelvis with the muscles of your standing leg.
From there it is then easier to engage your abs to unify your ribage and pelvis. You then have a more stable base for reaching your leg to the front.
Also, because this can be hard to learn on the fly, especially after having done the hand grabbing variations, you might try resting after the first two variations, then go into the third variation with fresh legs.
You may find that you learn it faster and can then transition into it without resting.
Another option for this third utthita hasta padangusthasana position is to practice first with your lifted knee bent. Because this is easier, work up to holding for ten slow breaths. Then work at slowly (over the course of several practices) making your knee straighter.
The alternative is to practice suffering the pain of trying the full pose straight away.