5.26 is similar to 4.13.  You can try trig substitution, or Cauchy-Schwarz inequality.

5.28: The following is true: $$\frac{1}{a_{k+1}} = \frac{1}{a_k} - \frac{1}{a_k + 1},$$ try to go from there.

5.30: If you use the hint, each equation can be converted to an equation related to trig functions, using Law of Cosines.  Alternatively, you can try to solve the system by multiplying the three equations, and then the resulting equation is divided by the square of each of the original equations.  Then you get three new equations, which should be simpler.