What is cos theta of tan theta= 8/5?

I'd suggest you write:

":What is cos theta   IF   tan theta= 8/5?

tan theta = opp / adj = 8 / 5, so we know that opp = 8 and adj = 5.  Then the length of the hypotenuse is sqrt(8^2+5^2) = 9.43.

Thus, the cosine of theta / cosine of this angle is

adj           5
= , or about 0.53.
hyp       9.43

The tangent ratio is the side opposite the reference angle over the side adjacent the reference angle.  If you set up a right triangle in the first quadrant where both x and y are positive, the side opposite the reference angle is 8 and the stationary part of the angle is on the x axis as 5.  We are now lacking the hypotenuse measure of this right triangle.  Find it with Pythagorean's Theorem.  c^2 - 8^2 + 5^2.  c^2 = 64 + 25, so the length of the hypotenuse is sqrt 89.  The cosine ratio is the side adjacent over the hypotenuse, so the cosine of that same angle is 5/sqrt89.

Your pictures are upside down