No. Triangles AOC and BOC are not similar. Only two side there are equal. To find AC you shoul recall the following property:

A right triangle inscribed in a circle must have its hypotenuse as the diameter of the circle (the reverse is also true: if the diameter of a circle is also the triangle’s side, then that triangle is a right triangle).

Thus, ABC is a right tringle right angled at C. Now, using Pythagoras theorem:
AC^2 + BC^2 = AB^2;
AC^2 + 9 = 16;
\(AC =\sqrt{7}\)

The area of the square \(side^2=(\sqrt{7})^2=7\).

The two quantities are equal

Answer: C.

it should be D, because it is not mentioned the AB is the diameter of circle

Please see the explanation above. They are there for aa specific reason

I do not get that how we have assumed AB to be the Diameter ?

Formal definition: In geometry, a diameter of a circle is any straight line segment that passes through the center of the circle and whose endpoints lie on the circle

The question tells us that the point O is the center of the circle.
Since AB passes through point O, it must be a diameter.