The problem that I want to solve is:
"find all the continuous functions $f\colon \mathbb R\to \mathbb R$ such that for every $x$, $f(f(f(x))) = x $ , I know that f(x) = x is an answer but how can one find (and prove) all the continuous answers ? And more generally what are the steps to find similar problems ?
$\endgroup$ 21 Answer
$\begingroup$If $f(x)$ is a polynomial of degree $d \ge 2$, let$f(x) = ax^d+g(x)$with degree of $g \lt d$. Then$f(f(x)) =a(ax^d+g(x))^d+g(ax^d+g(x)) =a(a^dx^{d^2}+...)+g(ax^d+g(x)) $so$degree(f(f(x)) = d^2$(leading term is$a^dx^{d^2}$) and$degree(f(f(f(x)))) =d^3$so $f$ can not be a polynomial of degree $\ge 2$.
If $f(x) = ax+b$then$f(f(x)) =a(ax+b)+b =a^2x+ab+b $and$f(f(f(x))) =a(a^2x+ab+b)+b =a^3x+a^2b+ab+b $so$a^3 = 1$and$a^2b+ab +b= 0$.
If $b = 0$then$a^3 = 1$which has one real and two complex solutions.
If$b \ne 0$then$0 =a^2+a+1 $so $a =\dfrac{-1\pm\sqrt{-3}}{2} =a_1, a_2 $.
Therefore solutions are$f(x) = ax$with $a^3 = 1$(so $a = 1, a_1, a_2$) and, for any $b$,$f(x) = a_{1, 2}x+b $.
So, if the solution has to be a polynomial with real coefficients, it can only be $f(x) = x$.
$\endgroup$ 8