Good questions!  The topic of Quadratic Residues is worthy of another class of at least 3 hours in number theory, after you have at least grasped Fermat's Little Theorem, Wilson's Theorem, and Chinese Remainder Theorem.

We will talk a little more about it when we are on the number theory topics in a few weeks.  Here I will explain a little bit.

1) Why is $-1$ is not a square in mod $p$ when $p\equiv 3\pmod{4}$?

Let's assume $-1$ is a square (i.e. quadratic residue) mod $p$.  In other words, there exists a number $a$ such that $a^2\equiv -1\pmod{p}$.  Hence we have $a^4\equiv 1\pmod{p}$. According to Fermat's LIttle Theorem, $a^{p-1}\equiv 1\pmod{p}$.  Therefore $p-1$ is a multiple of $4$ (think about why this is true: consider $p-1=4k+r$ and see what $r$ can be).  This means $p\equiv 1\pmod{4}$,  In conclusion, if $p\equiv 3\pmod{4}$, we know $-1$ is not a square.

2) Based on the definition of $f(x)$, 

$$\overline{f(\bar{x})} = \prod_{k=1}^{(p-1)/2} \left(x - e^{-2\pi i k^2/p}\right).$$

For each odd prime $p$, among $1,2,3,\ldots,p-1$, half are quadratic residues (QR) and half are quadratic nonresidues (QNR). The QRs are $k^2$ for $k=1,2,3,\ldots,\dfrac{p-1}{2}$. The rest are QNRs.

For $p\equiv 3\pmod{4}$, since $-1$ is a QNR, so is $-k^2$ for all $k$.  Thus the roots of the above polynomial $\overline{f(\bar{x})}$ are all distinct from the roots of $f(x)$.  If $f(x)$ and $\overline{f(\bar{x})}$ are put together, along with $x-1$, they cover all the roots of unity of degree $p$.  The rest of the solution follows.