Start with the SVD decomposition of $x$:
$$x=USigma V^T$$
Then $$|x|_*=tr(sqrt{x^Tx})=tr(sqrt{(USigma V^T)^T(USigma V^T)})$$
$$Rightarrow |x|_*=tr(sqrt{VSigma U^T USigma V^T})=tr(sqrt{VSigma^2V^T})$$
By circularity of trace:
$$Rightarrow |x|_*=tr(sqrt{V^TVSigma^2})=tr(sqrt{V^TVSigma^2})=tr(sqrt{Sigma^2})=tr(Sigma)$$
Since the elements of $Sigma$ are non-negative.
Therefore nuclear norm can be also defined as the sum of the absolute values of the singular value decomposition of the input matrix.
Now, note that the absolute value function is not differentiable on every point in its domain, but you can find a subgradient.
$$frac{partial |x|_*}{partial x}=frac{partial tr(Sigma)}{partial x}=frac{ tr(partialSigma)}{partial x}$$
You should find $partialSigma$. Since $Sigma$ is diagonal, the subdifferential set of $Sigma$ is: $partialSigma=SigmaSigma^{-1}partialSigma$, now we have:
$$frac{partial |x|_*}{partial x}=frac{ tr(SigmaSigma^{-1}partialSigma)}{partial x}$$ (I)
So we should find $partialSigma$.
$x=USigma V^T$, therefore:
$$partial x=partial USigma V^T+UpartialSigma V^T+USigmapartial V^T$$
Therefore:
$$UpartialSigma V^T=partial x-partial USigma V^T-USigmapartial V^T$$
$$Rightarrow U^TUpartialSigma V^TV=U^Tpartial xV-U^Tpartial USigma V^TV-U^TUSigmapartial V^TV$$
$$Rightarrow partialSigma =U^Tpartial xV-U^Tpartial USigma - Sigmapartial V^TV$$
egin{align}
Rightarrow\
tr(partialSigma) &=& tr(U^Tpartial xV-U^Tpartial USigma - Sigmapartial V^TV)\
&=& tr(U^Tpartial xV)+tr(-U^Tpartial USigma - Sigmapartial V^TV)
end{align}
You can show that $tr(-U^Tpartial USigma - Sigmapartial V^TV)=0$ (Hint: diagonal and antisymmetric matrices, proof in the comments.), therefore:
$$tr(partialSigma) = tr(U^Tpartial xV)$$
By substitution into (I):
$$frac{partial |x|_*}{partial x}= frac{ tr(partialSigma)}{partial x} =frac{ tr(U^Tpartial xV)}{partial x}=frac{ tr(VU^Tpartial x)}{partial x}=(VU^T)^T$$
Therefore you can use $U V^T$ as the subgradient.
参考：这里