A general class of non-smooth and non-convex optimization problems is discussed. Such problems arise in imaging analysis, control and inverse problems and calculus of variation and much more. Our analysis focuses on the infinite dimensional case (PDE-constaint problem and mass transport problem and so on). The Lagrange multiplier theory is developed. Based on the theory we develop the semi-smooth Newton method in the form of Primal-Dual Active set method. Examples are presented to demonstrate the theory and our analysis.