We present an energy formulation of continuum electro-elastic and magneto-electro-elastic materials. Based on the principle of minimum free energy, we propose a form of total free energy of the system in three dimensions, and then systematically derive the theories for a hierarchy of materials including dielectric elastomers, piezoelectric ceramics, ferroelectrics, flexoelectric materials, magnetic elastomers, magneto-electric materials, piezo-electric-magnetic materials among others. The effects of mechanical, electrical and magnetic boundary devices, external charges, polarizations, and magnetization are taken into account in formulating the free energy. The linear and nonlinear boundary value problems governing these materials are explicitly derived as the Euler-Lagrange equations of the principle of minimum free energy. Finally, we illustrate the applications of the formulation in soft matter physics, concerning (1) electrohydrodynamics (EHD), (2) strain-mediated magnetoelectricity, and (3) effective piezoelectricity of electrets.