We discuss instabilities of fluid films of nanoscale thickness, with a particular focus on films where the destabilizing mechanism allows for linear instability, metastability, and absolute stability, depending on the mean film thickness. Our study is motivated by nematic liquid crystal films; however similar instability mechanisms, and forms of the effective disjoining pressure, appear in other contexts, such as the well- studied problem of polymeric films on two-layered substrates. The analysis is carried out within the framework of the long-wave approximation, which leads to a fourth order nonlinear partial different equation for the film thickness. Within the considered formulation, the nematic character of the film leads to an additional contribution to the disjoining pressure, changing its functional form. This effective disjoining pressure is characterized by the presence of a local maximum for non-vanishing film thickness. Such a form leads to complicated instability evolution that we study by analytical means, including the application of marginal stability criteria, and by extensive numerical simulations that help us develop a better understanding of instability evolution in the nonlinear regime. These simulations, carried out on GPUs, allow for considering large domains so that instability development is not influenced by the domain boundaries. This combination of analytical and computational techniques allows us to reach novel understanding of relevant instability mechanisms, and of their influence on transient and fully developed fluid film morphologies.