Adjoint and self-adjoint filter operators are introduced, such that large-eddy simulation (LES) with a spatially variable filter width satisfies important physical properties: Conservation of momentum and dissipation of kinetic energy. The combination of an arbitrary nonuniform explicit filter with the Smagorinsky model leads to a new model of the turbulent stress tensor, which includes backscatter, while the total subgrid dissipation is still positive (analytically). Nonuniform filter theory is further developed, in order to provide a more solid foundation of practical LES. The paper distinguishes between three sets of equations: The Navier-Stokes equations (which are physical conservation laws), the filtered equations and the modeled large-eddy equations. It is shown that general filtering of the Navier-Stokes equations destroys their local and global conservation properties. However, it is proven that the adjoint of a normalized filter is conservative. As a result, the filtering equations are globally conservative, for special nonuniform (e.g., self-adjoint) filters. Implications for six subgrid-models that require explicit filter operations are considered, such as dynamic, similarity, filtering multiscale, and relaxation models. Incorporation of the adjoint filter analytically ensures several models to conserve momentum and dissipate kinetic energy. Examples of adjoint and self-adjoint filters are also provided, including a "three-points" self-adjoint filter and an adjoint filter that is applicable on unstructured grids. In addition, it is shown that positive nonuniform (self-adjoint) filters satisfy mathematical smoothing properties. The focus is on kernel filters, but projection filters are also discussed, and nonuniform self-adjoint Laplace filters are defined. The (orthogonal) projection operator is proven to be a nonuniform kernel filter.