We consider the problem of optimal rate allocation and admission control for adaptive video streaming sessions in wireless networks with user dynamics. The central aim is to achieve an optimal tradeoff between several key objectives: maximizing the average rate utility per user, minimizing the temporal rate variability, and maximizing the number of users supported. We derive sample path upper bounds for the long-term net utility rate in terms of either a linear program or a concave optimization problem, depending on whether the admissible rate set is discrete or continuous. We then show that the upper bounds are asymptotically achievable in large-scale systems by policies which either deny access to a user or assign it a fixed rate for its entire session, without relying on any advance knowledge of the duration. Moreover, the asymptotically optimal policies exhibit a specific structure, which allow them to be characterized through just a single variable, and have the further property that the induced offered load is unity. We exploit the latter insights to devise parsimonious online algorithms for learning and tracking the optimal rate assignments and establish the convergence of these algorithms. Extensive simulation experiments demonstrate that the proposed algorithms perform well, even in relatively small-scale systems.