Colloidal assembly at fluid interfaces has a great potential for the bottom-up fabrication of novel structured materials. However, challenges remain in realizing controllable and tunable assembly of particles into diverse structures. Herein, the capillary assembly of magnetic ellipsoidal Janus particles at a fluid-fluid interface is reported. Depending on their tilt angle, that is, the angle the particle main axis forms with the fluid interface, these particles deform the interface and generate capillary dipoles or hexapoles. Driven by capillary interactions, multiple particles thus assemble into chain-, hexagonal-lattice-, and ring-like structures, which can be actively controlled by applying an external magnetic field. A field-strength phase diagram is predicted in which various structures are present as stable states. Owing to the diversity, controllability, and tunability of assembled structures, magnetic ellipsoidal Janus particles at fluid interfaces could therefore serve as versatile building blocks for novel materials.