Non-linear State-space Model Identification from Video Data using Deep Encoders

Research output: Contribution to conferencePaperAcademic

65 Downloads (Pure)


Identifying systems with high-dimensional inputs and outputs, such as systems measured by video streams, is a challenging problem with numerous applications in robotics, autonomous vehicles and medical imaging. In this paper, we propose a novel non-linear state-space identification method starting from high-dimensional input and output data. Multiple computational and conceptual advances are combined to handle the high-dimensional nature of the data. An encoder function, represented by a neural network, is introduced to learn a reconstructability map to estimate the model states from past inputs and outputs. This encoder function is jointly learned with the dynamics. Furthermore, multiple computational improvements, such as an improved reformulation of multiple shooting and batch optimization, are proposed to keep the computational time under control when dealing with high-dimensional and large datasets. We apply the proposed method to a video stream of a simulated environment of a controllable ball in a unit box. The simulation study shows low simulation error with excellent long term prediction for the obtained model using the proposed method.
Original languageEnglish
Publication statusPublished - 2021
Event19th IFAC Symposium on System Identification, SYSID 2021 - Virtual, Padova, Italy
Duration: 13 Jul 202116 Jul 2021
Conference number: 19


Conference19th IFAC Symposium on System Identification, SYSID 2021
Abbreviated titleSYSID 2021
Internet address


  • Non-linear State-Space Modelling
  • Deep Learning
  • Pixels
  • Multiple Shooting


Dive into the research topics of 'Non-linear State-space Model Identification from Video Data using Deep Encoders'. Together they form a unique fingerprint.
  • Nonlinear state-space identification using deep encoder networks

    Beintema, G. I., Tóth, R. & Schoukens, M., 2021, Proceedings of Learning for Dynamics and Control, 7-8 June 2021, The Cloud. Jadbabaie, A., Lygeros, J. & Pappas, G. J. (eds.). PMLR, p. 241-250 10 p. (Proceedings of Machine Learning Research; vol. 144).

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    Open Access

Cite this