Stimuli-responsive properties of soft materials originate from dynamic structure changes. Layered polydiacetylene (PDA) shows color-change properties with the application of external stimuli, such as thermal and mechanical stresses. Layered organic materials possessing intercalation capabilities and dynamic properties have potentials for tuning their structures and properties by the intercalated guests. A variety of sensing and imaging devices can be developed by control of the stimuli responsivity. Here, the key structures, processes, and mechanisms for tuning the stimuli-responsive color-change properties of layered PDA are studied by in situ analyses with heating and cooling. The in situ analyses indicate that heating initiates thermal motion of the alkyl side chains of the PDA main chain in the host layers around 60 °C, regardless of the type of intercalated guests. Further heating induces torsion of the PDA main chain leading to color changes at different temperatures, depending on the types of the intercalated guest. Then, the layered structure is irreversibly deformed with lowering the crystallinity. The results indicate that the stimuli-responsive color-change properties are controlled by the rigidity of the layered structures consisting of the host layers and guests. The control strategy based on rigidity tuning can be applied to a variety of soft materials with stimuli responsivity.