Applications of new technology in travel surveys have demonstrated the possibility to obtain good quality activity data than traditional survey methods. However, the quality of the imputation diary data highly depends on the predictability of data processing algorithms, which are not fully ready yet. Narrowing the gap between imputation results and true activity–travel patterns is necessary to improve the ease of data confirmation in prompted recall surveys and develop fully automatic data imputation systems. This paper proposes an algorithm to decrease the discrepancies between imputed activity–travel diary and the so-called ground truth. Based on the activity–travel pattern obtained using a Bayesian belief network model, the algorithm takes into account the consistency of the full activity–travel pattern within a day in the sense that the activity–travel sequence is represented in terms of a hierarchical set of tours, and the transportation modes within a tour are logically consistent. We explore three different approaches based on the frequency at the trip/tour level and imputation probability at the epoch level, for each transportation mode. Results obtained based on the test using GPS data in the Netherlands show that the new algorithm significantly improves the imputation accuracy of transportation modes compared with an algorithm that does superimpose these pattern constraints.