Designing LinguaBytes : a tangible language learning system for non- or hardly speaking toddlers

Technological advances are moving the field of industrial design towards intelligent products, systems and services. Today we, designers, have the possibility to create environments that are aware of, and responsive to our actions, needs and wishes. As such we are getting closer to what is commonly known as Ambient Intelligence—a world of intelligent products and systems that are embedded, adaptive, context-aware and anticipatory. Such a world could considerably enrich our daily lives, under the condition that it fully understands our actions, needs and wishes. This is where the promise of Ambient Intelligence (AmI) gets difficult. Our actions, needs and wishes are not always straightforward; what we do is not necessarily based on logic—the condition-based perspective of machines—but on what we find meaningful. Consequently, if intelligent products and systems are to blend into our daily environments they should not only be able to understand what we find meaningful, but also respond in a way we recognise as meaningful. Here lies the role of the industrial designer of the future. This design dissertation is based on the conviction that meaning emanates from how we interact with the world, an idea inspired by Gibson’s theory of perception and the philosophical school of phenomenology. As meaning emanates from how we interact with the world, it is fully dependent on how we are in the world and therefore by definition subjective. Consequently, the only way to design Ambient Intelligence is to be respectful to human diversity. But how can we do this? How can we design for diversity? How can we, designers, deal with the complexity of heterogeneous user groups? These are the central questions in this dissertation. These questions are addressed in this dissertation using a single, three-year case study, aimed at designing LinguaBytes: a tangible, language learning system for non- or hardly speaking toddlers. Given the fact that these children (with ages between one and four years old) show great diversity in their being-in-the-world, it is crucial that this, to-be-developed, LinguaBytes system perfectly fits the way in which these young children perceive and experience the world, understand the world and give meaning to the world. Consequently, LinguaBytes should be highly flexible to be able to support this diversity. In this research the question ‘how to design for diversity’ was explored through live Research-through-Design cycles. The first cycle (Chapter 5) was aimed at defining theoretical foundations for this research and at getting grip on the research context. The theoretical foundations are presented in Chapter 2, which also includes a description of the research method. Chapter 3 and Chapter 4 include additional theoretical foundations and starting points with regard to the users and context of use, and learning (most prominently Social Constructivism) and early language development, respectively. As this research was done using a Researchthrough-Design method, the information presented in Chapters 3 and 4 was not a given at the start of the project, but accumulated and refined throughout the project in a continuous transaction between scientific knowledge and design action. The second goal of this cycle—getting grip on the research context—involved gaining empathy for the LinguaBytes users and context of use, in order to determine design strategies as well as a global positioning for LinguaBytes in relation to existing products and systems aimed at stimulating early language development. For this, several actions were undertaken, which resulted in a global definition of the LinguaBytes design space. This space was defined by two characteristics: (1) the system’s presence in the physical and/ or virtual domain; and (2) the prominence of language, sub-divided into foreground or background language. It was concluded that LinguaBytes’ design should allow for adjusting the balance between its physical and virtual characteristics, as well as the prominence of language in order to make it suitable for children of different ages and developments. As an exploration of this design space a set of four experienceable 3D-sketches were developed and tested in a Wizard of Oz setup, with two children from the target group and their caregivers. This exploration resulted in the conclusion that the success of intelligent systems such as LinguaBytes, pivots on the ability of these systems to interpret the user, on the accuracy of this interpretation, and on the ability to accurately interpret the interpretation itself. In short, the success of intelligent systems depends on their adaptive competence. It was concluded that, in order to be able to design a sufficient amount of adaptivity in LinguaBytes: (1) more insight was required in the situated factors to which LinguaBytes should become adaptive. It was decided to focus on children with Cerebral Palsy and on the context of speech therapy; (2) strategies were needed that would help design and implement this adaptive behaviour in future LinguaBytes designs. These issues were addressed in the second cycle (Chapter 6), through an extensive brainstorm on adaptivity, which resulted in two important conclusions. The first conclusion was that adaptation can be sub-divided using two criteria: (1) Are the adaptations made to the user or to what he or she is using? In other words, is the user being made suitable for what he or she is using or the other way around? (2) Who does the adapting, the user or what he or she is using? In other words, does someone do the adjusting or are the adjustments automated? In this research these two situations are called system adaptability linguistic exercises. Click-It supported eight exercises, divided over three linguistic categories: phonology, semantics and syntax. Compared with KLEEd, Click-It’s input and output functions were divided over more modules and more, reusable, input materials. In total, Click-It consisted of two modules that were used across applications, four application dependent interface modules and around 25 RFID tagged input materials. Click-It was experimentally tested with a total of twelve children at two child rehabilitation centres. The children used the prototype individually, accompanied by their usual speech therapist, in two speech therapy sessions separated by a week. Although this design was far from perfect, its higher level of realism allowed for identifying a first set of guidelines for adaptivity. Firstly, it was determined that the final LinguaBytes system should include at least two detection systems to be able to adapt its setup to the needs and preferences of individual children: one to detect the modular setup, and one for the identification of the child using the system. Secondly, LinguaBytes should keep track of a child’s motor behaviour and adjust its own actuation behaviour accordingly. Thirdly, LinguaBytes should behave differently in different contexts of use (for example, the home environment, the context of speech therapy) and exchange information between these contexts of use. Despite these apparent opportunities for adaptivity, this experiment also revealed scepticism towards adaptivity: no matter how intelligent LinguaBytes would ever become, the participating therapists did not believe it could ever obtain the required social intelligence to equal a human’s. They indicated that they wanted to be able to overrule LinguaBytes when necessary, which could help LinguaBytes learn. In addition, it was concluded that the system should allow for even more flexibility, for example through the possibility of creating personalised input materials. The fourth cycle (Chapter 8) therefore showed strategic system changes: (1) to increase speech therapists’ control over (the timing of) Click-It’s content the desired ‘control module’ was added; (2) programmable RFID labels were included, with which a child’s own, familiar materials could be turned into system input; (3) two tangible thematic backgrounds were included to increase instant adaptability. In general though, the fourth cycle was one of refinement as the Click-It prototype functioned predominantly well. Click-It was redesigned to solve functional and constructive issues, but most effort was invested in expanding the system to increase adaptability. This resulted in a prototype that supported a full set of stories and exercises within one semantic theme. The prototype, called Click-It 2.0, consisted of two cross-application modules, four application-dependent interface modules and almost 50 RFID tagged input materials. It contained two interactive stories and 39 exercises. Click-It 2.0 was experimentally tested at the same two child rehabilitation centres as in the third cycle, with a total of nine children, accompanied by their regular speech therapists. The children used the prototype in up to three speech therapy sessions separated by a week. In addition to individual sessions with Click-It 2.0, some children also used it collaboratively. From this experiment it was concluded that Click-It 2.0 approached a level of adaptivity required for supporting the diverse language developments of most children from the user group, but also that two actions were needed to thoroughly ascertain this: (1) a longitudinal study with multiple prototypes; (2) a exponential expansion of content and input materials. The fifth cycle (Chapter 9) was aimed at developing the final LinguaBytes design and producing three fully functional prototypes for longitudinal testing. Click-It 2.0 was therefore redesigned with an eye on stability and safety, and its content was expanded. The resulting design (described in detail in Chapter 1 of this dissertation) encompassed a set of approximately 500 core words, distributed over 16 stories and 220 exercises. To interact with these stories and exercises the design contained a wide range of input materials—16 story booklets, 236 input figures and 31 word cards—and five modules for input, output and control. The final design, called LinguaBytes, was tested at four different locations with a total of 65 children and 19 caregivers. LinguaBytes was used for ten months on a daily basis, up to multiple times a day. Most often LinguaBytes was used in individual sessions with a speech therapist, but also in group-sessions with up to four children under the supervision of therapeutic preschool teachers. The longitudinal study showed that LinguaBytes indeed possessed the required amount of content and materials to support the diverse language developments of the targeted user group and resulted in various guidelines for system adaptivity. These are included in this chapter, in the form of design recommendations. In addition, this chapter includes a reflection on the apparent Catch-22 of Research-through-Design: in Chapter 6 it was concluded that, when doing Research-through-Design, the quality of the conclusions are fully dependent on the richness of the prototype. However, this study revealed that there is a price to richness; there is undeniably a trade-o$ between the necessary quality of a prototype and the consequential time investment. Chapter 10 presents a reflection on the LinguaBytes case study, focusing on the theoretical foundations of this research, on methodological issues and on the research question. This chapter includes demonstrations of the contributions of this research, as well as a list of guidelines on ‘how to design for diversity’.