#Cognition #Clash in the #IoT #SXSW

Thank you to everyone who attended our (Karl Smith and Thom Heslop) talk at SXSW, it’s the start of a long road into a really complex and contextual problem. But being silent in the crowd as the King walks by with no clothes on is not an option, peoples lives, futures and prosperity is at risk, not to mention the risk of multi-trillion dollar lawsuits that can follow by knowingly distracting people who are engaged in critical tasks.

Cognition Clash in the IoT at SXSW16
Cognition Clash in the IoT at SXSW16

The IoT – Internet of Things (Ubiquity) is the next great opportunity for commerce to engage with business enterprises and customers. However, there is no unified approach to the mental load between physical interaction, mental interaction and digital interaction. This cognitive landscape is inhabited by associated experiences that gel human behaviour and machine interfaces through, touch, mouse and keyboard. The usage of sight, voice and thought create new complexities and risks which have until recently been the subject of defence technologies (battlefield and strategic), where clear outcomes and prescribed mental models exist.

IoT clash girl dies
IoT clash girl dies

The diversification of these touch points and multi-point human logic models clash and derail human thinking patterns.

We are looking for people and their knowledge to help create an Ubiquity Open Standard. We are doing this because no one else has noticed this fundamental error in thinking, the hoping that product based companies will work together in creating common standards that are driven by an understanding of human thinking capabilities, cognitive models, relational thinking and machine interactions is unlikely.

While product manufactures continue with supremacy attitude to other ecosystem products and services,

“the human voice and our needs and desires are subjugated to simply another component”

albeit the one that is constantly paying for everything without any input on how it works.

Some Foundations (the rest will go in a technical paper)

Distributed Cognition studies the ways that memories, facts, or knowledge is embedded in the objects, individuals, and tools in our environment. According to Zhang & Norman (1994), the distributed cognition approach has three key components: Embodiment of information that is embedded in representations of interaction Coordination of enaction among embodied agents. Ecological contributions to a cognitive ecosystem.

In Embodied Interaction Dourish -everyday human interaction is embodied; non-rationalising, intersubjective and bodily active.  User, not designers, create and communicate meaning and manage coupling. Not just concerned with what people do, but also with what they mean by what they do and how that is meaningful to them. It reflects the sets of meanings that can be ascribed to objects and actions over those objects as part of a larger task or enterprise

Cognition the key to the mind, how people understand what they can do is by comparison a Diagnostic Methodology (goals, adaptations, conventions) with what they already know by accessing the Active Narrative patterns they have created in their own minds according to Smith (2005).

Cognition Patterns Cognition Clash in the IoT different people think differently
Cognition Patterns Cognition Clash in the IoT different people think differently

Cognition Groups create a communication paradigm, they carry intention, meaning, risks and benefits.

  • Some Cognition patterns are common, shopping basket etc.
  • Some Cognition Patterns are social by Family, Sports Team etc.
  • Some Cognition Patterns change without notice

Guided Interaction, existing websites offer guided interaction – simplified cognitive pattern encapsulating a plethora of interacting technology and data systems: Shopping Basket – This representation allows for distributed cognition > appropriation > cognitive pattern forming understand– once a user has used a shopping basket they will understand how to use them and generalize: transferable cognitive pattern

Some of the issues with the IoT

  • There is no standard of interactivity for humans in the IoT – not a problem if passive background machine-to-machine. A very big problem if actively interacting with humans, who are all different and can create their own meanings for example LOL.
  • How does a user form any cognitive patterns from an invisible system?
  • IoT combines known patterns as hidden machine-to-machine communications that can create mistrust and security fears
  • Detailed component view we have constructed around daily interactions is no longer valid

Some of our initial research

IoT Design Principals

  • What is device / service for?
  • Where will it be situated?
  • When will it be triggered?
  • What other devices will it be interacting with?
  • Where can it clash?
  • Security? – * Lack of security – Shodan
  • Design Principal: “Do No Harm

IoT Design Risks

Context is critical

  • Situational interaction problems for consideration

The following barriers reduce our ability to understand the situation

  • Perception based on faulty information processing
  • Excessive motivation – over motivated to the exclusion of context
  • Complacency
  • Overload
  • Fatigue
  • Poor communications

A possible solution

  • Avatar (can be visual, sound, texture, smell, taste or a combination) – smart use of Artificial intelligence (AI), where the users cognitive interface is patterned on their unique cognition pattern through a learning algorithm
  • This avatar should be directional and instructional like digital signage
  • This avatar should respond to the users behavioural interaction and should fall away gracefully as users behaviour becomes more ‘expert* In effect it should be a learning system – learns from the users rather than based on static rules
  • For example the AI that George Hotz has built into his self driving car while not the answer points to the kind of thinking required to find the answer, don’t tell the machine to watch and learn from a human and then carry out your task (from 3.33 to 5.04) “the point is to drive naturally like a human, not some engineer’s idea of safety“. For anyone who then thinks this is the final solution, please let us know why you think driving a car is like cooking dinner or navigating the street?

The Full SXSW Talk is on YouTube

Connect to the speakers on LinkedIn here Karl Smith and Thom Heslop

Tagged : / / / / / / / / / / / / / / / / / / / / / / / / / / /

#UX #cognitive #interactions patterns for #IoT by #Gestalt


Pre-accepted and trusted visual standards are vital to user acceptance and experience as they encourage adoption of technology systems. This is vital so that users don’t need to learn new or counter intuitive interaction behaviours.


Just as the creators of hypertext transfer protocol (http) were able to attribute their invention to Vannevar Bush’s ‘Memex’ so user interface architects are able to attribute the key concepts of user interface structures to principals defined by Gestalt. The following explains the key principals of user interface design as key patterns based upon Gestalt principals.

Karl Smith’s Research

The psychology of visual location, shape  and colour are critical  to enable user to understand and interpret their location and expectations of use in any given area. My research from 2002 defined additional aspects as ‘biographical templates’ that establish key perceptions and personal drivers which I linked to persona’s.

Key patterns

Law of continuation

Continuation is the eye’s instinctive action to follow a direction derived from the visual field. For example, in Figure 1.1 our eyes follow the rail tracks from the left of the picture to the top or vice versa, with Figure 1.2 the eye follows the text box layout.

Rail tracks directing users view
Figure 1.1: Rail tracks directing users view
Text boxes directing users view
Figure 1.2: Text boxes directing users view


Law of figure-ground

We distinguish the foreground and background in a visual field.  Two different foreground colours let the viewer perceive different things from the same illustration, as illustrated in Figures 2.1 and 2.2. If our focus (foreground) colour is black, then in the Figure 2.1, you can see a vase.  In Figure 2.2, when the background is black, we see two faces.

Figure 2.1: Vase
Two Faces
Figure 2.2: Two Faces

Law of closure

Open shapes make the individual perceive that the visual pattern is incomplete and the sense of incompletion serves as a distraction to the learner.” Our minds will tend to close gaps and complete unfinished forms. In Figure 3 the letters used to form the word “INCOMPLETE” are sliced into parts but our minds complete the unfinished forms.

Law of Closure
Figure 3: Law of Closure

Law of balance / symmetry

A visual object will appear as incomplete if the visual object is not balanced or symmetrical.  A psychological sense of equilibrium, or balance, is usually achieved when visual ‘weight’ is placed evenly on each side of an axis for example, Figure 4.1 illustrates visual balance but in Figure 4.2 the image appears unbalanced.

Balance Figures Blocks and Web page template
Balance Figures 4.1: Blocks and Web page template
Imbalance Blocks and Web page template
Imbalance 4.2: Blocks and Web page template

Law of focal point

Every visual presentation needs a focal point, called the centre of interest or point of emphasis. This focal point catches the viewer’s attention and persuades the viewer to follow the visual message further. Figure 5.1 shows how a differently shaped element appears to protrude out from among other elements and draws attention, 5.2 create high impact.

Changing Shapes
Figure 5.1: Changing Shapes
High impact
Figure 5.2: High impact

Law of isomorphic correspondence

All images do not have the same meaning to us, because we interpret their meanings based on our experiences.  If we were to see the image in Figure 6 on a computer screen, we would interpret its meaning as a help or question icon, even if we could not understand the German word “Hilfe” because we associate a question mark with ‘help’ based on past experience.

Help Icon
Figure 6: Help Icon

Law of proximity

The law of proximity states that items placed near each other appear to be a group. Viewers will mentally organise closer elements into a coherent object, because they assume that closely spaced elements are related and those further apart are unrelated. In Figure 7, people mentally arrange the sign in component together as a form.

Hotmail login mind base joining of form
Figure 7 Hotmail login mind base joining of form

Law of unity / harmony

Unity implies that a congruity or arrangement exists among the elements in a design; they look as though they belong together, as though there is some visual connection beyond mere chance that has caused them to come together.  If the related objects do not appear within the same form, the viewer will consider the separate objects to be unrelated to the main visual design, leading to confusion. Figure 8.1 and 8.2 are examples of unity in presentation where all of objects are arranged together into a unified form.

Hotmail, password problems
Figure 8.1: Hotmail, password problems
Apple, password problems
Figure 8.2: Apple, password problems

Law of Similarity

Similar objects will be counted as the same group and this technique can be used to draw a viewer’s attention. Below in Figure 9 the viewer can recognise a triangle inside the square, because these elements look similar and thus part of the same form.

Figure 9: Similarity creates a focal point

Law of Simplicity

When users are presented with visuals, there is an unconscious effort to simplify what is perceived into what the viewer can understand. The simplification works well if the graphical message is already uncluttered, but if the graphics are complex and open to interpretation the simplification process may lead to unintended conclusions. The example below Figure 10:1 shows the Plough star grouping which people can naturally join together, while Figure 10:2 just shows the Sky

Star group the Plough
Figure 10.1: The Plough
The sky
Figure 10.2: The sky


Chang, D., Nesbitt, K., V., Australian Computer Society, 2006. Developing Gestalt-based design guidelines for multi-sensory displays. MMUI ’05: Proceedings of the 2005 NICTA-HCSNet Multimodal User Interaction Workshop – Volume 57 , Volume 57.

Kearsley, G., Campbell, R., L., Elkerton, J., Judd, W., Walker,  J., SIGCHI conference. 1998. Online help systems: design and implementation issues (panel). CHI ’88: Proceedings of the SIGCHI conference on Human factors in computing systems.

Flieder, K., Modrritscher, F., CHI Montreal 2006. Foundations of a Pattern Language based on Gestalt Principals.

Author Links

Tagged : / / / / / / / / / / /