I talked about my journey as CEO working to commercialise technology within UR.

My key advice as a startup CEO to keep moving forward was;

1) Very clearly know what makes you different and how you can make good money from it
2) You need to keep making decisions regardless of the noise and uncertainty
3) Keep Executing and finding fuel (from revenue or investment)

My key advice to keep moving FAST is to tend to your teams;

1) Inside your organization, your execution team
2) At the governance level, your strategy team
3) External team – the people who fill the gaps, make the connections,
   inspire you, provide you services, government agencies and the business community

Cheers
David

Google Glass provides both tactile and voice input. It provides a roughly 640×360 single eye display and a 5 mega-pixel camera. There is also a really cool voice api and timeline / card metaphor for building apps and consuming content.

Google Glass really does a lot with what is possible with today’s technology. However I feel like it is a misstep that the usage model is as a tool for your daily life rather than a specific productivity tool. This steps into a few social minefields. For example a user can look like they are paying attention to you, but instead be checking their news feed, or where a user captures images or video in inappropriate situations.

I feel like the key usage scenarios where fully augmented glasses can really shine involve a much more intensive usage model. Where you put on your glasses to do work or virtual collaboration. These glasses would replace big screens and your whole input stack. They would provide stereo images, eye tracking, gyros, front facing cameras for augmentation and front sensors for 3D hand and finger tracking.

The idea is to immerse yourself in a fully integrated 3d space where your visual system, voice input and hand gestures are all integrated. This setup will provide a massive productivity boost while working and communicating.

The ability to integrate finger tracking with synthetic 3d imagery opens up the possibilities of amazing user interfaces – combined with eye tracking and voice recognition you get a massive boost in the ability for the system to provide prompts and respond appropriately to voice commands.

There is no doubt that the ability to augment our senses in this kind of way is absolutely the most natural and compact way to interact – but just like societies have had to work out social norms for smartphones use, one of our challenges will be working out how wearable computing devices fit in.

Very exciting to see our new multi player instruments product get launched along with the amazing new Dell XPS 18 at SXSW in Austin Texas.

Dell announcement

Ubergizmo Review Link

This application really shows off our framework and the ability to manage not just multi touch but multi user scenarios. It’s also a huge amount of fun!

Here you can see the precision and accuracy which is possible using this sensor. Below I am trying to learn how to play “twinkle twinkle little star” on a 3D piano. Notice the ring ‘shadows’ which are added to help me locate my fingertips in relation to the piano.

The Leap Motion camera also lets me modulate the volume of notes using several enveloping techniques.

Check out this example where the drumkit is modulated based on the angular velocity of my finger as I imitate a drumstick.

Here is an updated view of our 3D modelling tech. This is able to paint diffuse texture, specular texture, bump maps and geometry simultaneously or individually based on the brush type.

We also have morph brushes which let the user paint between pre-designed whale models (like the robot whale in this example).

The UMAJIN app builder lets you create applications for Windows, iOS and android without writing any code. Use the powerful editor, pick your templates, lay out your rich content and deploy immediately to devices for quick feedback and testing.

The interface is simple and powerful, it helps you manage the many possible sizes and aspect ratio’s your application will need to run on. You can use the snapping and reflowing tools – or even individually specify the layout where required.

The action system allows you to create interactivity and the pro version can be used by developers to extend the existing actions with new ones. This makes working with developers extremely productive.

Input Styles

Keyboard : Qwerty keyboards and simpler TV style remotes with cursor keys both have a useful role for data entry. We will see this being replaced with voice more and more over time.

Voice : Voice commands are less explicit than a keyboard and preparing suitable command sets and potentially in multiple languages can still be a challenge. As natural language processing and voice recognition improves voice will become more flexible as an input type. Combining voice and a pointer input like gaze tracking or hand tracking make a powerful combination as now voice commands can be give context on which to act.

This branches out into voice recognition, natural language processing and agents. The idea of an agent is to provide a user the ability to have a conversation in a specific subject domain and be able to complete tasks for the user. An agent in a car might support route finding or phone dialling.

Mouse : This is one of the most accurate pointing devices and has evolved to include secondary sensors like a right click and a wheel. Clicking and dragging form the basics of most interfaces and are considered the core interactions for pointing inputs. Another useful feature of a mouse is the ability to move over items without clicking, this ‘hover/inspection’ capability can be used to bring up information indicating what is under the cursor and previewing what the result of a click will be through hints like the mouse cursor icon.

Touch, Multi Touch & Near Touch : Touch and multi touch form the basic click/tap and drag function for most interfaces. Near touch improves this with hover/inspect capability. Measuring the area of the fingertip on the touch surface or direct pressure information can be used for enveloping. One challenge with touch is that generally all touch points are stateless and you do not know which hand or user they come from. This means all touches must be treated as equal.

Pen : Pen is generally more accurate than touch for positioning due to the smaller tip, it will often provide hover/inspect capability and pressure for enveloping. Some pens also provide tilt information. Pen and touch will become a very important combination. This is because touch can be used to manipulate (pan, scale or rotate) and pen can act (cut or draw). Shape and handwriting recognition can add value to sketching or filling out forms.

3D input : Consumer systems generally have only one sensor so they generate 3D data from that single point of view. The sensor volume is a trapezoidal frustum starting some initial distance from the sensor and increasing in area until it reaches a maximum distance. The accuracy, noise level and type of information varies based on the technology used. There are three major configurations of 3D input.

1) Full body : Camera pointing at the full body from a distance, narrow field of view,
good for detecting simple skeleton, head and hand position

2) Close range torso : Camera pointing at the users torso, wide angle,
good for detecting head direction, finger positions and hand poses.

3) Close range hands : Camera pointing up at the users hands, wide angle
good for detecting accurate finger positions, angles, hand poses and a stylus

Option 1 struggles to provide full pointing capability because although the position of a hand/cursor can be determined with precision there is not enough information to reliably determine the fingers pose for detecting the equivalent of a click/tap. The workaround for this is to use x/y/z information only. For example the alternatives to a click/tap in order of usability are a) dwell on a button, b) a vertical or horizontal swipe over a button, or c) z push towards the camera on a button.

Dragging is not easily supported, so scrolling and panning is usually managed by moving to the edges of an area and dwelling.

Option 2 can detect hand poses so it is simple and natural to use the spreading/closing of the hand. This means it can provide full pointer support with an open hand being a hover. The closing hand is a pointer down, a moving closed hand is a drag and an opening hand is a pointer up. Scrolling and panning can also benefit from the concept of moving to the edges of an area and dwelling.

Option 3 can detect individual fingers in some detail. This means while you could use the hand just like option 2 you can go further and use each finger.

This works by exploiting the angle of an individual finger to determine it’s up or down state. For example a hand and fingers held flat/parallel over the sensor will have the fingers all in hover/inspect mode (shown as hollow ellipses). When a finger is pressed down at an angle relative to the palm it will be considered active (the ellipse becomes more opaque to indicate it is active/down).

While there is no surface to press against to generate a variable input for enveloping there is tilt and motion information from the finger. So the velocity or angular velocity can be used to determine how ‘aggressive’ this down and up motion was and provide modulation.

Because the technique used to identify the position of the fingertip measures its dimensions and angle, a stylus can be determined as different from a finger. This allows for pen style interactions.

Basic gesture terms

Click/Tap : The idea is a tap is only triggered when a pointer down and a pointer up occur with movement below our default threshold (e.g. 5 mm in any direction).

Release : Similar to a tap but allows the pointer to move anywhere as long as the pointer up occurs on the original object.

Drag : This is where the pointer is placed pressed down and then moved. Drag events should be triggered immediately and flicks calculated as a secondary event. When the user releases the pointer the drag stops. In some cases physics and hysteresis may come into play to continue the motion under momentum or magnetic snapping may be applied.

Hover : This is where the pointer is moved but not pressed. Hover events provide useful inspection capability to give users feedback on the purpose of the item under the pointer. Additional information may only be displayed after dwelling for a set period of time.

Pointerlock : This is a behaviour rather than a gesture. It describes how some objects when manipulated will appear physically locked to the pointer until the pointer is released and it is free to spin/move/slow based on the velocity of the users gesture. Reaching out to press the object again will stop its motion as it locks back on the pointer.

Flick : This is where the user moves in one direction. The thresholds are if the movement is far enough (say 30 pixels) and fast enough (say 30 pixels in 200ms). This event is then raised along with the direction of the flick. This is usually followed by an animation of the object succeeding or failing to be flicked depending on the application.

Pinch : This can generate Resize, Rotate and Pan actions. These can be performed with any two fingers and combined or for some interfaces be measured separately. These should animate the object being manipulated directly. Both contacts being on the same object and within a threshold of proximity will ensure the gesture was intentional.

Dwell / Right click : This is the process of holding the pointer down in one spot below the minimum movement threshold (e.g. 5 pixels in any direction). After a set time (say 500ms) a visual affordance must be displayed to indicate a dwell is being measured. This affordance should show the progress (say for 1000ms) until the dwell time is reached without triggering the movement threshold. Now the action or menu can be shown. The right click is used with a mouse and some pen systems to trigger the secondary menu.

Double Tap : This can be performed with mouse, but is much harder with pen, touch or 3D input. To avoid latency it is recommended to raise the initial tap event and then if a second tap is detected within the threshold (say 300ms) to raise a double tap. This is best used as a shortcut where the first tap selects and the second executes the default action.

Multi User

One of the important concept is that of multiple users not just multi touch. As UMAJIN runs as a simulation with its event driven core multiple users can be interacting.

For example in a photo app users could pick up, resize and rotate photos with perhaps the ability to drop photos into a favourites folder. The challenge with multi user is to minimize or remove modal interfaces. In this scenario, if the first user selects the upload button to for the favourites folder then this should trigger an asynchronous upload progress bar. This is then animated without stopping the second user resizing the photo they are looking at.
 

Interfaces

“Avoid unnecessary reporting. Don’t use dialogs to report normalcy. Avoid blank slates. Ask for forgiveness, not permission. Differentiate between command and configuration. Provide choices; don’t ask questions. Hide the ejector seat levers. Optimize for responsiveness; accommodate latency. Never bend your interface to fit a metaphor. Take things away until the design breaks, then put that last thing back in. Visually show what; textually tell which.“
Alan Cooper, “about face”

These guidelines outline how to make the BEST possible casual software. This is a broad category of applications, utilities, games and experiences which do not require significant investment of effort from the end user to enjoy. This is software which will delight and engage people. Our perspective on casual software can be broken down into three key concepts.

1) Function – Purpose delivered with a minimum of features
2) Gestural – Minimal interfaces, using physics and direct manipulation
3) Cinematic – Playful Interaction Design, Art/Sound design and effects

Function is the backbone and there is a balance which must be reached in designing an experience which delivers the key purpose without weighing the user down in options, learning, terms, menus, dialogs or steps.

Gestural is all about providing a great experience the first time, and each time thereafter. Gestural should be a contrast to tiny atomic GUI’s with many, many small buttons and menus. Gestural is all about broad simple motions whatever the input devices. This means minimal interface, obvious purpose, broad motions and clear feedback.

Cinematic is great presentation, providing animation on completion of actions and providing a clear state and flow for the experience. We want out users to know what they are doing and where they are.

Software is achieving these three points when a user can start an application for the first time, understand its purpose and intuitively use it. The software should feel natural, be able to be explored without fear, clearly illustrate actions that the user initiates and look and sound great.

Gestural introduction

The continuum of interaction for NUI interaction is very large. NUI (Natural user interfaces) describes a range of philosophies which centre on making the user interface invisible and obvious. NUI interfaces can be used with many input devices including keyboard, mouse, touch, multi touch, near touch, voice, pen, gaze and 3D input.

The use of machine learning, machine vision, speech processing, big data queries and natural language processing are all part of making these inputs seem to be more natural as we can increasingly take more types of ‘human’ input.

To achieve the goal of gestural software;

Broad motions are important. This means users should be able to use software by sweeping or touching with mouse, hand, pen or finger. They should not be required to find and locate very small parts of the screen. Double clicks and right clicks are for shortcuts or accelerators only, not for core functions.

Minimal interfaces have a small number of elements and they present a clear purpose. Generally this means when designing a layout to be reductive, reducing the number of individual elements.

To allow for applications which still present very rich content use should be made of compound elements. While these exist as only one element on a layout, they can contain many homogeneous content items which behave the same. An example would be a grid or glass shelf containing many album covers.

In terms of interaction with a compound object these usually support more content than is visible at first glance, often supporting navigating through scrolling, panning or zooming. This means the individual items within the compound element should be limited to being able to be selected – this means they will not conflict with any navigation.

Tactile interfaces provide a very important mental connection to the physical world for users. There are three ways we work to improve the ability of our applications being more tactile.

1) Direct Manipulation: two fingers to resize and rotate, multi finger crumple gestures, dragging, pointing and flicking – essentially providing the user with a direct rather than abstract interface.

2) Physics of momentum, friction and elasticity: This is where elements in an application can slide, bounce, and move in the way users are used to seeing in the natural world.

3) Enveloping: This is modulating an action based on a variable input such as pressure/angle of a pen input or the proximity/area of a finger. This can result in natural effects like line thickness while drawing or the volume of a virtual piano key. This adds significant richness to user interactions.

Ergonomics are very important to applications being approachable. Just like in the process of designing physical products, the layout, size and interaction of elements/controls is very important.

Each type of input sensor that a user may choose to use has different trade-offs and limitations with the human body and the environment the interaction occurs in. For example requiring people to hold their arms up on a large wall mounted touch screen can cause fatigue, requiring a user of a phone to pick out small areas is frustrating and putting the save button right next to new game might drive people mad.

Testing with external users can be very valuable. Testing internally is also important. It’s even better when your own QA, developers, designers and artists can be users of your software.

Another big challenge in ergonomics is the increasing layout challenges of different resolutions and aspect ratios on phones, tablets, notebooks, desktops and consoles. This is a significant task to manage laying out content so it is easily consumed and to make the interface handle these changes gracefully. A snapping and reflow system along with content scaling which can respond to physical display dimensions is a critical tool. It should allow suitable designer control to make specific adjustments as required for the wide variety of devices now available.

LINK Gestural guidelines

Cinematic Introduction

Performance 60fps frame rate and 0fps idle frame rate is our goal. Make things fast and smooth when we are using them and drop the app down when we are not.

Touch, file, audio, network and memory latency are always there. We need to work to understand and minimise these within the engine as much as possible – then to come up with guidelines for app developers where performance gains can be made.

Database – use transactions, use in memory mode for inner loops
Images – stream images, provide levels of detail
http – use async mode and show users a progress indicator

State is a really important thing to communicate to users. It might be the pen selected in a painting program with a clear highlight, and perhaps pulling it out of place. It may be a choice is being presented so we need to refocus the user on the foreground and drop out the background. By making state, and the resulting choices or action options really clear users will find our applications more intuitive and less frustrating.

Actions are often how we get things done in applications. We need to think about how we can show users these actions have been performed with direct feedback. It might be animating photos into the slideshow when you flick them, it might be having a photo disappear into particles when you delete it, it might be an animation of photos going into a cloud when you upload, but making actions clear as to their behaviour and their success will make users much more comfortable with the application. The aim is to make this action feedback fun rather than annoying and certainly not to slow the user down.

Consistent is important. We need to strive to share metaphors and components across our families of applications. It’s important we provide a consistent mental model for users. When they reach out and touch a screen we want it to work the way they expect.

Rich Content is appealing. We need to try to bring our content, our partner’s content, our USERS content to the front and centre. Full screen views of family videos, large thumbnails when browsing through files, great movie trailer posters filling the screen.

Content is the centre of the experience – the application is what enables and presents it.

Worlds are an important part of the human brain. We remember where our street is, where our place is and where our room is. This does NOT imply we make big villages to navigate – the abstraction of the getting to the web in a browser rather than having to walk down a virtual hallway to the library is clearly a benefit. Actually 2D planes are still our best interaction model with a 2D screen. The premise is that 2D workspaces can be layed out in a 3D space and transitions can be used to help user build a mental model of where they are in the overall space.

e.g. as you select a sub folder the previous folder animates past the camera and you appear to drop one level deeper into the hierarchy. Then pressing up a level will animate back. This provides users with a strong mental model to understand they are going in and out of a file system.

The task of processing raw 3D data to establish the fingertips and eventually the pose of the hand is quite a challenging problem.

We have built a few systems for providing this level of detail where the SDK’s provided don’t do exactly what we need.

This video shows using the position of the fingertips and then the angle of the fingertip to navigate a menu system. We are also using a swipe gesture to hide the menu.

UMAJIN “You Imagine”

The purpose of the UMAJIN script is to be a domain specific language which makes developing rich interactive applications easier and more natural.

The goals of the framework are

• Fast development
• File size conscious
• Native internet
• Application performance
• Rich content (such as 3D)
• Rich inputs (such as touch, voice, pen, depth camera)

What

This diagram describes the basic architecture of the three parts of a project. These are the creative assets, the script and the engine.

Creative Assets

The UMAJIN engine is able to load, play and render a wide range of content. Most common formats for images, audio and video are supported. There is support for fonts, 3D content, animation and Unicode text. The idea is to support the creative efforts of designers to create the most rich and interactive applications possible.

UMAJIN script

The UMAJIN script is a language for describing objects and logic. The language is designed with a low pattern syntax and object oriented features for lightweight reuse. This means that there are only a few keywords required and the syntax follows standard patterns.

For example the declare pattern is;

[keyword] [type] [name]

This pattern can be used to declare visible objects, abstract classes, properties or local variables;

• instance image my_image
• define image my_imageclass
• property int my_propint = 5
• int my_localint = 5

The language is kept simple to master with only 10 core keywords;

• Instance
• Define
• If
• Loop
• Method
• Raise
• Property
• End (ends all blocks)
• In (for time based method calls, or assignments)
• Tween (used for time based assignments which will be animated)

The object orientated syntax allows for very simple encapsulation of composite objects with custom methods and events. The key aim is lightweight reuse.

The base objects and composite objects built from them – support declarative programming. This is where other users (potentially using the GUI) can create functional visual programs by creating instances of your objects, setting properties and binding logic to their events.

The language is strongly typed, but allows coercion.

UMAJIN engine

This is runtime system which the language binds to. The engine supports the creation and management of visual objects and simulating, temporal assignments, animation, media and physics.

The rich object model is very important to productive development and includes items such as;

• Image
• Video
• Database
• http
• http_server
• Sound
• 3D
• Shapes
• Text
• Web
• I/O
• Timer
• Webcam

How

The Language

Declarative programming – Objects form the skeleton of declarative programs. The programmer creates instances of the different classes of objects to form the visual basis of a document or application. The programmer can then set the properties of these objects to adjust their appearance or behaviour.

Objects can be ‘nested’ into hierarchies. For example a button can be placed inside a panel. As a result visual properties such as position and scale are inherited. When you move the panel, the child button will move with it.
Objects also have methods. Methods contain code which performs operations. Methods can also bind to events which are raised by users. For example clicking the mouse.

An important concept in this style of programming is called sub-classing. This is where a new ‘sub’ class is created from a parent class. This is also known as the new class ‘inheriting’ from its parent. This is used to create new and more complex classes from the original object model.

New programmer created classes can also have custom properties, methods and events.

Glossary

Object : A discreet bundle of components – often relating to a visual element (a button) or an abstract real world analogy (a customer). The keyword INSTANCE is used to create an instance of a class, otherwise known as an object.

Class : A class is the definition of a new type of object. The keyword DEFINE is used to create a sub-class which is based on an existing class.

Property : An attribute of an object represented by a value which has one of the UMAJIN types.

Method : An action that can be performed on this object. This method will take parameters (like a function) to help describe what it should do. A method contains code and can return values.

Event : A class can raise an event which triggers a method in an instance of this class (an object).

Code : An ordered list of instructions which are performed and can modify objects and properties and call methods.

Declaring

UMAJIN script is a declarative language. This means you can literally layout the hierarchy of objects and their properties you want the engine to create for you.

Instance
Make an object of class image. This makes an actual instance of an apple.

instance image apple
   .filename=’apple.png’
end

Nested instance
Make an object of class image, make another instance inside it.

instance image apple
    .filename='apple.png'
    instance image worm
          .filename='worm.png'
    end
end

Define
Define a new sub-class of image. This does not create an actual object.

define image button
    .filename = 'button.png'
end

Define and Raise
Bind the new class to handle a mouse down event and in turn raise a custom on release event when an instance is created from this new class.

define image button
   .filename = 'button.png'
   event on_release()
   method on_mouse_down(int x, int y, int mod)
     raise .on_release()
   end
end

instance button mybutton
   method on_release()
   end
end

Logic

Logic or code is a series of instructions which each perform basic tasks which when combined together can create complex behaviours.

Types
Several basic types are available, these are simpler than full objects and include;

• Int (int8, uint8, int16, uint16,int32, uint32, int64, uint64)
• Real (Real32, Real64)
• String
• Bool
• Vec2, vec3, Mat4
• Color
• Datetime
• Var

Expressions

Programming starts with expressions and assignments. This is where a variable or property is assigned a value. The value can be an expression as simple as a single number or contain operators like addition, or calls to methods which return values.

Real velocity = 2.3			(2.3 is the expression, the = is the assignment)
Real speed = velocity * 1.87		(velocity * 1.87 is the expression)
Int length = sqrt((x1-x2)^2+(y1-y2)^2) 	(sqrt is a method inside the expression)

Conditions

Conditions are decisions where code can be branched based on the success or failure of a comparison. If, else and elseif are the structures provided. The condition needs to evaluate to non zero to be true and succeed. Comparison operators can be used to test values. These include == (equal), != (not equal), > (greater), >= (greater or equal), < (less than), <= (less than or equal).

Example 1 : Simple If block

if velocity>2.1
  //place code here
end

Example 2 : If/Elseif/Else block

if velocity>2.1
  //place code here
elseif velocity==1.5
  //place code here
else
  //place code here
end

Loops

Loops allow repeating of a block of logic/code multiple times. This code will run until a condition is met. Loops can be pre AND post tested.

Example 1 : Simple pre tested loop (Loop if)

int a=0
loop a<100
   //do something many times...
   a++
end

Example 2 : Simple post tested loop (End if)

int a=0
loop 
   //do something many times...
   a++
end a>100

Arrays

Often used with loops these are a great way of processing lists of data. Umajin supports numeric and string indexes. To access the specific item in a condition, expression or assignment the index is simply specified between the square brackets.
Multi dimensional arrays can also be declared and used by having multiple sets of square brackets.

Example 1 : Declare a basic array

int my_array[]

Example 2 : Accessing an array

string my_array[]
my_array[1] = “cow”
my_array[2] = “sheep”

Example 3 : Use the array in a loop

int my_array[]
int a=0
loop a<100
   my_array[a] = rnd(100)
   a++
end

Example 4 : Declare a 2 dimensional array

string my_big_array[][]
my_array[1] [“name”]= “cow”
my_array[1] [“age”]= “13”
my_array[2] [“name”]= “sheep”
my_array[2] [“age”]= “3”

Glossary

Expression : a combination of numbers, variables, methods and operators which evaluate to a value.

Assignment : setting a property or variable to a new value (potentially from an expression)

Condition : based on a comparison succeeding or failing branch the flow of the program

Loop : repeat instructions until a condition is met

Type : Applies to the type of an object (e.g. image), or a basic variable (e.g. string, int, real)

Array : A variable which hold more than one value declared by adding square brackets []