The Morphic 3 Project

Morphic 3 is a research and development project aimed at building the next standard in 2D user interfaces. It will be fully zoomeable, and will do mathematically proved alias-free rendering. In order to achieve these objectives, several techniques and design features are described. Some of them are new and others are not, but have never been consistently applied to a 2D GUI.

New! You can see the Cuis and Morphic 3 talk at the Smalltalks conference in Buenos Aires here.

History of the ideas behind Morphic 3, and project landmarks:

  • Independence of Display resolution (1991)

  • Modeling of coordinate systems as objects (4/2007)

    • Demo 1, 6/2007. Reified CoordinateSystems and Locations

  • Alias free rendering based on Signal Processing theory (8/2007)

  • Allow and encourage the use of textures and photos (10/2007)

    • Demo 2, 12/2007. Smalltalks-2007. High quality image resampling. Non-linear stuff

  • Zoomeable User Interface (7/2008)

    • Demo 3, 9/2008. New rendering algorithm. Uniform use of the anti-aliasing technique

    • Demo 4, 4/2009. Modeling of shapes as a trajectory of a pen

I have been interested in the construction of high quality user interfaces for several years. These design ideas reflect my current understanding on the desirable properties for a 2D user interface framework. Many of them are not new, but have never been applied to a 2D GUI in a consistent way.

Work on Morphic 3 started in mid-2007. Since then, several prototypes were built, as experiments and demonstration of the concepts. They were later discarded, after teaching me things I used on the next iteration. These demos are some landmarks in the path already walked.

Let me tell you about each of these ideas and landmarks in more detail.


Key idea 1: Independence of display resolution (1991)

This means that the code for application should not care about the pixel resolution or pixel size of the display device. In the original context, an application should use all the available screen space in the best possible way without the programmer needing to code for this, no matter if the application is run with Hercules, CGA, EGA or other graphics hardware.

This is the contrary of the recent trend in "DPI awareness". "DPI awareness" besides requiring developers to know about pixel resolution, it also requires them to know about pixel size, making programming more complicated and error prone.

Morphic 3 is simpler. In Morphic 3 there is no concept of pixel. The pixel is considered a low level primitive, like a cpu register in a high level language. The GUI is thought at a higher level, in the coordinate systems that makes most sense to the developer. This application specific geometric space is projected on the actual display at runtime.


Key idea 2: Modeling of coordinate systems as objects (4/2007)

This implies the separation of the handling of coordinate systems from the morphs themselves. A morph should only need to say "I want to use a log scale", instead of needing to convert every point it draws to World coordinates himself. Every Morph defines a space and coordinate system. Its #drawOn: method and the location of its  submorphs are expressed in its own coordinate system.

Although this might sound fancy for a programmer, we do this kind of things all the time in the real life. We do it when we do any kind of drawing, or when looking at a map or street guide. We also do it when we try to fill several glasses with the same amount of water from a bottle. We even do it when thinking on how much grass we have left to cut in our backyard, or when playing with a doll house inside our real house with our daughter. For all these things we use specific coordinate systems we immediately discard, without even thinking about them.

Each coordinate system (and not the transformations it performs) is an object. The coordinate systems are 2D but are not restricted to Cartesian or linear. Useful nonlinear coordinate systems include polar, logarithmic, hyperbolic and cartographic projections. Unfortunately, handling nonlinear coordinate systems precludes the use of many optimizations that assume that only affine transformations are made. Therefore it is an experimental feature, whose value/cost tradeoff still needs research.

Separation of the defining properties of coordinate systems from volatile information. The area a morph is using in its owner is volatile as it can change anytime as the user or the application moves, rotates or zooms objects. This volatile information is stored in Location objects that specify how and were a morph and its coordinate system are located. A CoordinateSystem object together with a Location object specifies a geometric transformation between the inner space they define and the outer space they live in.


Demo 1, 6/2007. Reified CoordinateSystems and Locations

Implementation of CoordinateSystem and Location objects. Implementation of a parallel Morph hierarchy that uses them.

These new morphs are an integral part of the existing Morphic 2 framework. Rendering is done via the existing Balloon 2D vector graphics engine in Squeak. Done to show the value of the ideas of independence of display resolution and explicit models for coordinate systems.


Key idea 3: Alias free rendering based on Signal Processing theory (8/2007)

The methods to do anti-aliased 2D images currently in use are Super-Sampling and Pixel-Coverage. The Balloon engine in Squeak I used for Demo 1 does super sampling.

For supersampling to effectively avoid aliasing, the supersampled display should have high enough resolution to be above the Nyquist frequency of the image. Usually this is not the case, the super sampled display will also have aliasing (although less), and aliasing is reduced but not completely avoided.

Pixel coverage is no better, as it behaves as an infinite resolution supersampling, but using a very bad filter: A step filter in the spatial domain. The frequency response of such filter has no real cutoff frequency, and therefore it doesn't bound aliasing.

The quest for higher quality resulted in the idea of applying the Sampling theory (the basis of digital audio and digital photography) for all rendering. This allows for mathematically proved alias free rendering. It is done by modeling all the objects to be drawn as continuous functions that specify color at each (x, y) point. This functions are properly filtered and sampled at the actual position of the pixels.

Please see the following image in an LCD screen:

The leftmost image shows much aliasing. The second image uses an approximate pixel coverage. It is better, but it still shows visible aliasing, especially at high frequencies. Image 3 uses supersampling with a very high resolution and careful filtering. It looks quite nice, but it takes about an hour to render! One of the challenges of this project is to find way to model filtered continuous functions that are fast enough to render in real time.

Finally, to really follow Sampling theory, we also need to take into account the final steps: Digital to Analog conversion: The display screens and the human eye comprise the digital to analog conversion and the analog filtering stages. Image four takes this into account, and does subpixel rendering.

Besides, as LCDs don't do a real analog filtering of the images, some times it is good to use a digital filter with a cutoff frequency that is slightly lower than the Nyquist frequency. This will mean a bit softer images, more pleasing to the eye, but with a little loss of detail.


Key idea 4: Allow and encourage use of textures and photos (10/2007)

To everybody, resolution independence means vector graphics. Look for example at Cairo, or Anti Grain Geometry. This produces a toy-like or cartoon-like look. Instead, include implementations of high quality image filtering and resampling operations to fully support the use of textures, photos and photorrealistic art as integral part of GUIs.


Demo 2, 12/2007. Smalltalks-2007. High quality image resampling. Non-linear stuff

Implementation of high quality image zoom based on "Interpolation Revisited", and support for non-linear coordinate systems.  You can read more at Morphic 3 at Smalltalks 2007.

Based on the original image below (with parallels and meridians as straight lines), it can render the two at the top, using appropriate coordinate systems.



These images are created from the smaller one at the top, and they show the effect of other non-linear coordinate systems.


Experimenting with this system made me realize that a good design should separate the specification of the shapes (the drawing primitives) from the anti-aliasing strategy done by the rendering engine.


Key idea 5: Zoomeable User Interfaces

Morphic 3 will support zooming the whole user intarface by small fractions, to accomodate it to the user preference regardless of the display hardware. It can also support zooming over a very wide range of zoom factors, like Google Earth, Microsoft's Seadragon, and Jef Raskin's Archy. Note that these are specific purpose applications, even more, for two of them, their only purpose is to demonstrate the very concept of zooming user interfaces. Instead, Morphic 3 is a general purpose framework to build any kind of applications.


Demo 3, 9/2008. New rendering algorithm. Uniform use of the anti-aliasing technique

Experiments with a new rendering strategy that only paints each pixel once. It also eases the use of non-linear coordinate systems and extreme zooming. This new way of drawing the whole Morphic world also allows multiple views (EyeMorphs) at different zoom factors, looking at the same or different areas of the world. It also supports many Displays, perhaps on different machines. In addition, it makes the use of whole pixel or subpixel anti-aliasing a property of the framework. Handling the anti-aliasing at the framework level allows the user to customize the applications to the type and properties of the actual display hardware in use. It also simplifies the implementation of the drawing primitives, as they are just the continuous functions that the engine will filter and sample as needed.

This requires reimplementing all the display primitives, and is not compatible with the existing Morphic 2 drawing strategy. Therefore, it includes a first sketch of a Morphic 3 as a separate framework from the existing Morphic 2.

Implementation of basic drawing primitives based on Sampling theory. Implementation of high quality sub-pixel anti-aliasing for them. Implementation of MorphicEye and a OldMorphicEyeMorph to integrate Morphic 3 worlds in Morphic 2.

Take a look at the next picture. This image applies all I learnt about aliasing, and it looks better than anything the usual techniques can produce.



Demo 4, 4/2009. Modeling of shapes as a trajectory of a pen

Experiments for a model of shapes that can be modeled as the trajectory of a pen. This makes many kinds of shapes fit into the new approach to anti-aliasing using just one implementation. These will include lines, geometric figures, and curves. 


Project plans

Morphic 3 shows a lot of potential for enhancing the quality of GUIs. It also makes evident the need for a huge amount of work to reach its objectives, including several interesting open problems that need real research. But I need to build a system that is ready for application development in, let's say, six months.

In 9/2009 I focused on the problem of finding a way ahead for Morphic 3. One that can be realistically expected to succeed, but without giving up on the new and desired properties, as they would greately simplify the design of graphical applications while enhancing their features and quality.


Morphic 3 in Morphic 2: Build a first usable release

The approach is to build a new hierarchy of morphs that can be used in the existing Morphic 2 world, without changing existing morphs. In this way, additional new morphs and rendering primitives might be added as needed. This also means that some of the more ambitious goals are left for later work.The objectives for this first phase of the project are (for the new morphs):

    Independence of screen resolution

    New geometry model, restricted to Cartesian systems and affine transformation

    High quality anti aliasing, based on Signal Processing therory

    Use of textures and photos as key parts of GUIs

    Integrated in the existing Morphic world

The first phase is already started, and I expect to have a usable release around mid-2010. Later, I will be adding morphs and drawing primitives until they cover all the functionality in the system and all the older morphs can be discarded. Then it will be the time for the next phase:


Morphic 3 as a separate framework

The objectives for later steps in the project involves a complete rewrite of Morphic, to allow for:

    Non linear coordinate systems

    New rendering algorithm for the world

    Zoomeable User Interface

    Multiple EyeMorphs, Hands, Fingers and Displays

This rewrite of Morphic will use the morphs done for the first phase, so all the tools, rendering primitives, etc will already be done. Most of the work will be done at the framework mechanics, although non-linear systems will require some enhancements to the filtering of the functions that specify the shapes.

There will be a lot of activity in 2010, so stay tuned!




 
 
   2006-2010 Juan M. Vuletich