Smartphones, 4G, IP, and IMS – Part II: Why is this G like no previous G

Making a phone call is an activity that seems almost as natural as walking, or drinking, but it has a history. About 120 years ago the layout of the phone terminal, with transducers for speaking and listening, was a settled matter of design. In another 20 years the matter of dialing phone numbers was settled as automatic electromechanical telephone exchanges were introduced. A phone call is a blend of humans making accommodations to machines and machines being designed for humans, and much of the early outcome of that blending persists today.

If you look at the software in any mobile phone, the key thing that makes it a phone is the call-state management. This is what keeps track of the state of phone calls (and, in 2.5G and 3G, data connections, too). It enables the user interface of the phone to display the correct state visually and play the right call progress tones (the ones that originate in the handset) at the right time. It also hooks into power management to light the LCD backlight at the right times, the switch that detects if a flip phone is closed, the keyboard lock, and performs other operations that create the user experience of a telephone.

The design of all mobile call state management is based on having one bearer channel to a circuit switch, plus SMS and data. The first implementations of mobile call state management had minimalist requirements: Display the on-hook dialed phone number on a numeric display, play call progress tones the user would find familiar from the landline user experience, and enable control over a small set of switch-based functions like call-waiting, where the call on the single radio channel to the phone is switched to a second call. It is familiar enough to be accepted by customers and accommodating enough to the way the network works to be efficiently deployed.

There are some differences between CDMA, GSM, iDEN, etc. but they are close enough to share a lot of the implementation. 2G, 2.5G, and 3G differ in the richness of the circuit switched signaling (but not so much that users are aware of it), and in whether and how much data you have. GSM and UMTS/WCDMA data and HSPA differ from CDMA data in detail, but, again, there's not much user awareness of the differences. Color LCD displays on phones make text messaging nicer, but the impact on the mobile phone call user experience has been limited to presenting a contact list and call log.

4G is the first fundamental break from the roughly 25 year old model of cellular mobile telephony. In 4G, the radio is part of a network interface on an IP network, like a WiFi radio. There are no bearer channels dedicated to voice calls. You either do IP telephony using SIP, in the IMS case, or you simulate circuit switched telephony on top of IP in the VoLGA case. There is no practical limit to the number of voice calls and other kinds of communication sessions a 4G device can juggle. Packets traversing the carrier's network carrying voice to other endpoints are no different from packets flowing between the endpoint and a third party or private gateway/switch with a telephone network interface. Identity, communications services, media (how voice or video is encoded), etc., all become decoupled from the network.

So we are entering a product development phase where the winners will be, in part, determined by how well they can make compelling new communications features and capabilities based on this change without confusing the end-user. What it means to be a phone is now infinitely malleable.

Surely, at the bottom of this a phone call has the same limitations as a face to face conversation plus the further limitations of not being actually face to face. No, even this can be called into question. The ability to redefine the phone call experience is comprehensive and the potential is limitless. And the potential ranges from the obvious – what the user sees on a screen – down to the subtlest microsecond-level manipulation of the human voice.

But will this potential be realized? That is more in doubt than that the potential exists. So far, the modern general purpose pocket-sized telecomputer surfs the Web and plays games. It will take the realization that new communications products can be created, and sold, to motivate reconsideration of how phones enable communication.

How a Customer's Question Frames the Definition of 'What Does a User Interface Architect Do?'

My title is Chief Architect, and the short description of what I do is that I designed the mCUE user interface and telephony middleware systems.

That's not a very satisfactory description for several reasons:

1. The product named mCUE is difficult to pin down: “Unified communications” and “middleware” are industry terms calculated to form meaning in the eye of the beholder.

2. When people hear “user interface” they think “draws pretty pictures”

3. The term “architect” is also nebulous – what is there to architect in a user interface? It's just presentation.

So, when asked, what can I say about the job I do? Recently, a customer's question brought the description of what I do in focus, and also helped clarify the nature of the product I designed. The question was:

How do you ensure consistency of customer experience from platform to platform?

The question was asked because we showed the customer our product in a very portable implementation that depends only on a reasonable Java VM and a small subset of the Java AWT classes, but they needed our product in an environment that requires use of that platform's UI classes and application framework. Why, in this new implementation, should they expect it to work like what they evaluated and liked?

That question gets right to the heart of what it means to design a mobile user experience, and, indeed, right to the heart of the whole project that I have been working on.

My work here started with discussions I had with D2 Technologies' founder David Wong, who is a pioneer in voice over IP technologies (VoIP) and DSP software, on the proposition that VoIP user interfaces were low-quality compared to mobile user interfaces, and that nobody had really thought about VoIP user interfaces the same way as mobile user interfaces.

That is, nobody had thought of VoIP user interfaces as a really important part of the product. VoIP users were fewer, and had fewer options than mobile users. VoIP users were generally enterprise users, accustomed to ill treatment by user-hostile software. VoIP user interface was immature because the resources applied to it were small, and the stakes in the market were correspondingly small. What if we took it seriously and put more mind-share and resources into it than our competitors? Could we make a breakthrough in VoIP user interface and, perhaps, ride that up the value hierarchy to a place among mobile handset technology providers, especially as they found they needed IMS endpoint technologies?

That is how the project that became the mCUE product started at D2 Technologies. It was about VoIP, but, specifically, about what we hoped would be the logical next step for VoIP endpoints: Instead of desk sets that tried to look like PBX phones, VoIP would logically be moving to mobile handset form factors and use WiFi to connect to VoIP systems.

That VoIP user interface market, however, proved to be too small even for a specialist company like D2: WiFi handsets are, still, a tiny fraction of VoIP endpoint market share, and desk sets are still saddled with crude user interfaces. Nor could we make an immediate jump to providing IMS endpoint technology. Only tier-1 OEMs were active in defining their IMS endpoint architectures, and these were mostly prototype efforts.

Nevertheless, we changed our design goals to aim at the intentions of IMS: To make every mode of communication a first-class citizen in the mobile endpoint. And, beyond that, we realized that IMS is not the be-all/end-all of mobile communication. We also made every service a user might want to use a first-class citizen alongside the mobile network. That is, we made IP communication in all modes and media able to either replace, as in a purely VoIP system, or stand alongside mobile service, without relying on an IMS network implementation to aggregate every conceivable IP communication service.

And that gets to the answer to our customer's question: How do we ensure consistency of customer experience from platform to platform? The answer is: we have to, otherwise we could not implement our goals. We must make a new “world” of communications objects that replaces the call-state management in a conventional mobile handset with a model of communication that encompasses all modes, all services, and all networks. And, if user interaction with this much-expanded communications model is consistent in all implementations, the user experience is consistent, no matter the UI toolkit we use to implement the presentation of the user experience.

So, what does an architect of mobile user interfaces do? He makes a model the user is comfortable manipulating, and presents that model in a consistent way across multiple platforms. This sounds a bit general, but aspects of this task are especially critical in mobile user interfaces: The mobile user interface is inherently a multimedia user interface – it used to be almost entirely presented as audible call progress tones and the billions of users of telephone systems worldwide have a strong expectation that the conventions of a telephony user interface and a mobile user interface will be preserved in an interface that brings in other modes of communication.

The mobile interface itself cannot be pulled in new directions very easily, yet assumptions have to be questioned: What, for example, can be done to enhance the task of starting a conversation? On-hook dialing was invented to keep the user off the precious resources of the mobile network until the last possible moment. Now that always-on data connectivity is inherent in VoIP networks and widely available in mobile networks, the “don't touch resources until you absolutely need to start signaling the network to connect a call” assumption is obsolete.

A mobile user interface designer not only makes the presentation of the user experience, he has to be able to design the model manipulated by the interface, and he has to know about the network, and even the business of operating the network, to make a presentation, a model, and a consistent set of functions that work together. And he has to know it all well enough to know the difference between a rule you have to observe and a rule you have to break in order to innovate.

This seems daunting. It is, at least one of the most interdisciplinary pursuits in software design. It is full of both constraints and opportunities. If you are unaware of the constraints you will make mistakes that make you look amateur. If you are unaware of the opportunities and what it takes to exploit them, you only advance by luck. The reward is that good work in mobile user interface design expands your mind the way few tasks can in the increasingly specialized world of software design.

Organizing the Pantheon of Multimedia Communication

Recently I heard Garrison Keillor quote Joseph Campbell, in one of his essays on writing: “A computer is like an Old Testament god, with a lot of rules and no mercy.”

It doesn't help that the rules are rational and sanitary. No stoning. No beheading. Bacon is not a problem. Just cold opprobrium that is made even colder by the certainty that You Did Something Wrong.

Fixing this is the central task of user interface designers.

Clearly, then, it isn't good enough to enshrine the olde thunderbolt-slinger in a fancy temple < cough >smartphone< /cough >. That won't make him any easier to deal with. We have to reform the rules. More, we have to reform the language the rules are written in and the definitions of the nouns.

That is what initiatives like MMTel and RCS require, but, so far, are not getting from user interface designers. Your 2.5G phone is a simple minor deity with simple rules. Everyone understands the propitiations and few are therefore intimidated. But the pantheon is getting crowded with both the bureaucratic jinns of IMS, and the anarchic dryads of the Internet.

Messaging is now multimedia messaging. Email and IM bring two other rites of messaging. IM presence and social network updates make it a group event, along with your push-to-talk circle of initiates. And plain old voice conversations can travel over the righteous roads of the synchronous digital hierarchy or the twisted but beguiling paths of the Internet.

Putting all this in separate little shrines, with separate sets of rules and liturgies is bound to put us at risk of hellfire and damnation, and the use of Strong Language. Consequently, the people are divided into the zealous, who are undeterred by complexity and proudly wear their scourges in a belt pouch, and the agnostic, who are driven to indifference.

The successful interface to IMS-based capabilities will solve this problem. It will put all the media for communication under a single set of simple rules and inside of one interface – not a set of applications, but one unified application for communication, just as using the 2.5G voice network requires just one user interface.

The problem is one of levels of abstraction. How can you take protocols as divergent as SIP, XMPP, a few dozen proprietary IM systems, some with VoIP and/or voice chat, push-to-talk, video calls, email, social network update feeds, and circuit switched telephony and give the user one set of tools, one set of contact information, one history, one set of shortcuts, and one consistent set of verbs and nouns for manipulating these capabilities?

The radio interface layer, and user interface on top of it, that turns your key presses into phone calls in one direction, and state-changes reported from the radio baseband into indications of call progress in the other direction is not enough. But how do you keep it just as simple for the user?

The notion of a conversation has to evolve to encompass the new media of communication. Key objects: people, conversations, connections, addresses, services, identities, status, and presence have to be made simple enough for humans to manipulate through the same 12 keys and handful of other buttons, and their visual manifestations have to fit on screens small enough to keep discreetly in one's pocket.

And this is the hard part: We have to create a new world of communications objects that spans media, networks, and protocols, and we have to make the human customer instantly fluent in the language of these objects. This layer, consisting of many new protocols, access to many new services, and quite a few fundamentally new capabilities has to be inserted between the keypad and the radio without the user noticing that the amount of software in the system has gone from bottle-rocket to space shuttle in one step.

The zealous don't care: They are happy to stuff a computer into a belt-pouch. Multimedia communication is easy if the user is comfortable with multiple applications. But, to give multimedia communication mass appeal, the masses must be appealed to: The result, for them, must be that their phones do more, but don't ask more from them.

Yo mama is a PDA

What is a smartphone?

Frequently, the question comes up “What is a smartphone?” Usually, this question is mixed up with other questions like “What makes a good phone UI?” and “Why don't all phones use a smartphone OS like Windows Mobile?”

The confusion comes from the fact there are “good” or “nice” UIs that can be classified as smartphones, like iPhone, and clumsier UIs that are clearly in the smartphone category, but that are hung up on a design legacy that prevents them being competitive with products like iPhone. Are they all smartphones? Do we need more categories? What about the Linux-based 3G phones in Japan? What about Android?

The PDA design legacy

The idea that handheld devices need an operating system emerged long before mobile handsets became the dominant handheld device. It emerged before one could expect all devices to include a wireless network. It emerged before Web browsing and Web applications became central to handheld device use cases. It emerged before we had a choice between a circuit switched voice network and packet switched data networks. It emerged before the disaggregation of communications services and network access.

Handheld device operating systems originated with PDAs. There are several problems that arise from this design legacy: PDAs started out as non-network-connected “sideloading” sync-oriented devices for carrying your contact list and schedule around with you. Was that even a good idea to begin with? Evidence is, no: The worldwide market for PDAs is less than one tenth of one percent of the mobile handset market. But, like some string of selfish DNA with a will to survive, PDA operating systems began to infect mobile phones.

An awkward relationship

This was never a satisfactory combination: PDA phones were heavy, expensive and sucked batteries. Some PDA users were happy with the ability to keep a large contact list and a synchronized schedule with them, but folding up Microsoft Outlook and putting it in your pocket is not a goal for most people.

The smartphone has had a long, slow growth trend in the market. Only three smartphone OS suppliers remain, one of which is on life-support, one of which can afford products that don't turn a profit for many years, and one which has as their almost-sole customer the largest handset maker in the business

The new need for a better phone OS

We are now long-separated from PDA use cases as key drivers of handheld device UI design. Multimedia, the Web, and a diverse combination of IP and mobile communication on multiple networks, using multiple services now drives the requirements for handset user interfaces. This is why old school smartphone OSs are facing new competition: It was easier for Google to make Android than it would have been to turn Palm OS, or Symbian and UIQ, into a platform that meets Google's requirements for a mobile device OS.

The new requirements:

• An OS that is non-exclusive, cheap to license, and adaptable on my schedule, not the OS vendor's.
• An OS that brings the Web front and center in the user experience. The “mobile Web” is a crock. Everyone wants the real Web.
• An OS that is undemanding of the user: Direct manipulation, touch connected to action, no multi-step operations, no “dialog boxes” - communication is not like filling out forms. In other words, it can't be harder to use than a non-smartphone, and it can't be less intuitive than a media player's UI.
• An OS that accommodates new forms of communication while retaining the simplicity of the mobile phone. Access to all of our modes of communication, all of our networks, and all of our services are converging into one device. The next phone UI should not make this a burden on the user, but a benefit to the user.
• An OS that makes presence central to the user experience. The “start page” of your phone should tell you about your contacts' availability.
• An OS that makes non-verbal communication a first-class citizen in the user experience. All forms of messaging should be treated uniformly, and a single interface should organize all messages.

None of these top requirements existed when PDAs were designed. Your to-do-list is not the most important thing for your mobile device to display. Your calendar is important, but it isn't central to communications tasks.

It is also worth noting that no mobile device meets all these requirements, especially not the communications-oriented requirements. The PDA heritage is being discarded, but what will take it's place? This is, as yet, unclear.

The new wave of smart device operating systems

Symbian (plus UIQ or S60), Palm, and Windows Mobile are the old-timers of mobile device operating systems, and the reason they are under attack from a new wave is that they cannot shake off their PDA heritage. Nokia is making the best of it by putting Symbian on some very capable hardware. But if you want to know why Nokia's amazing phones are not challenging the iPod, Symbian and S60 are the prime suspects. You simply can't build a user experience that is competitive with iPod using a platform born from a PDA (Yo mama!).

So what do we call the new arrivals? Android, the newest arrival, puts the question into focus. Android does away with most of the clutter of the PDA-style interface in favor of a friendly application “dock.” Sure, you could build a busy, multi-tab, form-filling-centric application in Android, but that's not what Google has done. Android's map application is full-screen, clutter-free, and all about direct manipulation. So is Android a “smartphone?”

I, for one, welcome our new Android overlords

I would venture to say that Android's creators hope it isn't thought of by end-users as a smartphone OS, even though it is targeted to the same big-screen hardware platforms as Windows Mobile. Android is for everyone who wants to surf and search on the go, not type-A email addicts who need to check their to-do list every time they glance at their phone. If you have the goal of building a better communications tool for everyone, don't look in a businessman's hip-pouch. You won't find it there any more than you would find inspiration for any consumer mass-market product there.