Because Google Asked For It: Project Glass Suggestions

No matter how many times I watch this video. I still have a hard time believing that Project Glass will someday be a real, tangible product that I can strap to my face. A heads-up display is pretty much the holy grail of computing interfaces. If Google can deliver, it'll be mind blowing. There's just so much to talk about. How often does an entirely new computing form factor come along?

In fact, you could probably say that a HUD is the last UI form factor. Computer augmented vision is as integrated with a computer as you can ever get. Sure, eventually the tech would move from glasses, to contacts, to a brain implant, but the UI will essentially be the same.

And boy, is this a wild form factor. The keyboard-and-mouse to touchscreen transition is pretty straightforward: Touch input maps to mouse commands fairly easily, and you even get a virtual keyboard. A HUD is an entirely different matter. There is no keyboard and mouse. You don't get a touchscreen either. You're pretty much going to have to throw out the last 40 years of computer interface concepts. Your primary forms of input are now your voice, your vision (via a camera), and some accelerometers in the glasses. Scary.

A total reinvention of the computer is a big deal, and a huge challenge. I'm sure it's with this in mind that Google has put out an open call for Project Glass ideas. Every time the video is posted to G+, it's always accompanied by the same closing question: "What would you like to see from Project Glass?"

What would I like to see from Project Glass? Oh, about a million things. To the mockup machine!

The Killer App: Facial Recognition

Facial recognition is the killer app for AR glasses. Anytime the camera detects someone in your contacts list, it could display the appropriate, customizable information. Just think about the possibilities there for a moment.

How much could something like this change your life? Are you bad with names (like me)? Well, you aren't anymore. Do you always forget to ask that guy about that thing, only to remember 10 minutes after you've said "goodbye"? Just set a reminder for the next time you (literally) see that person.

Facial recognition could connect all the info normally hidden away in a smartphone to the real world. You could surface some relevant info, like your last text/IM with that person (for the forgetful), or their latest status update. If you're at a loss for something to talk about, you can casually browse their social network feed.

Professionals would be more interested in someone's name, employer and title. Just something simple like that would instantly be +1000 to networking skills. Again, it would need to be customizable, maybe even with something like G+ circles. For good friends, you wouldn't want their name and employer poping up every time you look at them, that would be for the "professional acquaintances" circle.

This picture is for the fun, consumer scenario, but just imagine how facial recognition + a database query could revolutionize customer service. A repeat customer walks into an establishment (retail store, doctors office, you name it) and the employee instantly knows that persons name and their history with that establishment, even if that employee has never seen that person before.

This is the perfect blend of human and computer. Forgetting things is purely a human problem, computers always remember. Allowing me to tag people (and places) with information fills a big gap in human ability, and it perfectly emulates how our brains work. Normally, looking (or talking) to Paul would make me remember information about Paul, so displaying "Paul" information when I'm talking to him would be perfectly natural.

Google Goggles Tech: Text Recognition And Image Search

I've suspected for a while that Google Goggles was all about teaching a computer to see, and now it's time to use it. Not having a keyboard is going to be a serious problem for data entry, and voice isn't ideal for every scenario.

Imagine a professional meeting someone for the first time, and he wants to program the facial recognition feature for this new person. The new guy hands over a business card, but speaking all the information into your glasses would be time consuming and fraught with errors (Voice typing is never going to nail "Shailesh Nalawadi"). Goggles has the technology, right now, to scan a business card and input all that delicious data into your Android contacts. So, in an ideal world, the glasses could auto-detect the new face while you are talking, and after the interaction is over, a casual snap of the business card and this new person is now in your "memory" forever. The next get-together could be 6 months down the line, but you would still "remember" Shailesh.

Goggles also aims to answer the question: "Hey, what is that?" just by aiming a camera at something. When said camera is attached to your head, that ability becomes a lot more convenient and useful. Goggles was made for a device like this. Sell a bunch of Project Glass devices and suddenly Goggles becomes Google's most important product, similarly to how smartphones have made mobile incredibly important. Up until now Goggles has been a side project, but now is the time to throw more manpower at it and make the whole world search-able.

Augmented Reality And Dealing With Information Overload

The biggest missing feature in the video (other than facial recognition) was the complete lack of augmented reality. When looking through computerized glasses, the world should be a seamless blend of real and virtual, but the video showed a regular world with a flat, 2D computer space on top of it. Anytime the glasses displayed information it did so front and center, in a big, white box. A heads-up display can be so much more than this.

Maps is an excellent example of "doing it wrong." An overview map is fine, initially, but if I'm trying to navigate somewhere, why not paint the navigation line on the sidewalk?
Navigation the way it's supposed to be done. I also added time and distance to the popup, because, why not?

Real and virtual should live side-by-side, in 3D. It's not just incredibly cool; it's a much less disruptive way to display information. Take my facial recognition mockup, for example. It would be annoying to have a box like that constantly in your face, but imagine if it "floated" next to Paul's head. So if he were farther away, it would look like this:

Paul is further away from you, thus the information about Paul is less important. Since the info is anchored to his head, it automatically becomes smaller and maybe even more transparent. If you turn and look away from Paul, the white box would leave your view completely. This will automatically and naturally solve a lot of the information overload and popup-swatting depicted in the Project Glass parody videos, and it allows the UI to "offer" information in a less intrusive way than a popup. Not everything needs to be "pinned" to my vision.

Microsoft Office does something similar to this with the Mini Toolbar. It pops up when you right click highlighted text, and fades in and out in relation to its distance from the mouse cursor. Office basically knows what you're looking at via the mouse position, and tones down the toolbar if it doesn't look like you want to interact with it. Project Glass should work the same way. It should use all the information available to it to try to figure out if I deem something important or not, and adjust the UI accordingly.

For instance, in the Navigation scenario, if I stop and grab a hot dog from a vendor, or stop to read a poster on a building, the camera should be able to tell that I've stopped looking towards my destination, and the navigation instructions should fade out until I look down the road again. How well Project Glass is received will depend greatly on its ability to know when I want to see information and when I don't. Anchoring information to things in the real world seems like an easy and natural way to deal with UI focus.

Blending the real world with virtual objects also opens up a great interface for "offering" information. I don't necessarily want my shopping list blasting into my eyeballs every time I walk by a supermarket, but I don't want to have to dig through a note app either. Imagine my virtual shopping list "hanging" on the door, the same way a promotional poster would. If I want my shopping list, I can grab it on my way in and pin it to my vision. If I want to ignore it, just do nothing; ignore it the same way you would any other poster. It's a great way for the glasses to ask "Hey, do you want this possibly relevant information?" without the user having to explicitly dismiss it. Again, this is a perfect combination of human and computer, because "Shopping list enumeration" is usually the first thing on everyone's mind when they walk into a supermarket.

Most of this should only require Kinect-level intelligence, but there's always the possibility that none of this was in the video because it's just too hard. Project Glass v1 might not have the processing power (or battery life) to decipher world geometry at 60fps. That's not really conductive to blogging though, or any fun. If version 1 really is that limited, file this section under "version 2."

An Easy Way To Share Information In Real Life

Currently, these things are going to kill casual, in-person, information sharing. I'm not carrying a screen around anymore, so "Hey, look at this" means I have to pull the glasses off my head, and hand them to my friend. I not going to do that. "Settings -> Bluetooth -> Search for devices -> enter pin" isn't going to fly either; my cat picture isn't that important. And no, I don't want to do it through Google+ (or email).

Information transfer between two headsets needs to be as simple as "handing a phone over" is now. Anyone on my sharing whitelist should be able to virtually "pass" me a picture or web site with very little friction. Even something as simple as finding a friend on my contact list and IMing them seems stupid when they are right next to me.

Speaking of sharing, ever see the Microsoft Holodesk? It lets you "hold" virtual objects using just a transparent screen, a Kinect, and some software magic. Handing over a virtual object would be way cooler than "Do you accept this item from Paul?" These are HUD glasses, after all, feel free to crank the sci-fi factor up a notch.

Well, that's about all I can think of for right now. It sure is fun to imagine an interface like this, isn't it? Hopefully we'll hear more about Project Glass soon. And, hey Googlers! If you ever need a beta tester, just let me know. =D

Thanks for reading.


I Want Ice Cream Sandwich for Tablets to Look Like This.

Today we're brainstorming about ICS. That's Ice Cream Sandwich. Tablet Edition:
Look at that for a moment. Think about how much stuff you could get DONE. This is what Android tablets should be. Less of a big phone and more of a small computer.

For starters, the Apps Button is now a permanent fixture next to Back, Home, and the Switcher. We're taking the lessons of desktop operating systems to heart: The Apps Button is analogous to the Windows Start Button. It should be accessible all the time. No more having to hit home just to get to your apps.

After the Apps Button, we're going to fill the bar with whatever the hell we want. This is Android, it's supposed to be customizable! Let us drag and drop things to the status bar just like the home screen. Pin your favorite apps, permanently have a music player down there, or a power widget, or weather, or your next appointment. Basically anything you can stick on your home screen, if it fits, should be allowed on the bar. Many 1x high widgets would scale well right now. A bunch of multicolored shortcuts would be distracting, but luckily Honeycomb can desaturate icons on the fly.

You guys know Chrome OS actually multitasks better than Android? Yep, that chat and video panel idea are directly stolen from Chrome OS. Floating windows would be a little much, so I want a Facetime/Chrome OS mashup: docked, minimizable panels that you can fling from corner to corner. Just think of all the cool things you could have in a popup panel: a small calculator, a notepad, facebook chat, a checklist. Better multitasking is the point of a bigger screen, isn't it? Oh, and this is a perfect spot for video. I've lost count of the number of times I've gotten a notification while watching a movie. I want to shrink the movie for a few seconds, look at my notification, and maximize the movie again.

Here's another big screen benefit:

Split screen. This needs to happen. 16:9 is almost 2:1. You could stick 2 vertical fragments next to each other like this with almost no UI changes at all. Honeycomb even gives you separate back buttons for each app. All you would need is a long press option to "Open in split screen" and a button to go back to full screen mode. Chrome and Windows 7 have a "snap to screen half" option, and some phones even have 2 screens just so you can multitask like this. We've got the screen space, the tech, and the demand for this feature. Make it happen.

I'll say it again, tablets should not be big phones, they are small computers.The big difference between a Tablet and a phone should be the ability to multitask. Make the software as capable as a desktop OS. Put the things I want at my fingertips. Make it customizable.

How about you? What do you want your next Tablet or Phone to look like?

Honeycomb: Lots and Lots of Info From D

Andy Rubin showed off a Honeycomb tablet at D. None of the lazy Android bloggers did a breakdown of the video, so I guess it's up to me. After all, video is worth 30,000 words per second. Keep in mind this is honeycomb on tablets, and, layout wise, might not be indicative of what the phone version would look like. It's also not finished.

The Lockscreen

Other than the font, (which I have lovely nicknamed "Comic Sans Techno") you wouldn't peg this as Android device if you just took a look at it. Crazy. The status bar is now on the bottom (more on that later), and we still don't have lock screen notifications. The tablet has new email and a calendar reminder and the lock screen isn't providing any info about them.

It's interesting that Comic Sans Techno survived. They phased it out of the market redesign and one of the UX guys for Android said the font "looks a little dated" (he's being nice). Hopefully they aren't done with the lock screen yet.

If you touch the lock icon, a bigger circle appears around it; dragging your finger anywhere outside this circle will unlock the device. So those that liked the "rotary phone" unlock style from Android 2.0 can have that more natural unlock motion back. (In a one handed swipe, your thumb wants to go in an arc. Moving it in a straight line is less ergonomic.)

The Notification/Menu Bar
Take a look at this crazy thing! There aren't any hardware buttons on this tablet, so the main buttons now live on the bottom bar. Just like the current notification bar, the bottom bar stays on top of all the applications. It reminds me of the Windows task bar. This picture is a composite of a couple video stills, but representative of the real thing. 1, 2, and 3 are on the left of the tablet; 4, 5, and 6 on the right.

  1. Back
  2. Home
  3. Menu right? Nope. One of the interviewers accidentally hit #3 while on the homescreen. It brought up the tiny window you see on the right. The top title says "Apps", I can't make out the first items, but the second is "Gmail" and the third is "Talk". They all have thumbnails, which makes me think these are running apps
  4. Notifications! Yep they're down here now. No idea how you would get to the normal pull down that we're used to. You can make out a VERY faint mystery item, Gmail, and a Calendar reminder.
  5. A clock. It inexplicably fades in and out. Why would you want the clock to fade in and out?

  6. Signal strength and battery? It's very faint and I can never get a good shot of it. Signal and battery have to be somewhere. The first icon changes a lot during the video, suggesting it's signal strength. What do these blurry masses of pixels look like to you? Is the first one a "3" and a vertical battery? (You have 3 bars?) The second a "4" and a battery? No idea.

Applications can add items to the bottom bar when they have
focus. When Rubin turned on Google Maps, a 4th icon appeared next to back, home, and the Apps popup button. Maybe this is what they're using instead of a menu on tablets; just populate the bottom bar with whatever used to be in the menu. Or maybe this is a menu button and it only pops up when you can use it. He never touched it, and it disappeared when he closed maps.

For about 1 second, the tablet received an email while on camera. Here's what that looks like:
The Gmail app looks just like it the tablet web version, so that's not really news. The notification is in the bottom right.

The notification icons, clock, and battery(?) fade out, and the email notification fades in. You get a head shot, name, and subject. The app icon for the notification is on the right. The headshot pokes over top of the notification bar and overlaps the app. (cool!) Then it quickly fades out and goes back to normal.

Home Screen and App Launcher
On the home screen there's a top and bottom bar. If you can't make it out, the top bar has Google Search and Voice Search on the left, and the app tray and mystery item on the right. The rightmost icon looks like a gear with a plus sign next to it, so does that make it "Settings" or "Add a widget"? In the middle of the screen you get a calendar widget that doesn't suck (yeay!),  a bookmark widget, and the familiar maps icon.

For about 3 seconds, we saw the app drawer:

You can filter by "All" or "Downloaded" at the top. The app drawer is a 7 x 6 grid. The Music icon is now yellow instead of blue. There's a few icons I can't identify, but with all the apps available for Android it would be pretty hard to guess. (If you want to try, remember, they're alphabetical. So the thing between "Camera" and "Clock" has to start with a C)

Google Talk
You've probably seen the picture of this, but it's here for completeness. Those green, square blurry things are video icons. Bonus points to Google for consistency; it looks just like Gmail.

That's everything I could find in the video. Hope you learned something!

Notification Panel, Version 2: Compromise.

Some lovely redditors brought up a couple issues with the first design.

First, despite all my previous objections, the clear button being stationary is a good idea. So I'm going to throw out all that bad UI stuff and put the clear button in the same spot all the time. Ease of use outweighs making sense. Also, the clear button gets a little too small when you stuff it into the "Notifications" bar.

Second, the carrier "advertisement" is actually useful for roaming and letting you know when you're connected to a femtocell. So I'm going to put that back too.

So, bigger, stationary clear button, and keep the carrier name, but make it suck less. I can do that:

I'm very interested in hearing any objections to this design. I think it should cover everyone though.

I've also fixed a big drawback of the first version: I don't have to murder the stock keyboard for framework resources anymore, the new location allowed me to do it with all stock UI widgets.

It's all down to just 1 XML file now. (like it should be!) So porting should be easier. In fact I bet it can be deployed to any CM6.1 rom as is.

Here's the source if anyone is interested. It's from /system/framework/framework-res.apk/res/layout.

If you are really, REALLY brave, AND YOU HAVE A NANDROID BACKUP: you can try this over your CM6.1 install. It will ruin anything that isn't CM6.1.

If the actually works, and your carrier name is black instead of white, go to Settings, CM Settings, User Interface, Date and Provider, and change Carrier PLMN Label Color and Carrier SPN Label Color to white (#ffffffff).

My only big problem is that it doesn't work with UOT Kitchen.... help?

Feedback welcome.

Update: It doesn't work with MDPI devices (G1, Hero, MyTouch, etc). Right now I'm focusing on getting it built into Cyanogenmod. Wish me luck!

The Clear Button.

The combination "carrier name and clear button" section in the notification bar has always bothered me. 
1) It’s a waste of space, especially in landscape.
2) It’s pointless carrier branding.
3) It’s bad ui.

1 and 2 you probably agree with, but #3? Bad UI?
Yes, bad UI. The best way I can explain it is in picture form:
Couple caveats: The biggest being distribution. If you want this I really have no easy way to get it to you. I modified framework-res.apk, and everyone's is different. So if you have the exact same setup as me (A Droid running CM6.1 RC5) you can have this. If not it would have to be ported over.

The second problem is that I had to kill the default keyboard in order to make this look good. The grey bar that says "Notifications" stops at the edge of the button, so I had to make a custom button with the grey background behind it. ANYTIME I try to add a new file to the framework I get boot loops (Please contact me if you can help with this). I had to edit an existing file, and nobody uses the stock keyboard.

With all those problems it's more of a proof of concept than anything. But if you really want to try running it, here's the framework-res.apk. it will probably blow up anything that's not a Droid on CM6.1 RC5.

And for the smart peeps out there, here's a zip of what I changed.