The Secrets of "Input"

Today, I would like to talk about the theme of input using keyboards. I have been involved in the development of UIs (user interfaces) for smartphones and other devices, but when it comes to keyboards, I would first like to hear from Mr. Yasuoka, who has written a book on the "QWERTY layout," which is the most common layout for Japanese computers.

To put it simply, the QWERTY layout happened by chance. Some books say it was invented by Christopher Latham Sholes, but the truth is that it was patented and became popular as various people went through trial and error in the 19th century.

Actually, typewriters were originally "receiving devices" or "output devices" rather than input devices. This is because the first typewriters were invented to record Morse code faster than handwriting. The profession of typist was also born to document (output) Morse code.

Later, in the late 19th century, when a mechanism that distinguished between uppercase and lowercase letters was invented, it became a tool for shorthand—directly typing spoken words. The QWERTY layout was finalized around this time. Even so, it was still difficult to input at the same speed as speaking, and it was reportedly used to create clean copies of shorthand notes. That is how it gradually spread.

There seem to have been many different keyboard layouts until the Typewriter Trust was established in the 1890s. Many records remain. The QWERTY layout was just one of them, so there isn't much rational meaning behind it.

Currently, in the field of UI, many studies suggest that layouts other than QWERTY would be better. As smartphones and tablets have become commonplace and opportunities to input with fingers or pens have increased, there are voices saying that the keyboard layout is difficult to type on. For example, proposals have been made at academic conferences to place letters that are often used together, such as T and H or L and Y, next to each other.

However, everyone finds it troublesome to learn a new layout, so as a result, software keyboards on smartphones and tablets still follow the QWERTY layout today.

So, was it like input and output were almost simultaneous on the first typewriters?

I think they started being used as "input devices" in the 1940s when people began connecting teletypes to computers.

It wasn't that far-fetched to use teletypes, which were originally communication devices, to replace Morse code; it was like outputting exactly what was typed. Since it was difficult to go through the trouble of printing just to make a clean copy of a handwritten text, the typewriter was invented when a machine that could convert text into something like print was needed.

I have never used a typewriter, but I have an image of the keys being very heavy. Is it true, as mentioned in your book, that people in the past typed as many as 170 words per minute?

It seems so. They apparently used various techniques to type fast, such as adjusting their strength to the point just before the character is printed without pressing the key all the way down, or changing the key pressure.

When people imagine a typewriter, they probably think of the "front-strike" type that prints from the top of the paper, but this mechanism appeared in the 20th century. Until then, the method of striking the type from below was common. It used the weight of the type bars, so it printed reliably, but it had the disadvantage that you wouldn't know if you made a typo. Furthermore, because the type was lifted from bottom to top, the keys were heavy, and it's surprising that they typed 120 to 130 words per minute with that.

Was the goal for typewriters at that time to have operability that allowed typing at the speed of human speech?

That seems to have been the case for shorthand typists. However, a drawback of old typewriters was that the typing sound was loud. It seems it was too noisy to type right next to someone speaking, so they couldn't bring them to the site of the dictation; instead, they used a method where they typed up what the stenographer had written down later.

On the other hand, in workplaces where typewriters were used to convert Morse code into characters, a device called a "sounder" was invented, and it seems they could type while listening to the telegraph even if the typing sound was somewhat noisy. However, Morse code has a slow communication speed, so no matter how fast they typed, it was at most about 40 words.

In my daily practice, I input into electronic medical records while listening to patients, but I can't type at the same time, and I often make typos because of the pressure to type as fast as possible. Fixing them is also a hassle, so I wish there were a better Japanese input method.

Japanese input requires conversion, so there are different circumstances compared to Western input, right?

In addition to that, Japanese has Romaji input and Hiragana input methods. I'm a Romaji user, but I heard that you, Dr. Ogawa, have been using the "Thumb Shift" keyboard, which inputs Hiragana directly, for a long time.

The first word processor I bought when I entered university was a Fujitsu Thumb Shift. I couldn't let go of it as I continued to use it, and I've been a "Thumb Shifter" ever since. However, since the keyboards that support Thumb Shift are limited, I now use an emulator (software that converts a standard JIS Kana keyboard layout to Thumb Shift).

The reason I go that far to keep using it is that Thumb Shift feels so good to type on. To use an analogy, it might be close to the difference between a violin and a piano. On a violin, only the left hand is used to control the pitch, but a piano is played using both hands. Thumb Shift also uses all fingers to input, and the tactile feedback is just...

By the way, the J, K, and L keys on the QWERTY layout are in the easiest position for the right hand for a Thumb Shifter, and they are assigned to "to," "ki," and "i" in order of frequency. Since these keys are rarely used in Romaji conversion, I feel it's a very unreasonable layout.

I think the frequency of K is quite high in Romaji input, though. As for L, recently more students are using it for small characters in class, but they type "kya" as "KILYA" on purpose, so I had to warn them (laughs).

Mr. Yasuoka, were you also researching Japanese keyboard layouts?

I have looked into them. Around 1984 or 1985 when I entered university, the keyboards in the lab were half "Thumb Shift layout" and half "JIS Kana layout," so it was inconvenient if you couldn't use both.

Was Romaji conversion a minority back then?

The keyboard market share was about half and half. At that time, I was researching Katakana layouts, and the current JIS standard Kana layout, which starts with "Ta-Te-I-Su-Ka-N-Na-Ni-Ra-Se," was the layout proposal from "Kanamojikai" (a private organization advocating for the exclusive use of Kana characters) when standardizing the Kana typewriter key layout in the 1950s.

The proposal from the former Ministry of Posts and Telecommunications was a layout designed for telegraphy, which gathered frequently used characters on keys handled by the index finger, but it was abolished because it lacked small Kana and was inconvenient. In the process of researching such history, I came to the conclusion that keyboard layouts fail if they are decided based on frequency.

Mr. Masui, you are the developer of flick input for smartphones, but do you use flick input yourself?

Actually, I'm not very good at using it myself. However, many people find Hiragana input more convenient, and flick input seems to be quite popular among them.

People who are fast at flick input are really fast. Watching their hand movements, they type at an incredible speed.

Nowadays, predictive conversion input has taken root regardless of whether it's a feature phone or a smartphone. Many people probably feel that even inputting with a finger or pen is slow. I think more people feel that choosing from candidates is faster than directly entering characters or symbols. In the US, it seems few people use predictive input, so I think it's an interesting situation where new input methods are still likely to appear.

On the other hand, on PCs, many people still feel it's faster to convert after inputting all the readings, so predictive input hasn't spread much yet. Although it is valued by people with hand disabilities or those who are not good at typing.

I use predictive conversion when inputting electronic medical records. It's very convenient that frequently used items come to the top because candidates are displayed based on history.

Is the input method you use, Dr. Ogawa, for medical use?

I have the ATOK medical dictionary installed. However, I'm always troubled when converting homonyms. I wish those functions were optimized for medical use, but general terms often appear as candidates.

That might just be because the system is poorly made (laughs). Even in normal Japanese, it's common that when you type "shiyou," it doesn't distinguish between Specification (shiyou) and Usage (shiyou).

Previously, there was talk about creating an input system for medical records, but the reason it didn't happen might be because users can use it without inconvenience if they optimize their own Japanese input dictionaries.

It's also a bit of a hassle to select the displayed candidates with a mouse when I have to input in a hurry, so I also think it would be great if there were technology that selects characters just by looking at them, for example. Is it still a long way off before such an input system appears?

Research on eye-gaze input has been conducted for decades, but old eye-tracking devices had low accuracy and were expensive. However, recently, relatively cheap and high-precision ones have come out, so it might be put into practical use eventually.

I've been calling it "predictive input," but to tell the truth, this system is a "search." It's just pasting dictionary search results into the input field, so if there's a good search system for a medical dictionary, you can just combine it with that.

I see. So the accuracy of the input depends on the performance of the dictionary's search system.

On old PCs, "phrase-by-phrase conversion," where you convert long sentences all at once, was popular, right? At that time, they competed on that accuracy, but now it's considered faster and more accurate to input little by little with a predictive input system using flexible search functions.

The drawback of long sentence input was that you couldn't tell at a glance where a conversion error occurred.

Exactly. If there's a conversion error, you have to fix it every time. Because that's a hassle, it changed to the current input method.

I use flick on my smartphone, but did the idea of inputting "a, i, u, e, o" by moving up, down, left, and right come out of various trials and errors?

Actually, rather than me inventing flick input, I combined things that already existed. There's something called a "pull-down menu," right? That's a method of choosing from candidate items lined up vertically, but there's a method called a "pie menu" where you choose by the direction you move the mouse or pen. The advantage is that you can select menu items by the direction of movement after displaying the menu.

I thought that if I combined a pie menu with dictionary predictive input, it could be used for Japanese, and that system was born. I'm often called the developer, but the truth is that I implemented and sold existing technology.

Mr. Masui, were you also developing keyboard input before you were involved in the development of flick input?

I once prototyped a flexible dictionary search system, and at that time, I realized that searching and inputting are the same. Since words can be retrieved surprisingly easily with a dictionary search system, I thought about combining this function with an input system, and that's how predictive input was created.

I didn't feel any particular problem with the keyboard layout, so I usually use QWERTY with Romaji input, and keyboards from cheap PCs are perfectly fine. I use a US layout keyboard, but since there's software to change the layout now, it's not like it absolutely has to be this one.

So you thought of an easy input method from the perspective of someone thinking about dictionary search systems.

That's right. As long as I can search for words immediately, it's fine, so the operation method can be a pen, voice, or gesture.

I hear that more people are making custom keyboards lately. Did you know there's a specialty store in Akihabara?

No, I didn't know.

That store only deals with custom keyboards, and they sell things where freely arranged keys are controlled by a microcontroller so they can be used just like commercially available keyboards.

It's like a keyboard with a CPU. If you can write a program, you can freely change not only the layout but also which keys produce what characters or symbols, so various enthusiasts are making various keyboards. That store seems to be very prosperous, and you can see that there are really many people with particular preferences.

I'm the opposite of Mr. Masui; I have particular preferences for things like stroke depth and key pressure. At one point, people around me recommended that "Topre" keyboards are good, and when I looked into it, I could choose from various weights. So I bought and tried a product with a pressure of about 35 to 40 grams, and the burden on my wrists and elbows was light, and typos decreased. I realized that typing accuracy changes even with subtle angles of the hands, so it's important.

What kind of keyboards do you two usually use?

I use the one attached to the laptop body. Basically, I like things with a shallow stroke that allow typing without moving my fingers much.

For that reason, I also change the position of the return key myself. I hate that I have to stretch my pinky to hit it in the normal position, so I use the semicolon key as the return key. With this, I can do most things without moving my fingers much.

Don't you have trouble writing programs without a semicolon?

Lately, there are many programming languages that don't require semicolons.

The master guitarist Eric Clapton is sometimes called "Slowhand" because his finger movements are so exquisite, and that efficient movement is cool.

So you like that kind of keyboard handling, Mr. Masui.

Exactly. I want to type with 'slow hands' like Eric Clapton. I like the idea of it looking like my fingers are barely moving, when in reality, I'm typing characters at an incredible speed.

Conversely, typewriters involve rather dynamic movements, don't they?

That's because they are mechanical. Originally, you were printing using the force of human fingers, so the stroke could never be zero. However, I do wonder why they didn't eliminate the stroke even after things became electrical.

When PCs first appeared, there were no membrane switches (thin sheet-like switches made by printing circuits and contacts on film and layering them), and physical switches that didn't react unless pressed to a certain extent were the norm. Even now, there is still strong demand for having a physical stroke.

The fact that there is still demand for deep strokes even after technical constraints have disappeared is probably just a matter of preference.

I'm fine with the shallowness of a laptop, but the cover-style keyboard on Microsoft's "Surface" makes me feel uneasy about whether I'm actually pressing it or not. I find it a bit difficult to use if there's no tactile sensation at all.

How about the MacBook? I find that level to be just right.

The MacBook is fine, but I'm not a fan of software keyboards displayed on a screen like on a tablet. If there's a pen, that's one thing, but tapping a display with my fingers just feels wrong to me. As long as there's some feedback, I don't mind if it's a bit light.

I'm not good with the 'click-clack' shallow typing of laptops. The IBM "ThinkPad" was good because it had a relatively deep stroke for a laptop, but I'm afraid that shallower ones might give me tendonitis. That's why I use my own external keyboard even with a laptop.

Actually, I use a keyboard with a deeper stroke for my UNIX machine for programming, but the feel I'm looking for there is different from the keyboard I use for Japanese input.

When watching old American movies, you see people typing very fast using only their index fingers. Are there still people like that today?

Apparently, Bill Gates is like that.

Is that so!?

There are students like that in universities lately too. When I watch them from behind while they write programs, they type very quickly, but they only use two or three fingers. There are many students like that.

Are they looking at the keyboard while they type?

No, they can generally type without looking. When I observe liberal arts classes where they don't write programs, there are quite a few students from the smartphone generation who aren't used to keyboards. But after letting them use one for a while, they start saying it's easier to type on than a smartphone. Once they get used to it, they seem to feel it's faster than flick input.

Mr. Masui's story about assigning the Return key to the semicolon key was shocking, but don't you make mistakes?

I don't make mistakes on my own keyboard, but since everything is customized, I have trouble when using other people's terminals. By the way, while Mac and Windows have software to swap input systems, Chromebooks don't.

On Chromebooks, I use a self-made IME (Input Method Editor) written in JavaScript. I replace the default Google input system with it, but I can't recommend this to ordinary people.

I use both a JIS keyboard for daily use and a US keyboard for programming, but what troubles me is entering symbols. These two have different positions for double quotes, "@", "+", and "*", so I get them mixed up. Ms. Ogawa, as a Thumb Shift user, do you have that problem?

I don't use symbols very often in electronic medical records.

Still, you'd need the "@" for typing emails normally.

I switch to missing symbols using an emulator. Sometimes the characters actually typed are different from the keyboard on the screen, though.

How do you input "@" with Thumb Shift?

On the original keyboard (Thumb Touch™), the Enter key is divided into upper and lower sections, and the lower one is "@". In the emulator, if you switch to half-width input, you can enter "@" from the right of the 'P' key, just as indicated on a JIS Kana keyboard.

People who have been writing programs on keyboards for a long time used things like Sun workstations, so their bodies have become conditioned to only being able to use US keyboards. Recently, US keyboards are becoming less common, so we're starting to feel a bit out of place.

That's very true.

Even among English-speaking regions, the layout of a UK keyboard is different from a US keyboard.

Is it different between the US and the UK too?

It is. If you accidentally buy a keyboard at a computer shop in the UK, you'll be in for a rough time.

On a UK keyboard, the "@" is to the right of the "P".

France has a different layout too, right?

France is unique. They use the AZERTY layout.

It's interesting that just because they use the same alphabet doesn't mean they share the same layout.

The hardware industry is tough because standards differ by region. That's exactly why Steve Jobs made the iPhone keyboard a software keyboard. It wasn't because touch panels were cool or anything like that; it was a matter of cost.

But what I thought was truly amazing about Jobs was that he said, "Doesn't a touch panel look cooler?" Thanks to that, everyone was fooled, weren't they?

Modern keyboards have many mysterious keys that I have no idea what they're for. I think they're remnants of IBM mainframes, but it's a mystery why they remain.

I agree. I always think we don't need this many keys.

It's because they were assigned different functions during OS transitions. Young people who don't know that history misunderstand them as keys meant for those functions, and since it's even written on Wikipedia, they believe it even more.

The Tab key was originally a key used for creating tables on typewriters, but now it's thought to exist for jumping to the next column in an editor.

I really wish they'd get rid of the Caps Lock key. I wonder why it's still there.

Caps Lock is also a remnant of the typewriter. When uppercase and lowercase were first distinguished, there was only a Lock key. When pressed, it switched to uppercase, but it wouldn't return when released; it required a different key (Unlock) to return to lowercase. Later, springs were added in the 1890s so that releasing the Shift key would return it to lowercase, but since people accustomed to the old Lock key would be troubled, it was kept in the form of Caps Lock.

What era was that?

Roughly around 1900 to 1905.

It's surely not needed anymore. Thinking there might be software that uses such a function, I once asked on Twitter, "What do you use Caps Lock for?" The most common answer was, "To undo it when I press the key by mistake." Nobody really knew.

There are probably many people like that. In that sense, the Alt key is also a special key. This originates from IBM, and I believe it was for assigning functions other than the "Ctrl + [key]" combinations defined by ASCII (American Standard Code for Information Interchange, established by the American National Standards Institute in 1967). It's used in combinations like "Alt+A" or "Alt+B"...

Was there such a menu on IBM mainframes?

It didn't exist on PCs in our era. It was originally a key on the IBM terminal side, and I believe it was created because some necessary function was needed when connecting a mainframe to a PC-type terminal.

I also don't really see the point of having 12 Fn (Function) keys.

This happens because they don't remove things after increasing them.

It's the same with TV remotes; once something is added, it's hard to remove. This is common to all sorts of products.

In that regard, the small number of buttons on the Apple Remote and Fire TV Stick was revolutionary. They must have thought it through from scratch. Perhaps we should do the same with modern keyboards.

Did smartphones switching to touch panels also have the meaning of resetting that kind of trend?

But smartphone keys will probably increase too. Someone will arbitrarily add a feature thinking it would be nice to have. It's really a nuisance when that happens.

How will input evolve in the future? You mentioned that the original purpose of the keyboard was to input at the speed of speech, but I wondered if the role of the keyboard would end if voice input like Siri becomes widespread, for example.

The accuracy of voice input is improving. In particular, it's said that the recognition rate is higher when you speak quickly. If accuracy improves further, it will be effective for creating book manuscripts directly from spoken content. In fact, the economist Yukio Noguchi, who wrote "The 'Super' Organizing Method," reportedly writes books using voice input.

However, the drawback of voice input is that it's hard to use in places where other people are present. Also, as you'll find if you try it, speaking without making mistakes isn't that easy.

Voice input is tedious because you can't correct mistakes using your voice again.

A master like Mr. Noguchi can write many books precisely because he can speak without making mistakes. But I think the recognition rate has become surprisingly good over the last three or four years.

Dictation has always been a rational method for writers. If machines can do that accurately, writing activities will surely become more efficient.

In the medical field, there used to be a role called a 'Schreiber' who would write down what the professor said.

I think voice input is essentially a machine doing that instead. However, while it might work for writing on a blank sheet of paper, current electronic medical record formats have various frames shared with nurses and pharmacists, such as prescription and test columns. I feel it might still be difficult to distinguish those differences through voice recognition.

There are those formatting issues, and technically, it still seems difficult for Japanese to distinguish between hiragana, katakana, and kanji. Even when typing on a keyboard, I often think it's quite unreasonable to have three different types of characters.

From a reader's perspective, I actually think having different types of characters makes it easier to read. It allows for skimming.

True, there is the advantage of being able to pick up words at a glance.

With voice input, would we have to manually correct the kana and kanji?

I wonder. Maybe AI will work hard to fix it... However, even if you say 'Keio,' it's difficult to distinguish between '慶應' (the university), '京王' (the railway), or 'KO.' It would be great if the machine could judge based on context, but that won't always be the case.

The differences in input methods between English and one's native language that we discussed earlier must exist in other countries as well, right?

Indeed, everyone is creating input systems through trial and error for their respective languages. Even in German, there are assignments for 'ä,' 'ü,' and 'ö' in addition to standard Roman letters.

What's even more complex are cultural spheres that use characters other than the Latin alphabet. Most of Asia is like that, and it's particularly difficult for the Kanji sphere, including us Japanese.

This is because Kanji input can only be distinguished by 'sound' or 'shape.' Input by sound is limited in variety, and there are countless homophones. People in China and Taiwan use ten thousand different types of Kanji, but since searching by sound alone is impractical, their keyboards have Kanji components printed on them from the start, and they select from those.

Which method is most common for Chinese input?

Looking at students from the mainland, 'Pinyin,' which can be typed on a Roman alphabet keyboard, seems common, but students from Taiwan use 'Wubi,' where you input based on the shape of the Kanji characters.

Neither of those is very old, right?

That's right. Both methods appeared in the 1990s. I feel like many students in Taiwan and Hong Kong use Wubi by sticking labels on their keyboards.

I wonder why they don't both just use Pinyin.

I'm not sure about Hong Kong, but in Taiwan, a system called Bopomofo (a phonetic notation system for Chinese used mainly in Taiwan, also known as Zhuyin Fuhao) was already in use, and the input method is different. The correspondence with the alphabet isn't as strict as in China, so a unique input method was created.

Thailand has the Thai keyboard, which has the Thai language and corresponding alphabets on the keys. Korea also has the '2-beolsik' system, where Hangul and its corresponding alphabets are placed on the keyboard for input. Both are similar in that they don't use the Latin alphabet.

Hangul in particular has only 21 vowels and 14 consonants, so the number of keys is limited. Furthermore, since it follows a pattern of consonant + vowel + consonant (batchim), the input method is simple.

With Hangul, you just enter consonants and vowels in order and don't need conversion, so it must be quite easy.

We've covered many topics, but I'm still sad that Thumb Shift will be disappearing in the future.

Maybe you should stock up on them while you can (laughs).

Since Fujitsu is no longer manufacturing them, I suppose the only option left is for enthusiasts to develop emulators and keep it going on a small scale. As long as emulators support current OSs, I want to keep using Thumb Shift.

Are you familiar with the TRON keyboard created by computer scientist Ken Sakamura? There was someone who loved it so much that they made about 20 units themselves by creating a mold because they wanted to use it for the rest of their life. I'm sure there are people like that for Thumb Shift too.

There's also a trend of DIY building lately.

Yes. You can put a CPU in the keyboard itself and program it, so it's perfectly fine. In fact, you might even be able to handle conversion on the keyboard.

Conversion on the keyboard?

Even if the OS doesn't support it, you can just build the Kanji conversion system into the keyboard. You'd still need a display, but we live in an era where such things are possible.

Input really is diversifying. As voice and eye-tracking input become more common, will input devices expand even further?

At conferences for computer-human interfaces, papers on input come out every year. A great idea might suddenly appear one day.

Is there a possibility that new ideas like flick input will emerge?

Whether they become widespread or not is another matter, but since there are a certain number of researchers and it's an era where hardware is easy to make, things like eye-tracking and voice input are definitely becoming more user-friendly little by little. I hope people will watch over these developments warmly.

The QWERTY layout was also a product of chance, so I'm certain that new devices and corresponding input methods will definitely emerge. However, since people find it hard to let go of what they've once learned, we won't know if they'll catch on until the next generation.

While input methods have changed significantly over the last 20 years, PC windows and menus haven't changed for about 40 years. Thinking about it that way, I feel like input still holds more possibilities.

I just hope the number of keys doesn't increase any further (laughs). It's clearly exceeded the number a human can remember.

But it's exciting to see what kind of input will come out next.

I'd like to stay involved as well.