Transcript

00:00:00 - Speaker 1: On the academic side, you’re very limited by your work has to fit in the box of like a peer reviewed quantifiable research paper and in the commercial world, it needs to be commercializable in the next, you know, probably a year or two, maybe, maybe 3, but all the good ideas don’t fit in one of those two boxes.

00:00:27 - Speaker 2: Hello and welcome to Meta Muse. We use the software for your iPad that helps you with ideation and problem solving. But this podcast isn’t about Muse the product, it’s about Muse the company, the small team behind it. I’m Adam Wiggins. I’m here today with my colleague, Mark McGranaghan. Mark, you reading anything good lately?

00:00:43 - Speaker 1: Yeah, just last night, I actually reread an ultra classic, you and your Research by Hamming, who’s a famous scientist, and it’s about how you build a really impactful research program over the course of your career, and I was inspired to reread it because it’s one of the chapters in the classic book, The Art and Science of Doing Engineering, which is about to be republished by Stripe Press.

00:01:05 - Speaker 2: Stripe Press is really on a tear these days.

00:01:09 - Speaker 1: Yeah, for sure, highly recommended.

00:01:10 - Speaker 2: And also perhaps relevant to our topic today, and I’m happy to say that our topic today was requested by a listener. So Fetta Sanchez wrote in to ask us, how do you get into the HCI slash interaction slash new gestures research field. So probably we need to start at the top there. Maybe you want to tell us what HCI is.

00:01:32 - Speaker 1: Sure, so HCI stands for human-computer interaction, and this is things like the way humans interface with computers, and also the way they use computers as a tool in their lives, how they get things done, how they learn. To use them, how they accomplish their goals, things like that.

00:01:48 - Speaker 2: And I did a couple of years of a computer science undergraduate degree that I did not finish. And during that time, I really remember everything in the curriculum was algorithms, databases, compilers, maybe some network type of things. And I only learned about HCI as a field a couple of years ago. And to me it was a bit of a revelation because this concept of How the user interacts with the computer and that being a whole field of study. Well, I was very excited about, but stood for me in very stark contrast to the System the algorithms oriented computer science that I sort of knew from my brief time in academia.

00:02:29 - Speaker 1: Yeah, likewise, it was pretty new to me, and it’s a whole huge world, you know, there’s conferences and papers and many professors who’ve dedicated their entire careers to it.

00:02:37 - Speaker 2: It was fun for me to dive in and learn about that world a little bit, and you and I were both part of this independent research lab called Inot Switch. Uh, and through that process, we began publishing and then made some connections with folks in this field, and then you and I went to a conference called Kai last year that I think really kind of opened the door for us there.

Maybe one thing that would be worth doing is um categorizing here a little bit.

There’s Human-computer interaction as a branch of computer science in the academic tradition, that is say mostly done in universities, sort of the the pure sciences.

Then there’s corporate R&D which is more associated with for profit businesses, but actually it’s where a lot of the HCI innovations that are maybe the most famous, uh, we think of places like Bell Labs or Xerox PARC, maybe today, Microsoft Research.

And then there’s a small but growing space of called them independent computer science labs, independent HCI researchers, of which I think we we had some contact with. How would you define the difference between those three categories?

00:03:39 - Speaker 1: Yeah, well, like you said, the academic side is grounded in these research universities, and this is often directed by a professor or graduate students, and there the values are really around evidence, rigor, review, publication and communication, and creating knowledge over time, which is a whole thing we should talk about. And then on the industrial side, it’s often more integrative because you need to consider. Not only the the pure HTI elements, but the business elements and the hardware constraints and the how easy the thing is to learn for the user and practice and things like that. And then on the indie side, this is a smaller domain, but that’s tends to be more experimental, free form. People can bring their own wild ideas to it and just try stuff. So it’s a nice injector of new ideas.

00:04:22 - Speaker 2: One way we can maybe make this concrete is to describe the path from let’s say the lab to commercial product.

And I’ve I’ve struggled to find full stories on this in many cases, I think this is something that happens behind closed doors a little bit, even though science does have open publishing, the exact story of how something went from basic research or early um HCI research to a product that’s in the hands of end users is not well understood or well or written down anywhere.

Um, I think the Xerox PARC case is one that has a lot of um, Fame and certainly in the tech circles that we run in, there’s there’s some books about it. There, they invented things like the modern GUI, uh, as well as what you see is what you get word processing, and was really a pretty special place.

And notably there was a branch of Xerox, the copier company, and they were looking for innovations. I think their theme was the Office of the Future. And they were looking for innovations around that and, and clearly, you know, this is the 1970s, they knew that would have to do with computers, personal computing was, didn’t really exist yet or was, you know, still just an emerging idea. So that’s one famous example.

Uh, maybe more recently, you have something like Microsoft Research, and I think, you know, I don’t 100% know what the path is for some, you know, for example, interesting innovations that emerged from Microsoft, to what degree were those laboratory projects versus some other path. Uh, one that I find quite interesting is what we now on the Apple platform, we talk about face ID on the Apple platform we use face ID rather. And that uses stereoscopic cameras and infrared, and infrared camera, which gives you depth sensing, right? So this is why you can’t fool your iPad into unlocking by holding up a picture of your face, because it can actually sense the the shape of it.

And that idea was first in Windows Hello, which sort of was the Microsoft implementation of facial recognition. And that in turn, the technology there, I think came from the Microsoft Kinect, which is actually a gaming. Device, um, and I’ve tried to like dig into the history on this. I don’t know if it came out of a Microsoft lab. I think it may have come out of some other independent place. So you often have these very winding paths where a promising technology like stereoscopic cameras emerges, but you’re still trying to figure out the application of it. And it’s actually quite a long distance between when these early researchers are doing the work, and it’s in the hands of consumers as a usable product.

00:07:00 - Speaker 1: Yeah, and I think honestly, that’s the best case that you have this long winding path, but it does eventually find its way into commercialization. I think one of the ideas we had originally behind the lab was these two domains are kind of spinning in circles. So it’s a lot of good ideas from the academic world that are getting stuck or don’t have the appropriate context from the commercial world, so they’re not transferring over. And on the flip side, the commercial world isn’t tapping into the academic tradition and the way that it should be. So you have a lot of like the, the Microsoft research and the, the Googles and so on, they do a lot of internal research.

00:07:36 - Speaker 1: Google X maybe is their, their internal lab, or they have a bunch of computer science just doing research on, you know, search and stuff like that, uh, some of which gets thrown out as papers and some of which doesn’t, but the kind of the classic path from uh academic labs through commercialization I hypothesize is actually weaker than it, it should be or could be and perhaps was in the, in the past. And one of our ideas with the lab was to help bridge that gap with something that was kind of in between with the with the so-called industrial research lab.

00:08:01 - Speaker 2: Actually, Google search is another case. It’s not an HCI thing, it’s more of an algorithms thing, but the founders of Google, they were doing academic research work at Stanford, if I’m not mistaken, came up with this page rank algorithm, which was a science paper published like any other.

At some point, I’m not super knowledgeable about the story, but at some point they decided to turn that into a working prototype. They set up this search engine, they found it worked way better than anything else out there, and they realized they could spin that out into a commercial.

Entity. And so those two individuals took it from that early lab work all the way through to a commercially viable product, but it takes pretty extraordinary individuals and probably extraordinary circumstances or at least serendipitous circumstances for that to happen. And so what you’re alluding to there with the the gap between The academic researchers who are exploring wild new ways we can interact with computers and commercial companies that can bring these to people in their everyday lives. Um, that’s, you know, in the Google case, these, these extraordinary individuals took it across that threshold, but what can we do to create more movement there?

00:09:12 - Speaker 1: Yeah, exactly. I think We’ll see as we get more into HCI specifically here, that the HCI domain isn’t as obviously susceptible to the academic tactics as other domains, so things like algorithms are very quantifiable, they’re very repeatable, they’re very discreet, and those are things that work well in the the traditional academic model of of measurement and confidence intervals and so on, whereas HCI is often much more multi-dimensional, maybe case based, maybe hard to quantify.

00:09:39 - Speaker 2: Yeah, for sure, I think how it feels is like a huge dimension of making interfaces, but that is something that is very hard for science to evaluate.

Uh, it’s something that is more of a taste or judgment call, but then science is and should be about rigor and the academic tradition and fitting into these and and sometimes I think that does mean from what I’ve seen of the HCI field.

Sometimes I read these papers where, I don’t know, one example was, um, I think it was also a Microsoft research project. They did an interesting thing where they rigged up some projectors where you could essentially put windows from your computer, uh, individual windows, whether it’s like a document app or something else up on the wall and they had projectors, so basically all the walls. We were 100% turned into these screens, but it was collaborative. So I could put up one window, and it’s not like, while I’m, you know, screen sharing, no one else can, someone else could put up their window and you had this shared space that was very spatial and that sort of thing. This sort of stuff was, was, you know, part of what was inspiring us and we were thinking about the new opportunity. But notably there. It’s a really interesting prototype, you can look at their video and look at what they’ve done and read the paper and think about how this might be applied in the real world, but they have to, it’s not enough to just build the thing and say, hey, we liked it or we didn’t like it, then you need to go and do some kind of quantifiable test. And they did a usability test or user test, which is as near as I could tell was just grabbing 7 random people that happened to be walking by in the office and having them use it for 2 minutes and then, you know, giving them a little survey and writing it down. And it seems like, OK, well, I guess that makes it science because you’re measuring a thing. But that’s not where we make great breakthrough new interfaces, but it’s very difficult because you just leave it to, well, did you like the thing you built? People always are attached to the things they built. They always like the thing they built. How do we, how do we measure that? That’s probably an unsolved problem a little bit for the academic side.

00:11:34 - Speaker 1: Yeah, I think so. Thinking about things that do work well in this space, reflecting on my own journey.

I started not so much with the HCI as like proposing a certain windowing system or a specific gesture model. I started more on the fundamental side.

So we think about human computer interaction, you need to understand the human body, like biomechanics and things like that. You need to understand the human mind, like cognition, and then you need to understand the computer science fundamentals, things like the graphics pipeline. So I found it very useful to go and study those fundamentals, both within. And outside the HCI literature, and there again, that area is much more susceptible to traditional scientific methods, so it’s very good information. um, and then you really understand that the fundamentals, the ground truth.

00:12:21 - Speaker 2: You know, the point about humans and computers are equal participants in this. And I think there is a tendency for computer people to focus on the computer.

Maybe one thing that HCI tries to do, or at least um some of the HCI teams that I’ve had chance to interact with, including this team out of UCSD that we met at this conference we went to, they try to have maybe a cognitive science person or behavioral sciences person on the team, and they are concerned more with that, how does the human mind work, how does our attention work? How does our how do our bodies work, and then, but you also have to connect that. Together with what’s possible with the technology, both in the moment and of course, also in the future where we think technology might go.

And I think, you know, for example, VR AR stuff is maybe in some ways a hot or buzzy space or maybe was, maybe that’s died down a little bit.

But if you go read a lot of research about that, you see that for example, one of the biggest problems with that is just a simple case of, OK, if you got these controllers, you’re waving around in the air as the main way you interact with it, your arms just get tired. And it’s, it’s like they, they’ve measured this, right? They, they put people in situations where they’re using these kinds of controllers for long lasting tasks and they see that after an hour, you got to take a rest and they’re they’re, they’ve tried lots of different things to try to make that to be able to let you do a full work day the way you would at a standard desktop computer or whatever, and they haven’t found a solution. And so if you’re coming in, if you’re a commercial company that’s coming in and wants to do something with this space, you probably want to read that literature and keep those, uh, keep that challenge, that unsolved problem in mind.

Yeah, one place to fill in more of the picture on the academic side, for me, the big eye opener was going to, uh, the biggest conference in the space, which is Kai last year, you and I kind of spontaneously both decided to go. This is when we were still within the lab, but thinking about the use. Idea and that was a really great experience because we both got to meet a lot of the professors and researchers that were working in this space, got to see how many people were there. I, I don’t know, it was 2000, 3000 people, there’s hundreds of papers submitted, many, many tracks of talks, and then we saw all of these people who are working really hard at thinking big and thinking future facing about what, what computers can do for us and how we can interact with them. Some examples of just for fun, I pulled up my old notes, uh, had a very early version of Muse. Uh, back then, a prototype that I was working with, and I was able to dig that out of my, my archives, or dig the the Muse board exports out of my archives. Um, we had, for example, there was a talk on peripheral notifications, and this is where they’re basically testing, OK, so if you have a slack notification or an email notification or something pop up, and it’s on screen somewhere. What can we do to put it in your peripheral vision so that it won’t break your state of flow, or a better way to put it is just trying to understand what what kinds of sizes and colors and motions and shapes for a particular notification in a particular place in your field of view, how likely that is to get your attention. And then as a person who’s implementing something that wants to give a notification, you can go read this literature and they have this very extensive data set. And if you say, hey, I want something that’s absolutely certain to grab your attention, you should do it like this. If I want something that’s more a little bit of a note to the side, but I don’t want to distract you if you’re in the middle of something, maybe you should use this shape and this color and be in this space in your in your field of view. And there’s things there about keyboards and different ways to improve typing on mobile, there was lots of things about wall mounted displays. Uh, there was, um, Ken Hinckley’s group, uh, which has been a source of inspiration for us at use. They do a lot of stuff with tablets, particularly around the surface platform. They had one that was, I don’t know, they attached a bunch of extra sensors, they basically strapped a bunch of extra sensors onto a standard consumer tablet and they use that to detect, I think what they called like postures, so they could tell better the grip, like how you were holding the tablet at the time and then they can make the software behave differently. And clearly this is not something you can use in production. They, this is the equivalent of a raspberry pi taped onto the back and a bunch of sensors, you know, kind of hot glued on around the edges. This would never work in commercial environment, but it suggested some things you could do if such a capability. Existed and I think that that is a good example of what um what I think this field of this best does is it it it gives you possibilities to draw from and then it’s the applied people, what we would normally call just people building products that can potentially go and draw from that pool of ideas and that pool of things, finding things that have been learned and use them to make potentially new products that solve uh new problems or old problems in new ways.

00:17:07 - Speaker 1: Yeah, this experimental slash prototype approach is probably the thing that we um most think of when we think of HCI.

Another type of work that I found very helpful is the ethnography, where you go and you understand how people actually work day to day and what’s worked for them and what hasn’t.

Couple of examples there. One is a book called, I think it’s a small matter of programming or the simple matter of programming. This is a study of uh end user programming in the wild, things like Excel spreadsheets, CADS, and what actually works there, and because they talk to these people who are actually doing work every day and and having success or not in these environments, they’re able to pretty deeply understand what is useful in the way, in a way that you probably couldn’t get with either theorizing or experiments.

00:17:50 - Speaker 2: And I’ll just interject to say that one was a big inspiration for uh Hiroku.

And it’s also a good indicator of how much the academic world is ahead of in a, in a strange way.

We think of maybe in the startup world or the tech world or whatever, oh, we’re so on the cutting edge of things, but a small amount of programming was written in 1993, if I’m not mistaken.

And this was 2006 or 2007 when I was reading this and and applying some of what it, um, some of the ideas that were in it went into Hiroku. And so at that point, the book was already 15 years old, but a lot of the research and understanding in it and ideas that suggested were still really bold, innovative, or just thought provoking, in a way that current technology and software products and certainly programming tools um had not taken advantage of or um learned from.

00:18:45 - Speaker 1: Yeah, a lot of the ideas that one tends to think of in HCI perhaps as as a supposedly novel interaction or approach has actually been tried before. I think it’s very important to understand that prior art, especially if it basically didn’t make it into the commercial world and like, why is that? Or else you’re liable to make the same mistakes again. Um, another example that I’m thinking of was the study. Of so-called folk practices with computer programs. This is like little habits or techniques that people have picked up to make themselves more productive with programs, and they found two examples.

One is lightweight version control by making copies. So if you’re in, if you’re editing a photo and you want to, you know, have some quick version control.

Uh, you might, uh, duplicate the item in your canvas, like in Figma, you know, make another copy of it, and then fiddle with the new version, and then you can kind of compare it to the old version, even if you don’t have like a, you know, get for Figma or whatever.

Um, another one was this idea of everyone likes to have a little scratch space where you can like put, you know, your little clippings and bits and things you’re working on, and that was one of the inspirations for. the shelf in the original Muse prototype.

00:19:47 - Speaker 2: Another book we both read around that time was The Science of managing our digital stuff, and they had a lot of insights, again, things that I think we borrowed from a little bit from Muse, but because they come into it from this ethnographic or academic perspective, they just want to learn, they want to collect the data, they want to understand users. They’re not coming in with the point of view of like, we have a product we want to sell you or or just a uh A product we believe in and we’ve already bought into the mindset of, they just want to learn.

And so one insight there was people who have been designing file systems, that is the way we store documents on our computers for decades have talked about the hierarchical file system, that is to say, folders that nest inside each other, uh, is no one thinks that way and hard drives get messy and no one wants that, maybe we want a tagging system, I think BOS had a version of that, um, maybe we want fast search or whatever. And these folks just did a bunch of studies of people including how they use Dropbox or Google Drive or their own hard drives or just the way they manage their files, and pretty reliably, people like putting files in folders. And they like pretty shallow hierarchies and they can remember where it is and it’s best for them if it’s only in one place. And you can sit there and talk about how that’s not the best solution or whatever, but they, they did a pretty broad survey and just saw this is what people want to do despite the existence of other ways of doing it and the other kinds of solutions, including search and tagging and so forth. At some point you have to acknowledge the reality of this is how humans behave, and even if we don’t like that behavior, we need to think about that when we build tools for them.

00:21:27 - Speaker 1: Yes, if you’re contemplating doing a search-based or tag-based information management system, please read this book. It’s, it’s super critical.

00:21:35 - Speaker 2: There’s an interesting tension there between, I think the academic world. is not only good at, but is science is essentially built on prior art and you’re building on what came before, right? Any paper that doesn’t start with a survey of other research that this is built on or related to or other people have tried similar things, and you’re you’re extending the tip of human knowledge, hopefully, by building on everything we already know.

Um, and so for that reason, the academic world is very good at the the prior art thing. And maybe the startup world is all about, hey, I’m a 24 year old that doesn’t know anything and I’m totally naive, but I have this wild idea for a thing I want to build, and 99% of that at the time, that turns out to be an idea that a bunch of other people tried, it doesn’t work and fail for all the same reasons as everyone else does, but 1% of the time it turns out that some assumptions about the world have changed, and it is that naivety, it is that. Not looking at why people failed before that it allows you maybe to find an opportunity. So there is, there is a bit of attention there, but sometimes the um I’m very appreciative of the what people have thought about this, they studied it in depth, there’s a lot of prior art here, like look that up before you start building things, um, and I think that that would be advice I would give to my younger self, I think at a minimum. Alright, so that gives us a little bit of the landscape of of HCI. Now the next part of the question was, how do you actually get into this field? I think that’s kind of a tough one, so I’m gonna actually say that for the end. Uh, but in the meantime, there was a follow on question here and Fetta says, how do you forget or ignore current patterns and come up with new ones? You have some thoughts on that, Mark?

00:23:14 - Speaker 1: Yeah, I come back to this first principles idea of really understanding the basis for all of this, the biomechanics, the cognitive science, the computer science, and then understanding the Um, assumptions or lemmas, uh, of the current design paradigms.

So, you know, for example, Uh, one thing we see with with phones is most apps are designed for only one finger to be used at a time, and it would be a mistake to translate that design constraint or design decisions over to a tablet, we think, but a lot of apps just kind of blindly do that do that because they’re both iOS and they’re both touch apps.

Um, another example even more relevant to use is the pencil. A lot of the gesture space of tablet apps can’t assume that the user has a pencil because Apple and the various app developers just aren’t willing to make that assumption. Uh, with, with muse, we realized that was, uh, assumption that people were making and one that you could take the other side of. So we’ve basically said you really need a pencil to use muse and therefore we’re gonna have some of the functionality behind that, you know, that, that, that physical gesture.

00:24:15 - Speaker 2: Yeah, the status quo is a powerful force for all of us, and we, we tend to act on not quite habit, but this stack of assumptions about the world and what the right way to do something is. And here’s where I like to think in terms of maybe a spectrum between on one far extreme is the research thinking, the out of the box, wild ideas, weird ideas, when you go to one of these HCI conferences, this is what you see a lot of just Sometimes frankly pretty wacky mad scientist kind of stuff. Now, um, but actually there’s only certain times where that is appropriate and in fact, doing research is a place where that is appropriate. Typically, if you’re making a product that you expect people to use in the real world, it’s actually a bad thing to have weird out of the box ideas, particularly about basic interactions. You want the status quo, you want the known path, they usually called the best practice.

And I’ve certainly run into this on. Teams where I don’t know, you’re building a basic e-commerce site or something like that, and there’s someone there that wants to do something fun and exciting and so they’re like, and so they say, why not, let’s try this wild idea, you know, instead of checking out like this, you you do this crazy thing and 99% of the time that’s just a bad idea. Please do it the way that other people do it.

And this is one of the things that I think tends to make software so high quality in the Apple ecosystems, both Mac and then even more so on iOS is you have this pretty stringent set of, you know, they call it guidelines, but in many cases are just outright rules to get your app approved.

They have this very extensive culture and set of principles and so forth in the human interface guidelines and in all the precedent with Apple apps and the wider ecosystem there. It’s all really good and it all hangs together and it works well and people know how to use it. And so most of the time you actually should do the boring, expected common known path thing. And it required, but it’s a shift in mindset, a fun one, but, but also takes some stretching of the brain, you challenge yourself a little bit to go into the research thinking mindset as both of us did, we went to to Ink & Switch.

00:26:29 - Speaker 1: Yeah, I think that’s an important point and a balance to strike. Another big source of inspiration for me has been the world of analog tools.

We’ve been thinking about how to build good digital tools for maybe 50 years or so. We have a couple of 1000 years of explicit and implicit study of how to create analog work environments, so things like personal libraries, uh, studies, uh, workshops, artist studios, in some cases, there’s explicit treatises about how you organize one’s library, but there’s also just a huge amount of implicit and embedded knowledge in the patterns that we use every day and that people have kind of habitually used to organize, you know, say the library.

So I like to look at the, the physical world and see, how can we just like, as a baseline, make it as good as that. So a simple example would be, if you use ink on a pen, it has zero latency. If you use ink on a really good tablet app, it might have 15 to 20 milliseconds, which is a lot. And if you use it on a bad tablet app, it might have 50 milliseconds. Um, so that’s a really basic example of how there’s a, there’s a simple bar to set. Uh, another one that I think about a lot is multitasking. So if you have a desk, and you have your main piece of work in front of you, and you have some notes to the side or uh up on the top of the table. It’s super fast and easy to multitask your attention, just like you kind of move your eyes or you move your neck and your eyes re refocus, maybe you lean into one side or the other, um, but it’s it’s super fast and lightweight. What you think about a typical iOS app, it’s like, you know, press next page, transition animation, spinner, loads, fonts come in, right? And so it’s it’s very discouraging to actually do this kind of multitasking work.

00:28:06 - Speaker 2: And maybe the flip side of that of taking physical world information practices, things from artist studios and offices, file folders. Scissors, rulers, pencils, desks, you do tend to get, especially the first time an analog process comes on to is digitized.

So you think it’s something like desktop publishing going on to computers in the 1980s or yeah, word processors was taking what was a typewriter or a typesetter and moving that onto the screen, spreadsheets that were that way, um maybe PowerPoint, uh taking overhead transparencies, bringing onto the computer in the late 80s, early 90s.

In all of these cases, they tend to be very literal. Like the first version of PowerPoint was a way to print out overhead print transparencies, and it wasn’t until much later that the idea of a slide deck that would be all digital and you would never need to print out and put on a projector, uh, showed up. And then often you when you look back at these first transliterations from the analog world to the screen, you see this thing where it’s, oh, isn’t this funny? You know, there’s the little, the little picture of the trash can and a little picture of the Um, you know, often very literal and kind of heavy handed and not taking advantage necessarily of what can be done in the new medium. Do you have a, I don’t know, a sense for the how we take the best parts and the things that work about the physical world, knowledge tools that we’ve been working with for so long and are so adapted to human needs, but not also get stuck in a weird rut of translating them directly so that we don’t get the benefits of the computer.

00:29:39 - Speaker 1: Yeah, I don’t think there’s a simple rule for that, but again, I come back to the fundamentals. A lot of the stuff is driven by the like the biomechanics or the cognitive structures of our mind, which isn’t going to change. So for example, we have a very realistic, deeply embedded expectation that when we like touch something and move our hands that it moves, and that I think is basically not going away, and it would be a mistake to think it’s going to go away. Uh, likewise, I think we have quite embedded cognitive arch. texts around both spatial memory and associative memory. I think those are basically baked in and they’re not going to go anywhere.

00:30:10 - Speaker 2: I guess that comes to mind because I feel like that tension or it’s not even the right word for the interleaving of try to draw the best parts of the physical world workspaces, but also really embrace this digital space and it’s part of the pitch, I guess, or the the value hypothesis for use as a product is that.

We are going to take taking something you previously did with Post-it notes and your whiteboard and your notebook and some printouts of some screenshots that you scribble on that are on your desk, and moving them into this expensive and fragile computing device. That it will have new capabilities and new powers that you couldn’t get. And so getting bringing those best parts across, which is, for example, that yeah, you touch something and it moves right away and there’s this instantaneousness to it, and then you’re not like looking at spinners and loading screens and whatever, um, but also taking advantage of all the Um, incredible capabilities and the great depth of possibility that exists within once you move to the digital virtual workspace.

00:31:18 - Speaker 1: Yeah, one idea for an exercise here and this kind of gets into our next question would be just to try to understand and catalog the properties of these physical workspaces that are interesting. So for example, I have a desk here that I think is 6 ft by 3 ft.

00:31:32 - Speaker 2: For our non-American listeners, that’s probably about 2 m by 1.5.

00:31:38 - Speaker 1: Yes, thanks, Adam. So you have this desk and imagine it’s covered with like textbooks and notes and photo printouts at, you know, say 200 DPI. What’s the resolution of that? And if you do that exercise, you’ll see that it’s like massively bigger than even our most advanced displays, it’s not even close, and just being kind of aware of those basic fundamental properties of the physical world and how they might or might not be reflected in your app, I think is a good baseline.

00:32:02 - Speaker 2: So we mentioned academic HCI work, which tends to happen in universities and funded by grant money and the output is published papers, and then there’s corporate R&D which is divisions, separated divisions, but still departments within some large company that has a lot of cash, like a bell, or a Xerox or a Google to throw at potential new innovations, but there’s a third category that Or at least I hope it’s a category now, uh, that it’s much more rare, but I can switch falls into this, and that would be the independent research lab. And the hypothesis behind I and Switch was what if we take the corporate R&D lab, but we cut off the corporation. And this quickly leads you into how does this stuff get funded and our um.

Our mutual friend, Ben Reinhard has a whole series of excellent articles about how innovation happens and particularly the different kinds of funding models that can happen and how it gets funded in turn leads into the incentives of the people doing it and there’s quite a, quite a rabbit hole there for those who are interested in it. But the concept behind it and switch was that we could get some grant money to do independent research. With the idea that it would generate called intellectual property. I don’t love that term, but basically, ideas that could potentially be commercialized and ideas with enough depth to them and research, and where we falsified ideas that were no go, and we had some really compelling ones. One of those turned out to be Muse, which we we went ahead and spun out to begin the commercialization project process.

But there There are a few others that I know of that are independent labs. One is um Dynamicland, which is sort of Brett Victor’s effort to bring computing and programming in particular into a more spatial, a physical spatial environment, not just on a screen.

And then another one that I know of is um maybe more in its nascent stages, but Andy Maze has done amazing work on mnemonic devices. And he’s, I think funding and stuff maybe started with Patreon and maybe led up to institutional funding kind of more of a kind of a, what’s the word for it, a nonprofit, more of a philanthropy type approach. But I think there’s no great answer for how independent research can get done, but I at least I hope that I could switch is an interesting example, if not role model for others that might want to see how they can push the frontiers forward in a particular space.

00:34:24 - Speaker 1: Yeah, that’s both the challenge and the promise of this third type of institution on the academic side, you’re very limited by your work has to fit in the box of like a peer reviewed quantifiable research paper and in the commercial world, it needs to be commercializable in the next, you know, probably a year or 2, maybe, maybe 3, but all the good ideas don’t fit in one of those two boxes. As hard as it is to collect them with this third organizational type, I think it’s worth trying.

00:34:47 - Speaker 2: It’s a great point. I think the time horizon is one of the key.

Variables, let’s say that defines what I would call research for for anything, but certainly for human computer interaction, which is, um, I believe Xerox Park actually had an explicit time horizon of 10 years. Which is definitely way beyond what a commercial entity would normally do. Um, and I think, you know, basic science even has a longer time horizon than that sometimes.

But yeah, when you look at maybe university labs, they’re thinking forward really, really far, um, maybe corporate R&D labs are thinking further than their commercial counterparts. And then if you talk about a startup, particularly something. combinator, you’ve got to build that MVP, get it to market, validate it, get customers. You can’t be building it on some shaky technology that one, you don’t know if it’ll work, and two might take many years of development yet to come to come to enough maturity that you can base something that people really want to build a product that people will depend on.

00:35:44 - Speaker 1: Yeah, and I also think you get a bit more wildcard energy in these independent orgs, you know, the, the academic institutions and the, the big commercial labs are just necessarily more constrained and structured, and you can have just more eccentric people doing stuff on the independent side, which sometimes leads you down weird dead ends, but sometimes you get really interesting results and it kind of injects a new idea into the mix.

I’m actually we talked mostly about like independent research labs or research efforts. I also consider like indie creators, artists, tinkerers in this bucket too.

One example that comes to mind is that the video game Braid, which is this amazing like time traveling based game where the time traveling is like very smooth and scrubbed frame by frame. Um, that’s actually been something of an inspiration for me thinking about like version control and time travel for productivity tools.

00:36:33 - Speaker 2: Yeah, I think that’s Jonathan Blow, and he also went on to make.

Other like category breaking games, uh, trying to remember the name of it, there was a puzzle game that was actually really nice on the the iPad that I played with my girlfriend at the time.

And then if I’m not mistaken now, he’s working on inventing a new programming language.

So yeah, so that the, uh, maybe it just takes a certain mindset, a desire to perhaps even a um a drive to think outside the box and do weird stuff.

And yeah, I certainly agree that Labs depend on weird, wild, I think I saw the word maverick used quite a bit when describing um there’s this book called Dealers of Lightning, which I think covers, covers Xerox Park and and kind of those glory days pretty well, and it talks about, yeah, there are these, I don’t know, kind of long hair types and, you know, don’t wear shoes in the office and of course those aren’t the qualities that make them good researchers, but it’s connected to this.

Maybe desire to do a weird thing to not conform to try stuff at the fringes, to be actually fascinated by things that are at the fringes, as opposed to, this is weird, who cares? I want to work on something more mainstream, let’s say, um, and not to say that that’s a better or worse approach to bring to your work, uh, just that it, it fits in a different space in the innovation cycle. Well, maybe that brings us around to the core of the original question, how do you get into this field?

00:38:04 - Speaker 1: Yeah, and I, I feel like there might be two different questions embedded there.

One is maybe how do you participate or contribute or even just kind of find, find out what’s going on, uh, and the other is how do you make a living doing it.

And, uh, I, I think making a living doing it is, is harder, but it’s maybe simpler to answer. There, there are two main paths right now. There’s the academic path and there’s the corporate path. Um, the academic path you you basically you go to graduate school and you get a PhD. Uh, but even after that, it’s, it’s quite challenging just because it’s so competitive in the corporate path, you become a practitioner and you, you do good, you know, engineering or product work and eventually you can enter this more researching ladder. But I’m not sure we have that much to contribute on that front because neither you or I have gone down those paths, maybe more of the how do you engage with the community where we should focus here.

00:38:43 - Speaker 2: Yeah, absolutely. Well then, you teed up really nicely. How should we engage with the community?

00:38:49 - Speaker 1: Well, step here I would say is start digging into the literature, you know, it sounds obvious, but I think a lot of people haven’t done this either they don’t realize it’s there or they’re intimidated by it. Um, but this reminds me of Rich Hickey’s classic talk, hammock driven development. He’s like, if you’re working on something like you. I think you need a hash function that does X going to Google Scholar type hash function that does X enter and see what comes up. Like there’s almost certainly going to be something there.

00:39:12 - Speaker 2: Well, maybe there’s a great chance to talk about something again.

I coming purely from the what what academics would call the industrial side, uh, yeah, working in companies that build products that they sell to people. That’s what I did my whole career.

And so things like the fact that all this academic work tends to be published as PDFs in a particular format, there’s a lot tech to formatted to column PDFs, they have a particular style of writing, they have this particular style of citations, you typically, they’re not always open access, but when they are, they’re PDF on a web page, and the search engine for them is something like Google Scholar.

I I actually didn’t know that. I didn’t know how to go find those things. And so as a Let’s say as a product developer, designer or engineer, I knew how to Google for stuff. I know how to find stack overflow. I read medium pieces, I read people’s blogs, I follow other folks in my field on Twitter, but the academic world of things was sort of a dark, yeah, was dark to me, except for on occasion, I would stumble across a book like the one you mentioned earlier, a small matter of programming.

And I feel like I discovered this incredible trove of knowledge from someone that came at the the problem space from a very different perspective.

And I think it also goes the other way, not as much, but I think academics are less likely to read the medium think piece posted by the product designer, the engineer, and basically the two, I think the two communities, if that’s the right way to put it. Uh, have different communications conventions and different ways that they share knowledge with each other and different systems for evaluating. Uh, importance and so on.

So it’s very hard to, um, if you’re, if you’re steeped in one, it’s hard to cross the world into the other.

So maybe that comes to all right, you find some hooks into this, you can follow some people, whether it’s on Twitter, whether it’s through their personal blogs, you can start to find some papers and Google Scholar on the topic, you can find some slack communities maybe that talk about this stuff and you can try to get hooked into it and and. Again, if you’re someone that comes from more the practitioner side, we might say, engineering products, design, uh and you haven’t been exposed to the academic side, going and and exposing yourself to that is a very good idea and maybe vice versa.

00:41:30 - Speaker 1: Yeah, and one other thing I would emphasize there is that you can do this citation crawling practice where you find a paper that you’re interested in, you can go look at the, the references, and this will refer to a bunch of other papers and sometimes books and in HCI it’s mostly papers, there are a few books, and then you can type those titles into Google Scholar and follow them that way. And a good way to kind of know if you’re getting your hand around the literature is if. When you read a new paper and like you basically recognize most of the citations or they’re kind of off the edge of your um your map in terms of your area of interest.

So you’ve kind of identified the full graph of relevant papers and then you’re, you have a good handle on the literature.

00:42:05 - Speaker 2: And I think this is something that’s very much you learn this in the academic tradition, which is if you want to advance the state of the art in a field, first you need to know all the things that humans already know.

And you do that by consuming all the literature, and you know when you’ve consumed all the literature exactly the way you described, kind of a crawling process, which is you start with a few seminal papers or you start with a few that are your starting point and you follow all the citations until you get to the edges of it and you feel like, OK, I’ve filled in this space now I know. in some kind of um general sense, what humanity knows about the subject. And now if I am, if I have novel ideas or I want to do new research or I see open questions that stand on top of this, now I can go do that in order to potentially contribute to this.

00:42:52 - Speaker 1: Yeah, and then speaking of taking that next step, it can be intimidating, certainly if you want to jump all the way to publishing in a peer reviewed journal, but I think you can take more incremental steps.

One example that comes to mind is Dan Lew’s work on latency in computer systems. Uh, he did a series of measurements and experiments to assess uh the different latencies like from your keyboard. Your monitor for when you move your mouse to something happening, and uh but he was able to publish this on his personal website, and it’s not an academic peer reviewed paper, but it’s, that work has been quite influential, and you can indeed reach the kind of the caliber of academic work, even if you’re not participating in that full pipeline.

00:43:29 - Speaker 2: I’ll note that um work, and if I recall correctly, it’s published on kind of a really basic HTML page with very limited formatting and and whatever feels very um homegrown and authentic.

But one of the things he does that’s so compelling is he says, he starts with this hunch, which is computer seems slower than I remember when I was younger, but then he goes to, you know, maybe the way if you don’t come at it from that scientific rigor position, you might go, you know, computers seems slower. I’m gonna like make some snap judgments. And then I’m going to go write a blog post and complain about it. But what he did was say, well, are they actually slower? And he got a, I don’t know, some kind of high speed camera set up and set that up and pointed it at the keyboard and the screen, and he recorded himself pushing a key, and then you can see on the camera when it appears on the screen, and then he, he wrote down exactly to the millisecond and he did that with a whole bunch of different devices, including some computers dating back to the 80s and then he put them all on the table and sorted them in order.

And that’s a simple. Application of the scientific method to in this case, a very literal human computer interaction. How long does it take when I press a key when it appears on the screen? And that doesn’t say how long it should take or what would feel right, but you can put now real numbers to this intuition that maybe computers are more sluggish than they were at a different time.

00:44:55 - Speaker 1: Yep, exactly. And then if you are looking to take that step towards uh participating in these peer reviewed journals, a possibility that we’ve had some success with is collaborating with an established academic in the space. Um, Adam, you’ve kind of spearheaded our collaboration with Martin, maybe you want to describe that.

00:45:12 - Speaker 2: Right, well, we were lucky enough to get to work with Martin Klepman, who’s a one of the world’s experts on, say data and data synchronization, particularly around another track of research we had in the the lab around um what we eventually called local first.

And he is someone who was in the indust, let’s say the industry world, he was doing startups and at some point felt that he can contribute more to the industry or the world by jumping over to the academic world to do more basic research around algorithms having to do with um synch data synchronization.

And so we were lucky enough to get the chance to work with him within the context of the you can switch lab on a kind of a light part-time basis.

And that led pretty naturally to, OK, well, we want to write a piece and publish it. And he wanted to publish some of his findings and he said, hey, you know, I think this could go into the academic format.

And I said, well, Well, how does that work? He’s like, well, basically we take this web page we wrote, we put it into a lot of tech, we change some of the wording to remove, make it less emotional, uh, we changed the links into the citations where that makes sense, and we, we had a whole process to make it into something fits this format that’s expected by the academic world, and then we submitted it to a conference, uh, where it was accepted and eventually I actually ended up going to present it for.

Um, various travel logistics reasons. Um, but yeah, that was a very interesting experience because the four authors on the page, uh, the paper, I think you and Peter maybe both have a good bit of academic experience, although I don’t know if you’ve published that way before.

Martin is extremely good at that stuff, and then I knew very little about that world, but working with someone that knows all the ins and outs of it was a very um rewarding way to to learn about it.

00:46:58 - Speaker 1: Yeah, exactly. And to be clear, we didn’t just jump right to that, you know, a collaboration with one of the world’s leaders in synchronization technologies. There’s a little bit of a.

00:47:09 - Speaker 2: Yeah, don’t email Martin and ask him whether he’ll write a paper with you, he doesn’t know who you are. That’s not what I’m advocating for.

00:47:15 - Speaker 1: There’s there’s a bit of a proof of work function here where if you do some of your independent research in the space, and especially if you publish something that’s coherent and compelling, it becomes much more.

You know, reasonable to establish a collaboration. Actually, when we did some of our publications around Muse and our latency measurement work, we had a few academics reach out to us and you know, say that’s interesting, maybe we should, you know, do some work together. I don’t think we’ve brought any of those yet to the point of writing a paper together, but it just shows that once you have some, some work out in the world that shows that you’re serious, that you’re engaging somewhat in the academic tradition that you’re aware of the literature, that you have contributions, um, it becomes a more feasible to have those collaborations.

00:47:54 - Speaker 2: Yeah, perhaps like any other intellectual or maker or tradition, this is a world or a community or a society that thrives on seeing what else you’ve done, and if you see that someone has done great work that overlaps with work you’re interested in, and that creates opportunity to connect, to learn from each other and then maybe lead to, can lead to collaborations.

And yeah, maybe it’s not such a huge leap from do a weekend hack project and write up your learnings about it to eventually doing something a little more deeper and a little more serious that brings you in the direction of um the academic recognized academic world.

Well, it’s interesting to note then that In doing the research lab, we came to it not from the perspective of how do we become a part of HCI, but rather we just wanted to see computers and computing interfaces get better uh in in some particular ways that led us to doing maybe some interesting experiments that led to some novel research that we we published about, and that in many ways opened the door to us to be more connected to this larger academic field. Is that something that a path you would recommend for others?

00:49:05 - Speaker 1: Yeah, there are certainly interesting paths there, you know, there’s this independent research lab path, and of course, there’s the academic and commercial path, and I think those are all interesting.

I would also say though that being a scientist or being an innovator isn’t a hat that you’re granted by some external institution. It’s a way of thinking, it’s a way of navigating the world.

You know, a scientific method is something anyone can use. Publishing is something anyone can do.

Everyone can read the literature. So if you’re interested in this, I don’t feel like you’re, you’re stuck because you don’t have some credential like a PhD. Anyone can step into this world, go on to Google Scholar and read literature, and then maybe you have something to contribute on top of that.

00:49:40 - Speaker 2: It’s hard to think of a better place to leave it there. If any of our listeners out there have feedback, feel free to reach out to us at @museapphq on Twitter or hello at museApp.com by email. We’d love to hear your comments and ideas for future episodes, and big thank you to Fetta for giving us this very uh intriguing and deep topic to explore. I’ll catch you next time, Mark.

00:50:04 - Speaker 1: Great, thanks, Adam.