Transcript

00:00:00 - Speaker 1: There are a lot of other projects that have very similar models to this dynamic land database, but it definitely pushed me to think a lot more in terms of having state exposed by default, ambiently, and the value of being able to make little quick debugging tools that can piggyback on this global state. That was a super influential model on the way I think about programming and the way I think about debugging, this idea of being able to make really lightweight tools or jigs to help myself as I work.

00:00:32 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for deep work on iPad and Mac. This podcast isn’t about used product, it’s about the small team and the big ideas behind it. I’m Adam Wiggins here with my colleague Mark McCrannigan. Hey, Adam. We’re joined today by Omar Rizwan.

00:00:49 - Speaker 1: Hi.

00:00:50 - Speaker 2: And Omar, I understand you have a collection of metro cards.

00:00:56 - Speaker 1: Yeah, I mean, I was just looking at this shelf above my desk, and it turns out I have this giant, basically the only thing on the shelf is this giant plastic pencil case that looks like a giant metro card. And so, I think a lot of people do this, but I’ve just started this habit of just filling it every time I get a metro card or transit card from wherever I go. So now there’s like, I don’t know, there’s a lot, there’s a lot of cards in here. It’s pretty full.

00:01:20 - Speaker 2: So let’s see, what must you have? It’s certainly a Bay Area transit card, and maybe, I don’t know, an Oyster card from London, or what, uh, you know, does this reflect kind of like a travel log of your places you’ve been?

00:01:32 - Speaker 1: Yeah, in a way, it’s kind of a nice, I guess we can connect it to one of the themes, which is that there’s like kind of an object for each place. There’s an octopus card from Hong Kong, there’s a card from Paris. It’s sort of like, instead of entries written down in a book, it’s like I have these like little cards that I can kind of pull out and look at.

00:01:50 - Speaker 2: Nice, and then it’s sort of like, I like the idea of keeping it around because it implies you’re gonna be back, right, that you’re a globe trotting, you know, person of the world, and you never know when you’re gonna need to whip out your Hong Kong transit card.

00:02:06 - Speaker 1: I think there’s also something like comical about the like very large metro card, like very large version of anything. It’s like, uh, you know, a prank we used to do in like middle schools. If people left their laptop unattended, we would just go and make the mouse pointer really big and like not do anything else and just like.

00:02:25 - Speaker 2: And you are an independent researcher with a very diverse set of interests, lots of things that overlap with the niche interests that Mark and I, and I think a lot of the listeners have, including end user computing and embodied computing, file systems, vintage computing, and so forth. But why don’t you give us a little bit of a summary of some of the stuff you’ve worked on over the years and where your interests in the computing world lie.

00:02:51 - Speaker 1: Yeah, I mean, so my background is mostly, you know, in programming, you know, I learned to program very early, and I sort of got interested in, like, new ways to interact with computers. Like, when I was a teenager, there was all this stuff on, like, building your own multi-touch table, and then I kind of got involved with Brett Victor’s work at Dynamicland, but also did a bunch of other different projects, kind of in that space, and just in general, I’ve always been interested in like, Different ways to interact with computing, both like future looking and also historical, like, what are their operating systems that people have done, what are other interfaces that people have done. And so, that’s my background.

00:03:28 - Speaker 2: And I feel like just looking down your portfolio the right way to describe your list of of projects, your research provocations, perhaps they’re quite varied, but they seem to have in many cases a sense of less of a like, here’s a, I don’t know, a library you’re gonna use or an application you’re gonna use and more of a Almost like an art project element of like, let me make you think a little bit here.

For example, one kind of near the top, at least at the moment is hijack your feed, and if I’m not mistaken, this was one you did together with uh Jason Yuan, is that right? Yeah, yeah, who we’ve had on the podcast before as well. And yeah, I feel like that’s as much uh asking questions about social media feeds and the place they fill in our life and how we can like take a little more control of our computing world.

But then you’ve got, for example, TabFS which mounts the open tabs in your browsers as files and lets you basically do, you know, The kinds of shell programmatic things that you can do with normal files, but with sort of your web browsing kind of history or current open topics. So I’m not sure how much, I mean, maybe I don’t know, Tabafest is in quote unquote production and you have people using it for serious things, but as I look down this list, I feel like they’re more of a, yeah, again, it’s just kind of like art project to make you think and question assumptions about the status quo in computing. Is that a correct conclusion to draw?

00:04:52 - Speaker 1: Yeah, I think that is a lot of them. And that’s also, I think a lot of what I do, you know, on Twitter or in my writing, or whatever, is sort of try to provoke people or come up with these really striking images. And I think, like, I’m often very skeptical when people sort of try to articulate this like philosophy of like what computing should be or, you know, explicit tenets of these are the things we want. I’m much more on the side of like, we should have a few very striking, like, concrete examples of like, things you might want to do, or like, interactions that are possible, and then those will kind of drive people in a certain direction.

00:05:28 - Speaker 2: I think the project of yours that was the first one I ever came across was Screenotate, which is essentially it seems to combine a couple of your interests here, including Provenance and OCR, but essentially it’s a screenshotting tool that makes it very easy to grab the text out. Now I’m not sure how much the latest changes in MacOS and iOS where there’s some of that built into the OS. Well, maybe we’re even inspired by what you did there, but that is well’s product. You can download it, you can pay for it, presumably you’ve been maintaining it for a while, so it’s not pure research in the sense of, and I use it also, you know, all

00:06:04 - Speaker 1: the time, that’s key, like I’ve probably taken 30 or I’ve probably taken like 40,000 screenshots in it, so, wow.

Yeah, and I think there is, with a lot of the projects you’ve mentioned, there’s also this theme, and this gets at this idea of folk practices a little bit. There’s this theme of like, this is very vague, but like, connecting different universes in unexpected ways, like this idea of like, there’s your browser and your file system, and you jam them together, or there’s this like social media interface, and there’s this idea of tasks or productivity, and you jam those together, or even the dynamic land stuff, I think, has a little bit of this, like, there’s the objects in your computer and there’s the objects in the real world. You kind of try to Combine those in some way where you can use operations that you are familiar with on one and apply them to the other. And I think that also something that connects really well with people, because you’re sort of familiar with both sides, and so you kind of immediately see the combination of them, and you’re like, oh, this is really cool or really interesting, or like, I can quickly imagine how it would apply, you know, to my life in some useful way.

00:07:06 - Speaker 2: So you have the anchor points of the two things that you’re familiar with, and the novelty or the provocation, or the picture of what could be comes from thinking about how those two would combine.

Yeah. So our topic today is folk practices, and this is a term Mark and I use quite a bit here on the podcast and even on our team as we talk about ways to look what people do naturally with existing tools or existing features.

In, you know, a product that we or others are building and then sort of extract from that what they’re trying to do and in many cases you can even shape a product or a set of features or an operating system to embrace those folk practices.

And I think Screen notate, the project we just mentioned, is one good example of that because the idea that like People sometimes complained, screenshots full of text, this is so annoying.

Why not have the core text, you’re spending way more data to represent it, you can’t reflow it or do other things you can do with the text, and to some extent, Folk practices, I think is a saying like, look, screenshots of texts are really here to stay, and there’s a bunch of reasons why that might be, but just empirically, this is a thing people do and they do a lot. And so maybe we should learn from that and find out how to kind of roll with it. Like, if you can’t beat them, join them kind of thing, rather than kind of, you know, basically complain that you’re not doing it right.

00:08:32 - Speaker 1: Right, or at the very least, you might not join them, but at least you should look at it and be like, OK, why do people do this, rather than lecturing people about, you know, you should do this other thing instead.

00:08:43 - Speaker 2: We were at a conference together recently and you did a little demo to the group, and this is called Screen Matcher. Can you tell us about, yep, that one?

00:08:52 - Speaker 1: Yeah, so this is a project that I’ve been working on a little bit this year, and basically the idea is it’s this Daemon, it’s this app that sort of runs in the background of your Mac continuously, and it’s constantly watching your screen, so like the screen on your computer.

And so this screen matcher, you can teach it to look for patterns on your screen. It’s like you’re taking a screenshot, like, you drag out a region of your screen, and then you kind of feed that torematcher, and it’ll look for whatever you took a screenshot of from then on.

The example I usually give is like, you know, in the corner of every window on your Mac, there’s these traffic lights to like close, minimize, and maximize. And so you can teach Screen Maer to look for that pattern, it’ll find it wherever it sees it. And then you can draw on top of it. So, effectively what that means is you can add like a 4th or 5th button to every window on your computer.

But, you know, there’s a lot of other things you can do once you have this kind of continuous screen matching mechanic.

Like, you can kind of just like add buttons or draw or scribble on anything on your machine, and have these like automatic behaviors.

So the other example I usually give is like, with the screen matcher, you can build like an alarm clock without traditional programming, because what you do is you’d be like, OK. I want to wake up at 7 a.m. tomorrow. So you’d set the clock of your computer into the future. You’d be like, pretend it’s 7 a.m. tomorrow, and then you tell Screen Matcher, hey, when you see this pattern in the top right corner of the screen, when you see it say 7 a.m. I want you to play a sound and wake me up. So there’s this idea of like, you can extend the functionality of your computer in a very natural way, and there’s this idea that you can do things you might normally take like programming or scripting or whatever, just by pointing at your screen.

00:10:25 - Speaker 2: It reminds me a bit, especially that example you gave there of an if this then that or a ZAPA or something like that, which do have this element of automation without real programming, but those really rely on APIs.

So you need to have an API integration that that means that the vendor, the creator of whatever the thing is, in this case would be the clock or the operating system or whatever needs to supply an API that you can consume through some probably fairly complicated procedure.

And I feel like a hypothesis or a concept that’s embedded in this project and maybe some of your others is to sort of say, well, look, it’s nice to have APIs on things, but realistically the output from computers is pixels on a screen. So if we want to give some kind of end user programming capability, basic automation, rather than trying to browbeat program creators into creating an API, just sort of give up, and maybe give up isn’t the right way to put it, embrace that folk practice or embrace that reality that That GUI interface exists and by the way, computer vision is really good now, and so something like recognizing the widgets in the corner of your window or a clock value is actually relatively straightforward, so therefore maybe that should or could be the sort of an everyday API.

00:11:45 - Speaker 1: Yeah, I think that’s right, that there’s this, instead of this closed world of whatever is available via API you have this open world, much like when you take a screenshot, you know, you can take a screenshot not just of things that are selectable text, but if anything on your screen.

Similarly here, you know, you can automate based on anything on your screen, not just things that happen to be an API.

But I think there’s also kind of like an interaction argument for this, which is that Even if you have all the APIs available from the end user point of view, it’s like, OK, I want to do this automation. I guess I have to like read the, like, dictionary of APIs and like figure out what the right APIs are, or if they’re even available, I have to figure out like what kind of input and output they take, and that it’s always felt to me like very disconnected from the actual experience of using the computer. Like, you know, if I want to make an alarm clock, why can’t I like point at the actual clock on my screen, instead of figuring out that there’s a clock API that’s like based on the same source as the clock on the screen. Like, it feels like you should be able to point at the actual things that you’re already familiar with, instead of having some like API dictionary that’s completely separate, that feels like this like skeleton of the app.

00:12:50 - Speaker 3: Yeah, I really like it. And Omar, so the idea with Screen Matcher that you can both sort of scrape the screen for input, but then also do, I guess you would call output of typing things and clicking things, moving the mouse around.

00:13:03 - Speaker 1: I think so. You know, the current prototype, basically what you can do is you can just add but so you like can search for a pattern and then you can be like, every time you see this pattern, I want you to draw these extra scribbles next to it, and then when I click one of these scribbles, I want you to run a bash command.

I see, I see, but I think it’s very easy to imagine being able to have other responses. To seeing things on the screen, whether that’s like playing a sound. I mean, someone proposed to me that you should have all the effects happen by drawing stuff on the screen and then Screen Matcher would like match those things and do the effect directly.

Uh, I don’t I don’t know if that makes sense, but it has like a very nice, like, kind of aesthetic elegance to it.

00:13:40 - Speaker 3: Yeah, I also kind of like the baseline of anything that you can do as a human, whether that’s things you can see or inputs you can do with the keyboard or the mouse, you can script. Yeah, that seems like a reasonable invariant. Yeah. And as far as that’s a floor on automation, so no matter how hard the programmers try to deny you the ability to have agency over your own environment, you can’t take away my eyes and my hands. And therefore, if I can control those things, you know, basically I have scriptability of them.

00:14:08 - Speaker 1: Right, right. You could imagine if you wanted something that could deal with keyboard shortcuts, like, let’s say every time I hit like control 9, I want the computer to send an email or something.

You can imagine a plug-in that actually maps your keyboard into like a larger like screen space. So you have your actual screen, but you can imagine you have a bigger virtual screen and you like map your keyboard into it, and there’s like a virtual keyboard on the virtual screen that lights up when you have keys. And so you can sort of imagine mapping any sensor or actuator if you go the other way. Into screen space, and that would kind of make this like an entire programming system in a sense, cause you’d be able to address any kind of IO which, again, I don’t know if that’s useful, but it’s kind of like a cute idea, and I think this is like an interesting programming model, and it is in some ways a lot clearer than traditional programming, because like, if it goes wrong, you’re like, OK, it didn’t match the right things, like, that’s why it didn’t work.

00:14:57 - Speaker 2: Well, well, almost by definition, everything is what you are seeing, the computer is also seeing, and then what you are responding to that with by yeah, drawing something else or playing a sound or something like that.

I mean, that’s one of the things that makes programming so incredibly difficult. It’s obviously very abstract, but the connection between the set of symbols and the thing that’s actually gonna happen as a result of it is so disconnected, and that’s what makes kind of professional programming, professional software engineering.

Particularly really complex systems, you just have to model so much of what the computer is doing in your mind, that’s almost the hard part of it as opposed to just expressing concepts and symbols, for example, right?

00:15:37 - Speaker 1: And here, I think you sort of by default, get this ambient awareness of what the computer is doing. Which I think is something that’s also true of dynamically to some extent. I mean, I think that’s something that’s true of a lot of interesting programming systems, is like, you don’t have to go in and like, inspect what the computer is doing because your program didn’t work. You just like, look at the screen and you’re like, oh, that’s why that didn’t work, even though it may be a little wasteful from a sort of traditional programming point of view to be running all this state through the screen.

00:16:03 - Speaker 2: Mark, your earlier point about they can’t take away your eyes and your hands reminded me of another dimension of folk practice, which is what’s usually referred to as the analog hole when you’re talking about DRM digital rights management, where, OK, we’re going to give you this music, you can download this music and listen to it, but you can’t copy it, for example, but in the end, you can always basically just like take a recording device and hold it up to the speaker, and that’s the analog hole that no matter what you do with the computer.

And screenshots are, I think, an even more pervasive and useful version of that. It actually happened to me just the other day. I think someone sent me a PDF maybe a financial document. I can’t remember what, but I need to copy paste something small out of it and I don’t know, the PDF you or said something like, oh, you have to have the master password to unlock the whatever to copy paste, and I’m like, cool, man. And then, you know, took a screenshot and immediately use the OCR to just like copy paste it out, right? There’s a version of this in the Kindle app and whatever, and they’re just working so hard at it, but like in the end, it’s like I’m looking at the words on my screen. In a really worst case scenario, I could just manually type them out if I wanted to. And so it feels like a lack of acknowledgement of the reality of I’m looking at it and part of what my computer can do is manipulate images. So how in the world are you really going to stop me or anyone else? It feels like a weird denial of reality. Now talking about the debugging visibility that you might get from, for example, an on-screen keyboard or just the fact that all of these things are flowing in the, let’s say the concept that’s suggested by this project that you sort of see everything and that visibility is going to make it more approachable and more comprehensible to sort of non-professional software engineers. I noticed one of the notes or prompts you put into our little shared notes document here was whether visual programming was overrated. I feel like those are related. The appeal of visual programming is if you can see everything, it becomes more approachable and more comprehensible, but it seems like you have some feelings on that subject.

00:18:09 - Speaker 1: You know, I think there is a notion of visual programming, which is like, you put together blocks on the screen, or like boxes and wires, and I think this like, Uh, especially blocks, I think that like doesn’t really have that much to do with the kind of visual programming that’s suggested by the screen matcher, because in the Srematcher, the visual things are actually the data, you know, you’re like, these are the patterns that the system is matching, and then these are the things I want you to produce. Whereas in block-based visual programming, the visual elements are like actually like if statements and for loops and stuff like that.

It’s actually like not normal programming.

But I think they’re actually fairly different in the model of what is visual.

And I think it’s a very easy thing to fall into that like, there’s a lot of people who don’t like normal programming with a text editor and a compiler and whatever, but that doesn’t mean that they all have the same conception of what programming. It should be. Like I think there are actually many different ideas that are not necessarily compatible, and I think you know, visual programming is maybe too broad, at least it’s maybe too broad a category to be useful, and we should talk more specifically about what kind of visual representation you want for programs. And I think the other criticisms of visual programming that I think about a lot are One, it’s just like really annoying to manipulate visual elements on your screen with a mouse, compared to manipulating text for the keyboard. Like, you have this sort of bottleneck of like, oh, I have to drag things one at a time, I have to select things from a toolbox. I think this is part of the appeal of the dynamic lens stuff is it’s much, much easier to manipulate things on a table than it is to manipulate individual items on your screen. That might be better on an iPad or a multi-touch display. I think there’s like a lot of interesting work that somebody could do there, but I think that is actually a very serious problem, and I still don’t see it talked about enough, that it’s just like the ergonomics of visual programming are not that great compared to the ergonomics of text, on like current computing hardware.

00:20:03 - Speaker 3: Yeah, and I think the typical use of quote unquote visual programming tends to conflate a few different things. One is using the visual medium for high bandwidth feedback, which I actually think is really good. I’ll return to that.

But another is it kind of forces programs to be structurally correct often, you know, the visual programming blocks, like you can only put the circle inside the circle and stuff like that. But then it also necessarily enforces the sort of 2D program, which is very limiting, usually catastrophically so.

So I think some of those things are better than others, and also you can get some of them without going to a, what we typically think of as a full blown visual programming.

So for example, the idea of things being visible, taking advantage of the enormous bandwidth that you get over the visual channel, I think that’s great. I use it all the time and you can use it without using one of these typical visual programming language and actually leads me to a couple of my favorite folk practices. I know, a very simple one, but a super common one is just print after debugging, you know, it’s like dumping a huge amount of visual information from your regular program.

Another is this idea of shelves or scratch space and the related idea of lightweight copies. So an extremely common pattern that we see with great professionals is they’re working on something like a design for a web page. And they want to explore a branch, you know, a variant, and the proper programming way to do that is like get branch and so on. What people actually do is they select it all and they copy it and they paste it, you know, next to it, and they go fiddle with that. And if it works well, they delete the old thing and if it doesn’t work, they delete the new thing, and they’re off. That’s a very lightweight branching, but critically, you have both of them visible and it’s not like implicit in this really weird like get graph thing.

00:21:44 - Speaker 1: Right, right, like that’s another bottleneck cause like your git raff can only point at one thing at a time, and it’s hard to do comparisons unless you go into like comparison mode.

Yeah, I mean, like, with your example of like high bandwidth visual information, I’m constantly like, oh, I wish I could print off like a graph this graph of the state of my program.

And there are people on Twitter who do this regularly and have a good practice. But like, is that visual programming? I mean, it’s not like normal, you know, text programming with like string print off, but it’s also not block-based programming.

Like, it’s somewhere in the middle, and I think there are a lot of things that are in that space of like, you can’t do it on a traditional. You know, stack where you’re running in a terminal, run your compiler, running your program, but it’s also not like you threw all that stuff out and you have this sort of your dragging and dropping workflow.

00:22:32 - Speaker 2: Wulf and Julia to the engineers on our team recently were debugging a pretty complex, essentially there’s an in-memory graph structure that’s used and things were getting complicated once we added linked cards within the app and they ended up dumping it out, I think, to JSON and then there’s a tool I say it’s called Mermaid maybe that does a nice diagram visualization, and it was actually like fun to look at. It was really interesting.

Usually when you watch someone debugging, it’s like picking through these like monospace font logs and scrolling through the IDE but these visualizations were compelling and easier to understand, maybe for someone who is not someone deep in the problem space, like they were.

So, yeah, there’s a lot to be said for that.

I will point listeners to the classic Meta Muse episode with Maggie Appleton, where we talked about visual programming, and she makes this exact point that that is a label that is very broad. It covers a lot of things.

There’s some good taxonomies, but her basic concept for it, an argument for visual programming is a thing to explore more is you start From hey, how do we make the whole program out of, I don’t know, boxing and arrows, but you start from how do we just make more visual parts of programs we already have today, things like the DOM Inspector and the browser is one possible example, and you could imagine those as we get better and better at visualizing both running programs and at rest programs and code paths and Get branches and whatever else that there’s an accumulation of making a more accessible programming environment because it’s more visible and more tangible and can be interpreted in different ways other than just reading the code, it’s sort of mentally running it in your head and that for her is kind of the argument for visual programming.

00:24:14 - Speaker 3: This is making me wonder if there’s powerful primitives we could add to help with.

Leveraging the visual channel for debugging.

So, OK, it seems obvious, but actually having the standard of a single stream of MySpace font logs is huge. We can’t take that for granted, but we would be totally down in the water if we didn’t have that as programmers, right? But you can also imagine some other really simple basic printers that could help a lot. So one would be in the browser environment, you get this thing where if you log like JSON or a JavaScript object. It sort of gives you a nice rendering of it where it automatically expands or contracts when you click it and it kind of pretty prints the stuff and it highlights it with different colors.

00:24:55 - Speaker 1: Right, it doesn’t flood your console if it’s a giant object, like things like that that make it, yeah.

00:25:00 - Speaker 3: And often in web-based environments, there’s this pattern of like, you basically use a web page or a piece of the web page as the debugging panel, and you have HTML and CSS and I almost wonder if that could be almost like a standard, like in the same way that you have the log output, you have a little HTML page output and it has to be like HTML and CSS, but then you could write your own little debugging panels with like heat maps and graphs and stuff like that. I don’t know.

00:25:26 - Speaker 1: Yeah, I mean, I’ve played with things like, you can actually console log like a bitmap image, and so you can do these really twisted things where you like render something and then like, console log it out, and even that, you know, can be very useful depending on what domain you’re working with.

Like, if you have some domain object that’s like you have like a graph or a map or whatever, and you want to see that or like compare different instances of it, if you like log a bunch of sequence, that can be very, very useful, I think.

I would also say, and I think this gets at the point you’re making also, that I think another probably unheralded issue with this whole space of visual programming, visual debugging, it’s just it’s just like very, very hard engineering. It’s like you have to reinvent a lot of stuff that you get for free if you’re using normal text, if you want to do visual stuff, you have to invent your own editors, you have to invent your own consoles, you have to come up with interactions that work, you have to make sure they can post correctly, it’s like quite hard.

00:26:20 - Speaker 3: Yeah, and it feels like there could be a little bit of an easier layer there. One example I’ll give is, I’ve often wanted to have terminal output that was in the, what’s it called, in cursive style. That means that instead of each line coming one after the other and scrolling, sort of replaces the screen as if you’re using a command line program. But oh my goodness, that’s a whole ordeal in a lot of languages. Like you’re looking at these weird libraries and you’re admitting these like crazy control characters and it’s a whole mess. It feels like it could be a lot easier.

00:26:46 - Speaker 1: Yeah, and I think about that. I thought about that specific example before too, where I think the nature of terminal output where you’re like logging one line at a time, it’s like, if you have a program like a game engine or like a web browser or something that’s live, that’s interactive. And your console logging, like, you just end up with this flood of console locks, right? Like, a lot of the time, the logging model you actually want is to see this live view of whatever the variables in the system are, and then they just like update immediately, rather than this sort of log that just like will spill out because you’re running at 30 frames per second, or 60 frames per second, or whatever. And I think the terminal makes it really hard, like, you have to do a lot of extra work to get to that point, just cause the model is not really compatible with interactive programs.

00:27:27 - Speaker 2: One term we’ve touched on here a couple of times and I think is known to the audience of the podcast here’s end user programming, but Omar would be very curious to hear what does that mean to you or what’s interesting about that space. So I think the audience here has heard Mark and I and our take on it, but I’m guessing you have a different perspective.

00:27:46 - Speaker 1: Yeah, it’s funny cause I was kind of asking this question on Twitter a few months ago.

There’s something I think a lot of people are very attracted to about the idea of end user program, like, it’s almost this like charismatic concept of like, oh if only end users could program their computers. I mean, I think in a sense, everything end users do on the computer is end user programming, like programming is sort of an artificial concept, right? Like, if you’re using Microsoft Word or Microsoft Excel or PowerPoint, like these are all kind of like subsets of programming in a sense.

And so it’s, it’s one way to think about it is is it’s just a question of like giving even more agency to the computer user um uh than they have right now. I mean, I mean, I think part of my Thoughts about this come from this dynamic land context where I think, like end user programming was very deeply built into the system.

Like the idea is if you showed up at a dynamic land, a lot of the way in which you use the system is by programming it.

And so, you know, if you had a community of people built around a dynamic land, they would all know how to program in the same way that we all know how to read and write.

Some of that comes from the technical architecture of the system, but I think some of it would also just come from the social expectations. Like, it’s not particularly easy to learn how to read or write, but we do it because it’s useful to operate in the society that we live in.

And I think part of the premise of dynamic plan was that you would kind of construct a context in which that was true for programming.

00:29:07 - Speaker 2: Maybe it would be worth taking a sidebar here to talk about dynamic land for a minute. I know that’s a topic of interest to a lot of our audience. I know you were there for a while. I think it was a pretty formative experience in your career to date. Maybe you could briefly just tell us for those that don’t know what is that and what did you do there and what were the kind of core concepts.

00:29:26 - Speaker 1: Sure, so this was or is research lab started by Brett Victor in Oakland, California. Basically, the idea of dynamic Lane was to build this physical computer, where, like, there was literally a room or an office that was the dynamic lab, and you would show up. And you would have these pieces of paper, and each piece of paper was basically a computer program. And the idea is you would have a computer where you interact with the computer by manipulating real objects like pieces of paper or eventually like cups or like handwriting or like things that actually exist in the real world, rather than having, you know, current computers where you have a screen or a mouse or keyboard or a touch screen. So you have this completely different mode of interacting with the computer.

And I think importantly, it’s, this is a programmable computer. So not only do you use the computer. By moving real objects around, by manipulating objects, by pointing objects at each other. You also program the computer in this way. So you could actually do almost everything you wanted to, you could build software systems without needing to bring your laptop, without needing to bring your smartphone. So it’s this completely kind of self-contained end to end system in which you could do computational work.

00:30:34 - Speaker 2: And notably, I think everybody in the room is kind of in the same computer, if you do have a, I don’t know, a hackathon and everyone brings their laptop, they have their own. Discrete systems and I guess we’re all connected to the internet or you could connect to a shared server or something like that, but here if the room is the computer and we’re all in it moving the elements of that computational environment around where we’re all participating in the same computing environment. Do I understand that correctly?

00:31:02 - Speaker 1: That’s right. So basically, well, number one, there’s the physical element of like, you could see what other people are doing and kind of like go over their shoulder or work with them in that way, but there is also If you and I were around the table programming, each programming our things, there would be shared memory between our programs. So we could kind of insert things or respond to things in the same sort of room scale database.

00:31:22 - Speaker 2: And what were some of your either contributions on that project or maybe takeaways, especially now if you’re on the other things like, what were some of the core ideas that you carried with you?

00:31:33 - Speaker 1: Yeah, I mean, I think this idea of programmability is very, very important, and I think that’s something that’s missing in a lot of other physical computing work, whether it’s ARVR or also projection mapped or a lot of that kind of stuff, I think is from more of a traditional HCI uh or game development or whatever perspective.

Like, in some ways, the dynamic line system was less advanced, you know, in any particular respect, like, less advanced in computer vision, less advanced programming languages, but like combined, it was a novel system because you could program that, and because it was a platform on which you could do lots of different physical computing stuff.

So I think the program melody is uh is very important. I think that the sort of dynamic database architecture was really interesting and hasn’t been written about that much. It actually has a lot of close Relatives and I think a lot of what people are trying to do now with state management on the web or uh with distributed systems. There are a lot of other projects that I think have very similar models to this dynamic land database, but it definitely pushed me to think a lot more in terms of having state exposed by default, ambiently, kind of like in the screen matcher, and the value of being able to make like little quick debugging tools that can piggyback on these global state. So, you know, if you’re writing a program in dynamic, and it’s an idiomatic program, you would not use like variables and functions. You would kind of run everything through this database. And so, other programs could also respond to the state of your program just by querying the database, and everything would react live. So that was like a super influential model on the way I think about programming, and the way I think about debugging this idea of being able to make really lightweight tools or jigs to help myself as I work. And this idea of the value of like ambient state by default.

00:33:19 - Speaker 2: Jigs and visual ambient state, both of those concepts where I could see the thread into something like screen matcher even though that’s on the screen, because one takeaway you could have from the, I think it’s what we usually talk about as embodied computing, physical objects, you’re interacting with the physical world, you’re getting away from the glowing rectangles that Fundamentally are the core part of the computing experience that we all know and mostly love, and instead replacing that with something that’s more physical and in the world and humane, as Brad Victor puts it in one of his talks. But maybe for you, the takeaway was less the embodied computing and more some of those things like ambient. Visualization of state or programmability or you also have interest in the embodied computing, I think in some of your RFID work so I don’t know, maybe you’re just sort of following those threads in different projects.

00:34:11 - Speaker 1: Yeah, so the RFID work, we’re just getting underway, but we’re excited about that. I mean, I think there are a lot of directions. Like, I think this is a huge open space, and that was also one of the takeaways is that there’s just a lot to do, and there are a lot of problems with the dynamic client system, and there are a lot of areas where I think we were technically constrained, where I think there’s a lot of interesting things to do. And so I think the RFID stuff is kind of getting at that in some ways.

00:34:36 - Speaker 3: Yeah, and that reminds me of something that I thought was really important about Dynamic land, and this relates to the end user programming discussion.

When people talk about end user programming, they usually focus on how you program. Now, here’s the IDE, here’s the programming language, here’s how you debug. What people care a lot more about is what you’re programming.

And everyone cares about their physical environment. So that alone like almost immediately makes dynamic land.

A huge win.

And I remember, I walked into the room and just had this sudden urge to start programming stuff. You know, I want, you know, when this door opens and I want when this light turns on, I want to do this and that. It was a very natural urge. And by the way, one of the emerging end user programming use cases like the, the smart home, automated home, again, it’s because people care about certain things. And if you look at the history of successful end user programming environments, Unix, spreadsheets, SQL, MySpace, game scripting.

A, these are all environments that people have absolutely fanatical interest about. It’s basically the center of their lives or one of the most important things in their lives, and B, it is an enormous pain to program these. You think about SQL, for example, like you’re going to send a single string to your production database that, you know, who knows what it does and It’s gonna give you back a result or like spreadsheets where entire pillars of the financial economy are contained in like a 500 character formula in a single cell, highly questionable, you know, programming language design, but people get through it because they really care about the data.

00:36:04 - Speaker 1: Yeah, I mean this is one of the lessons, and if you talk to Maggie, this is one of the lessons in the Bonnie Nardi book where she does like ethnography of end user programming. It it’s like, yeah, you know, Excel, it just has these formulas which are just like this, you know, it’s literally a text-based syntax that you type in, like, people will learn it because they want to learn it, like, and so this is also maybe another sense in which the visual programming is not quite, at least it’s not like the only thing you need, where it’s like, yeah, you can make it as easy as you want, but like people are willing to learn, even if it’s really hard in the same way, you know, people are willing to learn to rewrite or whatever. Like, if there’s value in it, I think people will be willing to learn it, even if it’s not, you know, pedagogically like the best thing ever.

00:36:42 - Speaker 3: This does to my mind imply a sort of lesson to aspiring end user programming environment designers, which you got to start with the environment, I think it’s so tempting to start with.

I want to design a new end user programming language or IDE. It’s just, it’s really hard to get traction beyond like the educational and academic use case, but if you find something or create something. That people want to program. OK, OK, here’s an example. Minecraft. The way you program Minecraft is like you place these little blocks around in 3D space, and then you make your character walk around and poke them, like what? But it’s one of the most important programming languages in the world right now because people love that stuff, right? So you got to create an environment that people care about. Mhm.

00:37:20 - Speaker 2: Another one I like to point to is an end user programming success is Flash, because it did start from this kind of animator use case. You start from these animations and then you kind of use the dynamic medium of computing, and you go from static animations and something that become sort of games or full programs. Omar, I noticed you had some thoughts on software as a cultural thing, perhaps connected to that programming environment.

00:37:47 - Speaker 1: Yeah, well, first I think something that’s interesting about Flash and about Excel is this idea that like, it’s a useful system, even if you don’t get into the programming part, you know, like in Excel, you can just like write a list, and that’s a useful thing. Like, you don’t have to write formulas to feel like you’re being effective with Excel.

And in fact, if you do want to write formulas, it’s a relatively, you can just do that in one cell, it’s a relatively quick ramp up, and the same is true with Flash, right? Like. You can just use it as a drawing app, and then you can be like, OK, maybe I want to animate a little bit. So I think that is like an interesting common element between those.

But yeah, I mean, I was thinking about this, you know, I’m sure you all remember when the iPhone came out and it didn’t support Flash, there was this whole Sort of like Steve Jobs wrote the letter about how like, yeah, about how, you know, Flash is terrible for battery and you can do everything in it on HTML 5 anyway, and so we’re not going to support it.

And of course, I think, you know, what is it 12 years later, it’s just like that was completely false. Like people don’t do in HTML 5, the stuff they were doing in Flash, and in fact there was an entire sort of flash. Cultural ecosystem of like new grounds and mini clip and all these other places and people making flash games and like being inspired by the flash games other people have made, that was completely destroyed.

Like it just does not exist anymore, kind of partly as a result of that.

And so I was tweeting about this and some people were like, Well, how can we make a new flash? Like, we could make an animation ID? And I think I see the appeal of that, but I also think, even if you made exactly the same IDE and it did exactly the same things, without that sort of culture, community, ecosystem. Of people, you know, playing flash games that they like and being like, I wanna make a game like that. I think it’s hard to replicate the same thing.

Like, I think the IDE and the technology is only part of a I don’t know if you all know Max Kraminsky on Twitter, they had a good comment that I think you see this in a lot of programming systems, or even just like creative systems, people, I think they were talking about twine games, like twine is this sort of like interactive fiction creation tool, and they were like, you know, my students are not that excited about it. And then I show them some twine games and then they get more excited about it because people want to feel like they’re participating in this conversation with other people who have been working in the same medium as them. They want to feel like there’s like a canon of things that they can aspire to. They wanna feel like they’re placed in some kind of culture of stuff. And so I think, you know, when you’re thinking about making programming tools or creative tools, that’s a really important thing to think about is like, you know, if somebody looks at this, are they able to participate in some like medium or conversation or canon of things that are already out there?

00:40:19 - Speaker 2: Do you think that that’s something you can design for in creating a tool or is culture something that emerges kind of not quite serendipitously, but it’s some mix of things going on in the broader environment and what people want to do and to your point about the, you can make a flash style animation authoring environment for the web or that outputs to quote unquote HTML 5, probably people have, but Something about the way the world is now, probably you wouldn’t get that same kernel that then develops into that flash game culture that was so influential.

00:40:57 - Speaker 1: I mean, I think you can fail to do it, like, I think a lot of HTML 5 stuff has this property where, you know, like you can output stuff, but it’s just a web page like any other web like it’s sort of not constrained enough to constitute a medium in a way like I think you probably want something that has a more distinctive aesthetic. And then that kind of creates a distinct medium where people can look at like examples in that medium.

00:41:21 - Speaker 3: Yeah, I think something that supercharges this social propagation is being able to take some discrete artifacts and share it with a friend or they can copy it or fork it.

So the classic example is a spreadsheet.

And critically, when you copy a spreadsheet, you get both the output and the source code. And I think early web pages had this property where back then, you know, when I was a kid, the HTML and JavaScript and CSS was readable, so you could copy the source and paste it and then edit it yourself. But then to your point about these newer programs. It’s like this miniified compiled, you’re basically hopeless, so you can see the output like that’s cool. We have no agency to copy and fork it yourself, right?

00:41:59 - Speaker 1: Or I mean, with iPhone apps is another example, it’s like, yeah, you can’t copy an iPhone, or you could make an iPhone app, but it’s a huge process and like compared to, you know, making a web page back in the day where you just like make a dot HTML file and you put it online somewhere, it’s very easy to see yourself as a peer of the other people who are making stuff.

00:42:19 - Speaker 3: I’m gonna reiterate this, I think it’s so important. If you look at the successful end user programming environments, they all propagate this way. We gave the example of spreadsheets. The way SQL works in practice, it’s not like someone reads the SQL manual and then sits down at their company database and types out a query. It’s Mark has a query and he shares the query, and then Adam varies the query and then Henry varies the query from that. It’s like this like tree of life of SQL queries propagated socially.

00:42:46 - Speaker 1: Yeah, and that almost tells you that it’s something that’s genuinely useful and that’s like immediately useful, whereas it’s like, I feel like one of the problems with traditional programming is you have to learn how to program, like you have to go and like take a class or like work through a book or whatever, whereas with spreadsheets or SQL or whatever, you know, you can just copy and modify and like you’ll have something that works and it’s like a few lines.

Something I was thinking about with this screen matcher thing that I think is interesting in this general area, is this idea of like trying to unlock, like latent demand.

So, there’s a system called buttons in the early 90s, there’s like paper about it. It’s sort of like, I think of it as a predecessor to the screenmaer work, where they basically added this capability to this OS where you could stick buttons on the screen and make them do things. And that was the the only extension capability, like, it was not like a plugin system, it was like, we just added this concept of buttons.

Maybe you could like record things into them or whatever, but it’s really interesting reading their reports of how that affected end users thinking, because now, once you have this concept of buttons, you can be like, oh, I wish there was a button to do this. Like, you can, I wish there was a button to do that, like, because before you didn’t have any way to articulate the fact that you wanted to automate something, but now that you have this like, actually fairly weak concept. There’s sort of all this demand for like, oh, I wish my computer could do this, I wish my computer to do that, that you can now talk about in terms of buttons.

And so I think that’s one of the hopes for the screen mattress stuff is that, you know, having this automation capability brings out some kind of latent demand for things that people might already have been thinking about in an undirected way, but now there’s like a sort of means or um concrete way to talk about it.

00:44:23 - Speaker 2: as we think about the input and output of computers and that our ability to automate things which exactly as you said earlier, is just an extension of our agency, our general ability to control computers, and so we want to enhance that for people hopefully rather than reducing it or having it stay the same. And so, you know, here we’re talking about the IO of pixels. I know another one that you think about here is FFI is an underrated kind of problem area. Can you tell us about that?

00:44:52 - Speaker 1: Yeah, I mean, I think partly it comes out of This sort of frustration with like, If you get a programming language, whether it’s Ruby or Haskell, or JavaScript or whatever, it’s usually really easy to take in text and output text, like that’s built into basically every programming language.

But if you want to like take in images or output images, or if you want to like respond to multi-touch gestures, or if you want to, you know, put up a web page that other people can browse like, basically any actually interesting capability, you need to talk to other parts of the computer in ways that are often not available in whatever programming language you’re working in. And so I think in practice, You know, at least I personally, I’m like, oh, I can’t use like most programming languages because I actually like want to do things that are not just like computing things and taking in text and putting out talks, computing Fibonacci numbers.

00:45:41 - Speaker 2: Yeah, yeah, it was uh I’ve been in the position a number of times in my life where I’ve either encourage people to learn a program because I think they’ll find it interesting that they have the right kind of mind for it, maybe because career potential for them. So I’ve seen folks go through this over the years. I actually think there was kind of a golden age, at least web-wise in the era of PHP, HTML, and FDP, where there was this very simple mapping from files that would save out of a text editor and those mapped pretty 1 to 1 to URLs and the concept of query parameters would come in and you could start to sprinkle in dynamism through the little PHP tags. A friend of mine went through a just Of an intro, it wasn’t even a boot camp, it was more just kind of like a little intro to programming course, and I was really curious what they were going to show them, and it turned out they did Python at the console, which means, of course, that they’re teaching these folks how to like boot up the, you know, these are like most people are using Windows, they’re loading a DOS console and installing Python to run Python programs so they can use, you know, essentially printF and get from the console. And this actually is a totally foreign interface because most of the folks taking this class have never done that kind of terminal input output, but it’s just such a good fundamental way to get started, exactly to your point of take some text in, do something with it, and then spit it back out compared to what you would actually want to do is let me make an app on my phone. Or let me make a web page, or yeah, let me like take an image and like, you know, turn it into a cat meme, but that stuff is just like a wild tool chain of dependencies and moving parts and who wants to even get into that, that’s just not the place to start, even though those are the things you would actually want to do as a person that’s dabbling in programming.

00:47:32 - Speaker 1: Right, like, there’s this weird tension between like, OK, what’s good pedagogy, what’s simpler, and like, what is the actual well motivated thing? And then, I mean, this is very similar to our discussion earlier, where it’s like, the things that are well motivated are the things that you’re already seeing around you. Like, I go to web pages all the time, I use apps on my phone, but those things are so complicated that you kind of end up having to learn by doing these things that you’ve never seen before, and like, not having any sense of why this is interesting or important.

00:47:59 - Speaker 2: Yeah, it’s probably unreasonable to hope for, but I’ve certainly a future I would dream of is something where the average person with their phone would have the option to, I don’t know, long press an app on their home screen, and one of the options down at the bottom is like, make a copy of this and edit its functionality.

00:48:14 - Speaker 1: Right, right. And I think those are important at a cultural level, to like communicate to people that this is the thing you can do.

00:48:21 - Speaker 3: Yeah, this connects to a very long running theme on the podcast around the system’s problem.

And I usually describe that problem as something like, you want to be able to write a program in an end user accessible language that has full capabilities into the system, and that is also fast and secure.

But because of the way that we structured our systems to date, we’ve kind of boxed ourselves out of that. And indeed, if you want to write in an appropriately high level and safe language, there’s almost no way to avoid. Reduce capabilities and high latency and inability to be promoted up into the proper application or even proper OS level. So I’ve long advocated that a very important research project that we or someone else should undertake is trying to squash all these layers down, so you would have. A programming environment that has direct access to all of the critical IO, so visual, sound, keyboard, mouse, pen, and it all comes in in a very direct and clean way. So for example, the touch screen should not give you just XY coordinates. It should be a full heat map of the pressure sensor at every point on the touch screen. Yeah, but it just comes in as a simple two-dimensional range, your programming language. So it’s not some weird API that you need to go through. And likewise with graphics, oh my goodness, graphics. I don’t know if you all have tried to do. Graphics programming from scratch these days. You know, it used to be, they had a pixel buffer and you would put an RGB value into the pixel buffer and it would show up on your screen. Now you gotta like, instantiate the driver and initiate the shader compiler and compiler and give it the vectors and start the pipe. It’s incredible.

00:50:04 - Speaker 1: Yeah, I tried this and then I gave up because I spent like 3 days straight trying to like install Vulcan or like, if you look at the Vulcan example to draw a triangle, it’s literally like 3000 lines of C code.

00:50:15 - Speaker 3: It’s absolutely wild.

00:50:18 - Speaker 1: I had a professor in college who, his doctoral thesis was about this concept he called exokernel, and he wrote a paper called Exterminate All Operating System Abstractions, which you might want to check out if you hadn’t seen it, which is basically the title communicates the message of the paper, which is that like operating system should, like, that sounds up my alley.

Yeah, you know, it needs to like multiplex, like, the different programs can use the same resources, but it shouldn’t like turn your disk into files or turn your touchpad into XY coordinates, it should just like give you access to the underlying buffer and like do the minimum needed to multiplex it. And then if programs want a higher level interface, they can just like link that in, like, that should be the program’s responsibility, and not the operating systems. Yeah.

I think it’s partly because of the kind of projects I’m interested in. You know, a lot of my projects are about pushing some system to the limit of its capability, like the web browser or the operating system, even the dynamicle stuff, it’s like, you know, we had to talk to webcams and we had to talk to projectors, you know, and like a lot of that stuff, if you want to do it well, you have to go to a pretty low system level. You want to like, get these buffers and not have to copy them, all this other stuff. And so I think from that experience, my default these days is usually like, well, I guess if I’m in a browser, I’m gonna write in JavaScript, and if I’m on like the desktop, if I’m in Unix, I’m gonna write and see, cause then I know I have all the capabilities. Whereas if I write in anything else, it’s like, OK, I have this third party like bindings, and maybe they’re not up to date or like, maybe they don’t expose the right things, like, it’s just a mess. The only guarantee you get is if you like, write in these super low level languages.

00:51:46 - Speaker 3: Yeah. Now it’s also the case that if you write in one of these lower level languages currently, you might have an intractable amount of work to get up to the full capability and richness of an app.

So for example, if you wanted to write like an iPhone app on equivalent hardware up from C, it would be an enormous undertaking.

That to me points to a really fundamental issue here, which is that a lot of programming language, I don’t want to call. Design, but like programming language, bringing into its existence and programming environment bringing into existence is an economic problem and not a technical design problem.

The amount of resources you need to actually build out one of these new programming environments is enormous.

You know, I don’t know. Maybe it’s a billion dollars, maybe it’s $10 billion maybe it’s $100 million. You know, it’s a lot of zeros, right? And so the only way that you can realistically get there is to have some multi-step strategy.

And I feel like not enough people are kind of considering that, cause like, I wish, you know, we had this ideal programming environment where you could do X, Y and Z, but you gotta have some way to start. And by the way, I think a lot of it goes through like toys, games, fun stuff, you know, playing around with your home.

Programming environment, that’s kind of a way to get some initial bootstrapping and resources, which is why I keep advocating for doing experiments in that direction. We could probably do a whole podcast about economic thinking at some point, but I just wanted to mention that I think you got to consider this resources and incentives and motivations angle.

00:53:07 - Speaker 1: Yeah, probably you all have seen, you know, there’s the whole famous essay about Unix worse is better, but I think like one of the interesting arguments, I don’t think it’s quite an argument against it, but it’s pretty close, is, you know, the reason Unix succeeded was not because worse is better, it’s because AT&T like gave it out for free to universities and like that meant that everybody learned it in their university, and then it was kind of like the model operating system that you would base your computer around.

00:53:31 - Speaker 2: I’m thinking of the meme first time founders think about products, second time founders think about distribution, and in this case you know the distribution.

00:53:39 - Speaker 1: And that is kind of like a weird artifact of like 1950s like US antitrust. Like it doesn’t really, I mean, I don’t know if there’s really a lesson there, but like, yeah, I think like thinking too much about the technical construction is maybe a mistake compared to thinking about the economics.

00:53:54 - Speaker 3: Yeah, and this reminds me, I recently saw someone asking on Twitter, why don’t more programming languages have this nice property, and I think the property was there’s a small number of primitives.

00:54:07 - Speaker 1: Orthogonally applied to many problems which Oh, I saw this, yeah, I think that was Patrick Groy maybe, yeah.

00:54:12 - Speaker 3: OK, nice. Yeah, we’ll like the tweet in the show notes, you know, and by the way, it’s a thing that I’ve advocated for many times on the podcast, it’s very nice, but the correct observation was that this basically never appears in practical industrial programming languages, and my thought on that was that.

Well, unfortunately, as we’ve discussed on this podcast, the success of programming language is not determined by their design quality. It’s primarily a matter of what they’re programming against and the economic resources behind it.

So why did JavaScript succeed? That’s basically nothing to do with the programming language design and everything to do with it was the scripting language for the browser, which is incredibly important.

You could basically done anything there, I think, and it would have been enormously successful.

And so the reason why we don’t have these nice properties in programming languages is because They’re very hard. You have to constantly fight against entropy and accidentally bad designs, and given enough time, you know, basically no one could do it. And so you just have these high entropy programming language designs out there, basically by accident, I would argue.

00:55:07 - Speaker 1: Yeah, I remember making this joke a few years ago that, you know, imagine if we all sat down around a table and we’re like racking our brains, we were like, why is Objective C been so successful in the market? Like, what did they do in it that made people want to adopt it so much? And it’s like, obviously it’s because the iPhone, like, yeah, and it’s like really nothing to do with the programming language design other than like what made Apple willing to adopt a.

00:55:28 - Speaker 2: Right, and there I would say that the programming kind of stack for the Apple world, especially now with SWIFT and Xcode, and certainly all the APIs and stuff that’s there, is one of the better ones that captures some of the qualities of using kind of a, in this case, it’s a native language that’s sort of fast and you’re not going through some abstraction layer to get to the capabilities of the platform.

But at the same time, Apple provides just a huge number of APIs that are relatively high level for doing all the various things you might want to do, both hardware-wise, but just sort of capabilities of the system.

But yeah, you want to talk about economic incentives.

Well, OK, it’s that the App Store and Apple’s cut of that and the huge success. of apps on this platform just means that there is great incentive to or there’s a lot of money to sort of make it be successful, and also kind of coincidentally, Apple really cares about design and craft and put a lot of effort into making it pretty good as programming environments go, but You know, it could have been that you had this incredibly successful platform, they needed to make a programming environment for it, and it was not a company that cared about that stuff, and then you would end up with something that’s just much more random. So it’s sort of not a well designed programming language in a way, it’s not a sort of like a fitness trait in in any particular way other than for, I don’t know, programming language, you know, design nerds.

00:56:55 - Speaker 1: Yeah, I mean, I think the Apple stack is, I mean, most of my background was in Web programming.

I’ve done a little bit of stuff on the Apple platforms, you know, with the screenshot stuff and with other things, partly just cause that’s what you have to do, like, you know, if you want to be taking, monitoring your screen or seeing what’s underneath the screenshot, you know, you better be writing an objective C like you don’t really have a choice or Swift.

But I think if you have a web programming background and you haven’t looked at the Apple stuff, I would definitely recommend it cause it is like a very, just from a cultural point of view, it is like just a very different ecosystem. It’s like actually designed in a way where, you know, there’s one company that built the ID that built the language, that built the APIs, and they can kind of unilaterally make changes. I don’t think it’s all good, but it’s like, it’s different and it’s interesting.

00:57:40 - Speaker 3: Omar, I’m looking at our shared notion doc and you’ve written wiggly computer. What does that mean?

00:57:46 - Speaker 1: Yeah, I, it’s a good question. I feel like I’m trying to figure that out too, but Like a lot of things on the computer are like these buttons and toolbars and commands that you run, and they’re very much like, you hit a button and it’s this very discreet way of interacting with the computer.

And so I think there’s a question of how can we make computer systems that are wiggly, where instead of these discrete actions, you kind of can continuously move things, like the motion that you do as a human, where you’re like dragging something around or pointing out something or like wiggling something to highlight it. Like, how can that motion be carried through into the computer and into the application, and like, what are interactions that work like that, instead of interactions that work like you’re tapping or clicking something or hitting a key on your keyboard. I don’t have like a super well thought out philosophy of this, but I do think that that’s whenever I encounter things like that, it feels really nice, and I think there’s actually a lot of power there in terms of like, you can simplifying your computer system. By not having a lot of options and buttons and functions, and just sort of trying to carry through human movement.

00:58:53 - Speaker 2: A reminded mark of your probabilistic gesture input system where the system is sort of simultaneously guessing which gestures you’re starting and assigning probabilities, which there is some amount of in touch systems, so some of that’s built into iPad and what have you where it’s sort of a particular finger down could resolve into a pinch or a drag or something, something else, and it doesn’t quite decide which one it’s going to be until you are.

Was in, but I think you had a version of this which goes even beyond like a simple heuristic of, OK, it’s two things until that second figure comes down within 100 milliseconds and instead is much more of a fuzzy guess that eventually resolves and maybe even post hoc rewrites which you’re seeing visually to match what it now has decided to have done.

00:59:39 - Speaker 3: Yeah, I think there’s a lot to that, and by the way, it would interact very well with this visualization of programming state idea that we’ve been talking about.

You can imagine some little corners up on the top of your screen that appear and get brighter and dim, you know, that has like a two-finger gesture or a double click gesture according to what you’re doing. And also, by the way, These could be parameterized and end user adjustable. My belief is that basically any user impacting parameter in the code should be user adjustable on a slider. So two that already are typically are font size and key repeat speed. So when you hold down a key on your keyboard, it like eventually starts repeating and the delay until the repeat and the repeat rate is in good environments, it’s configurable. In the best environments, you can like hold down a key and then drag the slider back and forth and see how fast it’s going. That’s cool, until you get it right. And I think that’s how all UI should work, at least at the developer level. So something that we often try to do with Muse is we have like a debug menu, and there’s some parameter like, you know, ink curve, delineation, you know, variable or whatever. And instead of hard coding that into the code and asking a developer, oh, you know, I think the curve is too curvy, can you make it less curvy? That should just be a slider. Maybe there’s a detente, which we think is the current correct value or the default value, but then you could be drawing with one hand and sliding around this variable with the other and immediately seeing how it reacts.

01:01:00 - Speaker 1: Yeah, I really like that probabilistic gesture thing. I was actually thinking about something similar too. Cause like, right now, basically there’s like a gesture recognition layer, and then there’s the actual programs and they respond to the gestures.

But if you had a sort of end to end program that like could do the gesture recognition and then like generate a probability distribution, and then like run the program on the entire probability distribution, and then like, you wouldn’t have to resolve anything the program does until you’re like done with the gesture. Yeah, I don’t know if it’d be useful for anything, but I think it would be cool to see, like, you sort of see these overlays of like different universes, and then they kind of fade in and out as you move your finger. I mean, I thought about this a little bit in the dynamic line stuff of like, well, a lot of recognition systems are like probilistic, like you’re not totally sure what you’re seeing, so maybe you can kind of have this super position of like different things you recognize and what the effects of those are, and then you wouldn’t have to resolve them until the end.

01:01:47 - Speaker 2: Wiggly. Now one thing we mentioned earlier, but haven’t talked about much is your project TabFS and this is something where you essentially use a fuse file system to mount the open tabs in your browser as folders that you can then browse through and do programmatic operations on them. Now that’s interesting because that ties a little bit back to that we talked about the Unix kind of everything or the Unix philosophy and kind of text as input output pipes, but of course another piece of the Unix philosophy is everything is a file. And yet we do live in this world that is more kind of increasingly with mobile platforms, files are just kind of mobile and cloud basically means you know files are on their way out. How do you think about files and particularly in the context of this TFS project, do you think of that as something where you want more of your system to be Controllable through files, or do you see that, you know, I don’t know, files are more of a retro computing thing in the same category as a, you know, a Game Boy emulator.

01:02:47 - Speaker 1: Yeah, that’s a good question. It feels somewhere in the middle to me, like, I don’t have like a deep attachment to files as like the interface of the future or anything, but I do think that there is, at least like files are objects, and like, you have operations that you can do to them.

Like, you can copy them, you can move them around, you can cap them. You can look at them in Finder, you can look at them in Emacs, you know, you can grab them, you can watch them and do things when they change.

I think with the web, without that, there’s nothing like that on the web, like, you kind of have to build everything from scratch, and so mapping things into files is a step up from what we have now, even though I’m not like 100% committed to files as an interface.

01:03:26 - Speaker 2: Yeah, for me that agency that you get from these very, I guess, uniform operations, which is, you can always move it, you can always delete it, and you can always duplicate it.

And duplication is nice for backups.

I’m a big fan of, I think we talked a little bit earlier about the iteration process, whether it’s Git branch or something that’s more like a, you know, copy paste to kind of like riff on a few different variations of an idea of a thing that you’re working on, duplicate has that capability where I can say, OK, I want to try something out, but I want to be able to return to my current state. And I just know in the old school world the files, you know, I close the word processor, I copy my file.doc to my file to. doc or my file experiment.doc. I open that file and I know I can do stuff to it and I know. Won’t touch the other one and that’s something that’s often missing in cloud services, for example, where it’s like, OK, I want to do this big operation to like reorganize my email or something like that, but I’m not sure if it’s going to get messed up. So can I just snapshot the current state, but like, no, there’s no concept of that. That’s a good point. It’s some database somewhere, I guess some DBA who’s not me and works for a company that I’ve never met, could potentially do that, but it’s just not within my control as a user, right?

01:04:47 - Speaker 1: Like there’s no like Omar.gmail file that I can like duplicate.

01:04:51 - Speaker 3: And this is sadly actually one of the dying folk practices.

So, especially in games, they used to save the state of the game often as a SQL database, and then often the game files was compiled a code plus images.

So there were two ways you can like basically go in and poke at your game. One is you can look at the. To like save file and read it or even write to it, so you could like find the road that’s like your sword and like increase attack by 1000 or whatever. So you could also go in and edit the images. So people would do all kinds of stuff. Like, for example, if you had trees in your game, and it was Christmas time, you could like turn it into a Christmas tree so that you would be in this Christmas wonderland for your game. All kinds of stuff that people used to do when you could go in there and poke at your files.

01:05:35 - Speaker 1: Yeah, and it’s like, even if, you know, you have a cloud service and it has like a little pseudo file system, and you can do things like that, you don’t necessarily know that you can do things like that, right? Whereas like, you know, not everybody knows how to use the file system, but if you know how to use the file system, you know, your knowledge will generalize to anything, any application that uses the file system. There’s an element of like, are the operations available, but there’s also an element of like, you know, do users know about the operations, which I think is also important.

01:06:02 - Speaker 2: Yeah, I think the simple mental model of files, which on one hand, I think files and file management is one of the things that confuse, let’s say, non-power users, but basically average people with computers. I think that and the like Windows and the difference between like minimizing a window versus like closing an application.

These kinds of distinctions and so I think that was a reason why both mobile and cloud essentially getting rid of the idea of closing an application or managing files or worrying about your hard drive or backing up your hard drive, that stuff was just tough for non kind of power us. or computer professionals, but for someone who did go a layer deeper and understood the basic mental model, it was very simple and easy to understand that it didn’t matter if it was a Photoshop file or a spreadsheet or a text file or whatever. The duplicating, moving, renaming, deleting is kind of always the same.

And once you grasp that simple set of operations, it feels very empowering.

Yeah. Well, maybe as a place to end. I’d love to hear what projects are on the horizon for you, where are your interests drifting to next.

01:07:18 - Speaker 1: Yeah, I mean, there’s a bunch of different stuff.

I feel like I have queued up at the moment.

There’s the screen matcher work. I mean, I think with a lot of these projects, it’s like more of a question of like getting it to the point. Where we can at least publish something and like, get people excited about it, because I think there is like a huge, even with TFS, you know, there’s a lot more that could be done on top of that, I think. And I have a lot of things written there that this would be cool to do, that would be cool to do. But yeah, so like, getting some of those things released, you know, looking at some of the physical computing stuff, the RFID stuff, but, you know, I’m always on the lookout for interesting projects to do also.

01:07:54 - Speaker 2: It does beg the question is, what for you is finished in the sense of one ready for release, and then 2, the degree to which these are things you’re going to maintain or extend or improve over time versus, you know, when I think of a pure research, you know, like an Ink & Switch piece, we rarely build on kind of the prototype because the point was to publish about it. Yeah. Have the discussion, but then a future project, even it’s gonna research on the same kind of area, you might start from scratch or do you just might start in a different place. The goal is not working software to maintain over time.

01:08:30 - Speaker 1: Yeah, I mean, I think it varies and it also varies with the project, you know, like the screens is working software and I use it regularly. I mean, I think often the way I put the goal is like, I would like to get it to the point where I think that People reading it will understand the point that I’m trying to make. And I think, you know, that implies a certain level of polish, that implies a certain level of interesting examples or demos that may imply people should be able to download it and try it, but it doesn’t necessarily imply that it needs to be a fully working product or that I’ll maintain a.

01:09:01 - Speaker 2: Well, certainly, that would also say that the storytelling or explanation or yeah demo, whatever it is, is equally important, if not more so than the software itself.

01:09:13 - Speaker 1: I sometimes make a joke that I write a lot of these projects as an excuse to write the read me for the project, which is basically like an essay.

01:09:22 - Speaker 2: That makes sense. And in some cases, I guess it depends on exactly how much is conveyed through the project through a simple video, through installing it yourself, through a screenshot, how much like longer description is needed, but we had Jeffrey Lead and Max Schoening on recently talking about their income switch project. They basically said, you know, we spent Longer, I think on the writing and the trying to understand what we learned from doing this weird thing, and then of course writing itself is this whole own giant production process, making something comprehensible and deciding on the terminology and all that sort of thing.

01:10:00 - Speaker 3: Yeah, well, Omar, your projects have definitely had a big impact on me. You’ve gotten some very interesting messages across, and I think that’s the case for a lot of people who follow your work, so really looking forward to what you come up with next.

01:10:11 - Speaker 1: Thanks.

01:10:12 - Speaker 2: Well, let’s wrap it there. Thanks everyone for listening. If you have feedback, write us on Twitter at @museapphq or via email, hello at museapp.com. And Omar, thanks for challenging us and inspiring us with your combinations of unexpected things.

01:10:29 - Speaker 1: Thanks for having me on. Thanks, Mark. Thanks, Adam.