Transcript

00:00:00 - Speaker 1: Maybe the past generation of programmers were sort of subjected to a really awful version of visual programming, but it would be really sad to let that frustration that other people have felt ruin what visual programming could be in a much better design format or just done in very different ways, done in ways that combine graphics and text.

00:00:24 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for thought on iPad. But this podcast isn’t about Muse the product, it’s about Muse the company and the small team behind it. I’m Adam Wiggins, joined by my colleague Mark McGranaghan. Hey, Adam, and our guest Maggie Appleton from Egghead. Hello. Thanks for joining us, Maggie. Now, listeners of the podcast will know that Mark and I are fans of computing history. We think sometimes the technology industry is a little unaware of its own history, and I understand you recently visited a museum on that topic in your area.

00:00:58 - Speaker 1: I did. I got last weekend to go up to Cambridge, which is about an hour away from London, and visited the Computer History Museum of Cambridge, which was mostly a warehouse that some people had put an old N64 in and an old Mac and got to play a bit of Pong and Space Invaders, which is terribly educational, but probably not a comprehensive history of computing.

00:01:20 - Speaker 2: Now, depending on what generation you grew up with, you know, so now I’m old enough that I go to a computer history museum or sometimes there’s, I don’t know, like the pinball museum in Alameda there in San Francisco or some of these arcade ones, and I remember this stuff when it was new, maybe. Kid, so in that case, it can be as much a sense of nostalgia or a comparison with your sort of adults perspective on something that loomed large as a child. Were there things there you recognized, or was this all like essentially stuff that predated your time?

00:01:52 - Speaker 1: I was definitely of the N64 generation and definitely the, I don’t know which version of the Mac it was, but I have a very clear memory of going into a friend’s house who had the Mac. Windows 95 was maybe what we had had at home. I mean, I was grew up sort of in the naughties, so I’m that generation.

00:02:08 - Speaker 2: Is that what we’re calling it now? I’ve heard the knots.

00:02:10 - Speaker 1: I know. There’s no good word for it.

00:02:15 - Speaker 2: I mean the century, maybe we could go with. So, before we get into our topic, I’d love to hear a little bit about your background, Maggie, as well as a little bit about what you do at Egghead.

00:02:26 - Speaker 1: Sure, so, yeah, Egghead is where I currently work and it’s a web development education platform, so we kind of will joke it’s Netflix for developers, it’s mostly video tutorials, teaching anything, mostly on JavaScript, lots of React, Angular view, that sort of thing.

And I’m the art director, designer, illustrator there. So essentially I handle most of anything that is visual design, and I also do quite a bit of UX design there.

It’s sort of a small company, so you get to be a bit of an all-rounder. And because it’s a very developer focused company, I also develop, mostly working in React. I like to call myself a mediocre developer, so I’m not intense, you know, I don’t really like dig into, you know, the deep code, but I know my way around, you know, front end database. So yeah, that’s kind of me.

00:03:11 - Speaker 2: And it seems like you came maybe from the, I don’t know, illustration or design side, and then that led you to kind of the development side, or was it the other way around?

00:03:20 - Speaker 1: Yes, it was definitely design and illustration first. So I had grown up in a very technological household.

My parents were both programmers. I had a lot of access to technology quite young. So I learned, you know, HTML and CSS, but I wasn’t really introduced to proper programming young and you know, there wasn’t any like IT education in school.

So I went more into design and illustration in my early twenties and then learned programming, mostly out of employment necessity, you know, I was, I was designing websites and, you know, starting to learn JavaScript a bit and figuring out how to get a model to work.

And then, of course, once I started working at Egghead, got a lot more into programming and development.

So I very much come at it from, I struggle to learn it. I’m maybe not.

I don’t want to get into natural types of people, but of course some people have an affinity for being able to do abstract reasoning and they’re very good programmers because they can hold a ton of cognitive context in their head about what component is connected to our database, you know, where messages are being passed, and I am not strong on a lot of those fronts, so I do development. As someone who needs things visually displayed in front of me, I need a lot of explicit feedback from the machine telling me what’s happening and what things are connected. So my perspective, I kind of bring to programming is, I’m the person that struggles with it. So I really, really want tools that actually help you program if you’re not naturally inclined to abstract reasoning.

00:04:43 - Speaker 2: I think it’s interesting that you saw basic programming through HTML and JavaScript as being almost sounds like a table stakes for your career, and I think that’s the nature of the computing medium.

You can imagine doing visual illustration, design work in print, for example, and there you probably need to learn some things about printers, but for the most part you’re fine just doing everything in pixels on the screen.

You hand that over and There you go, and that’s probably true in other realms as well, but once you’re on the computer, now you’re working in a dynamic medium, and your ability to express things is very limited if you’re stuck with what’s static.

I think that’s going to become more and more true here in the future.

00:05:22 - Speaker 1: Yeah, I think it was good. I was ignorant of the fact at the time that I was in the generation where people were designing websites in Photoshop, and I didn’t know that was an option. So I went and learned CSS and JavaScript because I couldn’t figure out any other way to build a website. I didn’t know I could just mock up pixels and hand it to a developer. So I’m very glad I was ignorant of that.

00:05:41 - Speaker 2: So our topic today is visual programming and also to some degree, its relationship to the learnability of programming. And since you suggested this one, Maggie, maybe you can explain a little bit for the audience what the basics is. What is visual programming, why is it interesting area?

00:05:59 - Speaker 1: Sure. So visual programming is sort of a catch-all name for a whole bunch of approaches to trying to do programming differently. I want to say I think the name itself is problematic, which we can get into a bit, but it’s essentially trying to explore graphical interfaces for programming in various ways and then There’s been a long history of people trying to do this, and a fair amount of pushback to versions of it that have gone quite badly, which we can also get into. But yes, it’s essentially trying to explore ways of doing programming that is not just based in text, you know, linear text in a text editor, which is kind of the default nowadays.

00:06:36 - Speaker 2: Yeah, certainly when people think of programming, they think of it as being synonymous with coding. Think of that Hollywood hacker view where they’re sitting in front of the terminal, the text is probably green on black, and it’s a bunch of text in this computer language. Now, is there a well-known example of what would count as visual programming?

00:06:58 - Speaker 1: The two that jumped to my mind. I’d say most famous is probably MIT has a programming language called Scratch that is technically designed for children in order to teach kids programming sort of as a first programming language.

And I think this gets referenced a lot in visual programming, um, circles because it gets rid of needing to use sort of arbitrary syntax symbols to define things like objects and functions.

Instead, it uses drag and drop blocks with colors that correlate to the type of each element. So it’s a very interesting example of how you can kind of lower the bar to people coming in and doing programming since it is designed for children.

So that’s one that gets famous, but others that I would count as being some form of visual programming that will be very familiar to the audience are tools like Zappia, if this then that, Integrama now, the sort of the whole low code, no code movement where people are essentially building node-based logic flows where they’re saying, you know, if this happens, then do this, else. Do that, they’re doing, you know, programmatic logic, but they’re doing it in a very graphical interface. And I count that as visual programming, but I think there’s also a wide diversity of, we can talk about, you know, the amount of power each of these platforms might have, the amount of flexibility and control they give to the user. But yeah, those would count, I would say, as visual programming.

00:08:21 - Speaker 2: One that comes from my childhood, thinking of kids learning programming is there’s a game called Rocky’s Boots. It was like a 1980s, ran on the Apple II or something, and it showed you how to put together logic gates, the same kind that you use in electrical engineering, to make circuits and solve problems.

And that was one of the things that maybe introduced me to this sort of form of logic that eventually led to my interest in programming.

But one thing I think is interesting about that is In fact, electrical engineers designing circuits work with a form of visual programming. You make a diagram that shows where the resistors and the capacitors and the LEDs and whatever else go in this form that is very much, I guess it’s less of a flow diagram and more shows where the electricity flows, which may or may not exactly represent sort of the abstract logic, but I think that’s a professional context where there’s kind of a visual programming, at least representation.

00:09:16 - Speaker 3: And we can also talk about examples of visual augmentation. So with Scratch, kind of the whole environment is inherently very visual.

Other examples we could look at the spreadsheets where you have the cells themselves are like text formulas, more like a typical programming language, but it has this visual element of layout and spatial based variable naming.

Another example from when I was learning programming is the I think it was called Doctor Scheme. I think it has a new name now, or maybe it changed to that from something else, I forget, but it was the scheme development environment, which first of all was all in one, which is great. She’s got this like one executable you could run and do everything, but also when you hit errors, which is the dreaded case of a new programmer, it just drew lines to everything you needed to look at, which sounds so basic, but just connecting the dots between this is your call site and this is your function definition, and there’s something wrong, you know, check where these two lines terminate and see what’s up. We could talk about all kinds of examples of visual augmentation as well.

00:10:09 - Speaker 1: Yeah, and I know you guys are also familiar with a few maybe more powerful visual languages like origami, and I think X code is quite visual in the way that it allows people to program. So yeah, there are fully fledged IDs that have visual elements to them and might not be fully fledged visual programming environments, but the hybrid is actually much more interesting than this idea of pure visual programming, but we can get there.

00:10:32 - Speaker 2: Our colleague Leonard uses origami for some design prototyping. We even put some screenshots in one of the beta memos we did.

I’ll link that, but It’s interesting because he’s doing design work and typically he can do that in a tool like BigMA and pretty static mockups and then we go from there working with U interface engineers.

They connect the static screens to behavior, but there are certain things, for example, when we were working on this whole infinite canvas flex boards feature where just how it felt. and how it behaved was the whole design and you just couldn’t evaluate that by looking at a static screen and furthermore, just the standard click prototypes like you get in a lot of modern design tools just won’t cut it. And so he ended up doing some fairly complex programs that you could then in turn run on the iPad and really feel how it worked. It’s quite impressive.

00:11:25 - Speaker 3: You know, it’s gonna be very tempting to market this episode with a D&D style alignment chart for visual programming tools. It’s like lawful good is, you know, scratch and what’s the other corner? Chaotic evil is a meme generator as a programming tool or something like that. We’ll see if we can pull it off.

00:11:43 - Speaker 1: Yeah, I’m thinking of other ones I’ve seen. I think there’s a category of them that I think was very popular in the 90s, like I’m specifically thinking of Lab View gets a lot of negative press, I’ll say, as maybe not being chaotic evil, but it’s one of the infamous ones that I think was proprietary, so you kind of got locked into the system.

And then there’s a whole Tumblr. We can put it in the show notes when I find the link to it, but I can’t remember the name now. That is just screenshots of awful literal spaghetti code because it’s a node-based visual programming language. So it would just be thousands of boxes and lines running between all of them, and literal, just a visual spaghetti that you can’t tell down from up.

But yeah, that would be somewhere in the evil dimension.

00:12:22 - Speaker 2: Yeah, the boxes connected by lines also reminds me of another big category that I’ve come across in my, I guess not my day job would be the way to put it, but music production is something that’s been of great interest to me, particularly earlier in my life.

And so I worked a lot with software like Logic Audio, for example, sequencers or something like Reason, which is a kind of an all in one synthesizer plus sequencer, and these typically have various ways to essentially wire stuff together, but I I think one reason that the visual thing works well there is you have this clear kind of input output flow.

The output is always going to be an audio stream or in some cases there’s video tools as well where there’s a video stream, but it’s a single sort of time-based flow and so like a music track or something like that.

You just got 7 minutes of music output and you can in logic, for example, To the quote unquote environment and there you can just string together, OK, I want to take the source material. I don’t know someone playing a guitar and I’m going to put it through these filters and I want to fork it off this way and then the end output just goes out through you know your master volume and obviously that is very much designed for professional very serious use cases, but these are non-programmers.

00:13:33 - Speaker 1: Yeah, I’ve had the visual effects industry has a lot of those same sort of interfaces, so the node model box and there I was just connecting different things together and really just showing data transformation over time, which is when a flow diagram actually comes in useful.

00:13:47 - Speaker 2: And then another one I think could be remiss not to mention is iOS has an automation system built in called Shortcuts, which was originally a very cool indie app that Apple acquired called Workflow, and this essentially gives you sort of a scratch style model but probably Even simpler in the sense that you don’t have a lot of nesting and what have you, you just sort of assemble these blocks top to bottom.

You can do it on a phone, which actually is pretty interesting.

It’s possible to do reasonable programming on a phone, and again, I think it does rely on a fairly linear input and output.

You’re going to apply your shortcut to this image. You’re going to do some stuff to it. Something comes out the other side, maybe it’s a transformed image, maybe it’s something else, but that’s a good example of a place where Visual programming maybe is actually the mainstream in a way that a lot of folks don’t even acknowledge.

00:14:38 - Speaker 1: Yeah, it kind of transitions us into the idea that visual programming ends up being this really large umbrella that I think a lot of the debates around to get into people just not having an agreed definition of what we’re really talking about.

And so I mentioned that we would come back to it being that it’s kind of a terrible name because to call something visual programming suggests that it is something visible with human eyes, which would most certainly include text or a screen. So that would include all programming. So when people say visual programming, I think what it’s really getting at is, this is my sort of opinion on it, I suppose, is it’s trying to take things that would otherwise be implicit in programming, so parts of the program we don’t see, so right, data transforming over time or a variable being passed through a function.

And making it visible in some way.

And this is kind of where it gets into the balance that visual programming platforms or languages have to kind of grapple with is how much do you make visible and in what context, because you can’t make everything visible because the point of programming is abstraction and working with higher level primitives and sort of taking away things we don’t want to always deal with, right? Like where memory is allocated, you don’t want that visible in a higher level programming language.

So I always see it as how we take programming and we make different aspects of it visible in different contexts depending on what we’re trying to do.

00:16:03 - Speaker 2: Is there a better name, or if you were in charge of the rebrand on this somehow, what would you go for?

00:16:10 - Speaker 1: I feel like graphical programming would at least more accurately describe what a lot of these platforms are trying to do. I mean, a lot of them are simply just trying to apply better UX and interaction design to programming, right? They’re just trying to make it a more enjoyable experience that gives the user or in this case, you know, the developer what they need at the right time without overloading them with information, you know, really just being a good cognitive tool.

For them to get their work done.

So yeah, but graphical programming, I think would more accurately describe node-based interfaces or drag and drop, you know, ones like scratch, or even debuggers really, I think are a form of visual programming in a way, ones that really walk you through how the program is executing over time, although I think it’s kind of wild that you wait till something goes wrong until those tools come into the picture. Like, why isn’t there an inline debugger in every IDE, but you know.

00:17:06 - Speaker 2: Actually one of the first, sort of one of my mentors when I was getting started with the professional programming, he strongly felt that you would go and put break points at every interesting point in the code the first time you ran it, and you wouldn’t run it and just see what happens.

You’re gonna step through, inspect the variables. Watch every single point, and only once you’ve been through it all and watched it all execute from the inside, and everything’s what you expect, then you can look and see what lands on the screen, if that’s what you expected or not, but that’s kind of the final step.

That’s more your victory lap, not your first test of whether what you did worked.

00:17:40 - Speaker 1: I think you mentioned that console logs, right, are one method of sort of putting out or making the program more visible at certain points in time. I was trying to think of a good metaphor for what console logs are, right? It’s like the stream of the program and it’s like you’re shooting up a small bit of information at a point in time.

00:17:57 - Speaker 3: Yeah, it’s sort of like if you have a map of a territory and you take a journey along the territory that we go on loops and you cross back over your own path and you make jumps and stuff, kind of captures all of that in a linearized form, and I see it as a compliment to the map, you know, the map gives you the nature of the ter not people gonna get mad at me on the internet for saying that, but you know, that the map has some correlation to the territory hopefully, but also the linear view of everything that you encountered is also very useful.

00:18:26 - Speaker 2: One idea I would raise for the rebrand question I asked earlier was on the Ink & Switch piece we did on end user computing, and we can talk about the relationship of visual programming to end user computing, but we talked about this property of embodiment and it’s exactly what you said, the idea of taking something implicit and potentially very abstract and making it visible. And I think a lot of times in discussions I’ve seen about this, which are often among people who are already professional software engineers, and that includes both people who are very enthusiastic about end user programming or just increasing the accessibility of programming, but they themselves are still.

Professional programmers, I have that background mark does as well, and whether or not they’re pro or con, they’re coming from that perspective of someone who, for whatever reason found it was drawn to or was good at holding all of that abstraction in their minds. And so in some ways, I wonder when I see the objection of, I don’t know, there’s so much visual spaghetti here, but that spaghetti all exists in symbolic code as well. You just can’t see it, and you have to mentally model it, and some people are unbelievably good at that, and a lot of those people are people who go on to be really great programmers.

I’m probably kind of in the middle on that, my complexity threshold is Not that great, and I tend to always want to like factor my code in a way to make it more comprehensible and give myself more visibility and that sort of thing. So I wonder how much the needs of visual programming or this umbrella we’re calling visual programming, those serve sort of hypothetical people that today are not programmers but maybe would have that capability. Which I think is the other dimension of this, which is the learnability side of it.

00:20:10 - Speaker 1: Yeah, I definitely agree with that, that I do like to be the representative for, yeah, mediocre programmers or people who aren’t naturally inclined to it. And that I have heard a lot of the debates around it of people saying, Well, I love them, don’t take them away from me. You know, if you try to make it visual, I won’t be able to work in my crazy text editor with all my customized hot keys and I’ll say like, textual programming’s not going anywhere, you know, it’s like, this is all gonna be. Additional affordances built on top of a textual paradigm, you know, text is not leaving.

But people say they can’t understand why we would ever need it. Whereas my perspective coming into programming, you know, starting when I was 20 or so, when I was first shown VS code and the JavaScript tool chain that I was supposed to work in an NPM and the command line, I just went, you can’t possibly be serious. Like this can’t possibly be the way that I’m going to do this. Pranking you. Yeah, what? My computer looks like it’s just stepped back into the 80s. Like, I don’t understand. You know, I’ve grown up in the rich graphical user interface that gives me what I need visually and it was just shocking. So I do think that people who are programmers or professional programmers are maybe ones that, I don’t know if they want to say they’re more willing to step into that world of invisible connections and implicit logic. And then it is everyone that maybe would shy away from that or would find that more stressful or just less intuitive, they end up being like oh OK well programming’s not for me.

00:21:36 - Speaker 3: That really resonates with my experience, and I think there’s a few things going on here.

One is this issue of accidental complexity, where you have aspects of the experience that are just not really essential to doing the actual programming work. Basically, the tools are designed poorly and unnecessarily so. There’s too many things going on, some of them don’t work, they’re unnecessarily confusing.

I had a similar experience when I first wanted to do. Web app at a time, the state of the art was like these Java serflet things. This is like full on enterprise Java wildness. And when I encountered that, I was like, there’s no way I’m gonna do this. Sorry. And so I went back to, you know, my doctor scheme where everything is in a nice little bubble.

Eventually you got Ruby on rails and everything was fine and so forth.

But there’s this other issue of communication and information theory. So by that I mean, A lesson that I learned from management is whenever you’re communicating something, something is said, and then something else is received on the other end. It’s like a lossy process, and I think the reason a lot of programmers like text based interfaces is that they have very low loss there. They kind of understand what all those symbols mean, and furthermore, they’re very dense, whereas a new programmer, and I was once one of these, basically it all gets lost. It just looks like a garble of like alien symbols. So one of the reasons that visual programming environments could be so appealing is they have lower underlying information density, but they’re constructed in a way such the optical clarity is higher, right? More information gets through to the other side, and I think ultimately that suggests that there’s this end game of you get both the very high bandwidth potential of textile interfaces with the augmentation that you can get from other visual aids. I don’t know exactly what that looks like, right, but I think there’s something there.

00:23:12 - Speaker 1: I definitely, I think, discovered that when, as part of my work for Egghead, a lot of my work is illustrating programming concepts, right? So it’s very much just educational material. We’re talking like articles or animations that kind of go along with video lessons, where, you know, I’m visually explaining things like, you know, how graphQL works or how React’s hooks work. So it’s just taking concepts. And what I believe I’m doing is making visible the mental models. I tend to use the word me.

Metaphors, because I’m from more of a cultural anthropology background, and I’ve read a lot of thinkers like George Lakoff and Mark Johnson, who talk about embodied metaphors and sort of the fact that almost everything that we talk about in the abstract realm is based in our embodied experience of the world. So we have this inherent understanding that like, up is good and is more and is happy, we’ll see the stock market went up and the stock market didn’t physically move upwards, right? It, like, had more money, move into it. But we use the term up to think more, good, better, right? And we’ll say like, you know, I’m feeling down today, means I’m feeling sad. You’re not physically moving downwards. But the point is that we have these metaphors that are based in being humans with bodies that have like fronts and backs and ups and downs, and we map all our abstract reasoning on top of that.

And when you have to draw like a programming function. You really have to look at the language we use around programming functions and really figure out like, OK, there is a physical metaphor baked in here because all things are based off physical understanding, um, and then drawing it, you know, you draw, OK, there’s a component that’s a higher order component in terms of a lower order component it’s passing data, so it’s, you know, mimicking this idea of physical objects moving through space.

So it was very obvious to me in trying to draw programming concepts that it’s based in physical knowledge. And visual programming languages or visual interfaces, just make that explicit, right? They like actually draw a box that is above another box or it is to the right of another box, which suggests it’s in the future because we think the future is to the right and the past is to the left, at least in English, Western speaking countries.

That’s how we map time. So I’ve always just been kind of fascinated by how We can sort of use these embodied metaphors to help people understand programming concepts because everyone has a body, so you get these things for free. Like we all get the concept of front and back, and we can use those in interfaces to communicate to people, you know, the relationship of these otherwise very abstract lines of code that are just encoded things on a hard drive that would otherwise be incomprehensible.

00:25:41 - Speaker 2: That’s super interesting and also it actually connects to our design ethos for Muse. A lot of our concept there is we want to tap more into, for example, your spatial reasoning, which is very strong for most humans, as well as things like the direct manipulation and touch and so on, basically taking advantage more of our bodies and our minds, natural, you might call them talents or just built in capabilities because of course so much of what we do with computers we are really Bringing ourselves into an abstract intellectual realm, and we can learn to do it, but it is very much a learned and a learned over a long period of time skill.

00:26:20 - Speaker 3: Yeah, and I think this idea of understanding, whether that’s through metaphors or physical embodiment or mental models is really important because ultimately most programming errors or failures are ones of understanding and not ones of like mistyping, and I think a lot of our tools, they focus on The mistyping issues, and it’s much harder, yet more important to get the ones of understanding.

Like if you think about the most gnarly bugs you’ve dealt with, it’s usually because you misunderstood something about the world you’re modeling or how your program worked, not because you type out something. If you did something that kind of gets fixed quickly.

So anything that can help us understand what actually is, is very powerful, which, by the way, is another reason why I’m a huge fan of logs because they are a sort of ground truth with very high information density of what actually happened in your program.

00:27:07 - Speaker 1: Yeah, I’ve actually recently been working on a project that really sort of emphasized to me how important this is, right, of the mismatch between the program creator’s mental model and the person trying to write the programming language.

So, of course, I’ve mentioned JavaScript because that’s just kind of the world I work in. So this is a collaboration project that I worked on with a programmer named Dan Abramoff, who’s a really wonderful kind of well-known JavaScript developer who’s on the React core team. And so we’ve been collaborating to help create a course that visually, diagrams and a. Mas the underlying mental models of JavaScript, the way that it actually executes, which helps people get over a lot of the bugs that are written into JavaScript code simply because they don’t understand what happens when you say, let banana equals fruit or whatever. And then you reference that later, but if it’s been reassigned or it’s inside an object and it’s been passed through a function, they aren’t tracking in their head exactly where the variables gone and how it’s changed over time because it’s not visible in the program. So anyway, this project is all about showing people literal, explicit diagrams, showing them what happens when certain types of code executes in JavaScript and it corrects a lot of the mental models that are incorrectly taught in a lot of JavaScript textbooks. And this is where working with Dan and being able to help visualize this course, I completely reformed my understanding of how JavaScript code executes. But there’s no IDE that was going to show me that. So it was just one where I went, it’s great to make educational materials for this stuff, but it’s not built into the tool. Exactly.

00:28:35 - Speaker 3: It reminds me one of the earlier programming courses I took was in C and it was kind of notorious because there was this one point in the course where a lot of people hit a wall, which was when you introduce pointers. You know, it’s this incredibly important and fundamental thing that totally changes how the variable behaves, but just a little star, you know, in front of the variable name or whatever. And it kind of resonates with what you were saying about the mismatch between how people are thinking about what’s happening versus what is actually happening, the importance of correcting that.

00:29:04 - Speaker 2: How do you go about coming up with a metaphor when you’re working on something like the just JavaScript course or any of these egghead pieces? Presumably you need to develop your own understanding of it either from your own experience or from the course materials, but then you want to go to translate that to some kind of visual thinking or create a metaphor, hopefully when that’s correct and useful. How do you do that? Can you give us an example?

00:29:29 - Speaker 1: Sure. I’ll say there’s sort of a spectrum. I mean, I like spectrums anyway. There’s a spectrum of visual programming, and there’s a spectrum of metaphors, I’ll say.

So the just JavaScript project with Dan probably leaned a lot more on very direct spatial metaphors. It was very much like what is connected to what, what is above what. So it’s sort of circles and our. in a very classic diagrammatic way.

But a lot of the other work I’ve done for Egghead is illustrating programming concepts and what I would say is on more towards the poetic spectrum of metaphor, where it’s not technically, like, you know, the most accurate, it doesn’t exactly match to syntax showing what happens at each point. But I’ll make a lot of illustrations that will be metaphors that give you an understanding of like, oh, OK, if it’s a waterfall image, it’s like, OK, this is something about data flow, that sort of level of poetic metaphor that gives you more of the gist of an idea more than the literal syntax interpretation.

I’m thinking of one of like, I did one for Redux, which is a state management system for React. And I use like a little joystick with space aliens, like shooting, cause it has actions, so it’s like you can like shoot out actions from redux from like a single state source. So that’s one way, it’s a bit more fun. Obviously, it’s a lot more visually interesting than just doing a box and arrow diagram, but it gets the concept across to people that you have a central place and it’s shooting actions out from it. And the the process I kind of do for that is whenever I’m given something to design a metaphor for, I will read docs, you know, I’ll read programming articles, I’ll go see what people are talking about on Twitter related to the thing, stack overflow questions, and sometimes if the community is already speaking about it in certain metaphors, I just use that because they clearly already understand it. If they’re trying to explain it to each other on Stack Overflow and they’re using these metaphors, that’s usually what I’ll base them on. But a lot of the time, I’m just kind of doing basic reasoning of going, OK, what are the functions of this thing, what are its qualities, and then you do a bit more creative lateral thinking to say, OK, what else in the world has those functions and those qualities.

00:31:26 - Speaker 2: Yeah, I think that that points to another reason to have this kind of illustration in learning materials, which is just to make it fun and break things up, and yeah, visuals to break up, for example, long streams of text, or, you know, if it’s more of a video format, long strings of speaking, or you’re just looking at a screen, that kind of looks the same the whole time, you know, it’s just a code editor window or something. I’m reminded a bit of Wi Poignant Guide to Ruby, which was a beloved introduction to programming for the Ruby language and had these adorable cartoon characters and kind of a web comic style, and, you know, a lot of it was not specifically helpful for helping you learn it. It just made the thing just more fun to consume and more memorable because you connect chunky bacon with, you know, some of these more practical concepts that are being taught to you.

00:32:17 - Speaker 3: Chunky bacon, that’s a throwback, man, wow.

00:32:21 - Speaker 2: Now, I can’t believe we got this far without mentioning Brett Victor’s seminal piece, Learnable programming, and maybe somewhat to the point you’ve both made in various ways, which is, it doesn’t have to be either or. The idea of creating embodiment or visualization or showing more of what’s happening, the abstract pieces, how things are connected, how things flow in time can also go. With symbolic code and this piece from some years back now, but I think has been influential in terms of, well, people lamenting we don’t have programming environments that are a bit more like this, but demonstrates lots of interesting and inspiring ways to potentially make, for example, control flow more visible.

00:33:04 - Speaker 1: Yeah, the Brett Victor piece has kind of been a seminal launching point for this whole visual programming industry.

Of course, it was alive and thriving well before. I mean, you can kind of trace it back to, of course, Alan Kay and Xerox Park and Small Talk, and there’s a lot of precedents for this, but the learnable programming piece, I think, is the one I’ve seen referenced the most and has gotten the most people, you know, maybe in the last 5 to 10 years, more interested in it and to get them to go exploring and to go, OK, maybe the way we program now doesn’t have to be the way we’re always going to program. And the thing I really loved about Victor’s piece, I mean, of course, it’s a long article and there’s many aspects to it that we could kind of dive into.

But this idea of just that because code when it comes down to it, it’s something being executed in full on machine code and bytecode, and with so many layers of abstraction up from there, it’s not like we only have to represent it in one way. When you get into the higher level programming languages, syntax is one way of displaying what that machine code is actually running. And right in the rendered UI is another way of displaying what that machine code is running. And I don’t know why there’s not a reason why we can’t flexibly display the same set of machine code in different ways and turn it on and off and, you know, say, OK, I want to look at it in visual data flow, set. Or I want to look at it in a way that shows me the component architecture, if it’s like a component tree kind of context. So just flexibly displaying it in different visual formats, rather than saying, OK, syntax is the only way I’m ever going to interact with this code until it becomes the final output.

00:34:39 - Speaker 3: Yeah, interesting. And now that you mentioned that, I feel like, as is often the case, the gamers are ahead of us, particularly the game designers, like I’m not an expert in this, but my understanding is that the game engine environments, like they’re very sophisticated. There’s like the code view, then there’s like the object graph view and you can like, you know, turn on different pieces of the visual environment and there’s all kinds of performance based views for looking at memory and CPU and GPU usage and so on. So there’s perhaps something to learn from that industry.

00:35:07 - Speaker 1: I also think the data visualization industry is one we could really draw a lot from, like I know Brett Victor had mentioned he had researched a lot in Edward Tufty’s work, who’s a information designer I suppose is the best term for him, but he’s written a number of books like visual explanations and beautiful Evidence that cover.

Kind of the best practices for how you can represent data as visual charts in a way that it actually respects the original data and presents it in an accurate way that isn’t bending it or being too manipulative about the way that you’re presenting it.

But he has, yeah, a wonderful set of sort of principles for how to do it well.

And Victor had talked about how he read him before writing learnable programming. But the data this community in general has done a ton of work on thinking of different ways to visualize numbers, and they are fairly well known for it.

Do you guys know on dashboards you can sort of turn on line chart or you can turn on bar chart, you can sort of jump through different modes of being able to look at the same numbers and that’s sort of in my head, what.

We might be able to do with syntax in sort of maybe IDE plug-ins in the future.

Of course, there’s plenty of IDE plugins that do help you look at your code in different ways. But more of that is kind of what I would love to see is like more plugins that just are like, I’m just gonna show you the architecture side of your code. I’m just gonna show you how this variable changes over time in your code. I’m like one of those where you’re kind of like, am I missing something? Like, why isn’t this happening?

00:36:32 - Speaker 3: Yeah, I totally love to see more of it, and I think it can come both from the IDE language plug-in layer, but also bottoms up from individual projects. I feel like I must have at some point done one of these things where you write a little script that like takes your code or your system and compiles a dot graph so you can visualize nodes and edges with one of these standard dot visualization tools. And I feel like there’s a lot of opportunity to do that in individual projects like you feel the need to have a different, more graphical view into your project. Well, you’re a programmer, you can make it so.

00:37:01 - Speaker 1: And it kind of to bring us back to what end user programming. I mean the the strange thing about programmers, right, is they are end user programming, like of all the industries, they’re the only ones where they are both the users who have the ability to actually build their own tools and to actually get in and change their own applications and no other, I guess, community that uses computational tools is able to do that.

00:37:22 - Speaker 2: Here you’re talking about a programmer writing their own VS code plugin, something like that.

00:37:26 - Speaker 1: Yeah, exactly. Or even if you use a certain language or a framework, you can build a library that adds on to that, right? Even if you’re not on the core team and you aren’t going to PR into the main repo of that framework or tool, you can build extensions off it, you know, it’s the beauty of being able to fork and change and collaborate on these things means that there’s so much flexibility in in developers having their own authorship power.

00:37:49 - Speaker 3: Maybe we need more culture of jig making and software as there is in woodworking. Woodworking is interesting, it’s one of the rare fields where maybe the only one I can think of where they have that ability to kind of build their own tools in wood, and there there’s this whole culture of making jigs to help you build the furniture. In fact, you might spend most of your time making the jigs and the furniture is kind of this final, you know, culminating step. Perhaps we need more of that in the software world.

00:38:12 - Speaker 2: Some of the ways to visualize your code or your program differently that you both have mentioned like in the game world maybe something like Unity where you tend to build stuff into the game that lets you look at the scene graph in different ways and I think we have that a little bit in the web world. Dev Tools is one that we tend to give a lot of love to. I think we talked about that a good bit with Jeffrey recently here on the podcast, but there it is an example of what you said. You see the output.

But there’s also the kind of hover over elements and highlight which is where and what the extents are, and then there’s also the margins and padding stuff that kind of goes down in the lower right corner, but then you’ve also got an expanding hierarchical graph and you can see those all at the same time and even as you’re musing over its highlighting, you’re looking at exactly the same thing in three different ways, and you can spot different problems or understand by visualizing it in these different ways.

And I would further argue that we see that a little bit in more the native development world with toolkits like Flutter and SwiftUI where there often is even a live side by side. Here’s your interface output, here’s the code, and often you can change things on both sides and it flows. Bidirectionally, but I think in all of those cases it’s already something where it’s a visual output and so you’re interacting with some of the under the hood stuff around that visual output. It feels very natural to visualize that. When you come back to the more abstract things like program flow over time, that’s where maybe it’s less obvious to do that, or maybe it’s just less clear how you would do it in the first place. But yeah, I think that visual debugging or visual understanding of something that is already a visual output to begin with, is actually a pretty rich place for the stuff.

00:39:55 - Speaker 1: And especially, I mean, you’d mentioned the two way or bidirectional aspect of it is super important. I mean, one thing I’m really liking is there’s kind of a new rise in state charts and state machines, but visual ones that are flow both ways.

I’m specifically thinking of there’s a tool called X state that works for React where you can change either the diagram or the code and they’re side by side and they mirror each other.

Whereas if I want to debug some CSS thing in the browser, it’s like, I’ll go test it out in the browser. But then I still need to go back to my ID and like actually change the actual code. And it’s just that like slight friction of you’re not actually changing the thing itself. You’re like mocking it up in one, and then you go back to the actual source. We’ll know and, you know, interaction design, any amount of friction you put into a tool or an interface is just causing that much more cognitive load on the user.

00:40:42 - Speaker 2: I’m looking at Xtate right now. I hadn’t seen this before. It is very cool, like, uh, code and flow chart diagram side by side.

We’ve done a bit of research on the bidirectional editing capability as part of Incanwitch, and it can be very, very hard to do that well if you have something like maybe a tool like Apparatus would be a good example. I can link that in the show notes, but something where you have the ability to do direct manipulation with dragging something with the mouse or your finger in one kind of pain, and the other you have code that you can update.

But having not one be the end state, the output, I don’t know the bad version of this is Dreamweaver, right? What the output it writes is just completely incomprehensible and you can’t kind of go back from there. So that bi-directional thing I think is sort of a holy grail in one way, but also it’s very hard to do technically, and I think it also depends a lot on the nature of the programming language itself and the tools you use to edit kind of the symbolic representation.

00:41:40 - Speaker 1: Yeah, I usually take the long stream of, maybe, I don’t want to say failures, but disappointments in the wuzzy wig world. You know, everyone that’s ever dreamed of, yeah, dream era, you draw the website and you can change the size of the div and the code, you know, reacts appropriately. The fact that no one solved it after this long and that many, probably billions of dollars, I’m not quite sure. It shows that it must be an incredibly hard problem, you know.

00:42:05 - Speaker 2: Yeah, indeed, and I know in the X code world where it ships with this interface builder, which is essentially a drag and drop GUI system for setting up your interface, but very often I’ve seen iOS developers, they prefer to say just don’t use InterfaceBuilder, write everything in code because Or or later you’re going to be basically working in code anyways and if you have some of your program is represented in this visual thing that’s edited one way, some of your program is represented in code, the two interact with each other in a bunch of complex ways, you’d rather have just the one medium that’s actually better long term.

Now another great article that I came across by total coincidence, just as you and I, Maggie were talking about, possibly doing a podcast on this topic, is an article that is called Hackeroo’s Folk Wisdom on Visual Programming. And it essentially is a pretty large scale analysis of every single time something visual programming related has come up on Hacker News and sort of like distilling out the common themes and then starts with the really common and fairly shallow reactions and then goes into some much more nuanced and interesting discussion. So there’s lots of here including some we’ve already talked about, but is there any pieces of this article that are especially interesting to you, Maggie?

00:43:20 - Speaker 1: It’s funny, I had been following a lot of those same threads, and there’s a fair number of them, and they’re the sort of hack and news topics that go for hundreds of comments, right? People who just go deep on this stuff. I had been reading a lot of those same ones and passing through them right when that article was published, and I tweeted the author, I said, you know, thank you so much for doing the hard work of distilling that all for us.

But I think the thing I found most interesting about it, and just from reading the threads is, I mean, the strong opinions people have on it, or you really get into people have very Emotional, I want to say, like reactions to visual programming and sort of the cultural anthropologist side of me just finds that quite an intriguing thing to look at, where many of them, I want to say, have almost been like emotionally scarred by past visual programming languages, like the stories they’re telling in the comments of the time they had to work on the most horrendous proprietary visual programming platform, and no one could find anything and the literal spaghetti code is just all over the Place and they just wanted to go back to text.

So I think there’s some very legitimate critiques that should be acknowledged that maybe the past generation of programmers were sort of subjected to a really awful version of visual programming, but it would be really sad to let that frustration that other people have felt ruin what visual programming could be in a much better design format or just done in very different ways, done in ways that combine graphics and text. Yeah, so that was my main takeaway from it.

It’s almost like you have to do some historical healing before you can kind of move forward with this concept.

00:44:47 - Speaker 2: It is interesting that it can be such an emotional topic because I think if we were to boil it down to a bit of a duality, you have both people like I think those of us here talking right now who really believe in some kind of end user programming future or programming being more accessible or just computing as a medium is so. Important for humanity’s future, and we need just more humans and a broader array of types of humans, different perspectives and different backgrounds to be able to access this medium.

And so that’s maybe something we feel strongly about.

Certainly I feel very passionately about.

On the flip side is maybe as you said, first of all, the bad experiences that have scarred people, but also you do come back to this again, the hacker news comments are people who many and most are probably professional software engineers. There are people for whom the pure text model and classic text editors and IDEs and so on serve well, or maybe it’s less about being served well and more than they’ve had to go through the pain and suffering to get over that hump, to learn all those abstractions and manage a whole. All that stuff in their head, and then in a way, maybe they would feel robbed if someone else can just kind of skip over that.

I actually had that feeling. I learned a little bit of electrical engineering many, many years ago doing Burning Man art projects. So I wanted to make stuff that lit up in cool patterns with LEDs and so on. And I had to go through the whole kind of classic electrical engineering process, and I found it very challenging. It didn’t really fit with how my brain worked. I thought it would be relatively easy because I’m a computer programmer, but circuits are a whole other thing.

And just a few years later we had the raspberry Pi and the Arduino, an increasing number of these kits and Ada fruit and all these much more accessible things. I remember talking to a friend who said, Oh, you know, I’m going to set up my project that will have, you know, these LEDs or whatever. It’ll color cycle like this, and I’m like everything you’re describing there, that’s going to be a lot of work. I hope you’ve really budgeted some time. And he bought these kits. You know, I ate a fruit, and a week later he had this amazing thing working and I went in and asked him about it and I realized that he just didn’t need to learn all this stuff about debouncing with capacitors or whatever that I had to fight through because there are these just more accessible platforms. And there’s a knee jerk, I think I had that then, a knee jerk kind of like, wait, I had to walk uphill both ways in the snow to learn to do this. Other people should have to too, or I wasted my time or something like that. But of course, I think it’s much better that there are much more people that can now do things like make cool light up projects because these technologies are more accessible. I feel like there’s probably a version of that for today’s programmers versus people that might access it in the future if it was a little bit more learnable.

00:47:32 - Speaker 1: Definitely, there’s also another I suppose trend in people that are a bit anti-visual programming or I’ll say resistant to it or have some, you know, skepticism and reservation around it. Is this, I have found this sort of a duality where people lump themselves into going, OK, I’m a math logic person and art people are on the other side of campus, and I am not an art person, and everything visual is art.

And this idea that they’re not visual thinkers, I hear that all the time from people who are in programming, and I can’t really fathom what that means in a lot of cases because I go, Well, visuals are about embodied when you have a body, and I think, you know, you could probably draw a box and an arrow and communicate some visual stuff if you And trying to draw some detailed drawing.

But yeah, I think the, the, I’ll say yeah historical divide of art from science and visual from logic and this idea that math happens in textual syntax is a bit of a hangover that people just think, well, visuals don’t belong in math and programming.

But I mean this is why I think cultural anthropology, one of the sort of points of it is to sort of show people the vast diversity of expressions that human. Life and culture can take and that everything we have was invented at one point in history, and we are one very specific narrow manifestation of possible human culture, and there are many others, and we could switch to them or move towards them and we don’t have to live in some world where math and art live on alternate sides of the dichotomy.

00:48:54 - Speaker 2: That’s a really powerful point, Maggie. I feel like we should wrap it up there, but for people who are wondering about examples, they want to see what this looks like because indeed it is a visual topic beyond the ones we’ve already mentioned. Are there some good resources you can name that allow someone to flip through and see the variety of different experiments folks have tried for visual programming?

00:49:15 - Speaker 1: Definitely, there’s some people who have done a lot of wonderful labor compiling screenshots of I think every possible visual programming language or platform that’s ever existed. And what I want to point to that we’ll put in the show notes is a visual programming codex that Ivan Rees put together. Which is a GitHub repo with just a wonderful array of examples. And there’s another one called Whole Code Catalog. So a nice play on the Whole Earth Catalog that has, again, a nice sort of categorized collection of different programming platforms and alternatives and the sort of the different qualities each of them has all nicely organized in a beautiful website.

00:49:52 - Speaker 2: Well, thanks everyone for listening. If you have feedback, you can write us on Twitter at museAppHQ or on email, hello at museapp.com. You can help us out by leaving a review on Apple Podcasts, and Maggie, I’m really glad you’re advocating for making programming more visual, more embodied, more learnable, and I’m especially inspired by the idea of having a less bifurcated world between the art thinkers and the math thinkers. I think we need that. That’s a beautiful vision for me.

00:50:21 - Speaker 1: Thank you so much for having me on. I really appreciated getting to come on the show, and I’ve mentioned to you I I adore the podcast, I think I’ve listened to every episode up until now, so it was really wonderful to get to finally come be a guest.