Transcript

00:00:00 - Speaker 1: To borrow the Hobbit, one does not simply build a new sync layer. You start with a product and you want to build a sync layer, and now you have two products. You had a huge undertaking to build this kind of a system on the server and on the client and should not be a default answer, I think, for anyone, and it was not our default answer, but I think it worked out and was the correct decision for us in the end.

00:00:27 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for deep work on iPad and Mac, but this podcast isn’t about Muse product, it’s about the small team and the big ideas behind it. I’m Adam Wiggins here today with my colleagues, Mark McGrenigan. Hey, Adam. And Adam Wulf.

00:00:44 - Speaker 1: Hey guys, it’s great to be back.

00:00:46 - Speaker 2: Now Wulf, you took a little staycation recently, and I understand you’re working on a little side hack project there called Developer Duck. Tell us about that.

00:00:56 - Speaker 1: Yeah, it was kind of nice because the vacation just happened to align with all of the chat GPT AI magic that’s been released lately.

So I spent a fair bit of the vacation just working on a prototype for a developer tool that uses AI to help developers kind of build faster to take over some of the tedious tasks so the developer can work on the more meaningful tasks. So it’s been a really fun adventure. It’s got, you know, X code integration and it’ll edit your source files for you or add comments or explain code you don’t understand or fill out code and replace comments with code and it’s got a chat feature as well, just like chat GPT but it’s focused and prompted more specifically for developers, so it’s a bit less for both and a code highlighting and all sorts of stuff. So it was just a really fun. Yeah, adventure to kind of see what can these AIs do now and is the hype real or is it really just someone behind a curtain?

00:01:59 - Speaker 2: And I think the name there is a reference to the uh rubber duck from the pragmatic programmer, am I wrong?

00:02:07 - Speaker 1: No, that’s exactly right.

It’s a really common metaphor for programmers to talk to a rubber duck and just verbalizing the problem out loud and kind of pushing it through the language center of our brain helps us clarify what it is that we’re actually looking for.

And so this is that sort of a thing.

You can chat with the robot, express your problems, it’s able to prompt and kind of guide you towards solutions sometimes. But it’s really a great first step before you go interrupting a coworker and pulling them out of their flow state. You can stay in your flow state and talk to the rubber duck and hopefully get a solution without much delay in either your day or your coworker’s day.

00:02:52 - Speaker 3: Yeah, I found rubber ducking to be unreasonably effective, and so I can only imagine with the supercharging of Jet GPT it’s even better.

00:03:00 - Speaker 1: Yeah, it’s actually been really fun because a lot of times I’ll know what I need to do, but I don’t know the right jargon word to search Stack Overflow or to search Google.

And so sometimes even just that, you know, the 1st 5 or 10 minutes of searching around is just understanding what is this problem actually called? Like, what is it that I actually need to do? I know the problem I need to solve, but I don’t know what the name of the solution is. And so just typing the problem into something like developer duck, it actually prompts back with, oh, these are actually great bunch of keywords and ideas and even possible solutions that I should go try, and then it’s something I can go and dig deeper on on Stack Overflow or Google or whatnot to find the final answer.

00:03:44 - Speaker 2: So our topic today is local first, one year later. So this is a reference to or let’s call it a sequel to an episode, the 3 of us recorded about a year ago where we dive deep on the technical architecture from Muse’s sync system, and I’ll link that episode and show notes.

Now, as a reminder, local first is this idea that we want to get the benefits of cloud software. Think of Google Docs, where you can share and collaborate really easily with other people. But also gives you the benefits of call the more traditional style of just saving a file to your hard drive, right? It’s always fast, it’s available offline, and then you just have more data ownership generally.

And part of how we do this is using a technology that comes pretty recently out of the computer science world called CRDTs. So at the time we we recorded them, we were still in beta with this device syncing, we’ve launched Muse 2.0, which included that and lots of people are using that in production. And now we have our next iteration, which is Muse for Teams, which uses the same local first sync technology, but now for multiplayer that you can have many people on a team, each with multiple devices that are all in a single board or single workspace. So I guess the prompt for this episode then is what have we learned in the past year of running this system in production at scale? And maybe as a starting question, I’ll ask each of you, how does it compare to your experience working on call them traditional client server apps like a web app or an iOS app that calls out to a retrographQL API.

00:05:11 - Speaker 1: I think the biggest difference. Just using it day to day on the client. I how nice it is to just be able to work with data as if everything is 100% local on the device.

The network has been almost entirely abstracted away, and so there’s no waiting for the rest API call. There’s no error handling if the API is down. There’s no HTTP error 404 verse 401 verse 500, like none of that exists, which is So pleasant to work with because you just, oh, I’m gonna load some data, and then I edit some data, and then I save it back and I’m done, and all of the network stuff is handled.

At a lower level, completely abstracted away, which meant that building new features. Has been dramatically faster than when I build stuff with a traditional rest API or traditional server-based API. It’s just let us iterate. Much, much quicker than we could otherwise, I think.

00:06:17 - Speaker 2: That was a surprise to me, which was Julia reported, you know, she does more of the interface engineering, she’s essentially a client of the persistence layer you’ve created.

So when we use that persistence layer to replace core data, which of course is so well established and has decades of development, it’s well hooked into all of the iOS interface APIs.

She’s worked with it for a long time and I thought, OK, well our homegrown system is going to be Almost certainly less nice to work with, but she was really pleased with how easy it made everything and how few error states you need to deal with, and it just simplifies things because it exactly as you said, it feels like writing an old school program that just loads and saves stuff to the disk and hold things in memory, and just all of the complexities of distributed systems that you’re essentially forced to deal with one way or another through network errors and things like you mentioned. Aren’t a consideration in building your client side software.

00:07:15 - Speaker 1: Right, exactly. The core data has a number of just kind of programming niceties for how to annotate which data gets saved in the database and which data is just kind of ephemeral in memory.

And The way that we architected this sink layer is Really modeled on a lot of how the developer interaction with core data works. It ends up being almost the exact same.

Interaction at the code level with our sync layer as it is with core data, which is really nice because then it meant that we could just Switch some model files from core data annotated model files to now sync annotated data files, and it was almost a 1 to 1 mapping, so it was really easy to get in on board.

It’s really easy. To take a lot of the experience that developers already have after years of working with core data, a lot of that mapped almost 1 to 1 with a new system. And so that minimizing developer learning. I think it’s been probably just as important, if not more important than Some of the technical details of how it physically works behind the scenes, making sure that that interface for the developer is easy and convenient. It’s let us work much faster than we could otherwise.

00:08:37 - Speaker 2: Mark, what have you learned in a year of running these systems? Particularly interested in things that maybe were surprising to you.

00:08:45 - Speaker 3: Well, the good news is it hasn’t been that surprising. And by that I mean, I think we’ve realized all the benefits that we expected and hoped for.

It’s, you only need to update code in one place. It obviously doesn’t require the device to be online. All the data is there on the device when you need it, and so on.

And also, we’ve been working through all the challenges that we expected to deal with versioning differently, performance becomes more sensitive, you’re downloading a lot of data, you know, all these things that we basically expected from our research in the lab. So, I think we’ve got a lot more texture around all those benefits and challenges, especially the challenges.

But no, to me, real big surprises.

One thing that has played out a little bit differently than I had hoped and expected was how content aware the server is. So the very earliest research prototypes we had, the server really had no idea what was going on. You created these very abstract like channels and mailboxes, and the server had no idea how that mapped to people who were collaborating or documents or anything like that. You basically just shuffled bits around totally the direction of clients. And the motivation for that was to keep the server as simple as possible and to retain the option of doing straightforward and and encryption. And we tried going down that road for a while, but it’s proven really tough. I think the two reasons are, one, it does put a very big burden on the clients to have to basically route all their own mail everywhere and not being able to rely on any server, for example, saying, you know, these people are working together on this document that’s where these packets go to these people. And also, of course, it does become challenging if the server can never know anything about the content. You know, if you want to send an email notification about an update, server needs to know about that unless you do something really wild. So that’s been a little bit different. It’s not a huge difference, but Something I wanted to mention.

00:10:34 - Speaker 2: Yeah, I think we’d hoped both from the perspective of call it general kind of privacy principles, but also from the perspective of the simplicity of the server that it could be a completely dumb pipe for the data and we talked about this in our episode with Martin Kleppman about even the idea that someday you might bring your own sync service, that you know, AWS and other providers offer you kind of a pipe or a sync service that you could plug into any application the same way that a file system could plug into any application.

And I still like that idea, but it does restrict things you can do server side.

You mentioned the emails notifications is another one like, I want to get notified when someone replies to my comment thread when none of my apps are logged in, but like there needs to be some system somewhere on the back end that can make some kind of basic interpretation of the data in order to offer features like that.

00:11:26 - Speaker 1: I think one interesting thing that I’ve seen from kind of the outside of the server. Is that a lot of those abstractions are still true.

It is in still some ways a very simple pipe that just routes data back and forth. Just now it’s kind of a pipe with a window, so to speak, so you can kind of look inside the pipe on the server and look at things that you really need to look for.

But the other thing that, as I understand it, has been challenging is a traditional server architecture will have The state of the world inside of its database, done. So you just kind of query the state of the world and you get the answer back and you know exactly who’s talking about what. But because we’re working primarily on local data, It means that the server doesn’t necessarily have one single view of the world, one single, what’s the latest state of the world for any particular client or team, but the server needs to actually kind of Go back and read all of the logs from all of the different devices recently to kind of recreate the state of the world and determine what’s changed. Mark, maybe you can talk a little bit about that and just kind of what’s been different. And how it’s architected versus kind of a traditional rest API architected to do things even as simple as notifications.

00:12:46 - Speaker 3: Yeah, OK, so there’s two things going on here. There’s, do you have an event-based or a state-based view of the world, and is it a push versus pull? So in a traditional database you have a state-based view of the world and it’s pull. So for example, you have an object that represents a cart in a database and what’s in the database is basically just the current value of the cart.

00:13:13 - Speaker 2: And then when a client needs a car, it will ask the database for this card snapshot, and the surfer will reply, select star from shopping carts where card ID equals whatever and you get back a bunch of products at the end, right.

00:13:19 - Speaker 3: So the event-based variant of that would be instead of a cart, you have a log of cart changes. Add this item, remove this item, change this quantity, and if you roll up all those changes over time, you can build a view of the current cart. You don’t necessarily persist that.

And so we use this event-based approach. I found that to be fine. It does take a little bit of work and as we’ll talk about in the client, it does become more challenging if you want to do queries like select all cart where it contains pencils or whatever, that becomes more challenging.

That’s not too bad in my mind.

Then there’s also this push versus pull thing. So it’s I think relatively easy on standard servers when you get a pull request, it says, give me the cart with this ID while you go to your index on disk, you crawl a bee tree, and there you are, you’re at a cart with an ID, whereas in our model, what happens is you receive a cart change event. And then you need to figure out all the devices in the world that are potentially interested in cart change events for this cart and send each of them that update. And by the way, they might not be logged in right now, so you gotta do it, you know, now versus later and keep track of who’s received it and when and stuff like that. So that that part has been challenging, you know, it’s basically it’s an engineering problem, it’s not insurmountable, I don’t think, but I do think it’s harder than traditional poll style database.

00:14:41 - Speaker 2: Yeah, perhaps partially harder just because it’s less well precedented. We’re operating on a model that is, I think in the long run is and could be superior, we’ve already seen benefits from that, but we don’t have the same tooling or knowledge about it that we do on these more classic state-based systems.

I’ll give a little anecdote where sort of there’s a let’s call it a user benefit or as a person on the team who’s not part of the engineering and but I am testing internal builds and things like this. I’ve been very impressed by how good the data integrity is, and I think we talked about this in our first episode because essentially you’ve got the servers just shipping around these bundles of data. That it doesn’t have any insight into what’s in them, but even on the client, there seems to be a network layer.

Wulf, you can probably correct me if I’m interpreting this wrong, but it seems like it receives those bundles and then once it saves them, then it figures out what to do with them.

And in some cases if you have a new app schema, so that is to say we’ve added a new data type or something or changed the data type, the newer clients can interpret that, but actually if you’re on an older client.

It doesn’t know how to interpret it, but it still leaves the data laying there and so you may be able to interpret it in a future client.

And so we’ve had situations, obviously we’ve had internal builds for testing, whatever scheme has changed, etc. etc. and so you might have something where you’re on the wrong version of the client and you can’t see some of the data because it can’t interpret it, but it’s all there.

It’s receiving it through the network even if people are in real time doing edits and your client is receiving them, it doesn’t know what to do with them, but it just sort of leaves them sitting in the local data store and then when Get the new version of the client, now you can see all the new stuff and that’s basically resulted in any time there’s been data problems, it’s always transient, it isn’t like people being on different client versions results in some kind of data loss. It’s just sort of confusion and how it’s viewed. And that actually it is really nice from the perspective of me having more trust in, OK, the data is always there, it’s just how it’s interpreted, how it’s materialized, is going to depend on sort of the client version and the current code that’s showing it to me.

00:16:48 - Speaker 1: Yeah, that’s exactly right.

I think one of the really important things we did really from day one was clarify how the client talks to the server and make sure that that what we call the network schema was well versioned so we had a very clear definition of kind of how we ship boxes to and from the client.

And then we also have the app schema version, which is how do we version the data that’s inside of the boxes. And so what that lets us do is change and upgrade how the app understands the data model, regardless of how that data is sent to and from the server. And similarly, we’re able to upgrade and make more efficient how the data is sent to and from the server, regardless of the type of data that the app is sending. And so having a very clear definition about how the data is shaped in the application versus how the data is transmitted has been very helpful to make sure that everything that arrives on the client is in a very well known state, and if the app doesn’t currently know how to interpret that state, that’s OK. It just saves on disk. Next time the app updates it checks again, oh, I’m a new app. Look, I have some data here that I didn’t know how to look at last time. Let me try again. Oh yeah, now I understand it. OK, great, and it can continue on just fine, and that lets old clients on maybe a V9 or V10 of the app schema, continue to operate just fine cause they’re talking in The version 10 of their language, and whenever a client logs in with a version 11 of the A schema language, It can still understand all of the V10 things, and it starts talking in V11 language, and all of those V10 devices that are still on the network, hear that and go, I don’t understand this yet. Let me just save it. And then once they upgrade, then everyone’s talking the same language again and everything works fine. So it lets each device speak in the way that is comfortable to the other devices, interpret in the way that is comfortable from the other devices, and when it gets confused, that’s OK, it can just wait, and having those three states. Has meant that it’s been important as we’ve planned for new features to make sure that we’re planning with regards to those three states, that we’re planning with regards for backwards compatibility, and potential future compatibility, and What happens when two devices are talking with two slightly different languages and making sure that the correct updates are still visible while then fancy new updates, it’s OK for them to be invisible, and to make sure that the application still behaves in a correct way.

00:19:30 - Speaker 3: Yeah, and looking back, we’ve had a lot of dynamism on the so-called app schema. That’s what’s in the boxes, the boards and the ink and the text blocks and all that stuff. The network model of binary data plus transactional data plus ephemeral data, which I argued you could like intuit from first principles cause all applications look like that. That’s proven correct, like that basically hasn’t changed. Some small stuff around the edges to make it a little bit more efficient and support collaboration. But we’re still shipping stuff around in the same boxes we used to, more or less, even while the the application has changed pretty dramatically inside the boxes.

00:20:07 - Speaker 2: Maybe the comparison there is wire protocol to, yeah, database schema, but I think it is the case in the kind of cloud style client serverE applications that I’m familiar with and I spent a good bit of my career working with in the last couple of decades where, OK, we’re gonna add a way to turn things red, so therefore we need a new field called color that goes in our back end database which is SQL. The back end code needs to handle that. We need to add it to the API, which might be a crowd API, might be graphQL or something like that, and the client needs to interpret that.

It’s going to display that in the interface, and the reason that we get that dynamism, as you said, Mark, is the client can essentially update its app schema, add that thing in there, but know that it will be transmitted through without needing the back end to change, know anything about it.

And so you essentially only need to really change it in one place, which is the client side code. Now I think it does create a locus of complexity on the client for us, whereas maybe those client server applications, standard cloud applications we’re familiar with, to end up with more balanced complexity between the back end and the front end, but the not needing to coordinate. You know, OK, the back end engineer is going to design the API and the front end engineer is gonna build the interface and needs to consume that and all that back and forth for every little thing we wanna add. I think that’s a real boon to development speed.

00:21:33 - Speaker 1: Yeah, I do too. I think that’s been so important for us to be able to completely abstract away how data is transmitted to and from the server versus what that data means.

And the complexity is almost all entirely on the client, instead of being shared between the client and the server, but I don’t think we’ve ended up with double complexity on the client.

I think we have essentially halved the complexity of any app schema change, and so the client complexity ends up being kind of the same amount as it would have been with a rest server, but now we’ve could just completely removed a lot of the server complexity, and so the The actual amount of work, the actual amount of difference. Ends up being about half, in my view, compared to a more traditionalru-based classic server architecture.

00:22:28 - Speaker 3: Yeah, I think that’s right. And I think there’s a little bit of additional complexity on our client versus the typical one, but that’s due to the client being stateful versus stateless. And if you had a stateful arrest client, it would have the same issues with migrations and so on, which we’ll talk about.

00:22:43 - Speaker 2: Let’s talk about the operational side of it, so.

I’ve spent a good bit of my career, say, carrying a pager, although mostly that’s metaphorically, it’s notifications to my phone, but you know, you write code, especially in the early days of a startup, you don’t have a dedicated ops team and you set yourself up to receive notifications for your monitoring systems or your pingdom or whatever it is, that’s kind of just tracking whether the systems you’ve built are online.

You also have things like migrations where you need to, OK, we’re going to do it. I don’t know, let’s do it at 10 p.m. at night because we know the system needs to be offline for 10 minutes while we do this thing and it’s kind of stressful, and we don’t want to stay up too late cause we’ll be tired, but we also want to do it at night when volume is low.

And so one thing I’ve observed just kind of again from the On the inside, but outside the engineering team perspective that I have is, first of all, just the operational cost in general has been lower, that we spend less time on that. We have pager systems and what have you, but they don’t just seem to occupy as much of our teams. Mental and energy bandwidth.

But the other thing is, I think we’ve had to do one or two downtime just for migrations.

There was one recently and yeah, I remember it was longish, I think it was like 20 minutes, but when the only impact is your little filled-in circle turns to empty and Your devices don’t sync for a little bit.

Maybe that’s annoying if you’re in the middle of something that you wanted to use between two devices or a collaborative session, but you’re not stopped from working, you can access everything. It isn’t the whole O is slack down as notion down as GitHub down, my work is completely interrupted. You can really continue as is, it’s much more of a low-key event and both of those sides of it, like the team needing to spend less time and energy and the impact of customers for downtime. Being a little bit less. Both of those seem like dramatic wins to me, but I’m curious how you both see it.

00:24:36 - Speaker 1: Yeah, from my side, I think that’s been one of the really nice things is that the downtime that we do have has almost no client impact at all.

Certainly from the code perspective, because I can just save and read from the disk and the network is entirely abstracted away.

There are not any new error handlers that I need to cover. There’s no alerts or Error states with the data that need to be managed. I know that everything will eventually sync and everything will eventually come back down, and the state will be shared between all devices. And so a lot of that complexity is just kind of gone from the client, which is so nice. And like you said, the users can still work, they still get, you know, certainly more than 80% of the the benefits of used by having all of their data locally. So downtime is While still rare from use, very rare, it’s certainly not an impact for users when it does happen.

00:25:36 - Speaker 3: Yeah, the server downtime is less common than it would be otherwise. It’s less impactful on users and it’s much less stressful for the operators.

Which I think is actually quite valuable when you’re a very small team, you can’t actually staff a full standard pager rotation with a team of the size, so you either need to let that slip or have more flexibility like we do with the local first set up.

And I would emphasize something Wulf mentioned, which is In our client architectures, the offline case is the standard case. The way the client works is it basically saves data as if it was offline, and if it happens to be online, it goes and like does another step, which is send it to the server. And I’ve long been an advocate of this idea that if you don’t treat the error cases as a standard case that happens all the time, they’re just not gonna work. And we have the flip of that here where it’s the thing we’re doing all the time, so it works fine when the server goes out.

00:26:28 - Speaker 2: I like how again another reference to our episode with Martin Kleppman, he phrased it as, you always have latency, and that latency might be 50 milliseconds. If you’re lucky, it might be. second or a second if you’re further away, your network’s slow or something like that, but it might be 3 hours because that one of the systems in the way is offline.

And in a way you can treat those as all classes of the same problem, which is you just need to queue up what you want to send and send it and merge it in when things are back online again. That sounds so simple in concept, but in practice, when you have systems that are built around the expectation of things being online and needing a central server to resolve conflicts. Then you end up special casing all these different things, but if you treat it as classes of the same problems, which we do by assuming you’re offline and then saving it to a place that it can later be streamed, and of course in a format that it can be merged together, that’s a really Key element that’s obviously the CRDT technology, but it’s also to some extent this separation of the app schema from the data packets that are either on disk or in transit as something that can be completely disconnected from the application function transmitted at any time.

Well, maybe it would be a good time now to get into what we’re working on now and have been working on for the last 6 months or so, which is multi-user. So when we started on this path, we knew that we wanted the single player app, then it syncs to your devices and then it’s multi-user, and that indeed the same technology that syncs between your devices in this local first manner kind of offline as the default case could also be used for multi-user to get that either in real time that Google Docs figMA type experience, but also something more asynchronous and Indeed, that is what we have working now. So what was some of the big changes or what was some of the big architecture decisions to make going from, OK, we can sync reliably between a person’s Mac and iPad. Now we want to have several people connected to the same board or workspace with their Macs or iPads and have all that fit together. What did that transition look like?

00:28:52 - Speaker 3: Yeah, so we retained our same basic model, but had to add one layer of indirection. So recall in the single user case, the model was a user’s muse consists of basically all the edits they’ve ever done in one big log. And if you have a copy of that log, say on a different device, you can replay it and get the state of that news corpus on that other device and therefore, when one device writes an update, the update needs to be conveyed to the other device and so on. And when a new device comes online, it gets a full copy of just that log.

With the multi-user case, basically have two logs now, every user still has a log, which is like all the edits that they’re supposed to see, you know, all that’s dated, of course, but also at the edits that people who are on documents that they’re collaborated on made and each, let’s call it document for now, each document has a log that consists of all the edits ever made in, say, a board. And now the job is you need to maintain the logs for each of these users by like stitching together all of the logs of the constituent documents they’re collaborate on.

So that’s the layer of the direction. So you still have the same core model of you have immutable edits, you have logs, the users have a log, the devices catch up by maintaining a high water mark in the user’s log. That’s all the same, but then the addition is you now have this additional layer of abstraction with the documents and then threading those to all the constituent users.

00:30:23 - Speaker 1: The way that I’ve thought about it, having not worked on the server code, I’m curious, Mark, how much this is accurate or is a fairy tale.

But if there’s 1000 users and they each have 1000 records, then when a user talks to the server kind of in the old world, it was relatively easy for the server to say, oh yeah, you’re user A, great, you’re definitely within these 1000 records, no big deal. Let’s figure out which of these 1000 records you need to talk to.

But now because any particular user can be on a team with any other particular user, if you have 1000 users and 1000 records and a user logs in, instead of looking at just 1000 records, now you have to look at all 1 million records because you don’t know which other user might have written something to that person. So it’s really a huge kind of order of magnitude harder problem to solve now in many ways just because of that one extra layer of indirection that we ended up adding.

00:31:18 - Speaker 3: Yeah, the engineering is quite a bit harder on the server now because as you said, it used to be that you had N or N is equal to the number of users, totally isolated islands of data, you know, all the users' data, their devices, the high water marks, everything could be siloed off. Whereas now, if you think about these entries in a database are basically overlapping islands because you have, you know, a document is shared by many users, but then those users, some of them are also on other documents together, but those aren’t exactly the same. And furthermore, we’ve wanted to keep it general so that we can support future product changes, so we could have made it a little bit easier on ourselves by saying that there are K islands where K is the number of teams and used for teams, but then we’d have to go solve this problem again when we went to the fully general case so you can collaborate across teams. So yeah, it is pretty challenging.

One other example I’ll give here to add a bit of texture.

So it used to be that the only types of updates in the system were edits, like when you add a text or whatever, but now we have updates around the configuration of teams and permissions, so you can add a member to a team and you can add a team to a document and so on. So it used to be the kind of hardest thing for the server to do would be to get a new device online cause then that device needs to be caught up for all the Entries ever written for that user, but at least it’s already in one log, it’s not too bad.

Now the hardest case is you have a new user added to a team. So for that, the server needs to say, OK, who are all the users in this team, and what are all the documents that have ever been added to the team and what are all the updates in any of those documents and then zip together a huge log and send it over to that new user to be added into their existing huge log. So it’s just an example of how it can become quite challenging even though there’s just one additional layer of indirection.

00:33:04 - Speaker 2: The one thing I’m really interested to learn more about is what we usually call the unit of sharing.

Some of the things we’ve explored on the product side include things like sharing an individual board the same way that you can with a lot of products like a notion or craft or something where there’s a big share button and you can share kind of just that one document, something like a complete team workspace, which is the main thing we’re supporting right now.

But ultimately, as you said, there’s those islands, those overlapping islands that we need to think about it. I’ve heard terms fly around like scope and scope set, and I haven’t fully followed all that. So maybe you guys can be up to date, not just on how we think about the unit of sharing, but maybe what were some of the things we tried that didn’t work or how do we land on the architecture that we have now, I guess is what I’m curious to know.

00:33:48 - Speaker 3: Yeah, so the thinking with designing the sharing protocol was we wanted to keep all appropriate options open for the product, because this was before we even had sharing at all, you know, much less the variations of sharing that we’ve thought about, including sharing individual boards, whole team spaces, subspaces, boards, but their children as well, right? There’s all kinds of variants possible. So the idea was to keep that option open and we did that by creating this unit called a scope, and a scope is the smallest possible unit of sharing. It’s sort of like a physical analogy might be a book. And let’s do something crazy like go and cut the book out, you kind of got to give someone the book or not, but you can also choose to form those in the libraries and give people access to the whole library, right? So it’s not like the only thing you can give someone access to is one book at a time, there’s there’s ways to aggregate those things, but that’s sort of like the smallest unit that you would plausibly lend out.

And we specifically designed them and named them such that they weren’t any of the things in our system. They’re not boards, they’re not teams, they’re not nested boards, they’re not spaces. These are all things that might be defined as a set of scopes. But that’s not baked in. And then to emphasize when you’re sharing a scope actually in the system it’s what we call scope set. So you basically bundle up a set of scopes, which of course in the degenerate case can be one scope, but in full generality it could be any number of scopes up to and including the whole team corpus. And furthermore, the system does allow those currently to overlap, so you could share a team space with some set of people and then share a subset of that with a larger group of people, for example.

So basically the idea was to allow this full generality.

And so the key decision that we had to make at the beginning, which would be hard to reverse, as we’ll talk about, was what is the smallest unit that we might plausibly want to share in the future.

That is, this is the unit that’s kind of all or nothing when I share it with another person, and what we decided was the board. So you’re not gonna be able to share, or we didn’t foresee the need to share like a subset of a board. That seems plausible to be all or nothing, but we didn’t want it to be bigger than that, so we wanted to retain the option of being able to share just a board, even if we didn’t exercise that option in the short term or even ever. And I think that decision has proved pretty much correct. We have one little nugget in there with text blocks which Wulf can talk about. But I felt pretty good about that decision. I should say that there’s a bit of a downside if you allow for all this generality and then never exercise it. So say you had this scope system and you can share it down to a board, then the only thing you ever did was allow teams to share their whole workspace. Well, then you’re basically doing a lot of this accounting with scope sets and scopes and, you know, it’s a lot of records to deal with for never taking advantage of it. So you would hope that you would eventually get to using most of that granularity, but you don’t have to.

00:36:33 - Speaker 1: As you alluded to, I think the text blocks are an interesting case, and they are an interesting case only because of the history of the Muse database and the fact that we came from core data.

00:36:45 - Speaker 2: Maybe I’ll just interject here quickly and mention the text blocks are basically double click anywhere on the canvas and you can just start typing, including a copy paste, like a long form article. I’ve written a lot of longish articles this way with the concept of blocks is something that appears in Notion, Rome, others where you can kind of freely reorder them, drag them around. So in a sense it is sort of like an exploded long form text note, but it’s a really important data type for the use canvas.

00:37:13 - Speaker 3: Right, and furthermore, because they tend to be like lines or small paragraphs, you can have a lot on a board, you know, dozens, hundreds on a board.

00:37:20 - Speaker 1: Yeah, exactly. I think having scopes as the unit of sharing and translating that to boards has made a lot of sense for us because it’s given us enough flexibility while not being kind of too strict in what we need to do.

The way that we initialize those scopes is for boards, it’s very obvious it’s a board, but we also have lots of other data types. We have images, we have files, we have PDFs, we have note cards, we have text blocks which are kind of paragraphs of longer form text.

So there’s lots of different content types, and each one of those content types is a scope. And for historical reasons, back in the old core data database world presync, each of those types inherited from kind of its foundational document type. And so when we migrate it to sync. The most natural connection between core data and this new scoped sync world was to say, OK, every single document in core data now gets its own scope in the sync world, which works great because the scope and a document are essentially the same thing. A document was aboard, now a scope is a board. A document was a PDF. Now a scope is a PDF. But where that fell down for us was that text blocks, each paragraph, each kind of sentence on a board. was technically its own document in core data. And so then that meant we had this explosion of scopes in the sync world where a single board with a fair bit of text on it might suddenly be hundreds of scopes that we would need to synchronize.

00:39:00 - Speaker 2: And I assume that each scope comes with a certain amount of bookkeeping because of its ability to be a shareable unit.

00:39:08 - Speaker 3: Right, you gotta say this little bullet point needs to go into a set that can be shared with the users, this little bullet point, and so on.

00:39:15 - Speaker 2: Right, and really there’s no world where you’d want to be able to share these 3 text blocks, but not the 3 that are under it, you know, or share them with different people there from a user perspective, we know that those are going to be all or nothing.

00:39:28 - Speaker 3: Yeah, and furthermore, unlike big binary blobs like movies, there’s also no incentive to try to avoid copying, so you might want to put a blob in a separate scope, so if it moves to a different board, it can be changed by pointer only, whereas with the text block if it’s 10 characters, you might as well just, you know, rewrite it under the new scope.

00:39:49 - Speaker 2: OK, if I pick up a heavy video and move it to another board, including one that’s even shared with other people, that doesn’t require reuploading the video or something like that because it’s referencing that same blob, whereas if I copy a paragraph of text that just under this new system where a text block is not a scope, we might as well just bring all the text across. It’s fine that it copies it because it’s just a trivial amount of data.

00:40:15 - Speaker 1: Yeah, exactly, and importantly, all of that bookkeeping for text box being scopes needed to happen on both the client and the server, and so that explosion of scopes kind of Add a lot of extra headache for the entire development team on kind of every aspect of the Sinclair. And so that’s what I would consider a major migration of the app schema is unscoping all of these text blocks, so that way they become just very simple kind of attributes or many objects inside of the board, instead of an entire document and all of the kind of the heavy weight that goes with it, that it was before.

00:40:56 - Speaker 2: Well, speaking of unit of sharing, I’d also like to hear about, let’s call it edge cases. So in this local first world and this almost offline by default point of view, and Mark you described this event-based model and alluded to maybe that sharing, adding or removing someone to a unit of sharing a scope, I assume is itself an event. What actually happens or how do we handle, I’m offline and I remove someone from the team, but in the meantime, they’re online and they’ve been making changes. How do those things that are not really data edits, but are really more permission changes get handled in this world.

00:41:36 - Speaker 3: Yeah, there are quite a few interesting edge cases, and generally the way that we handle it is that the server decides the sequence of events. So unlike content edits, where we use a vector clock to allow clients to resolve them without the server needing to make any decisions, because semantically it’s kind of fine for people to be concurrently editing things.

With respect to permissions, there really needs to be one order that everyone agrees on, and the only place that that can happen is a server. So when events come into the server, not when they’re written on the client, when they come into the server, they’re sequenced monotonically and the server can use that to decide who can do what. If you think really carefully about it, you can get some weird behaviors out of that, I guess, but like with our experience on distributed handling of content edits, it’s just not a practical problem for us right now. As far as I know, we haven’t had any of these weird edge cases in practice. The real problems that we have are much more mundane. But it is there.

00:42:36 - Speaker 2: There are other things we’ve learned from, obviously we’re not running the multi-user stuff at anywhere near the scale of the multi-device stuff that’s, you know, live in a production product in the App Store, but still, we’ve had a good number of users and edits come through the system over the last few months that it’s been live, including our own team’s use, which is pretty heavy. What have we learned from that? Have there been surprises? Have there been things we’ve wanted to modify as part of, you know, working on this alpha slash beta software?

00:43:06 - Speaker 3: I’ll give you one little use case, it’s been a challenge in multi-user. This is what I call the link click experience.

So it’s very common with SAS that you’re working with a team on like a weekly planning doc, and the team leader in the morning goes and prepares the document, and then they paste a link, a URL to it in Slack, and then everyone goes and clicks on the link. So I really didn’t like the behavior of this in traditional SAS apps where what happens is when you’re a team member clicking on the link, what you see is a spinner for 3 seconds. I was incredibly annoyed by that.

We’re gonna fix that by doing local first. Well, sort of.

So now there’s a potential problem where you click on the link, it opens up your muse app and it downloads the whole weekend of updates from the entire team, right? Or the whole day of work from people in Europe, if you’re in the states. And you click the link, you expect like a normal SAA for it to go to the weekly planning board, but at first has to basically download and process all the updates for the day. And that’s an extremely critical flow for a SAA, and it’s one that, to be totally frank, don’t handle great yet. So that has been, that’s an example of a challenge that’s unique to local first.

00:44:10 - Speaker 1: I think that’s a really interesting one because I think the superpower of local first is that you always have your data and you can always work offline, everything always works as you expect.

But that same superpower becomes your kryptonite when you’ve been offline for a weekend or offline for a vacation, and you need to come back to the office and suddenly you have to wait for, you know, even if it’s just 15 or 30 seconds. When all of the rest of your work has been almost instant, suddenly that 30 seconds feels like a lifetime.

And cleaning up that user experience, I think is certainly on our list, but it’s definitely one of the weird kind of edge KC things of this type of architecture.

00:44:51 - Speaker 3: Yeah, and to be clear, it’s absolutely fixable. It’s not really a flaw of the approach, it’s an engineering deficiency, and we will fix it eventually. But you are quite susceptible when you’re trying to bring all the content in locally. If you are slow, for example, that becomes very apparent when you’re trying to bring in the whole corpus versus when you’re just trying to bring in a page in a traditional set up.

00:45:11 - Speaker 2: Yeah, I think we anticipated this in the research phases many years ago, and we always speculated that in our dream world, and we’ve talked about this with not only among ourselves, but many of our guests here, including folks like Martin Kleppman and Jeffrey Litt from the local first world that something built into the operating system where you essentially had a syncD, you know, Damon that’s running in the background or it’s part of your file system that it is always syncing those changes whether or not you have the app open.

And so when the app opens, it can just interpret kind of what’s already there on disk, that sort of thing where subscribing to events is something that’s part of the this kind of next generation file system. I think that’s what we always hoped for, and you could basically get a similar effect with Muse if you just left it open all the time and minimized so that you’re always streaming down the changes, as long as your computer’s kind of awake and online, and then when you click on the link, it just sort of brings it to the foreground versus needing to go get a bunch of updates.

00:46:07 - Speaker 3: Yeah, although, well, I feel like we need to look into this, cause whenever my muse is running in the background, the Mac has like no problems looking for system updates and all kinds of ridiculous stuff that I don’t want to happen, but I’m not sure if it’s always getting the updates from the sync server when it’s sleeping, basically, so we should look into it.

00:46:23 - Speaker 1: Yeah, absolutely, it’s one of those, uh, many things on the list. But certainly, as you said, a solvable problem, which is always nice.

00:46:32 - Speaker 2: This is a downside to pioneering a new architecture is that, you know, you gotta kind of build a lot of the foundations yourself and you might run into things that are sort of handled or solved or just standardized, let’s say clicking on that link and getting that spinner is pretty well established for the classic web sass, but we’re sort of reinventing some of the primitives of the universe that we’re working within, so that creates an ongoing stream of work for us.

I’ll also mention kind of on the user experience side, I think another closely related thing is not just the weekend, but if you take a vacation and you have a team that’s very active generating a lot of content, particularly again, if you use video or heavy PDFs or lots of ink, that sort of things, you can essentially fire up your iPad, fire up your Mac, and find you have a gigabyte of updates to download. And again, it’s not a huge deal, but it may be surprising for, you know, if you’re compared to cloud software where you can open any page and notion without needing to download all the rest of what has changed since the last time you opened it. So that’s a benefit to that like really shallow cache of the web kind of standard model.

Now of course the flip side of that is I go to open a page that I had opened 30 seconds ago and my browser’s already forgotten about it, but notion’s offline or I’m offline or something and I can’t load it and I just get a spinner. So you know there’s trade-offs there, but the way I see it is that part of what we’re doing here is seeing how this local first architecture can work in practice, what it means for users, both benefits, but also these edge cases or downsides, and what can we do to mitigate those, or are there places where there’s just truly trade-offs where this is going to be sort of a worse experience than other approaches like client server or even classic desktop software. So we’re in the process of exploring that and it’s part of what’s fun and exciting about doing this work.

00:48:24 - Speaker 3: One other fun one that I’ll mention that’s become apparent with multi-user is the dancing cards. So when you come back from vacation and you’re receiving this big sink down at the data layer, it’s pretty clear that we should be telling the user, like there’s a sync in progress, the little indicator on the bottom right can be flashing or whatever, and then when it’s done, it’s done.

But then what you display on the actual boards is a pretty interesting question. You could like make it impossible to interact until it’s fully caught up, but that doesn’t feel great.

What we currently do is that as we’re getting updates, we’re like replaying them and the cards are literally dancing around, reflecting the moves that they’ve made in the last week and the additions and subtractions and so on. So it’s kind of funny to watch. We still need to figure out what the exact right user experience is there, not to mention the technical problem of how often do you to bounce the re-renders and so on.

00:49:09 - Speaker 1: Yeah, I think there’s a lot of interesting things that we could end up doing that we’ve certainly talked about and planned for, compaction being one of them, where if I move a card and move a card the second time, then that first move, we really don’t need to store anymore because it’s just never gonna happen.

And so being able to remove old content that we know is going to be overwritten from the database, that’s gonna dramatically reduce the number of things that a user would ever need to download.

I think we can probably reorder the types of updates and so that way, instead of going from past to future, we can Possibly send back updates from future backwards to past, and so that way the most recent arrives on the device first. So I think there are a lot of technical things that we can do to minimize kind of the strangeness of some of these updates. But yeah, it’s definitely a new frontier and so we’ll find strange new experience. OK, now we have the technical solution, let’s implement that and then that just moves us to a different strange experience that’s slightly less common, that’s a little bit further down the road, and it’s just a continuing adventure.

00:50:15 - Speaker 3: Yeah, well then I’m looking forward to working on there as prioritization.

And we kind of alluded to it. Right now we do some prioritization because we prefer to download the packs relative to the blobs, which is good because packs are much smaller and they’re more important. They tell you what boards are where and what cards are where and so on.

But I would also like to see us prioritize blobs. You can imagine constructing and continuously updating a priority tree based on how close a blob is to your current position in the hierarchy, or other factors like how often do you open up a known board, and based on that be constantly prioritizing your blob download so that You, for example, prioritize the ones that are on your board, and also on the board that’s like 1 hop away and then 2 hops away, and that’s always being reshuffled according to where you are on the board. I think that’d be cool.

00:50:58 - Speaker 2: So as a closing topic or a potentially very substantial closing topic, a question I get fairly frequently from other teams is basically what should we do? We agree broadly with the principles of local first, we need to sync our application data as basically everyone does in this modern world.

How should we do that? What should we use? Should we build it yourself the way you’ve done it? There’s now an increasing number of commercial products that offer off the shelf solutions like replicache and live blocks and party kit. There’s libraries like Amerge that we worked on at Inco Switch or YJS.

But then of course, we chose to build our own and we talked a little bit about the motivation for that back in the first episode.

So yeah, I guess if a team came to you today and said that they’re working on a piece of application software that is in the productivity space and wanted to implement syncing between devices and multi-user collaborative support, both real time and asynchronous, what would we recommend them or what would you both recommend them in terms of the best way to do that today?

00:52:08 - Speaker 3: I’m shaking my head on the podcast cause it’s really tough.

There’s not an easy answer here. There isn’t, as far as I know, any satisfactory, fully integrated solution.

Now, we haven’t developed that ourselves, even for our internal use, and I can explain what the gaps might be.

And there are also some existing commercial offerings and maybe some open source offerings, but as far as I know they’re still quite a ways from the full solution.

So it’s tough.

What might I do if I was starting again? Well, first of all, it’s very hard to recommend that someone roll their own unless perhaps they’re extremely interested in it and they’re an experienced systems engineer.

I think if both of those aren’t true, you’re really asking for trouble, and then what do you do? I think it’s worth trying out the existing open source and commercial implementations and doing like a smoke test or a bake off where you try to build a basic app, like a to do list or something like that, and each of these things, I think you would find that none of them are there yet, especially if you do some reasonable anticipation of the things that you’re gonna need in a sophisticated production app.

So it’s tough.

I want the industry to keep trying, but there really isn’t an off the shelf solution that’s satisfactory right now. And I think for that reason, you’re just wearing your business hat.

I don’t think it’s fair to make the immediate turn on the decision tree towards local first, as much as I like it, right? I think you got to be open to the possibility that it’s not appropriate to spend your innovation tokens there. We can link to the talk for this, to spend your innovation tokens on the persistence technology. You might want to spend it on a new business model or a new market or something, right? Yeah, I mean, it’s a bummer to say that, but it’s still early.

00:53:41 - Speaker 1: I agree. I think there’s a lot more options today than there were when we started, what, almost 1.5, 2 years ago now, on our own sync engine.

00:53:50 - Speaker 3: I mean, I’ve been working on this for like 5 years.

00:53:53 - Speaker 1: Yeah, yeah. And it’s such a tough thing, it does keep getting better and, you know, like you said, every product has its own needs and will be able to accept slightly different trade-offs or dramatically different trade-offs than any other app, and so it’s In building anything, the should I roll my own question is almost always answered with, of course not, that’s generally a bad idea. You know, leverage what other people have done.

I think because it is such a young world for sync and for CRDTs and for local sync in particular, it is a reasonable option. I think if I went back and talked to ourselves a number of years ago, I think I would still recommend that we do this, cause we’re able to decide. Which trade-offs we care about and which ones we don’t, and make something that works really well for use and for our use case, but You know, to borrow the Hobbit, one does not simply build a new sync layer. You start with a product and you want to build a sink layer, and now you have two products. You had a huge undertaking to build. This kind of a system on the server and on the client and should not be a default answer, I think for anyone, and it was not our default answer, but I think it worked out and was the correct decision for us in the end.

00:55:14 - Speaker 3: Yeah, I’m, I’m happy we did it, and I think it was a good call. I just think it needs to be, you know, appropriately tempered for people ask us that question of what should we do.

Wulf your comment reminds me of something that the Hoku postgrads team told me back in the day, which is, I, I hope I’m quoting this right, but they said something like, plan for 10 years to build a database, and I’m not sure if that was their original coinage or if that’s due to the post grass community, but it’s been my experience that’s basically correct. I, I’ve seen now quite a few database projects attempted and You know, often in year 5 they’re kind of just getting their sea legs and starting to figure stuff out. So it’s not too surprising in that respect.

This is basically building a database of sorts.

I’ve been thinking in anticipation of this episode, like what would a satisfactory commercial solution look like, like what problems would need to solve, and I think it’d be fun just to enumerate my list here, so just to quickly run through it, I have.

Obviously you need basic persistence and networking to be able to get the data back and forth, but importantly, you also need built-in batch line pipelining and compression. It’s something that I think a lot of systems miss.

You need compaction and excision. Eventually you need anti-entropy. We don’t have that yet. You need the ability to handle this on in the background. You need some sort of two-way interface like an ORM comparable for both crude and rendering. You need versioning, migrations, handling all those cases that we’re talking about with the disjoint worlds. You need something around like queries, indexing, declarative views, something like that. You eventually need prioritization and for similar reasons you need the ability to declaratively load and unload, so everything isn’t either all or nothing on disk or memory. That’s a lot of stuff. It takes a long time to build. It’s actually quite hard to get all of that right, I think, unless you’ve seen the very specific problems in production. So it’s tough.

00:56:51 - Speaker 2: And I’ll add to that list developer experience, right? You’re designing an API, you’re often asking developers, you know, in the iOS world, they’re used to working with say core data or maybe something a firebase in the web world, they might be used to working with something like local browser storage or, you know, in the rails world, you do yeah ORMs on the back end, that sort of thing, and those things are really well established.

And developers know how to get their data out of whatever persistence layer they happen to be using and turn it into their rendering layer, and now you’re offering them this new as exactly as you said, a new database with new contours and new capabilities and behaves in different ways, hopefully better ways in the long run in terms of user experience, but that is a whole new surface area, a whole new API, a whole new experience for the developer, and certainly I know that that’s something that a lot of these projects like Amerge and YJS and the commercial ones I mentioned.

Spend a lot of their time on it’s just like what’s the right API that feels familiar and can be used in a way that’s maybe similar to other ORMs or persistence layers, but also offers the things that make this sync-based approach and local databased approach unique.

00:58:01 - Speaker 3: Yeah, and this developer experience thing reminds me, we’ve done this whole podcast basically talking about this world of Swift, which we use both for iOS and Mac, and we’ve also implemented a little bit of this client side protocol and node for testing purposes and on go for doing server side content inspection, but you gotta grapple with the idea of in a full Typical production Sa app, you need some answer for whatever language and environments you’re using, which typically will include at least the web, the Apple ecosystem, the Android ecosystem, and something like server backend.

And to be honest, we’re a ways away from a satisfactory answer there, especially when you consider that this needs to plug into the rendering side and that those rendering environments are unless you do something pretty wild, they’re all completely different.

So I don’t know what the answer is.

Maybe it’s something like a lip sync, which is written in C or rust, then you have a layer on top of that for the other environments. Maybe you basically have a template that you hand transliterate into the different programming languages. Yeah, a lot of work to do.

00:59:04 - Speaker 1: Yeah, that was one of the biggest efforts initially building this sink was some of the things you already mentioned, Mark, which was threading, and how do you make sure you’re doing things on the background thread versus the main thread and queuing things appropriately or balancing what’s in memory and what’s on disk, and how often do you load and save to make sure that you’re efficient. We’re still a long ways from the complete solution there, but getting that initial foundation for a strong developer experience, that’s easy enough for a new developer to use, but also efficient enough that we don’t, you know, shoot ourselves in the foot. Every other line of code is a very difficult balance to make and it’s something we’re still getting better at and There’s still a lot to do.

Core data, for instance, handles collections of objects, kind of the one to many relationship, much better than our sync layer currently does. There’s a lot of Code that we need to write to handle cards on a board or ink strokes on a board or some of the other collections that we have. And I think going on what, 2 years now, that 10 years for version one of a database sounds about right.

Even though we’ve done a lot in these 2 years, I think there’s a lot left to do to get to. Kind of a production, fully scalable, abstract Sinclair.

01:00:25 - Speaker 3: One positive note I’ll mention we’re glooming a bit here is I do think we’ve correctly landed on SQLite as the underlying sort of data storage layer.

01:00:36 - Speaker 2: All roads lead back to SQL light, right?

01:00:38 - Speaker 3: Yeah. Yeah, SQL is an incredible piece of software, one of my all-time favorites, as I said many times on this podcast, but you need some foundation to be able to persist the data reliably, and like one does not simply write to the file system, it’s a complete disaster trying to write directly to the file system for anything other than perhaps large blobs.

And obviously get all kinds of primitives like in the season and the stuff that are very useful in sequel light. So it’s a little glimmer of hope like you can imagine a lip sync that goes on top of sequel light, which of course runs everywhere, it’s very portable. So I’m glad at least we’ve got that little piece nail down at least.

01:01:13 - Speaker 2: Your mention of language bindings. Mark reminded me of the really great work the automerge team has been doing. You know, we evaluated automerge for use in use again when we started this, which was quite a while ago, and at the time they really didn’t have any good solution for non-JavaScript, non-web world stuff, which obviously we are.

Since then they’ve come out with a 2.0 release. There’s a really great blog post detailing the history of the project and where it’s going and everything like that in the show notes, but this includes a complete Read and write and rust by our good friend and colleague Orion Henry and many others who worked on this project to essentially not only make it many orders of magnitude faster, but also make it possible to link essentially with those C language bindings and potentially be accessible from any. Language.

So still, I think a long road to fill out all the rest of the infrastructure that goes with that since automerge just covers the CRDT part, not the networking, for example, and it’s still quite a challenge or you still need to be a real expert to integrate that into your app, but I do think there is a world where automerge plus some kind of networking layer, plus like an electron or Atari on the website or something like a native app on the.

Kind of MacSwift side is a plausible way to build a new type of application. So I think that kind of off the shelf best practice, whatever you want to call it, has come a good ways even since we were last talking about it. So I’m hopeful with a number of great folks working on this both in the research world and on the commercial side and the interest in it.

That given another year or two from now, it could be much closer to something where the average application developer could just say, yeah, of course I want to add local for sync between devices and real-time collaboration that works seamlessly and handles all the offline cases. I just plug in component X, Y and Z and bam, I can just go and work on my application and I get all this stuff with relatively low effort. Definitely not there yet, but I could see a world where that could come true. Well, let’s wrap it there. Thanks everyone for listening. Join us in Discord to discuss this episode with me, Mark Wulf, and our whole community, the links in the show notes. And you can follow us on Twitter at MAHQ. Mark Wulf, I’m really pleased with how far we’ve taken this so far, how well it’s performing in production, despite the challenges you inevitably discover from being on the frontier. And well, I hope we can do another episode a year from now and see what we’ve learned about multi-user since then.

01:03:49 - Speaker 1: Yeah, thanks for having me on. It was fun to be here again.

01:03:52 - Speaker 3: Yeah, right on that. I’m looking forward to it.