Xilem 2024 plans

Raph Levien, January 10, 2024

2024 promises to be an exciting year for Xilem. The big news is that Google Fonts is funding the work of four open source contributors to the project: Aaron Muir Hamilton, Daniel McNab, Matt Campbell, and Olivier Faure.

Aaron will be working on text layout, building on parley, text input including IME, and Android platform integration. Daniel will be working on GPU infrastructure and performance, with much of the work upstreamed to wgpu. Matt will be working on accessibility, and Olivier will be building the core widget tree, with a focus on developer experience.

Xilem is an ambitious project to explore the limits of UI performance, as described in my High Performance Rust UI talk. The goals include gathering knowledge about the viability of Rust for UI, and what patterns work well. While there is a research focus, we plan to gather this information by building a real toolkit, solving many of the hard problems. The current state of the project is a fairly rough, though promising, prototype, so there is much work to be done. It’s exciting to have the bandwidth and experience of these people working toward that goal, but perhaps the most exciting prospect is bringing together more of the community, people passionate about building high quality UI. Our goal as an open source community is to facilitate that, and to coordinate the work efficiently. What we have now is not a production-ready product, but we wish to move in that direction, including quality documentation - some of the consistent feedback we’re getting about the current state.

Another goal is to bake in accessibility from the start. Too often, it’s an afterthought, tacked on at the end. We’re excited to have Matt, primary author of AccessKit, on board.

Xilem is designed in layers, with a reactivity layer on top, a widget tree adapted from Druid, and with Vello as the 2D rendering engine. These layers are designed to work well together, but the individual pieces may be useful in other contexts. The funded work will focus on Android as a target, but is intended to work well on desktop platforms as well; this is one of the core strengths of the Rust ecosystem. We don’t have iOS as an immediate priority, but attention from the open source community could change that, and the foundations should support it.

The status of the web backend for the Xilem widget set is more complex. Simply put, it is not a priority for us right now. We have done some explorations into generating DOM nodes from Xilem, but doing it well would increase scope and complexity beyond what we’re comfortable with. For applications deployed primarily on the web, xilem_web (which generates DOM nodes) should be a viable choice, with the possibility of Tauri or Electron for desktop deployment. To enable online demos of native applications, it is most likely that we will implement a version which draws to a canvas element. This has limitations in accessibility and native look and feel - for example, text input and scrolling may behave contrary to user expectations. Until more work is done refining the web backend, we would not recommend this approach for production applications using Xilem.

2023 retrospective

Looking back at 2023, there was a great deal of foundational work. Much of the effort was on Vello, the 2D rendering engine that serves as the graphical foundation of the project. A particular strength is high performance path rendering on GPU, which enables fully dynamic text effects including animation of variable font parameters. Though we had a reasonably good implementation of path rendering in place, much of the second half of the year was taken up by “stroke rework” which computes all stroke styles on the GPU, and also improves the performance and accuracy of antialiasing for all paths. To this end, we invented new computational geometry algorithms for computing parallel curves, the newest based on Euler spirals, with some elegant math tuned for GPU evaluation. A writeup explaining the details is in the pipeline.

There was slow and steady work on the higher levels of the stack as well, including factoring out the reactive layer into a generic xilem_core module, and validating that with generation of DOM nodes for browser deployment. The widget tree saw some development as well, including integration with the Taffy layout engine.

Governance and community

Though we now are funded to work on the project, the open source nature of the projects continues unchanged. We’ve been building up infrastructure for open source community, including a Zulip instance, weekly office hours, wiki, and, as of this post, a shared blog. Our contributor guidelines explain a bit about our processes. Almost all decision making is by consensus. Only very rarely do I need to step in and tie-break. As we scale up, we’re going to be more systematic, starting to use RFC documents for major changes.

We’re intentionally designing our community to foster learning and collaboration. For example, discussions are on Zulip rather than Discord because it’s searchable, meaning that the same questions are less likely to come up over and over. Its grouping into topics lets participants quickly skim activity as opposed to getting drowned in chatter. It’s working well for us. We also invite others in the Rust UI ecosystem to use our Zulip and wiki. As the Kurzgesagt video The Internet is Worse Than Ever – Now What? observes, large social spaces are degrading, and the video recommends seeking out smaller, more focused communities. I hope ours is one such.

In any case, we are happy to share bits of infrastructure with other Rust UI efforts, and are always eager to learn from them. I see many of us working toward common goals rather than competing, even if there are different approaches and priorities.

Thoughts on the Rust UI ecosystem

I’ve been thinking about doing UI in Rust for about 8 years now, and have explored many, many approaches. I’m not alone in this, there have been well over two dozen projects started.

As mentioned in a previous blog, any given application will have a set of requirements. Is Web one of the main deployment targets? Will there be embedded video? Is there a need to integrate with some other subsystems, such as a game engine? Each of these has profound implications. Some problem spaces (compilers are an example) are “smooth,” in that continual refinement will lead to fairly similar outcomes no matter the starting point, but over time I’ve come to the conclusion that UI is especially lumpy. I believe this contributes to the continuing pattern of new UI toolkits coming out every couple months or so; the author surveys what’s available, finds none that match the specific set of requirements, and creates a new one.

In some ways, Xilem is in the same pattern, but there are some things we’re doing differently. For one, it’s explicitly a research project, with knowledge about how to build UI in Rust an outcome possibly as important as the codebase. We’re trying to be systematic about finding the best ways to do things, which ideally will make some solutions more general. And, increasingly, we’re designing things as modular layers that can be swapped out. With luck, even if people don’t end up adopting the entire stack, there will be some useful components for the ecosystem.

A good overview of the opportunities and challenges in Rust UI, particularly for making mobile apps, is parallel futures in mobile application development by Andy Wingo. The linked blog posts in that series also go into considerable detail about JavaScript and TypeScript based approaches, which have important lessons as well.

Declarative UI patterns

The trend in UI programming has been overwhelmingly away from a soup of mutable objects with interlinked references and toward various declarative or reactive patterns. That’s a good thing for Rust UI, because the former is very un-ergonomic in Rust.

In the JavaScript world, there is a Cambrian explosion of reactive techniques. React is by far the most popular, but there is a strong sentiment it’s possible to be better, particularly on performance, but also robustness, as the logic for deciding when to re-render is fickle, and can miss re-renders. Elm deserves special mention, as its avoidance of global shared mutability makes it particularly easy to adapt to Rust, so it serves as the reactive core of the iced, relm, and vizia UI toolkits, as well as yew on the WASM side. But there is also Ember Octane (based on Adapton), a number of fine-grained reactivity approaches such as SolidJS, and many, many more (not to mention the incredible diversity of solutions for state management).

We’re seeing much of that diversity mirrored into the Rust world. On the WASM side, we see Dioxus as a fairly faithful adaptation of React, leptos (also sycamore and tachys) adhering fairly closely to SolidJS, yew as an adaption of elm, and more.

A reactive layer such as Dioxus or leptos can be compiled to WASM and run in a browser to generate DOM nodes. More recently, we’re seeing that it can be decoupled from that environment, and a reactive core can drive a widget tree. Examples include Freya which uses Dioxus, and floem which implements fine-grained reactivity inspired by leptos.

Xilem doesn’t directly correspond to any JavaScript framework; it relies on static typing and is actually closer to SwiftUI than anything web based.

One of the central questions in Rust UI going forward is: which approach to declarative UI is best? It’s possible a clear winner will emerge, or perhaps there will be a different answer depending on the use case, or it might just come down to a matter of personal style, with a number of viable contenders (arguably that’s where JavaScript is). We’re taking a two pronged approach. First, we’re moving forward with Xilem. There’s a specific reason - Xilem is designed to support multithreaded rendering easily, while many of the other frameworks rely on thread local storage for access to a store of observables, an approach with high impedance mismatch to multithreading. Since performance is a goal, we didn’t want to give that up by adopting another framework. I also think Xilem is pretty good, but that remains to be seen.

Another way to gather more evidence on the Xilem declarative layer is to use it to generate DOM nodes in a browser, in other words decouple it from the widget set. As described in the Xilem Vector Graphics talk, we have been experimenting with that a bit, and a major development in the last year was to factor out the core declarative logic into xilem_core, and also develop a xilem_web layer for creation of DOM nodes. Early indications suggest that this may be a viable alternative to frameworks such as Dioxus and leptos, and we're eager to learn more from those explorations.

Xilem is untested, though promising, and it may be that some other approach is better. If so, we’re prepared to pivot. We’ll be designing the widget set so that it doesn’t have a hard-coded dependency on the Xilem declarative layer, and could, at least in theory, be driven from other frameworks. Olivier is author of the masonry crate, where being agnostic to the declarative layer was an explicit goal. He’ll be adapting many of the ideas from that.

Making a widget set agnostic has other potential benefits. While writing app logic in Rust is very promising, there are still compelling reasons to use other languages. A widget tree implemented in Rust with fast graphics and smooth interactivity, scripted in some other language, is compelling. I made a prototype in Python, which, though rough, shows promise.

Infrastructure

A UI toolkit depends on a great deal of infrastructure. We’re making careful choices around that, and also looking for opportunities to collaborate with the rest of the ecosystem. One choice I’m particularly excited about is investing in wgpu. I think WebGPU will move modern GPU usage forward considerably, especially compute shaders, which are heavily used in Vello. This technology will make truly portable applications requiring high compute throughput possible for the first time, and we’re shooting for smooth UI integration with the GPU usage in these applications.

Whether to use winit or build our own platform integration has long been a subject of discussion. We’re reopening that discussion with the winit maintainers as of this writing, but it’s still too early to say whether we’ll join forces, or decide that the goals of the project support having a separate codebase.

A huge part of any UI effort is text, and that’s also a long-standing interest of mine. I’m impressed with cosmic-text but ultimately we decided to move forward with parley, in part so we can test out ideas with variable fonts. The lower levels of the text stack will be rebased on the fontations work of the Google Fonts team, which is rapidly becoming a production-quality implementation.

Building common infrastructure for accessibility also benefits the entire ecosystem, and we're thrilled to see the increasing adoption of AccessKit. Indeed, on all these topics, we’re happy to share what we’re doing with other projects, as most of the infrastructure described above is general in application, rather than specific to our projects.

Conclusion

The potential for Rust UI is huge. The Rust language enables top-notch performance and reliability and is also expressive for high-level application logic. We plan to prove that out, based on encouraging existing research and explorations, as well as new research to discover the patterns for Rust UI that work best at scale. This involves work at all levels of the stack, including infrastructure we are happy to share with other Rust UI efforts.

Xilem is still an early prototype. For those who desire a “batteries included” toolkit for building UI, we ask your patience. For those who are interested, you are invited to come build this future with us.