Retrospective on the May-July roadmap

Daniel McNab, August 24, 2024

Linebender is an informal open-source organization working on various projects to advance the state of the art in GUI for the Rust programming language.

At the start of May, we created a roadmap for the following three months. We recently went through a process of analysing how well our progress was guided by the roadmap; this report summarises those results.

Vello (original)

Robust dynamic memory (original)

• Priority: Highest.
• Status: Experimental PR (vello#606), but hasn’t landed. Work required on the CPU side API, but the GPU operations are well-understood. This PR could not be released in its current form.

Reliable snapshot tests (original)

• Priority: High.
• Status: This was implemented in vello#610. This has already been extremely useful in our review process, as it allows changes to be validated in the files changed view. We also implemented an experiment to use Git LFS, which has been working really well to avoid increasing repository size.

Sparse strips (original)

• Priority: High.
• Status: We are continuing this work, but there is no code currently available. See #gpu > Sparse strip path rendering for details.

Add tests (original)

• Priority: Medium
• Status: This item had already been completed at the time the roadmap was written. This was also expanded upon in. During our retrospective meeting, we did determine some potential future expansions to testing, namely:
  • Running individual stages in isolation, to enable property testing (TODO: Make Vello issue).
  • Testing for additional platforms, such as DirectX on Windows and Android (TODO: Make Vello issue).

Progress beyond roadmap

We have also landed several important features which were not part of our planned roadmap. These include:

• Blending which uses more than four deep layers (vello#657).
• Preliminary support for embedding pre-existing wgpu textures (vello#636). This was driven by user needs. Further work in this direction is tracked in vello#664.
• Support for Emoji (COLRv1 in vello#615, bitmap in vello#641).

The stroke expansion paper documents a key part of Vello's pipeline. This was presented at ACM High Performance Graphics (HPG) 2024, placing 3rd Place in the Best Paper Award.

Masonry (original)

Complete text input (original)

• Priority: Highest
• Status: Not that much progress. This work is being continued, and indeed there has been significant progress as of the time of writing. This is tracked in parley#52.

Better tracing for debugging (original)

• Priority: High
• Status: xilem#384 implements most of this, the rest is deferred. There are known issues with the size of the log files created by default in very large apps.

Fix glaring documentation problems (original)

• Priority: High
• Status: The issues mentioned in this item were addressed in xilem#329 and xilem#503.

Finish repository port (original)

• Priority: High
• Status: This was completed. We moved all Masonry issues into the Xilem repository, and updated all references in code to use full links. This move was coordinated in #linebender > Moving Masonry into Xilem.

Write “widgets in arena” RFC (original)

• Priority: High
• Status: The RFC was rfcs#6, which was mostly implemented in xilem#396.

Write “pass order” RFC (original)

• Prioriy: High
• Status: rfcs#7 is in draft, and several important parts of the RFC have been completed.

Rewrite documentation (original)

• Priority: Medium
• Status: This work has been deferred until the "pass order" refactor is complete, as we expect Masonry's core to be quite stable at that point.

Improve focus handling (original)

• Priority: Medium
• Status: Not done.

Improve pointer status handling (original)

• Priority: Medium
• Status: Basic translation of touch events was added in (xilem#313). This unblocked using Masonry on devices where pointer input is not expected (primarily on Android). We have split the remainder of this task into two tasks, with a priority placed on completing the parts which unblock other work (such as scrolling on Android).

Progress beyond roadmap

Masonry is in a state of flux, however is starting to see a few exploratory users. To unblock these users, some additional important features were added outside of the roadmap, such as:

• Support for embedding Masonry inside a pre-existing event loop (xilem#417).
• Support for a gap property in the flexbox (xilem#437).
• Fixes on iOS, including xilem#418 and xilem#421.
• xilem#233 allowed our tests to be run on CI in a consistent manner.

Parley (original)

Create abstractions for text selection and editing (original) ?

• Priority: Highest
• Status: There have been useful discussions in parley#52. Work has started on this in parley#106 in response to this retrospective.

Inline block layout (original)

• Priority: High
• Status: A first cut of this feature was added by Nico Burns in (parley#67). parley#76 expands upon this, and has been approved in principle, but is awaiting thorough review.

Write adequate documentation (original)

• Priority: High
• Status: Some work was done towards this in parley#26, which unfortunately could not be merged. parley#97 added some documentation to Fontique. Further documentation work from Nico is currently blocked on the tree-like styles PR parley#76 being merged (to avoid conflicts).

Software renderer (original)

• Priority: Low
• Status: This was not completed in the form envisioned. However, Nico contributed two examples, showing how to use Parley with Swash (parley#54) and tiny-skia (parley#55). We have decided not to carry this forward into the next roadmap.

Progress beyond roadmap

parley#56 added support for Emoji, which is used with the Vello PRs.

Recently the harfruzz project, a fork of rustybuzz to be integrated fully with fontations, has emerged as an extremely promising direction for text shaping for the Rust ecosystem. We plan to use it as the basis for text layout.

Xilem (original)

Android bringup (original)

• Priority: Highest
• Status: Android is supported in several of our examples, with the primary work required for support completed in xilem#309. No non-boilerplate code changes are required to get a Xilem application running on Android. Some compromises were made, in particular cargo gives an un-silencable warning for each example which supports Android and cross-platform backends. We did reach out to the Cargo team about this use case, but did not get a positive response.

Update repository layout (original)

• Priority: High
• Status: This was completed in xilem#302. Kaur applied the same change to Vello in vello#590.

Rework xilem_core to be compatible with Masonry backend (original)

• Priority: High
• Status: This was completed in xilem#310. Philipp then applied this to Xilem Web in xilem#403.

Write documentation (original)

• Priority: High
• Status: This was not completed. The work in xilem#310 did document some of the core, but there is very little user-facing documentation of the Xilem level.

Re-implement async integration (original)

• Priority: Medium
• Status: The core of this task was completed in xilem#423 was the main implementation. Two-way communication with the async tasks is not yet implemented

Add unit tests (original)

• Priority: Low
• Status: This was not prioritised. We have raised the priority of this item in our next roadmap.

Add basic benchmarks (original)

• Priority: Low
• Status: There has been no progress on this item. This will be a "highest" priority item in the next roadmap.

Progress beyond roadmap

Xilem has also seen some very useful changes not tracked above, including many community contributions. Some highlights of these contributions are:

• Philipp Mildenberger contributed heavily to Xilem's core design, such as by restoring the ability to return ViewSequences by returning to a previous workaround for Rust's overly-cautious trait solver (xilem#472). He also extended the Xilem API by restoring OneOf (xilem#394), and implementing various Adapt nodes (xilem#401). Additionally, he picked up the work to expand Flex to use the new Xilem Core in xilem#428.
• Jared O'Connell contributed two new examples, a calculator (xilem#467), and a stopwatch (xilem#492).

There has also been lots of really good progress on Xilem Web, but we do not roadmap Xilem Web and so we do not cover that progress here.

AccessKit (original)

Matt Campbell's work on AccessKit has begun, and a proof-of-concept Android adapter is in progress.

At the time of writing there is not a permanent link to this work, but it can be seen in the android-basics branch.

Android specific work (original)

Subclassing View/Application/Activity (original)

• Priority: Highest
• Status: This work is in progress, and unfortunately has required more research than expected. We spent some time investigating DEX injection, which Matt has got implemented. However, it’s now looking like a custom Activity and View is the way forward, which requires close collaboration with upstream tools. We are continuing to prioritise this highly.

Handling IME (original)

• Priority: Highest
• Status: This work is ongoing. winit#3787 allows opening a virtual keyboard, however this does not integrate with full IME (so for example, it is incompatible with autocorrect). Real IME requires the Activity to support it directly and therefore involves content view subclassing.

Gesture recognition (original)

• Priority: Low
• Status: xilem#313 added emulated mouse press and move from touches. This unblocked use of Xilem on Android, but is not the intended final state. The priority of detecting a scrolling gesture has been increased slightly.

Conclusions

Overall, the roadmap provided a useful guide of what to work on, and we have implemented many of the most important features from it. One clear recurring issue is documentation - even where we have chosen to roadmap this, we have made little progress. Similar concerns are clear for testing in Xilem, although we have laid some important groundwork there. We firmly recognise the role of documentation as an aid to onboarding, which is becoming more important as we work towards alpha releases. I believe that we can make good progress in this area before the end of 2024.