Migration to codemirror 6?

[OyvindLGjesdal]

I was looking at shrinking the bundles of yasgui when I noticed the info bar on the current dependency of CodeMirror 5.

NOTE: CodeMirror 6 exists, and is more mobile-friendly, more accessible, better designed, and much more actively maintained.

I don't know whether this would be a relevant or wanted change, since I believe zazuko/yasgui is a patched/bug fixed fork of triplydb/yasgui, and ugrading would probably be a big departure, but I thought it wouldn't hurt to open an issue for discussion.

There is a migration guide at https://codemirror.net/docs/migration/

Using modules sounds like a reasonable change and could make the bundles created from yasgui shrink, and help to discard unused code paths for projects that depends on it.

[vemonet]

Hi @OyvindLGjesdal , I would be very much interested, and could contribute to it (we are building on top of Yasgui for endpoints served at the SIB: https://sib-swiss.github.io/sparql-editor/)

But as you said it it will be quite a journey! So better plan it well

The big question to me right now is: CodeMirror 6 or Monaco editor using a SPARQL LSP? (Language Server Protocol)

• Example of a SPARQL editor recently written with CodeMirror 6: https://github.com/aatauil/sparql-editor
  • No autocomplete yet, but it is on the roadmap according to the readme
  • Autoformatting could be done with https://github.com/sparqling/sparql-formatter
• There is someone in the Qlever team currently building an advanced LSP for SPARQL with soon autocomplete based on endpoints content: https://github.com/IoannisNezis/Qlue-ls
  • The most interesting it is that it is built with TreeSitter which allow accessing and "parsing" the query before it is valid for context-aware autocomplete (basically you know that the current subject is of a certain type so you only autocomplete predicates that are available for this type). Without this you need to rely on regex...
  • Auto formatting is already implemented
• There is a simpler LSP for various semantic web formats, including SPARQL, but not really advanced (can be a good source of inspiration for Monaco integration though) https://github.com/ajuvercr/semantic-web-lsp

I think ultimately Monaco editor (aka. VSCode in the browser) with a mature LSP would give more features in an editor familiar to more people, but the setup is more tedious, especially if we want to provide custom autocompletion functions. This will probably require to change the upstream rust code and rebuild the wasm binary, which would be fine for us because in our opinion autocomplete for a SPARQL endpoint don't need to be customizable, it should just reflect the content of the endpoint.

I would love also to move to a more modern build system (vitejs is so much better than webpack in term of dev experience and speed) and replace the require by import. Updating the editor would be a good opportunity to do so

[vemonet]

Hi @OyvindLGjesdal @ludovicm67 , I made some preliminary investigation

I have tried to migrate to CodeMirror 6 in the original Yasqe webpack project (see my branch https://github.com/vemonet/Yasgui/tree/experiment-migrating-to-codemirror6-with-webpack), commenting most of the constructor code and trying to just setup the most basic setup of CodeMirror 6 (to go incrementally), but I faced this error: Uncaught TypeError: ext is undefined. Which seems to be related to some dependencies conflicts: codemirror/dev#402

I created another package folder, this time using vite instead of webpack for compilation, with the exact same dependencies in the package.json. And I did not get any issues! (see this branch https://github.com/vemonet/Yasgui/tree/experiment-migrating-to-codemirror6-and-vite) Imo webpack is fumbling the dependency resolution

You can easily run it by going to the right branch and run:

cd packages/yasqe-vite
npm i
npm run dev

vite over webpack

I don't know what is your opinion, but I really don't like webpack, config is messy and it's slow. Every single modern JS project have been built on top of vite (solidjs, svelte, vue, remix, nitro...), there's a reason to it. The only major project still using webpack is nextjs, but it's hidden behind their own wrapper, and they are currently moving away to turbopack. I have been using vite for all my we components projects, and it's really good and pleasant to use to compile packages (it saved js web compilation imo :D), plus it has a ton of plugins with a mature ecosystem. So if we plan to migrate to CM6 (which uses modern JS modules, while CM5 uses old school common JS require stuff), I would also migrate to vite

What has been done already

For now I just reused the code from this project to get a basic SPARQL editor working:

• https://github.com/aatauil/sparql-editor/blob/master/src/index.ts, with sparqljs to parse the query
  • Note that last commit was about ~2 years ago so I don't think the author is planning on adding more features anytime soon, but it's already a really nice head start
• https://github.com/aatauil/codemirror-lang-sparql for the grammar and syntax highlighting.
  • It uses lezer, the parser of CodeMirror, It is quite simple and easy to read and maintain, we can easily move it to our repo if we need to make improvement. I would go for this implementation if it's complete enough, and if the tokenizer works for our use-cases
  • The previous implementation for Yasgui relies on perl, and is much harder to read, improve and recompile.

Already a few things are nicer with CM6 than with CM5:

• Some features are much easier to implement, and does not requires to do some of the complex implementations that have been done in the past. For example to mark an error in the SPARQL query syntax (the "gutter"), there is a specific linter class, it's about 10 lines, it uses sparqljs as parser, and you get a better visual cue: there is a nice red wavy underline on the word + a red dot on the line, meanwhile the previous code basically implemented the whole gutter system themselves drawing a svg, etc which was making the code quite complex
• We get really nice explanation and examples when hovering over each keyword

What needs to be done

There is still a lot of things that needs to be done to have an editor as good as what we currently have with CM5:

• Priority:
  • Proper autocomplete for prefixes, types and properties to work and to be easily configurable (hardest part probably)
• Should not be too hard:
  • Error when missing prefixes (for some reason it is not triggered as part of sparqljs linting)
  • Automatically load prefixes when typing it if in the prefixes list
  • Improve syntax hightlighting (prefixed IRIs should be orange, and variables should be blue)
  • Enable to properly fold pieces of the code (prefixes works, but folding groups does not)
  • Enable auto comment when hitting ctrl+/
  • Make it so the current indentation is respected when going to new line
  • Try to keep most of the existing features, but using the new system, e.g. handle all existing events
• Finishing:
  • Moving all other packages to vite (YASR, YASGUI, utils)
  • Probably need to update the documentation and republish it in this Yasgui repo (we can copy the existing doc to markdown, update the parts that needs it, probably using Docusaurus and deployment on GitHub Pages?)

I don't have time to work more on this right now, so if anyone is interested feel free to continue and update us here please!

Otherwise I am also still waiting for https://github.com/IoannisNezis/Qlue-ls to be mature enough to see if it would be a good candidate.