Bringing joy to Scheme programming

Andrew Tropin - Nick: abcdw, Site: https://trop.in/, Fediverse: https://fosstodon.org/@abcdw

Format: 22-min talk; Q&A: BigBlueButton conference room
Status: Q&A to be extracted from the room recordings

00:02.120 Introduction
00:23.280 Interactive development
01:18.180 REPL: Read Eval Print Loop
02:53.720 Long-lasting loops
04:07.600 Not interruptible
05:23.160 No protocol
05:51.480 Not scalable
07:25.860 nREPL
09:01.740 Arei, Ares, and how to try
10:34.180 Demo
11:27.640 Continuations
12:32.460 Reading from stdin
13:33.420 Fancy example with continuations
15:13.160 Guix API
17:42.060 Support
17:57.020 Future steps - Multiple simultaneous evaluations in different contexts
18:46.220 Tree-sitter integration
18:56.880 Full-fledged debugger
19:22.760 FAQ - Does it support other Scheme implementations?
19:58.380 Is it possible to use it with other text editors?
20:22.121 Conclusion
20:45.880 Contacts

Description

Emacs is usually a primary tool people remember, when talk about development environment for lisp languages. It serves the purpose great for Common Lisp, Clojure and of course Emacs Lisp, but what about Scheme? Let's talk about current state of the things, recent improvements, and emerging tools in this field.

My talk covers the following:

• What does a usual Scheme developer day look like? And how it can be made more enjoyable?
• Important developer duties, their automation and acceleration.
• Interactive development and its benefits.
• Emacs setup for Scheme development.

Links:

• https://git.sr.ht/~abcdw/emacs-arei
• https://git.sr.ht/~abcdw/guile-ares-rs

About the speaker:

Talk about Lisp development workflows, REPLs, and modern Scheme tooling for Emacs. Author of Guix Home, maintainer of rde, FOSS developer.

Discussion

Questions and answers

• Q: How much do you use these repels remotely ex using a server or desktop from your laptop?
  • A: I don't use it remotely at the moment, but it should work perfectly fine (except maybe lookup and other similiar functions). I also want to add a shepherd service for ares-rs, so you can connect to GNU Shepherd and systems based on it (like GNU Guix) from you emacs process and interact fluently with guile code.
• Q: Can this be integrated with eglot?
  • A: I'm not sure how this integration could look like. Theoretically, it's possible to expose many of ares-rs functions via LSP custom actions (or whatever it called). Anyway, contact me on IRC or https://trop.in/contact to discuss it in more details if you have something in mind.
• Q: How hard is it to add support for something else than Guile? Does it make sense to contribute at this early stage of development? I've written several packages for CHICKEN Scheme before and would like to try this one.
  • A: It's a matter of implementing the whole chicken-ares-rs Many of the code can be reused, but not all, unfortunately. emacs-arei doesn't need any (or almost any) changes.
• Q: (One day late sorry) Is nREPL more extensible than what SLIME/SLY use in Common Lisp world (I think it's comint.el) ?
  • A: Yes, it's. I was evaluating and considering SWANK protocol, but found nREPL to be more suitable and future proof. https://github.com/astine/swank-client/blob/master/swank-description.markdown

Notes and discussion

• brilliant work for scheme
• yeah, this is overdue. the only real alternative is slime-r7rs

Transcript

[00:00:02.120] Introduction

Hello and welcome everyone on EmacsConf 2023. I'm Andrew Tropin. I work on operating systems and programming languages. Today, we discuss Lisps, Schemes, REPLs, interactive development, and how to make your own cozy development environment.

[00:00:23.280] Interactive development

Let's start from interactive development. Lisps are famous for a nice Interactive Development Experience. They have REPLs. Emacs Lisp has its own Lisp machine, and a lot of cool IDE with different functionality is already here and providing a nice and pleasant experience. The question is, is it enough? In most cases, yes, but for some languages, we have some white spaces, some missing pieces. And for example, in Scheme world, we already have a few tools. We have REPL, we have integration for REPL in Emacs, but is it enough? Let's see.

[00:01:18.180] REPL: Read Eval Print Loop

We know that Emacs is very good for Lisps and REPL. Lisp and Emacs should be a perfect setup. But let's see how REPL basically works. It's an event loop which does three things. It reads an expression, it evaluates the expression, and it prints the result. We can take a simple expression, input it into REPL, and evaluate it and see the result. Very nice, very convenient. You can experiment and see immediately what is happening. You can even run a long-running process which does something. You can interrupt it and everything will be okay. But the problem appears when you start to develop a bigger project. And in most cases, you don't do your whole development in REPL. You do only a small part of it. In most cases, you just write the source code in text files, and after that, you run those snippets of code from those text files, or run the whole project. It's not very convenient to copy and paste every time the snippets of code to the REPL, see the result, modify the snippet of code, copy it again, and so on. So people invented some integration between REPL and your text editor. So you can evaluate expressions inside your text editor and see the result here.

[00:02:53.720] Long-lasting loops

Works good so far, but what happens if we run a long-lasting loop, which does a lot of operations. As you can see here with a simple example, the output of the function, stdout of the function is presented here, and the resulting value is here. If you run a long-running process, you don't see anything happening. And you see there's a watch instead of my cursor. Maybe you don't see it, but nothing actually happens, at least from the point of view of the user. But if we interrupt the evaluation, we will see that some process in the background was launched, but we didn't see anything. Because the REPL is a single-threaded blocking process, which reads stdin and prints stdout, make the integration between the REPL and your text editor is not an easy task. And even if you do it, you have a lot of downsides, usually.

[00:04:07.600] Not interruptible

First of all, the process is not interruptible. If you have a remote process which listens on the socket to which you connect from your development environment, and you run some infinite loop, for example, you can't interrupt it. Because interruption is done via signals, and signals to remote processes are not usually the thing in such integrations.

[00:04:38.760] Output is not interactive

Output is also not interactive. Usually, for example, here you can see when I evaluate the expression, the output is captured on the evaluation side, and after that, after the whole evaluation of the whole expression finished, I get the result, all the stdout at once. And if I run the process which evaluates for 5 seconds, I will see the first signs of the life only after 5 seconds of evaluation. Okay, what else?

[00:05:23.160] No protocol

When you do such integrations, you have no protocol, you have just stdin and stdout. You print to stdin from your text editor. You read from stdout of the process. It's hard to tell if evaluation is finished, if it requires stdin, and how to extend the REPL to make it more featureful, and so on.

[00:05:51.480] Not scalable

And also, such integrations are usually not very scalable. For example, if you want to have a completion, you type something, you have the completion. Cool. But if you run the process and at the same time try to have a completion, you don't have it, because the evaluation is in progress, and you can't calculate the completion candidates at the same time. To make it more obvious, I will start a completion here. You see the completion pop-ups. I start the evaluation process, and when I try to complete something, the evaluation freezes and there is no completion. Not very convenient. Usually, you have some long-running processes and you want them to continue while you have your go to definition, completion, and other things. Overall, those issues make it quite inconvenient to integrate REPL in text editors or development environments, so you need something else to make the work comfortable.

[00:07:25.860] nREPL

There is already a solution called nREPL. It's a synchronous protocol which allows to send operations to the server and receive responses in a synchronous manner. And here is a simple example of a few operations. First one is cloning the existing session, and as a response you will get a new session. Also you send the evaluation request with code that you want to evaluate, and you get two responses. First one says that output is captured and it's equal to "hi\n", and after that, you receive an "Evaluation completed", the value of this expression. This protocol was developed for CIDER development environment. It's a Clojure development environment for Emacs. It's very cool, featureful, reliable, and I would say production-ready. A lot of professional Clojure developers use it. The nREPL protocol is very simple. It has a few operations out of the box, and you can extend it with any arbitrary operation you want. I work a lot on Guix codebase and other Scheme projects, so the experience I had previously with nREPL was not satisfying. I decided to just implement nREPL protocol.

[00:09:01.740] Arei, Ares, and how to try

First of all, I implemented nREPL server in Guile. I called it guile-ares-rs, and used it with a generic nREPL client for Emacs. It worked. It had some rough edges, but overall it was okay. And after that, to add more features to make the implementation more complete, I wrote my own nREPL client for Emacs and called it arei. And I got almost complete Guile IDE in two months. So ares-rs is nREPL server implementation. arei is Emacs client, which uses the same nREPL protocol. It utilizes sesman package for managing sessions, the association of buffers with nREPL connection. It has some roots. The implementation has some roots in Geiser, CIDER, Monroe, and Rail. I took small snippets for some parts of functionality. I used the CAPF and xref infrastructure for completion at point and cross-reference capabilities. And by the time of conference, I hope that README will be complete enough so you will be able to try it yourself.

[00:10:34.180] Demo

Let's see what is possible with it already. Let's connect to nREPL server. After that, you can evaluate the expression. And you see the stdout and the result. Very nice, very convenient. You have different expression, you evaluate it, you get the value of the evaluation. You can run an infinite loop which prints to stderr and stdout and you see all necessary stuff. Very cool. But also, you can interrupt the evaluation, which is very convenient if you accidentally run an infinite loop.

[00:11:27.640] Continuations

Also, do you remember here we have a few more examples that we didn't try yet? For example, on usual REPL implementation, if I evaluate this expression, I get return value. I make a continuation and save it to this variable and I try to call this evaluation and I get an exception, because the environment in which this continuation was created was different and it has redefined stdout and stderr to capture it. But when I run it one more time, when I resume the continuation, the environment changed and it doesn't work. What happens in arei? I define continuation, I save the continuation for the simple expression and I resume the continuation with a new argument, and you can see at the top of the screen that it works perfectly fine.

[00:12:32.460] Reading from stdin

Also, with a usual REPL implementation, let's see what happens when we have a process which reads from stdin. I evaluate the expression and nothing visible happens. I can try to type C-g, C-c, and after some time it will say user interrupt. What actually I expect in such a case to have a minibuffer which prompts me for the input. When I evaluate the same expression in the arei, you see the prompt at the minibuffer and here I can tell, "Hello I'm a message from minibuffer". Cool. You will see that this message is printed to stdout, and unspecified was returned as a result of this expression.

[00:13:33.420] Fancy example with continuations

Let's make some fancy example with continuations. Continuations is a very cool mechanism which is not the topic of today's talk, but you can find a lot of interesting information in Scheme documentation or in related books, and I advise you to do it because it's really nice thing that is actually applicable in many different programming languages. Here you can see the infinite loop which just prints values increasing one by one. And here we save a continuation on each iteration. I can call the continuation and it will resume from the previous saved step. And you can see, it resumed from the same step

we interrupted earlier, but we provided a new value for it. another value for it.

[00:15:13.160] Guix API

Okay, what most annoying thing that I had previously with the usual REPL implementation that I have a quite nice Guix API where I can build packages, systems and other stuff. But if I evaluate this expression, I will get an error. Okay. I will get an error because I don't have an appropriate environment. But what I can do, I can connect to the remote REPL by creating a server with guix repl --listen command and connecting to it with geiser-connect command. And now I can evaluate this expression. Right? Wow. Okay. It actually doesn't matter for my example. I will explain how it doesn't work easily. This is a long-running process which prints something and it can take up to a few minutes. And for the whole few minutes I don't see any results, the same as with this infinite loop which prints to stdout but I don't see anything interactively. With arei, I can run the evaluation of the same expression, and you will see instantly that stdout is presented here in slightly yellowish color. I can interrupt the evaluation if I don't want to wait until it's finished, and just after that, I can evaluate another value. So that's cool. And let's see one more thing. We have an infinite loop and we have some completion here. And completion still works, very nice, while the infinite loop is running. Okay.

[00:17:42.060] Support

Actually it took me around two months of full-time work funded by my own savings, and you can support and help to the project using OpenCollective or by contributing on SourceHut.

[00:17:57.020] Future steps - Multiple simultaneous evaluations in different contexts

The future steps for the project include an experimental workflow where you have multiple simultaneous evaluation in different contexts. For example, you have Fibers, you have Goblins, you have some HTTP server or some other thing, and you want to run all of them independently in slightly isolated sessions, and you want to have the ability to still interact with them. For example, if they require standard input or something else, you want to be able to provide it. You want to see the stderr and stdout of those long-running processes and so on.

[00:18:46.220] Tree-sitter integration

The second thing is tree-sitter integration for better syntax highlighting, code navigation, and other features.

[00:18:56.880] Full-fledged debugger

And after that, probably we will do a full-fledged debugger so you can jump expressions one by one and see the results and see some intermediate values during the evaluation. And it's very possible because nREPL is a very extensible protocol and you can implement whatever you want on top of it.

[00:19:22.760] FAQ - Does it support other Scheme implementations?

I will answer two probably very frequent questions. Does it support other Scheme implementations? At the moment, it doesn't, but the Scheme implementation is not restricted. You have a server which is implemented in your language and you have a client--in our case, arei-- which communicates with this protocol. So if you implement nREPL server in a different language, it should work with already implemented arei client.

[00:19:58.380] Is it possible to use it with other text editors?

And is it possible to use the same functionality in other text editors, for example in VS Code, Vim, whatever? Yes, it's possible and the case is similar here. You have already implemented nREPL server and you can write your own nREPL client in a different text editor and it will work.

[00:20:22.121] Conclusion

I would like to thank the authors and maintainers and contributors of Guile, Geiser, CIDER, Clojure, and Emacs, and all other people who are somehow related to the work on those projects involved in this talk. And I hope the Scheme programming will be enjoyable.

[00:20:45.880] Contacts

If you want to contact me, join #tropin IRC channel at libera.chat, or drop me a message via email or feediverse using andrew@trop.in handle. I will see you in a bit in Q&A session.

Captioner: sachac

Q&A transcript (unedited)

Let's see, 1 asking how much Andrew uses these ripples remotely or versus on their own desktop. And another asking if this can be integrated with EGLOT. And I will note that it is very cool that this year we've had so many talks on Ripples. Just goes to show how powerful Emacs is and just how much or how far you can push it and how much you can do with it. And so see someone asking on IRC, if or how many people use GnuGeeks. Since we are talking about Scheme, GnuGeeks is a great platform slash operating system or distro for your test house, but also for servers and such. They do some impressive, amazing work. And it's all, pretty much all done in Gindugal's scheme. So very cool stuff. Bye. You I see another interesting question on the pad. How hard is it to add support for something other than Guile? And if it makes sense to contribute at this early stage of development? They said that they've written several packages for chicken skin before and they would like to try this 1 as well. I guess since Andrew isn't still here, and there was some chatter about GnuGeeks in the chat, maybe it might be nice for me to share my screen and plug Inukis for a little bit and introduce it or at least show its website to folks who may not have seen it yet so I'm going to try and do that now. You Okay, let's see if this works. Okay, so this is GNU Geeks' website. You can go to geeks.gnu.org. And they introduced it at the top. So it's a wholly free operating system or distribution of GNU Linux. Meaning that it only has free software packaged and no non-free packages, so it is endorsed by the FSF and the GNU project. As someone said in the chat, it's kind of like Nix, but instead built on GNU Gallop scheme. It has transactional upgrades and rollbacks. So if you do upgrade your system and let's say in the middle of it, your hardware fails or your power goes out, the likelihood of things being corrupted is very low because the upgrade is essentially prepared like in the background. And then pretty much atomically, the system is switched to it. And also if there is some kind of, sorry, I'm losing my voice here. If there is some kind of issue that makes your system unbootable, you could always go back to booting the previous revision of your system when you restart in the Grub bootloader. Yeah, so they have a nice blog where they regularly post updates and what's new in the project. You can go check that out. We also have a packages archive where you can see a list of all the software that has been packaged for GNU Geeks. It is an impressive list. I don't know how many tens of thousands of packages there are. Geeks has been growing very well. And you can search the packages here. And yeah, all kinds of things are packaged. Of course, GNU Emacs is packaged, along with many extensions or packages, GNU Emacs packages that are packaged as system packages for Geeks. Yeah, so definitely go check it out. You can use Geeks both as a standalone package manager, let's say on a Debian-based distribution like Triscale, for example, or you could install it like as a complete system distribution on its own. So the former is useful if you want to maybe get a taste for Geeks and try it out before fully committing to it and switching to it as your main distro. You can try it on top of any other distro pretty much and then you can of course install it on its own as well as a system distribution. Yeah, there are a bunch of manuals and reference cards and videos that you're welcome to watch. They have several mailing lists. It sounds like they have a wiki now as well. And the development is done on Gnu Savannah. If we go to savannah.gnu.org slash projects slash geeks, Yeah, the project is developed here and they have a bunch of repositories including the main 1 which is geeks.git itself. So yeah, folks are welcome to go check it out. Let's see, maybe we can go have a look at some package definitions, although I think we're almost out of time on the live stream. So, yeah, just quickly. Emacs to the CM has all the, Emacs packages or Emacs itself. And Emacs-xyz is where you'll find all the Emacs like ELPA packages, but package for use on GNU Geeks system or with GNU Geeks. And I think that's all the time that we have. So yeah, thanks for tuning in, folks. Please post your questions on the pad. We'll pass them on to Andrew. And yeah, hope you enjoy this. Definitely go check out Andrew's work and Gnu geeks as well. You are currently the only person in this conference.

Questions or comments? Please e-mail emacsconf-org-private@gnu.org