Posts on aliquote.org

January in review

Thu, 05 Feb 2026 09:32:43 +0100

Here’s a bunch of links that I didn’t manage to post on the micro blog in due time.

Time Machine-style Backups with rsync: I always forget about hard links and this post explains how to take advantage of rsync’s --link-dest option.
Yet another blog post about rsync: 4x faster network file sync with rclone (vs rsync). I once complained that rsync was very slow on my MacBook for my own backups, and it looks that I may need to rely on rclone.
Maze Algorithms: Astonishing online demo and algorithms. All the code is available on GitHub. It’s Coffeescript, long time I never heard of this.
Rust-like typedef for C as a short way of referring to fixed-length types: some C habits I employ for the modern day.
Bad apple but it’s 6,500 regexes that i search for in vim, if you want to give vim golfing another try… On a related note, Evan Hahn explains how he set all 376 Vim options and I’m still a fool.
Notes on Apple’s Nano Texture: I do not work outside but I can confirm that the Nano texture display on my MacBook looks way better compared to the previous glossy displays I used to use.
Why DuckDB is my first choice for data processing. I’ve tried it in the past and I always forget that often times it may provide a better alternative to SQLite, if only for dynamic column selection, queries that start with the FROM clause, function chaining, and list comprehensions, but see Even Friendlier SQL with DuckDB. For a more challenging review, see Andy Pavlo’s Databases in 2025: A Year in Review, although it does not deal with DuckDB.
OpenBSD-current now runs as guest under Apple Hypervisor. I need to give it a try with UTM at some point.
Python Numbers Every Programmer Should Know: interesting summary of latencies as found in web framework, coroutines, list and dict operations, etc.
Effective Simulated Annealing with Python, with zero dependencies.
The fastest way to detect a vowel in a string. On the importance of trying to swap the inner and outer loops.

Replicating human perception is hard, and it’s made harder when constrained to the limitations of display technology or printed images. You don’t have to be ashamed about adjusting a photo when automated algorithms make the wrong call. — What an unprocessed photo looks like

♪ Front 242 • First In/First Out

Micro posting in December

Wed, 31 Dec 2025 18:27:43 +0100

2025-12-05:
Quai de la Garonne, Toulouse, Dec. 2025
2025-12-05: /me is listening to “If I wanted someone” by Dawes
2025-12-05:

There seems to be some resurgence into the old web now, time will tell if it gains any real ground. It’s an uphill battle: Besides most online eyeballs now being glued to social media apps, we’re seeing AI take over the way people interact with the internet altogether. Back in the old days if you wanted to know more about something you’d google the term and start going through the websites Google said are most relevant. — Blogging in 2025: Screaming into the void

2025-12-05: Modeling temporal data with an unknown number of changepoints. #python
2025-12-05: System Reliability and Six Sigma in R and Python.
2025-12-07:

I’ve seen a number of editors grow popular and then wane. At various times I’ve seen Emacs, Eclipse, IntelliJ IDEA, Atom, SlickEdit, gedit, Notepad++, Sublime, NetBeans, Visual Studio, Xcode, IDLE, PyCharm, Android Studio, and more. Some of those aren’t around any more. I’m not worried about Vim getting eaten by a big tech company, or enshittified to feed advertising or AI ambitions. — Choosing Vim over VSCode

2025-12-07:

So, while I agree that it’s wrong to generalise that ‘Perl 6 killed Perl’, I would say that Perl 6 was a symptom of the irreconcilable internal forces that killed Perl. Although, I also intend to go on to point out that Perl isn’t dead, nothing has actually killed Perl. Killed Perl is a very stupid way to frame the discussion, but here we are. Perl’s decline was cultural

2025-12-07: Screenshots from developers: 2002 vs. 2015.
2025-12-08: osxphotos looks like a blessing to manage iPhoto library. The same applies to imessage-exporter. #apple
2025-12-09: QuickSort in 61 characters:

quicksort = # & @@@ {# //. x : {__} :> (## & @@ Reverse /@ GatherBy[x, (# < x[[1]] &)])} &

2025-12-09: Rainbow array algebra: I really like syntax-highlighted math notation and all those pretty diagrams. See also MathTools. #mathematica
2025-12-09: wolframite: An interface between Clojure and Wolfram Language (the language of Mathematica). It works like a charm with latest release of Wolfram engine on macOS. #clojure #mathematica
2025-12-11: Common Lisp, ASDF, and Quicklisp: packaging explained: It is important to realize that a “system” and a package are entirely different things: one is an entity in an add-on tool, the other is intrinsic to Common Lisp’s namespacing. #lisp
2025-12-14:
Cathédrale Notre-Dame de l’Assomption, Rodez, Dec. 2025
2025-12-14: /me is listening to “Zombie” by YUNGBLUD
2025-12-14: I Fed 24 Years of My Blog Posts to a Markov Model.
2025-12-16: /me is listening to “Gallowdance” by Lebanon Hanover
2025-12-16:

The speakers are pretty good. And for a laptop, they’re actually amazing. They have depth, the upmp, and are loud enough to watch films with comfort - can’t say this about any other laptop I’ve had before. — GNOME is better macOS than macOS

2025-12-16:

Why readable math still matters in a world aided by LLM-assisted code generation. — In Defense of MATLAB: Why Engineers Still Need Whiteboard-Style CodeI too am a firm believer in simple syntax (R, Mathematica, or even Scilab or Octave): “The math is identical. But in the Python version, the engineer is thinking about computer science concepts."

2025-12-16: Interesting discussion on matrix multiplication with Mathematica: Speedup matrix number multiplication. #mathematica
2025-12-18: New kid on the blocks: ty: An extremely fast Python type checker and LSP. I’m pretty happy with basedpyright at present, but I will check this new tool. #python
2025-12-18: Introduction to Software Development Tooling: Lot of good stuff and very clear handouts.
2025-12-23:

I agree that it can be difficult to make sense of a model containing billions of parameters. Certainly a human can’t understand such a model by inspecting the values of each parameter individually. But one can gain insight by examing the properties of the model—where it succeeds and fails, how well it learns as a function of data, etc. — On Chomsky and the Two Cultures of Statistical Learning

2025-12-23:

Note that this updates cells live in response to previous cells that they depend on. Not pictured is that it doesn’t update cells if their dependencies haven’t changed. You can think of this as a spreadsheet-like Jupyter, where code is only rerun when necessary. — the terminal of the futureStill I don’t really like web-based notebooks.

2025-12-23: Flocking Quadtrees.
2025-12-23:

Night road, Dec. 2025

2025-12-27:
Jardin Tino Rossi, Paris, Dec. 2025
2025-12-27: /me is listening to “Albatross” by Alela Diane
2025-12-27: Recent Python bookmarks:

How uv got so fast: uv is fast because of what it doesn’t do, not because of what language it’s written in.
profiling.sampling: statistical profiler via periodic stack sampling introduced in Python 3.15.
Exploring Mathematics with Python: mathematics with Python as a replacement to Turbo Pascal.
2025-12-29: Mole: Deep clean and optimize your Mac. #apple

Micro posting in November

Tue, 02 Dec 2025 17:02:16 +0100

2025-11-03: Recently added:
2025-11-05: By the power of grayscale!: A nice review of common techniques for image processing, with applications in C. #clang
2025-11-16:
Jardin des Plantes, Paris, Nov. 2025
2025-11-16: /me is listening to “Familiar Fields” by Duster
2025-11-16: Nice read on GNU Awk: AWK technical notes. It reminds me of all the good Awk programs I happened to find on Programming Praxis. #unix
2025-11-16: Not sure I haven’t bookmarked this one already, but in case: Deploying Common Lisp Scripts. #lisp
2025-11-16: Randomness Testing Guide: a beginner’s guide to randomness testing with a focus on simplicity and explainability.
2025-11-30:
Jardin Public, Bordeaux, Nov. 2025
2025-11-30: https://webr.sh. Reminds me of this old thread on CV. #rstats
2025-11-30:

If there is any reason why people use Vim/Emacs, it’s because at some point, when they let someone use their computer, they get to nonchalantly, seemingly in passing mention “oh, you don’t know how to use Vim?” while watching that someone struggle to exit it. Everyone likes feeling superior. — ZSH Tricks and Why I Use It

2025-11-30: The privacy nightmare of browser fingerprinting. See also Exposed By Default.

Micro posting in October

Tue, 28 Oct 2025 08:52:59 +0100

2025-10-20:

“Oh my god! They linked in more than 64k of floating point library to compute the sqrt of an integer that is always less than 128!? For gods sake, just use a for loop counting up to 12! Or if a for loop seems liek too much work, how about a 128 byte look up table?” — The IDEs we had 30 years ago… and we lost

2025-10-20: On kerning vs. letterspacing: kerning is more specific: it refers to special cases where pairs of adjacent characters are pushed towards each other.
2025-10-21: /me is listening to “Tell Me Who You Are Today” by Beth Gibbons
2025-10-21:

The biggest misconception about Rust is that it’s “harder” than other languages. But that’s only true if you measure difficulty by how fast you can start writing code. If you measure difficulty by how long it takes to debug, maintain, and make a system reliable, the playing field levels out. — Rust is not harder than other languages

2025-10-21: Advanced Matrix Multiplication Optimization on Modern Multi-Core Processors: Lot of micro-optimization like caching and kernel tricks.
2025-10-21: Algorithms for Competitive Programming (via Nima Aghdaii’s post on Suffix Arrays).
2025-10-21: Python 3.14 Is Here. How Fast Is It?: The 3.14 JIT interpreter does not appear to provide any significant gains in speed, at least not with my test scripts. #python
2025-10-21: Using deno as my game engine.
2025-10-21: Zippers: Making Functional “Updates” Efficient
2025-10-23: /me is listening to “And No More Shall We Part” by Nick Cave & The Bad Seeds
2025-10-27: /me is listening to “The Rip” by Portishead
2025-10-27: Homebrew nice update: download in progress now showing like other cli tools (npm, deno).

2025-10-27: Folding in Parallel: The input sequence can be arbitrarily partitioned (and recursively sub-partitioned, if desired) among workers, which would run in parallel with no races, dependencies or even memory bank conflicts. Such embarrassing parallelism is ideal for multi-core, GPU or distributed processing.
2025-10-27: Modern perfect hashing for strings.
2025-10-28: /me is listening to “The Suit” by Public Image Ltd
2025-10-28: I don’t think I need notmuch search and tagging facilities since mu and Neomutt cover all my needs. But in case I change my mind, I leave this links here:

the Arch wiki on notmuch
Gérer ses mails en local : étape 3 bis, du rab’ de mutt
Notmuch feature for Neomutt
Notmuch of mail a setup Part 2 - notmuch and Emacs- this HN thread

I especially liked reading Search-oriented tools for Unix-style mail.
2025-10-28: Applicative Sorting: On using Applicative to sort any collection. #haskell
2025-10-28: SSH to a machine behind firewall. On a related note, LocalCommand and RemoteCommand are also handy in some cases, e.g., LocalCommand osascript -e "tell application \"Terminal\" to set current settings of window 1 to settings set \"ssh\"" to change Apple Terminal theme when ssh’ing on a remomte machine, or RemoteCommand tmux new -A -s remote -n main to start Tmux right when connecting to a remote server.

Micro posting in September

Tue, 30 Sep 2025 09:59:16 +0200

2025-09-01:

If someone’s subscribed to your blog with RSS they can read the full text of your article in their RSS reader of choice. RSS readers are good at rendering articles. When you publish a summary with a link to the full article on your website, you’re making them come out of the RSS reader’s window or app into another. It’s annoying. Doubly so if you still have unread articles to return to. — Please publish the full article in your RSS feed

2025-09-01: The Most Important Machine Learning Equations: A Comprehensive Guide, using Numpy. #python
2025-09-02: /me is listening to “The Last Beat of my Heart” by Siouxie and the Banshees
2025-09-02: Good old Automator folder actions… I didn’t remember it was so tedious to rename default screenshots (there’s no option to add date and time, we must chain both actions together).
Not shown in this workflow, this AppleScript (without messing with file extension guard) to copy filename to the clipboard.
2025-09-02: Hacky way to get gf works with Hugo rooted inline links (e.g., /post/bla/):

setlocal includeexpr='~/Sites/aliquote/content'.substitute(v:fname,'/\$','','g').'.md'

I’ve tried several combination of path, suffixesadd and includeexpr with no success, so I ended up with this ugly harcoded combinaion of root path and extension as prefix and suffix. #vim
2025-09-02: Lot of VimL here…

~/.config/nvim
» tokei -e pack -s files
===============================================================================
 Language Files Lines Code Comments Blanks
===============================================================================
 Vim script 78 6095 5761 128 206
 Lua 17 1674 1567 37 70
 Scheme 2 99 91 1 7
 Plain Text 2 418 0 334 84
 BASH 1 8 4 1 3
 Python 1 1 0 1 0
 Swift 1 13 11 1 1
===============================================================================
 Total 102 8308 7434 503 371
===============================================================================

2025-09-02: HaScheme is a library that implements a subset of the R7RS’s libraries in a lazy way, using force and delay-force. When HaScheme is used by itself, it is a lazy functional programming language with the same syntax as Scheme, embedded within Scheme. #scheme
2025-09-03: If you installed the XCode command-line tools and you did notice there’s no longer /usr/local on macOS, you probably want to add the following to your after/ftplugin/c.vim for gf to work properly on included headers:

setlocal path+=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include

2025-09-04: /me is listening to “Beautiful is Boring” by BONES UK
2025-09-04: https://w3m.rocks: Site dedicated to the finer aspects of W3M, an open-source terminal web browser (and pager) available for any Unix-like OS.
2025-09-04: Comfy macOS setup: Neomutt running inside Tmux in a separate tab) so that I can completely wipe out Apple terminal and get back to it at any time. Also, setting beep_new = yes put a little bell on the tab when I got mail.

2025-09-11: /me is listening to “Papillon” by Editors
2025-09-15:

Why are bookmarks dismal? Because they tend to accumulate way too fast. — How I manage my bookmarksWhence the micro section on this website…

2025-09-16: Looks like Tahoe is ready, as well as another sequel of Sequoia. #apple

2025-09-16: New features available with macOS Tahoe: Apple Terminal gets Powerline glyphs and 24-bit color support. What else? More detailed review here. #apple
2025-09-16: Zstandard’s long range mode works wonders for genome sequences without newlines.
2025-09-17: /me is listening to “Let Us Be Loving” by Alice Russell
2025-09-17: I finally got this “rounded upgrade”, or macOS 26. As someone who spend almost all the day in a Terminal, it’s pretty okay since I’m only exposed to changes made to the menu bar (Dock has always been hidden on my laptops, and I don’t store files or folders on the Desktop). Safari 26 was no longer working with comact mode on Sequoia 15.7 (no way to interacti with the address bar other than with using keyword shortcuts). The floating buttons and panels are a minor inconvenience on most apps I use, but still it’s a drastic shift in macOS UI. Maybe this is becuase I still own a very old iPhone with a very outdated iOS, or maybe this is just me, like many others. Also, they revamped the nifty volume icons we used to have on the Desktop. At least we got a useable Spotlight again. #apple
2025-09-17: More than 5k pages left here.

2025-09-17: I just want an 80×25 console, but that’s no longer possible. Reminds me of my playing with virtual consoles all along on Linux distros. #unix
2025-09-18: How Container Filesystem Works: Building a Docker-like Container From Scratch.
2025-09-18: How to Debug Chez Scheme Programs. Note that this applies to Chez Scheme. #scheme
2025-09-18: Rendezvous hashing is an algorithm to solve the distributed hash table problem – a common and general pattern in distributed systems.
2025-09-19: /me is listening to “The Suburbs” by Arcade Fire
2025-09-19: Elements of C Style: Oldies but goodies. #clang
2025-09-23:

Since I’m back to work tomorrow I got it out to see if it was charged enough for the train journey in the morning. 90% remaining. After 3 weeks. It wasn’t turned off, just closed. Why I’m Spoiled By Apple Silicon (But Still Love Framework)

2025-09-23: Compiling with Continuations.
2025-09-24:

Why use a language with 0-based indexing for linear algebra other than putting an additional cognitive burden on the students learning the subject? This is a recipe for the nefarious off-by-one error. And these errors are sneaky. The code might run but produce incorrect results and it’s a nightmare for the students (or the poor TA helping them) to figure out why. — Is Fortran better than Python for teaching the basics of numerical linear algebra?

2025-09-25: Debugging Rust with Vim’s Termdebug. #vim
2025-09-29: Why Today’s Python Developers Are Embracing Type Hints. #python
2025-09-30:

A recommendation engine that looks at my browsing history, sees what blog posts or articles I spent the most time on, then searches the web every night for things I should be reading that I’m not. In the morning I should get a digest of links. I also want to be able to give feedback on which were good suggestions and which weren’t to improve the next day’s digest. — The 28 AI tools I wish existedBack in the days, I used to use Prismatic, and I was very happy with its recommendation engine. Recommendation engines and collaborative filtering are everywhere since then. What’s next?

Writing in Markdown with Neovim

Sun, 31 Aug 2025 09:50:36 +0200

I don’t use any plugin for writing Markdown in Neovim. I used to have some function to navigate between headings, select code block, or for folding sections, code block and things like that. But now all of this is builtin. To enable folding via Treesitter, just add:

setlocal foldmethod=expr
setlocal foldexpr=v:lua.vim.treesitter.foldexpr()

The [[ and ]] to jump between headers are available too, and vab will select a code block (even % works to jump to matching fences¹). I like to use reference rather inline links. Fortunately, Seth Brown’s formd still works, and I added a shortcut to put all link rightly numbered at the end of the document. I also have a dedicated function to put all heading in the quickfix list, which I find more convenient than folding all the document and navigating between headers with zj and zk. Previously, I was relying on ctags to generate tags for me, and I used fzf-lua btags to live query the table of contents. I can preview any Markdown file with Marked2.app – I have custom preprocessor and processor to render images from my website properly or to use the same custom CSS file that I use to export document to HTML. Finally, I use translate-shell for brief translation from French to English.

nnoremap <buffer> gra :call Formd("-r")<cr>
xnoremap <buffer> <silent> grn :!trans :en -brief -<cr>
nnoremap <buffer> gO :call MarkdownHeaders()<CR>
nnoremap <buffer> <silent> gl :!open -a Marked\ 2.app %<cr>
nnoremap <buffer> <leader># :-1 read ~/.config/nvim/templates/md<cr>

Some folks use the K mapping to call macOS default dictionary, but I use setlocal keywordprg=nvim-keywordprg\ dict, where nvim-keywordprg is a little shell script that handles several external doc commands (Stata, Lisp, R, Python) with w3m, although I only use it for Common Lisp actually. Here is the corresponding entry for dict:

term=$(echo -n "$2" | jq -Rr @uri)
curl "dict://dict.org/d:$term"

[2025-09-02]
Finally, I found it more convenient to use vim-mac-dictionary. With little effort, you can extract the useful part of the code. Be sure to sort your local dictionaries in the right order since the Swift code will only lookup the first entry. Since you can use any VimL or Lua function as keywordprg (when prefixed by a colon), you only need to modify the defualt command so that it reads: command! -nargs=+ MacDictWord :silent! call Find(expand('<cword>'))<CR>. Much more pleasant than the above ad-hoc solution.

To manage my bibliographic entries, I use Bibtex and a wrapper around fzf-lua which can be called in normal or insert mode. It was very useful when I was writing tutorial and slides, but now I rarely use it since I mostly write on this blog.

Treesitter now include both the markdown and markdown_inline parsers, which means you get highlighting for free everywhere, including code blocks. I do not use any linter, but prettier does a nice job when it comes to formatting the whole document:

compiler quarto
setlocal formatprg=prettier\ --parser=markdown\ --stdin-filepath

As can be seen, I also use quarto as my make program. Actually it’s not very elaborated,a nd I need to fix the errorformat, but at least it allows me to export Markdown or Quarto documents to PDF or HTML using make pdf or make html:

" compiler/quarto.vim
if exists("current_compiler") | finish | endif
let current_compiler = "quarto"

let s:cpo_save = &cpo
set cpo&vim

silent CompilerSet makeprg=quarto\ render\ %\ --to\ $*

let &cpo = s:cpo_save
unlet s:cpo_save

♪ Linkin Park • Castle of Glass

Or maybe not, I didn’t really check, but I still have this piece of code: let b:match_words = b:match_words . ',^```.\+$:^```$'. ↩︎

Micro posting in August

Fri, 29 Aug 2025 21:34:39 +0200

2025-08-02: I just noticed that Neovim now allow for full case-insensitive tab completion in command mode, which means I can now enter path as I do in Zsh. #vim
2025-08-10: 10 new zsh tricks you may not know….
2025-08-13: /me is listening to “There is a light that never goes out” by The Smiths
2025-08-13:

“Generally speaking, if you want to have very very good performance, we should avoid cache misses as much as possible.” But Python is prone to having a memory layout that is cache-unfriendly. — Python performance myths and fairy tales

2025-08-13: Rich’s sh (POSIX shell) tricks: lots of little details indeed. #unix
2025-08-13: journaling system cobbled together with nix, vim, coreutils. How to make things simple. See also I Tried Every Todo App and Ended Up With a .txt File. #vim
2025-08-14: Apparently, external evaluation cells in Wolfram 14.3 are in trouble with personal .Renviron file. If I remove mine, everything goes well, otherwise I get a permission issue with renv. My $HOME/.Renviron only modifies the path for renv cache:

R_LIBS_USER="~/.local/lib/R/library"
R_PROFILE="~/.config/R/Rprofile"
RENV_CONFIG_USER_PROFILE=TRUE
RENV_PATHS_ROOT="~/.cache/org.R-project.R/R/renv"

Once R virtual environment is created, I can reinstate my personal config file, and everything is okay, but I guess next time if an R expression that requires to install external dependencies I will face a similar issue. Otherwise, external evaluation cell and the whole ExternalEvaluate stuff looks pretty interesting.

2025-08-18: TIL about doxx, which is a MS docx viewer for the terminal that preserves formatting and provides an interactive experience.
2025-08-19: /me is listening to “Facility Girls” by Soft Cell
2025-08-19: Are you looking for an online NBT editor for minecraft? https://offroaders123.github.io/Dovetail/
2025-08-19: I got tired of NNW and I’m back with Newsboat. The bug I mentioned earlier on was due to a problem with my config file (probably related to multi-threading).
2025-08-19: I tried them all (recently, Ghostty, for a day). My preferred terminal emulator was probably foot, back when I was on Linux.

» ls .config/archived
alacritty/ ghostty/ kitty/ matterbridge.toml tig/ zed/
foot/ helix/ lazygit/ psql/ wezterm/

2025-08-19: TIL about Doppler. Drag’n drop and just press play. #apple
2025-08-19: TIL we can open multiple PDF files as tabs in Preview.app (h/t Burkhard Schmidt). On Sequoia, the corresponding setting is now in the Desktop and Dock panel. #apple
2025-08-20: I recently switched to vi keymap for Zsh. Here are some useful remapping to make it feels smoother:

bindkey -M vicmd '~' vi-swap-case
bindkey -M vicmd 'u' undo
bindkey -M vicmd 'U' redo
bindkey -M vicmd "?" history-incremental-pattern-search-backward
bindkey -M vicmd "/" history-incremental-pattern-search-forward

bindkey -M viins '^?' backward-delete-char # backspace
bindkey -M viins '^[[Z' reverse-menu-complete # shift-tab
bindkey -M viins '^a' beginning-of-line
bindkey -M viins '^e' end-of-line
bindkey -M viins '^h' backward-delete-char
bindkey -M viins '^k' kill-line
bindkey -M viins '^n' down-history
bindkey -M viins '^p' up-history
bindkey -M viins '^u' backward-kill-line
bindkey -M viins '^w' backward-delete-word
bindkey -M viins '^y' yank

autoload -U edit-command-line
zle -N edit-command-line
bindkey '^f' edit-command-line

I don’t use ^r for backward search since it is managed by Fzf directly.And in Tmux:

bind -T copy-mode-vi u send-keys -X halfpage-up
bind -T copy-mode-vi d send-keys -X halfpage-down
bind -T copy-mode-vi v send-keys -X begin-selection
bind -T copy-mode-vi C-l send-keys -X clear-selection
bind -T copy-mode-vi y send-keys -X copy-selection-and-cancel
bind -T copy-mode-vi 'C-v' send-keys -X rectangle-toggle \; send -X begin-selection
bind -T copy-mode-vi n send-keys -X search-forward "» "
bind -T copy-mode-vi p send-keys -X search-backward "» "
bind -T copy-mode-vi / command-prompt -i -I "#{pane_search_string}" -p "(search down)" { send -X search-forward-incremental '%%%' }
bind -T copy-mode-vi ? command-prompt -i -I "#{pane_search_string}" -p "(search up)" { send -X search-backward-incremental '%%%' }
%if "#{!=:$EDITOR,}"
set editor $EDITOR
bind -T copy-mode-vi C-f send-keys -X pipe-and-cancel 'xargs -I% tmux send-keys $EDITOR Space % Enter'
%endif

Which means in both case you can edit a command or a visual selection in your $EDITOR using ^f.
2025-08-21:

More importantly, though, feed readers put your subscriptions under your control. There’s no risk that the administrator of an RSS feed sells your details to spammers, or accidentally leaks them like the aforementioned world renowned expert on account information leaks, because there’s no administrator. If you decide you don’t want to see a particular feed any more, you just instruct your reader to stop fetching it. There’s no sending of mail or clicking of unsubscribe links. Nothing changes in the content creator’s system; they don’t even need to know it happened. — Really Simple Syndication

2025-08-21: No more Python or Bash or Javascript to fetch title or other metadata from Youtube link required if you have yt-dlp installed on your system:

2025-08-21: Recently I settled on glow and doxx. See also Terminal Trove.
2025-08-21: Mavericks Forever. This reminded me that I still have my old MacBook Pro from late 2013, which came with Mavericks pre-installed. I ended up my Apple journey with Mojave running on a Macbook 12”, which has always been my preferred version of OSX, and one of the best laptop I used to use over the years. Now the MBP is waiting for Ubuntu to upgrade thousand of packages (I haven’t used it for two years now) before I perform a do-release-upgrade.
2025-08-25: Tonight I updated my default colorscheme (quiet) in Neovim when I noticed there was a monochrome version. Since we cannot set t_Co in Neovim, I create a user colorscheme named morequiet using these specs only. I added a few tweaks for diff mode (I like the look’n feel of the quiet colors in this case) and TODO-like @comment.* identified by Treesitter. So far so good. #vim
2025-08-25: Everything Is Correlated. Great article, as always, full of historical references.
2025-08-25: Managing Local and Remote Filesystems with Vim and netrw. #vim
2025-08-27: I don’t use Cabal that often, and I manage almost everything with Stack. I accidently installed a package (bhoogle) that adds nothing to Hoogle, but there’s no cabal delete or cabal uninstall, you know. So I simply nuked the whole $HOME/.cabal directory that GHCup installed for me with an older version of Cabal. And now I can’t get it back. Instead, Cabal now follows the XDG Base Directory Specification, which means compiled stuff will end up in $HOME/.local/state. Let’s go for another env variable (CABAL_DIR) then. #haskell
2025-08-27: Now playing from Apple Music right into Tmux status line!
Put this script somewhere in your $PATH:

#!/bin/sh
unset title

title=$(osascript -e 'if application "Music" is running then' -e 'tell application "Music" to get name of current track' -e 'end if')
artist=$(osascript -e 'if application "Music" is running then' -e 'tell application "Music" to get artist of current track' -e 'end if')

if ${title+"true"}; then echo "$artist ♪ $title"; fi

If you don’t mind using the -e switch as your line separator; otherwise just write a .scpt file and load it directly.See this gist for a more elaborated solution that accounts for both Cmus and Apple Music.
2025-08-27: Rust Language Cheat Sheet. #rust
2025-08-28: A Flexible Minimalist Neovim for 2024. I wish people favor builtin stuff more often, as does the author here, rather than installing every single plugin available on GitHub. #vim
2025-08-29: While preparing a short tutorial on using Stack to manage Haskell projects, I noticed that my screenshots include drop shadow from macOS screenshot builtin app. I was reminded that the key is to ⇧⌘4, then press the Space bar as usual, but before clicking hold down the option (⌥) key. #apple

ArXiving on August 2025

Tue, 26 Aug 2025 16:22:28 +0200

Statistical methods: Basic concepts, interpretations, and cautions (https://arxiv.org/abs/2508.10168)

The study of associations and their causal explanations is a central research activity whose methodology varies tremendously across fields. Even within specialized subfields, comparisons across textbooks and journals reveals that the basics are subject to considerable variation and controversy. This variation is often obscured by the singular viewpoints presented within textbooks and journal guidelines, which may be deceptively written as if the norms they adopt are unchallenged. Furthermore, human limitations and the vastness within fields imply that no one can have expertise across all subfields and that interpretations will be severely constrained by the limitations of studies of human populations. The present chapter outlines an approach to statistical methods that attempts to recognize these problems from the start, rather than assume they are absent as in the claims of ‘statistical significance’ and ‘confidence’ ordinarily attached to statistical tests and interval estimates. It does so by grounding models and statistics in data description, and treating inferences from them as speculations based on assumptions that cannot be fully validated or checked using the analysis data.

This is a chapter from the 3rd edition of the Handbook of Epidemiology (Pigeot and Ahrens, eds.). It’s been a long time since I haven’t read papers or textbook on epidemiology, but I had this one on my bookshelf at some point in my life. This is all about the study of associations and causal relationships outside the field of randomized clinical trials, which are discussed upfront:

The methods thus provide logically sound inferences only in ideal cases, rather than correct answers for our actual imperfect applications. They are however widely applied and misinterpreted in studies that fall far short of their assumptions, as when actual participation and treatment are not random. These problems are only partially accounted for by discussions of study limitations; occasionally they are further analyzed with speculative models for the non- random (systematic) aspects of the causal processes generating the data.

But the simplifying assumptions are a recurring theme in the rest of the chapter, as well as that of a fictional world in which all assumptions hold. There is a very enlightning discussion on $p$-values, how they are often misinterpreted, and how they should be regarded in light of observed data (see pp.19 ff.). And basically, the whole paper is about statistical significance and how not to fool onto the trap of significant or non-signifcant $p$-values. Of note, Bayesian statistics are briefly discussed. I must admit I never encountered a single epidemiological paper using Bayesian inference.

A Comprehensive Comparison of the Wald, Wilson, and adjusted Wilson Confidence Intervals for Proportions (https://arxiv.org/abs/2508.10223)

The standard confidence interval for a population proportion covered in the overwhelming majority of introductory and intermediate statistics textbooks surprisingly remains the Wald confidence interval despite having a poor coverage probability, especially for small sample sizes or when the unknown population proportion is close to either 0 or 1. Using the mean coverage probability, and for some sample sizes, Agresti and Coull showed not only that the 95% Wilson confidence interval performs better, but also showed that 95% adjusted Wilson of type 4 confidence interval, obtained by simply adding four pseudo-observations, outperforms both the Wald and the Wilson confidence intervals. In this paper, we introduce a rainbow color code and pixel-color plots as ways to comprehensively compare the Wald, Wilson, and adjusted-Wilson of type $\epsilon$ confidence intervals across all sample sizes $n=1, 2, \dots, 1000$, population proportion values $p=0.01, 0.02, \dots, 0.99$, and for the three typical confidence levels. We show not only that adding 3 (resp., 4 and 6) pseudo-observations is the best for the 90% (resp., 95% and 99%) adjusted Wilson confidence interval, but it also performs better than both the 90% (resp., 95% and 99%) Wald and Wilson confidence intervals.

While two-group comparisons with a continuous outcome are generally easily understood and carried out using a $t$ or Wilcoxon test (although, in the latter case, with bad assumptions), estimating the precision of proportion estimates always rely on Wald confidence intervals, aka the “normal approximation”. In this case, the proprtion of success is estimated as $\hat p \pm z_{\alpha}\sqrt{\frac{\hat p(1 - \hat p)}{n}}$, where $z_{\alpha}$ is the $1-\alpha/2$ quantile of the standard normal distribution ($z=1.96$ for a 95% confidence interval). An alternative option is to rely on Wilson’s CI, and the well-known Agresti-Couli interval provides a very good approximation to it. You may recall that in the case of proportion, the standard error may be computed from the estimated proportion, $\hat p$, which yield Wald CIs, or from the proportion under $H_0$, $p_0$, which is the score test statistic (also called $z$-test in most statistical packages).¹ The Wilson CIs are more tedious to compute (see Formula 2 in the paper and this blog post). The authors describe adjusted Wilson intervals for 90, 95, and 99% confidence level. The color scheme retained for these analyses remain, however, quite surprising.

Better bootstrap t confidence intervals for the mean (https://arxiv.org/abs/2508.10083)

This article explores combinations of weighted bootstraps, like the Bayesian bootstrap, with the bootstrap $t$ method for setting approximate confidence intervals for the mean of a random variable in small samples. For this problem the usual bootstrap $t$ has good coverage but provides intervals with long and highly variable lengths. Those intervals can have infinite length not just for tiny $n$, when the data have a discrete distribution. The BC$_a$ bootstrap produces shorter intervals but tends to severely under-cover the mean. Bootstrapping the studentized mean with weights from a Beta$(1/2,3/2)$ distribution is shown to attain second order accuracy. It never yields infinite length intervals and the mean square bootstrap $t$ statistic is finite when there are at least three distinct values in the data, or two distinct values appearing at least three times each. In a range of small sample settings, the beta bootstrap $t$ intervals have closer to nominal coverage than the BC$_a$ and shorter length than the multinomial ootstrap $t$. The paper includes a lengthy discussion of the difficulties in constructing a utility function to evaluate nonparametric approximate confidence intervals.

For small samples and/or skewed data, the Bayesian $t$-test is a good option (see this unfinished post of mine). BCA correction for a bootstrap $t$-test usually undercovers the mean, and other problems appear when we consider the studentized rather than the arithmetic mean. In this paper the authors consider various techniques to overcome limitations arising from small samples and they show that the power law bootstrap $t$ and the beta bootstrap $t$ (decsribed in §2.5) provide good results, the latter being second order accurate.

See also, Hall, P. (1988). Theoretical comparisons of bootstrap confidence intervals. The Annals of Statistics, 16(3):927–953.

Dissecting Microbial Community Structure and Heterogeneity via Multivariate Covariate-Adjusted Clustering (https://arxiv.org/abs/2508.11036)

In microbiome studies, it is often of great interest to identify clusters or partitions of microbiome profiles within a study population and to characterize the distinctive attributes of each resulting microbial community. While raw counts or relative compositions are commonly used for such analysis, variations between clusters may be driven or distorted by subject-level covariates, reflecting underlying biological and clinical heterogeneity across individuals. Simultaneously detecting latent communities and identifying covariates that differentiate them can enhance our understanding of the microbiome and its association with health outcomes. To this end, we propose a Dirichlet-multinomial mixture regression (DMMR) model that enables joint clustering of microbiome profiles while accounting for covariates with either homogeneous or heterogeneous effects across clusters. A novel symmetric link function is introduced to facilitate covariate modeling through the compositional parameters. We develop efficient algorithms with convergence guarantees for parameter estimation and establish theoretical properties of the proposed estimators. Extensive simulation studies demonstrate the effectiveness of the method in clustering, feature selection, and heterogeneity detection. We illustrate the utility of DMMR through a comprehensive application to upper-airway microbiota data from a pediatric asthma study, uncovering distinct microbial subtypes and their associations with clinical characteristics.

I had to help a student two years ago for a microbial analysis, and she found an arcane R package on GitHub which relied on Spearman correlation of table of counts, IIRC, and allowed to build a network of co-occurring species, but this didn’t allow to take into account co-factors of interest. Unfortunately, the experimental design was flawed, so the analysis ended up early. It is interesting to see that covariates can be taken into account in the analysis of such large dataset .

What makes a study design quasi-experimental? The case of difference-in-differences (https://arxiv.org/abs/2508.13945)

Study designs classified as quasi- or natural experiments are typically accorded more face validity than observational study designs more broadly. However, there is ambiguity in the literature about what qualifies as a quasi-experiment. Here, we attempt to resolve this ambiguity by distinguishing two different ways of defining this term. One definition is based on identifying assumptions being uncontroversial, and the other is based on the ability to account for unobserved sources of confounding (under assumptions). We argue that only the former deserves an additional measure of credibility for reasons of design. We use the difference-in-differences approach to illustrate our discussion.

This papers provides a use case of difference-in-differences, and its relation with the parallel trends assumption, which usually holds if the exposure is randomly assigned. Random treatment allocation, however, does not mean we need to use a DID estimator, and post-treatment comparison should be preferred.

Perspective: An outlook on fluorescence tracking (https://arxiv.org/abs/2508.13668)

Tracking single fluorescent molecules has offered resolution into dynamic molecular processes at the single-molecule level. This perspective traces the evolution of single-molecule tracking, highlighting key developments across various methodological branches within fluorescence microscopy. We compare the strengths and limitations of each approach, ranging from conventional widefield offline tracking to real-time confocal tracking. In the final section, we explore emerging efforts to advance physics-inspired tracking techniques, a possibility for parallelization and artificial intelligence, and discuss challenges and opportunities they present toward achieving higher spatiotemporal resolution and greater computational and data efficiency in next-generation single-molecule studies.

For someone like me who deal with data coming from cellular biology, this provides an excellent overview of fluorescence microscopy.

♪ Stone Temple Pilots • Crackerman

See also Coin tossing experiment: Score and Wald tests. ↩︎

Mathematica shorthand notation

Sat, 23 Aug 2025 10:23:40 +0200

Like APL, J or Haskell, Wolfram comes up with a lot of shorthand notation, in addition to the prefix, postfix or infix notation. For instance, you can write Normal[•] or • // Normal, where • is any valid Wolfram expression. Here is a brief list of what I’ve collected over the years, from various sources. Usually, I tend to stick to plain text expression (so called circumfix notation), i.e. Length[{1,2,3}] rather than Length@{1,2,3} or {1,2,3}//Length. I tend to use the // postfix notation for things like Normal or MatrixForm. The only exception is for pure function and Apply (@@) or Map (/@). Sometimes I combine them all, like in the example provided on Wolfram Howto’s:

(#^3 + a) & /@ {1, 2, 4, 6, 5, 8}

Anyway, here’s the list of expression that I am aware of:

Expression	Alternative	Role
`@`	`f[•]`	function application
`/@`	`Map[f, •]`	map function to list
`//@`	`MapAll[f, •]`	map function to every subexpression
`@@`	`Apply[f, •]`	Apply list to function
`@@@`	`Apply[f, •, {1}]`	Apply list to function at first level
`@*`	`Composition[f, g]`	composition of functions, e.g. `f[g[x]]`
`/*`	`RightComposition[f, g]`	right composition of functions, e.g. `g[f[x]]`
`<>`	`StringJoin[•, •]`	concatenate string
`/.`	`Replace[•, •]`	replace expression by pattern
`//=`	`ApplyTo[•, f]`	assignment with function application
`+=`	`AddTo[•, •]`	like in C or Python
`++`	`Increment[•]`	like in C
`/.`	`ReplaceAll[•, •]`	applies a rule to transform each subpart of an expression
`;;`	`Span[•, •]`	span of elements from start to stop
`{}`	`List[•]`	create a list
`#^2&[3]`	`Function[x, x^2][3]`	example of a pure function

I don’t know of any shortcut for Thread or MapThread

♪ Evanescence • Bring Me to Life

Loess fitting in Mathematica

Thu, 21 Aug 2025 08:57:48 +0200

Wolfram, né Mathematica, 14.3 introduced a cool new feature for graphical exploratory data analysis: loess fitting. It is already discussed at length on Stephen Wolfram’s blog post about this last release, but I was interested in trying it myself. Needless to say I don’t do much applied statistical work these days. I miss compitational statistics a lot and I expect to come back to it at some point, though.

Without further ado, let’s consider data analyzed in this old course of mine, specifically the FEV dataset. It is available in R format, which is well managed by the omnibus Import function:

data = Import["fev.rds", "Dataset"]
data[1;;5, All]

We are interested in describing the relationship between the FEV (fev) and age (age) in male and female (sex). Here is a basic scatterplot:

data[ListPlot, {"age", "fev"}]

The Wolfram help center does a good job at explaining what is LOESS smoothing. Here we are interested in small departure from the linearity assumption in usual OLS.

There are various ways to displays a scatterplot smoother. First, we can just fit a local smoother:

LocalModelFit[ToTabular[data[All, {"age", "fev"}]]]

We use the tabular data structure introduced in version 14.2. It returns a FittedModel, like other statistical models in Wolfram (e.g., LinearModelFit), although there’s no best fitting model such that we cannot use Normal (despite what the online help says about the “BestFit” slot), We can obtained the predicted (interpolated) values using the “PredictedResponse” slot.

Here is another way to perform the same task: we can directly ask for a plot using ListPlot[data[All, {"age", "fev"}], PlotFit -> "Local"]. We could also work directly with a list of pairwise values, instead of association lists:

trend = LocalModelFit[List @@@ Normal[data[All, {"age", "fev"}]]];
Show[ListPlot[data[All, {"age", "fev"}]],
 Plot[trend[x], {x, 4, 18}, PlotRange -> All, PlotStyle -> Red],
 PlotRange -> All]

Finally, we can fit a separate loess curve for male and female using GroupBy on the structured dataset:

g = GroupBy[data[All, {"age", "fev", "sex"}], "sex"][[All, All, 1 ;; 2]];
male = LocalModelFit[List @@@ Normal[g[[1]]]];
female = LocalModelFit[List @@@ Normal[g[[2]]]];
Show[ListPlot[g[[1]], PlotStyle -> Blue,
 PlotLabels -> Placed["Male", Above]],
 Plot[male[x], {x, 4, 18}, PlotStyle -> Blue],
 ListPlot[g[[2]], PlotStyle -> Orange,
 PlotLabels -> Placed["Female", Below]],
 Plot[female[x], {x, 4, 18}, PlotStyle -> Orange], Frame -> True,
 ImageSize -> Large, FrameLabel -> {"Age (yr.)", "FEV (l/s)"}]

Of course, we could also play with the bandwidth parameter, as illustrated in this example on Stack Exchange.

♪ Cats on Trees • Sirens Call

XDG Base Directory and macOS

Thu, 14 Aug 2025 17:59:34 +0200

Last month I spent a few hours configuring my system (macOS Sequoia) to follow the XDG Base Directory Specification. I like the idea of having common paths defined for most applications. It worked quite well for me on Ubuntu, and I wanted to declutter my $HOME dotfiles.

[2025-08-26]
Two weeks later, I came across this blog post by Rebecca Turner on macOS and user configuration files. I share her thoughts about “When should you use ~/Library/Application Support”, and I always expected other devs do too. Hygg is a nice alternative to doxx, and in addition it supports PDF and EPUB, but its configuration and state files are written in $HOME/Library.

I already follow part of the XDG specs by putting everything I can in $HOME/.config. I also make heavy use of $HOME/.local to store additional programs (/bin), libraries (/lib) and extra stuff (share). To automate things a little I defined specific environment variables to point to the right directories. Here is what I have in my $HOME/.config/profile.d/00-path.sh which is sourced for every login shell (Apple Terminal use login shell by default):

export XDG_CONFIG_HOME="$HOME/.config"
export XDG_CACHE_HOME="$HOME/.cache"
export XDG_DATA_HOME="$HOME/.local/share"
export XDG_STATE_HOME="$HOME/.local/state"
export TERMINFO=/usr/share/terminfo/
export LEIN_HOME=$XDG_CONFIG_HOME/leiningen
export HOMEBREW_NO_EMOJI=1
export HOMEBREW_NO_AUTO_UPDATE=1
export W3M_DIR=$XDG_STATE_HOME/w3m
export SQLITE_HISTORY=$XDG_STATE_HOME/sqlite_history
export LESSHISTFILE=$XDG_STATE_HOME/less_history
export PYTHON_HISTORY=$XDG_STATE_HOME/python_history
export NCFTPDIR=$XDG_CONFIG_HOME/ncftp
export GNUPGHOME=$XDG_CONFIG_HOME/gnupg
export IPYTHONDIR=$XDG_CONFIG_HOME/ipython
export JUPYTER_CONFIG_DIR=$XDG_CONFIG_HOME/jupyter
export MYPY_CACHE_DIR=$XDG_CACHE/.mypy_cache
export MPLCONFIGDIR=$XDG_CONFIG_HOME/matplotlib
export CHICKEN_DOC_REPOSITORY=$HOME/.local/lib/scheme/chicken-doc
export CHICKEN_EGG_CACHE=$XDG_CACHE/chicken
export CHICKEN_INSTALL_REPOSITORY=$HOME/.local/lib/scheme/chicken
export CL_SOURCE_REGISTRY=$HOME/cwd/common-lisp
# export PKG_CONFIG_PATH=/opt/homebrew/lib/pkgconfig

Yes, that’s a lot of global settings. Also, please note that the aim is not to follow stricto sensu the XDG specs, just to organise things a little. The first four lines are what is of interest in this case. Things get a little more interesting when we read the documentation related to Python and its ecosystem as we learn that we can modify the default path for the IPython or Jupyter config directory, as well as Matplotlib, or get ride of Mypy stuff. I use $XDG_STATE_HOME to store all my histories (zsh, sqlite, etc.). By defining such environment variables I was able to remove almost everything in my home directory. I know we can move GHCUp, rustup, and other frameworks to other places, but I prefer to keep them in place. Here is the whole picture:

» ls -a ~
./ .config/ .juliaup/ .ssh/ cwd/ Music/
../ .cups/ .local/ .stack/ Desktop/ Pictures/
.cabal/ .deno/ .mail/ .Trash/ Documents/ Public/
.cache/ .DS_Store .Renviron .zprofile Downloads/ Sites/
.cargo/ .ghcup/ .roswell/ .zshenv@ Library/ tmp/
.CFUserTextEncoding .julia/ .rustup/ backup/ Movies/

» ls ~/.config
asciinema/ gh/ htop/ leiningen/ neomutt/ racketrc wtwitch/
bandcamp-dl.json ghc/ ipython/ matplotlib/ newsboat/ ripgreprc yt-dlp/
beets/ ghostty/ irssi/ mdlrc npm/ starship.toml zed/
clojure/ git/ isyncrc mpv/ nvim/ tmux/ zsh/
cmus/ gnupg/ jupyter/ msmtp/ profile.d/ uv/
ctags/ gnuplot/ latexmk/ ncftp/ R/ w3m/

» ls ~/.cache
bibtex-ls/ gh/ m2/ neomutt/ stack/ wtwitch/
chicken/ ghcide/ matplotlib/ nvim/ starship/ yt-dlp/
clojure-lsp/ hie-bios/ maven/ org.R-project.R/ textual/
common-lisp/ lua-language-server/ mu/ slime/ uv/

Everything zsh-related can be stored in XDG_CONFIG_HOME, provided you define the ZDOTDIR location. Here is what I put in $HOME/.zprofile:

ZDOTDIR=$HOME/.config/zsh

for file in "$HOME"/.config/profile.d/*.sh; do
 . "$file"
done

Note that the zsh config file (zshrc) will still need to be prefixed by a dot. I also symlink $HOME/.zshenv to $HOME/.zprofile to avoid issue with some CLI tools that won’t initialize zsh as a login shell (e.g., Neovim terminal). I guess it’s not difficult to avoid this issue, but I’m lazy as you know.

♪ Hooverphonic • Mad About You

I am a recovering Mac user

Wed, 13 Aug 2025 02:16:54 +0200

It’s been a little more than 6 months that I got back to using a MacBook as my daily driver. The laptop I use is a MacBook Pro M4 14″, with 24 Go of RAM and 14 cores (10 performance + 4 efficiency; also 20 cores for the inbuilt GPU). The screen is a Retina 3024 x 1964, which I use at its default resolution It’s just a beast among all of the MacBook I have own over my entire life. It’s also the first Mac machine I didn’t bought myself. The only thing I miss is the the lightness of the 12-inch MacBook, but I definitely do not regret the shitty keyboard on that machine.

Still, I don’t use most of what Apple offers nowadays. I don’t use Apple Intelligence (ChatGPT in disguise if I understand correctly), I don’t use iCloud sync for my Documents (just for casual PDFs or images I want to share with my iPhone or other machines via iCloud on the web), and above all I barely use any Mac apps except Apple Mail, Safari, Music, Photos and the Terminal. Other than that, I also use Messages (better to type on my MacBook, now that it has a decent keyboard), Preview (I only miss synctex support), and that’s pretty all! I usually have only one or two apps visible at the same time on my desktop and I rarely use the new window tiling facilities introduced in recent version of macOS.

Previous posts are all about the same story. Mots of what I need is managed by Homebrew and various command-line utilities I built over the years. I could easily replace Mail and Music with Neomutt and Cmus. In fact, I keep my emails synchronized in a xapian database managed via mu, and in macOS own mbox format. Oftentimes I find myself using Neomutt for boring tasks like deleting and archiving a bunch of emails, which I prefer to Apple Mail. I use Time Machine for a daily backup on an external SSD, but I also have hourly backups handled by Kopia, which never failed in the past 4 or 5 years.

I only installed the command-line tools, not XCode. The biggest app I have in my Applications folder is Mathematica. I think I don’t need to add an extra 10 Go slot for an application I won’t use except for Instruments.app, maybe. In sum, my setup hasn’t changed a lot, except I no longer use Emacs and I barely do any serious statistical stuff using R these days.

To sum up, the M4 Pro is a great MacBook (I’m happy I didn’t have to pay the bill myself), as much for its design as for its hardware capabilities. As I said, software coming along the MacBook are pretty okay and integrate well together. This is a big win compared to what comes natively on any Linux-based distro. Above all, I like how macOS feels like a rather unique operating system despite the fact that I barely use the mouse for interacting with the OS. I like things that look simple in macOS and not in other OSes, and I like that it doesn’t crash or update all way long. I’m definitely feeling older, and I think I can’t move back now. Also, my work habits and technical requirements have changed. I mostly work on remote servers these days, and do casual stuff that don’t require much CPU or GPU power. When I’m back at home, this laptop is a mean to process my email, write some code for fun, play Minecraft (I can now enable shaders or play full vanilla at 120 fps!), or listen to music relayed through AirPlay. Well, I’m no longer looking for performance, just stability, that is things that don’t break every three months or so. That’s also why I don’t have so many Neovim plugins (only 5), no Tmux or zsh plugin manager, and so on. I want to use my computer, not spend time tinkering my setup.

♪ Alice on the Roof • Easy Come Easy Go

Micro posting in July

Wed, 30 Jul 2025 11:26:36 +0200

2025-07-07: I often want to delete a line matching a pattern in Vim, as well as, say, the next one. Here’s how to do it: :g/pattern/.,+1d. #vim
2025-07-07: TIL about the Datastar framework. Looks good and a bit different from HTMX.
2025-07-07: Serving 200 million requests per day with a cgi-bin (via Simon Willison). I built many online quizzes and online survey using cgi-bin back in the days.
2025-07-09: cl-repl looks much better than linedit which is still broken, btw. #lisp
2025-07-10: If you want to disable syntax highlighting in Neovim 0.11+, you’ll need to disable lsp-semantic_tokens (lua vim.lsp-semantic_tokens.Stop(0, 1)), in addition to treesitter (TSBufDisable highlight; if you rely on Treesitter, it should disable stock syntax, syntax off). Better off using the quiet colorscheme as I do for quite a few months now. #vim
2025-07-10: New kid on the blocks: jank, based on Clojure but with an LLVM-based JIT backend.
2025-07-10: Uncommon Uses of Python in Commonly Used Libraries. #python
2025-07-15: /me is listening to “Pictures on My Wall” by Echo & the Bunnymen
2025-07-15: An old series of [blog posts](Debugging Lisp Part 1: Recompilation) on debugging Lisp. The author also covered series and various macros. Be aware there’s no index, though. #lisp
2025-07-15: MacPaint Art From The Mid-80s Still Looks Great Today. Oldies but goodies. #apple
2025-07-15: State of the Terminal: A nice review with lot of technical discussion on key encoding and the xterm-256color falaccy. #unix
2025-07-15: macOS defaults list: Useful if you’re looking for a particular settings that you’d like to update in your terminal. #apple
2025-07-16: I don’t even know what to think about this, but it was interesting to read: Solving LeetCode™ problems with Racket. #racket
2025-07-16: Since I haven’t looked back at my Neovim config for months, except for fixing occasional deprecations or API changes, I’m just reviewing how past projects evolved over time. I noticed that Lazyvim no longer relies on telescope but is now using fzf, which is great. It looks like Blink is the new rockstar for auto-completion. I only use builtin Vim stuff, but it looks nice. #vim
2025-07-16: Algorithms for making interesting organic simulations: Lot of interesting stuff, which reminds me of Anders Hoff’s work at https://inconvergent.net.
2025-07-17: /me is listening to “Unsent” by Minuit Macine
2025-07-20: /me is listening to “Salute Your Solution” by The Raconteurs
2025-07-20:

Nowadays, Apple includes the system upgrades in the upfront cost of its computers, so the incentive to constantly roll out ten or twenty or three hundred “new features” should be lower. Inexplicably, since the company adopted that no extra charge, yearly release cadence, it has seemingly been more reticent to do a disciplined “Snow” release, no matter how necessary. — Apple Needs a Snow Sequoia

2025-07-20: Cool Python Functions. I already came across this post a while ago, but it’s interesting to read or reread, especially on how to iterate backward or how to merge two dictionnaries. #python
2025-07-21: I spent some time figuring out why my Zsh startup time became so slow over time. Initially I thought about compinit, and I added this fix mentioned everywhere on the internet (read zcompdump once per day):

# autoload -U compinit && compinit
autoload -Uz compinit
if [ $(date +'%j') != $(stat -f '%Sm' -t '%j' ~/.config/zsh/.zcompdump) ]; then
 compinit
else
 compinit -C
fi

This helped, but the real culprint was NVM which considerably slows down Zsh startup with its loading script nvm.sh. I use the following trick. In retrospect, compinit isn’t accounting for that much time when checked agaisnt zprof, but Zsh now starts like a breeze again:

~/.config/zsh
» time zsh -i -c exit
Saving session...completed.zsh -i -c exit
0,03s user 0,03s system 90% cpu 0,073 total

2025-07-21: Still wasting time looking at possible micro-optimizations on the CLI side. The MacBook M4 Pro is a beast, and I found it reduced by a fair amount (~ 40%) Neovim startup time. Digging into how fast it starts now (using [vim-startuptime](https://github.com/rhysd/vim-startuptime) and hyperfine), I found the following:

scratch master* 2m44s
» hyperfine "nvim --headless +qa" --warmup 5

Benchmark 1: nvim --headless +qa
 Time (mean ± σ): 32.5 ms ± 0.4 ms [User: 19.0 ms, System: 7.1 ms]
 Range (min … max): 31.4 ms … 33.7 ms 85 runs

scratch master*
» vim-startuptime --vimpath nvim
Extra options: []
Measured: 10 times

Total Average: 22.782900 msec
Total Max: 28.338000 msec
Total Min: 21.359000 msec

 AVERAGE MAX MIN
------------------------------
14.312000 18.240000 13.348000: /Users/chl/.config/nvim/init.lua
 5.226100 6.687000 4.827000: require('fzf-lua')
 2.576100 3.206000 2.425000: require('nvim-treesitter.configs')
 2.393600 2.972000 2.255000: require('nvim-treesitter.query')
 2.110400 2.705000 1.979000: require('vim.lsp')
 1.709600 2.168000 1.511000: require('fzf-lua.config')
 1.443200 1.824000 1.330000: require('fzf-lua.path')
 1.227300 1.388000 1.181000: loading rtp plugins
 1.133800 1.385000 1.052000: require('nvim-treesitter.tsrange')
 1.064300 1.333000 1.004000: require('nvim-treesitter.compat')
 1.053900 1.290000 0.977000: require('nvim-treesitter.ts_utils')
 1.016600 1.270000 0.961000: require('vim.treesitter')
 0.990800 1.184000 0.943000: /Users/chl/.config/nvim/plugin/lisp.lua
 0.973200 1.165000 0.927000: nvim_exec2() called at /Users/chl/.config/nvim/plugin/lisp.lua:0

The lisp.lua custom plugin is just here to load slimv which sits as an optional plugin, and so I realized that there was no gain in this case. Better to leave it as a start plugin and remove the custom plugin itself. It makes no difference whatsoever. It’s still much higher than three years ago when I was using a proper package manager and much less Lua code in my init.lua. However, I found that using vim.loader.enable() introduced in Neovim 0.9 resulted in a slight improvement:

~/.config/nvim
» hyperfine "nvim --headless +qa" --warmup 5
Benchmark 1: nvim --headless +qa
 Time (mean ± σ): 25.7 ms ± 0.5 ms [User: 12.8 ms, System: 6.7 ms]
 Range (min … max): 24.3 ms … 26.9 ms 112 runs

Loading my init.lua is now two times faster as reported by vim-startuptime. #vim
2025-07-26: I didn’t even know that Apple dropped rsync and replaced it with openrsync.

» rsync --version
openrsync: protocol version 29
rsync version 2.6.9 compatible

2025-07-26: I ran into this issue when reinstating Cmus on my MacBook. The locale fix works for me. I also had to set softvol=true to change volume (don’t know why but I didn’t need this on Mojave, IIRC).
2025-07-26: It’s 2025, and here’s the state of Python on macOS. I only use virtual environments and uv, but I wish we could go all uv tool install. #apple #python
2025-07-26: TIL about macOS / binaries, to look up macOS binaries lying all around System Monitor. #apple
2025-07-26: 6 Techniques Hacker News Uses to Create Great Shell Scripts: Some good advices here and there. #unix
2025-07-26: SVGs that feel like GIFs.
2025-07-26: Yatoro: VIM-like Apple Music Player in Terminal. #apple #unix

ArXiving on July 2025

Wed, 23 Jul 2025 15:27:46 +0200

Multiple Hypothesis Testing in Genomics (https://arxiv.org/abs/2506.23428)

This analysis report presents an in-depth exploration of multiple hypothesis testing in the context of Genomics RNA-seq differential expression (DE) analysis, with a primary focus on techniques designed to control the false discovery rate (FDR). While RNA-seq has become a cornerstone in transcriptomic research, accurately detecting expression changes remains challenging due to the high-dimensional nature of the data. This report delves into the Benjamini-Hochberg (BH) procedure, Benjamini-Yekutieli (BY) approach, and Storey’s method, emphasizing their importance in addressing multiple testing issues and improving the reliability of results in large-scale genomic studies. We provide an overview of how these methods can be applied to control FDR while maintaining statistical power, and demonstrate their effectiveness through simulated data analysis.
The discussion highlights the significance of using adaptive methods like Storey’s q-value, particularly in high-dimensional datasets where traditional approaches may struggle. Results are presented through typical plots (e.g., Volcano, MA, PCA) and confusion matrices to visualize the impact of these techniques on gene discovery. The limitations section also touches on confounding factors like gene correlations and batch effects, which are often encountered in real-world data.
Ultimately, the analysis achieves a robust framework for handling multiple hypothesis comparisons, offering insights into how these methods can be used to interpret complex gene expression data while minimizing errors. The report encourages further validation and exploration of these techniques in future research.

An overview of different approach to control the false discovery rate in multiple testing: Benjamini-Hochberg (BH), Benjamini-Yekutieli (BY), and Storey’s method. The BH method is already available in R’s p.adjust(). The BY method is just an extension of the BH method that accounts for positive correlations among test statistics, while Storey’s technique estimates the proportion of null hypotheses $\pi_0$ among the tested CDS. Whether you rely on a nominal $\alpha$ level or $q$-value cutoffs, as in proteomics analyses, all those methods work equally well. The author’s conclusion is as follows:

BH is best for balanced performance between false discovery control and power.
BY is suited for avoiding false positives but sacrifices true positive identification.
StoreyQ is optimal for maximizing significant discoveries in exploratory genomics research.

Note that although this targets RNA-Seq analysis, where Multiple Hypothesis Testing is one of the many challenges raised by the big data omics era, this should apply to other contexts as well. It is worth noting that the authors included a simulation model for count data, using a Negative Binomial Model, which is what is usually found in pipeline like DESeq2.

I appreciated that the author provided a sidenote on statistical significance and biological relevance: “It is also crucial to recognize that statistical significance does not always imply biological importance, nor does a lack of significance disprove a biological hypothesis. In highly variable, small-sample contexts, meaningful changes may evade detection, reinforcing the need for well-powered experiments. Similarly, even a modest difference can be flagged as significant in extremely large datasets. These nuances highlight that DE detection should be interpreted within the broader scope of effect sizes, reproducibility, and biological validation.”

Controlling the false discovery rate under a non-parametric graphical (https://arxiv.org/abs/2506.24126)

We propose sufficient conditions and computationally efficient procedures for false discovery rate control in multiple testing when the $p$-values are related by a known \emph{dependency graph} – meaning that we assume independence of $p$-values that are not within each other’s neighborhoods, but otherwise leave the dependence unspecified. Our methods’ rejection sets coincide with that of the Benjamini–Hochberg (BH) procedure whenever there are no edges between BH rejections, and we find in simulations and a genomics data example that their power approaches that of the BH procedure when there are few such edges, as is commonly the case. Because our methods ignore all hypotheses not in the BH rejection set, they are computationally efficient whenever that set is small. Our fastest method, the IndBH procedure, typically finishes within seconds even in simulations with up to one million hypotheses.

Yet another paper on controlling the false discovery rate. As mentioned above, when there’s some correlation between tested markers, modification of the default FDR correction (Benjamini–Hochberg) is in order. The trick the authors used in this work is to take into account a dependecy graph for $p$-values, allowing to assume independence of $p$-values that are not within each other’s neighborhoods, and otherwise leave the dependence unspecified.

clustra: A multi-platform k-means clustering algorithm for analysis of longitudinal trajectories in large electronic health records data (https://arxiv.org/abs/2507.00962)

Background and Objective: Variables collected over time, or longitudinally, such as biologic measurements in electronic health records data, are not simple to summarize with a single time-point, and thus can be more holistically conceptualized as trajectories over time. Cluster analysis with longitudinal data further allows for clinical representation of groups of subjects with similar trajectories and identification of unique characteristics, or phenotypes, that can be investigated as risk factors or disease outcomes. Some of the challenges in estimating these clustered trajectories lie in the handling of observations at inconsistent time intervals and the usability of algorithms across programming languages. Methods: We propose longitudinal trajectory clustering using a k-means algorithm with thin-plate regression splines, implemented across multiple platforms, the R package clustra and corresponding \SAS macros. The \SAS macros accommodate flexible clustering approaches, and also include visualization of the clusters, and silhouette plots for diagnostic evaluation of the appropriate cluster number. The R package, designed in parallel, has similar functionality, with additional multi-core processing and Rand-index-based diagnostics. Results: The package and macros achieve comparable results when applied to an example of simulated blood pressure measurements based on real data from Veterans Affairs Healthcare recipients who were initiated on anti-hypertensive medication. Conclusion: The R package clustra and the SAS macros integrate a K-means clustering algorithm for longitudinal trajectories that operates with large electronic health record data. The implementations provide comparable results in both platforms, satisfying the needs of investigators familiar with, or constrained by access to, one or the other platform.

It’s been years I’m no longer involved in healths analytics and patient-reported outcomes, but this one reminded me of a post on Cross Validated about clustering of longitudinal data. Since I’m barely reminescent of those days, I remember one answer of mine which took up my entire morning and into which I put a lot of effort and attention.

Targeted tuning of random forests for quantile estimation and prediction intervals (https://arxiv.org/abs/2507.01430)

We present a novel tuning procedure for random forests (RFs) that improves the accuracy of estimated quantiles and produces valid, relatively narrow prediction intervals. While RFs are typically used to estimate mean responses (conditional on covariates), they can also be used to estimate quantiles by estimating the full distribution of the response. However, standard approaches for building RFs often result in excessively biased quantile estimates. To reduce this bias, our proposed tuning procedure minimizes “quantile coverage loss” (QCL), which we define as the estimated bias of the marginal quantile coverage probability estimate based on the out-of-bag sample. We adapt QCL tuning to handle censored data and demonstrate its use with random survival forests. We show that QCL tuning results in quantile estimates with more accurate coverage probabilities than those achieved using default parameter values or traditional tuning (using MSPE for uncensored data and C-index for censored data), while also reducing the estimated MSE of these coverage probabilities. We discuss how the superior performance of QCL tuning is linked to its alignment with the estimation goal. Finally, we explore the validity and width of prediction intervals created using this method.

Again, it’s been a long time since I ever ran an RF algorithm on real data points. It looks like the RF framework has been extended far and long, and even for quantile regression. In this regard, the out-of-bag sample is used to estimate the bias of the marginal quantile coverage probability estimate.

Tensor-product interactions in Markov-switching models (https://arxiv.org/abs/2507.01555)

Markov-switching models are a powerful tool for modelling time series data that are driven by underlying latent states. As such, they are widely used in behavioural ecology, where discrete states can serve as proxies for behavioural modes and enable inference on latent behaviour driving e.g. observed movement. To understand drivers of behavioural changes, it is common to link model parameters to covariates. Over the last decade, nonparametric approaches have gained traction in this context to avoid unrealistic parametric assumptions. Nonetheless, existing methods are largely limited to univariate smooth functions of covariates, based on penalised splines, while real processes are typically complex requiring consideration of interaction effects. We address this gap by incorporating tensor-product interactions into Markov-switching models, enabling flexible modelling of multidimensional effects in a computationally efficient manner. Based on the extended Fellner-Schall method, we develop an efficient automatic smoothness selection procedure that is robust and scales well with the number of smooth functions in the model. The method builds on a random effects view of the spline coefficients and yields a recursive penalised likelihood procedure. As special cases, this general framework accommodates bivariate smoothing, function-valued random effects, and space-time interactions. We demonstrate its practical utility through three ecological case studies of an African elephant, common fruitflies, and Arctic muskoxen. The methodology is implemented in the LaMa R package, providing applied ecologists with an accessible and flexible tool for semiparametric inference in hidden-state models. The approach has the potential to drastically improve the level of detail in inference, allowing to fit HMMs with hundreds of parameters, 10-20 (potentially bivariate) smooths to thousands of observations.

This is a novel approach for semiparametric inference in hidden-state models, which relies on incorporating tensor-product interactions into Markov-switching models, also known as Hidden Markov Model, a mix of two stochastic processes (one observed and the other latent). It’s dense and I need to read this when I’ll have more time. Now, let’s go back to holidays, FWIW.

♪ Within Temptation • Faster

July in review

Sun, 20 Jul 2025 19:49:55 +0200

https://news.ycombinator.com/item?id=11071754: It’s been a long since I deleted all my dotfiles on GitHub. Who cares? I have my own setup backed up on external HD and private repos if I need to deploy them on new machines. It has been working fine so far. Also, I must say I’m now barely maintaining a few CLI tools: Zsh, Neovim, Tmux, and Neomutt. Anyway, I’ve been using this approach for a while ten years ago and it’s pretty simple and efficient.
Lisp in Vim with Slimv or Vlime on HN. This provides a reasoned account of why a true REPL is better than any send-to-terminal plugin, like vim-slime. I always enjoy reading Susam Pal’s articles. It seems I didn’t bookmarked the right HN threads, but there are plenty other posts around this very nice post of him. See also this thread.
Ghostty 1.0 on HN. Yet another interesting thread where I learned a lot about pseudo terminal and technical details like missing damage tracking or the use of rendering threads as QoS background. In a similar vein, I highly recommend Gregory Anders’s discussion on the State of the Terminal to learn more about key encoding and capabilities as reported by infocmp. At present, I’m happy with Apple Terminal, especially since I mostly use B&W color scheme everywhere, ¹ and Tmux takes care of the rest for me. I may change my mind later.
I recently learned about tex2page which looks like an interesting alternative to Hevea.
Why your website should be under 14kB in size. I don’t really care how long it takes to load each pahe of this website as long as it is not too much. I really care about the fact that pages don’t look too much bloated and that they are readable on almost any browsers. I cry whenever I came across websites where we can barely read anything because of bad design or unreadable text with crazy background.
I’m Switching to Python and Actually Liking It: If you’re into uv, this post might be of interest. I learned about ty by reading this post. More than uv, the author mentions many tools and techniques that should be considered good practises nowadays.

I strongly believe the maximum quality of work is reached using the human+LLM equation. I believe that humans and LLMs together are more productive than just humans, but this requires a big “if”, that is, if such humans have extensive communication capabilities and LLMs experiences: the ability to communicate efficiently is a key factor in using LLMs. — Coding with LLMs in the summer of 2025

♪ LeAnn Rimes • How Do I Live

I became so lazy over the years that I didn’t bother to make a truly B&W theme for Neomutt. I just configured the mono color scheme and made an alias for Zsh: alias mutt="TERM=vt100 neomutt". ↩︎

The unquantified self #33

Fri, 11 Jul 2025 06:20:11 +0200

Hoodie, the last entry dates back to June 2024. In the meantime, a lot has happened: last year, I spent an uninspiring summer resting up a storm, before getting back into the swing of things at the start of the new academic year in September. Several scientific articles published since the start of the year and numerous students to supervise. A busy academic life. This blog has been quite dead for a while. It’s almost like since I switched to my new MacBook I stopped doing anything fancy on my own. So, where do we stand today?

Well, I’ll be soon be on vacation, so I’ve got nothing to do but rest, read and wander around. If I find enough motivation, I’ll get back to drawing, but in the meantime I’m still in this state of stupor in front of my pencils, which I don’t really like: either I have ideas and no energy to get started, or I draw without really wanting to and end up throwing everything away in the process.

I haven’t made any progress on the goals I set myself last year. That’s why I don’t like setting goals, other than as possible todo lists. I stopped watching Netflix and TV shows altogether. I just kept reading books (even for that I haven’t found much energy lately) and listening to my digital library. I haven’t processed all 25k tracks in my Apple Music library, but I listened several times to the same albums. I still use Beets to manage automatic tagging and renaming of all songs I grab from Youtube or Bandccamp from time to time. I haven’t downloaded any new songs since a few months now, I believe I’m almost done with my current library. Now it’s time to listen to every pieces of music again and again.

I gave Zed yet another try but I can’t stand by anything other than Neovim now. My config hasn’t changed since last year, which shows that I’ve finally managed to satisfy myself with a more or less stable and complete editor. More or less stable, I’m afraid, because I’ve still spent some time fixing deprecations and API changes since the switch to version 0.11. This is mostly related to LSP and diagnostic stuff, but at least it was doable in half an hour or so. Zed is really great, though. It is a well crafted and well thought piece of software and I believe it will become a strong contender to VS Code. I upgrade to Mathematica 14.2.1 a few months ago but I haven’t had time to look at the new features (other than the tabular data structure). I kept busy writing Python and Bash stuff for bioinformatics applications. I really want to go back to Lisp and Haskell side projects.

I don’t miss Linux, except for all those things I lost since my laptop crashed last year. I’ve recovered or rebuilt the essentials and set up a stronger backup system. Still, there’re a lot of photos missing from my digital library. My current setup is increasingly minimalist and consists of a combination of tmux and Neovim for everything related to text processing. Evrything is done in Apple Terminal. I have Neovim configured for all programming languages I use, for $\TeX$ and Markdown-related stuff (I still use Org for oldie stuff), and for Git as well. I have LSP and REPL configured as I want them to be, and I rely on the same 5 plugins (fzf-lua, nvim-treesitter, nvim-treesitter-textobjects, neogen, vim-table-mode). Other than Apple Terminak, I use Apple Mail and Safari all day (I still have a local copy of all my emails managed by mu and Neomutt), as well as NetNewsWire. Nothing fancy, really.

♪ Lady Gaga • Dance in the Dark

Micro posting in June

Mon, 30 Jun 2025 10:04:42 +0200

2025-06-08:

by failing to provide programmers with features they need to make the best use of modern hardware, programmers instead resort to enabling an option that is known to be blatantly unsafe. — Beware of fast-math

2025-06-08: Sad Sad Apple Tears: A thoughtful review and commentary on Apple’s evolution from a developer’s point of view. #apple
2025-06-10:

The fundamental difference between streaming and downloading is what your device does with those frames of video. — The Difference Between Downloading and Streaming

2025-06-10: I never could get used to Pandas data frame (while I’m able to use any Scheme or Lisp data frame frameworks, and Mathematica association lists or tabular data structure). Good points discussed here, although I’m a vanailla R user (no tidyverse): Why pandas feels clunky when coming from R. #python
2025-06-10: I was a happy user of newsboat until it stopped working on my MacBook, for whatever reason. It probably just works right now, but since I’m definitely out of sync with my RSS feeds, I was looking for a way to have it both on my old iPhone (this excludes de facto most modern readers, including NNW) and my MacBook. It looks like lire fits the bill, although there are occasional iCloud syncing issues (sync time and rate limit). Anyway, that’s still better than Reeder which does not sync to iCloud at all. #apple
2025-06-10: It’s not going to put me out of a job. This is from 2017 but still relevant as of today.

At least 75% of the actual job of programming is figuring out what a person means when they ask for something. This is not a problem that a machine can solve for you, for the same reason that it’s harder to build a robot that builds other robots than to just build a robot. Have you had words with Siri lately? Do you think Siri is in a good position to understand your intent, or it’s maybe just searching for command words in a limited lexicon? Do you really think it could find Henry’s stool? Or does it seem more likely that it would get confused and require a human… trained to translate… high-level human-language… into language suitable for… the machine?Why I’m not especially interested in machine learning

2025-06-10: When good pseudorandom numbers go bad: An interesting discussion about eigen decomposition with applications to sampling from multivariate distribution. #rstats
2025-06-10:

» brew info --cask container
==> container: 0.1.0
https://github.com/apple/container
Not installed
From: https://github.com/Homebrew/homebrew-cask/blob/HEAD/Casks/c/container.rb
==> Namecontainer
==> Description
Create and run Linux containers using lightweight virtual machines
==> Artifactscontainer-0.1.0-installer-signed.pkg (Pkg)
==> Analyticsinstall: 63 (30 days), 63 (90 days), 63 (365 days)

More info: Apple supercharges its tools and technologies for developers to foster creativity, innovation, and design. #apple
2025-06-12: LOL I just switched back to NetNewsWire as my default feed reader. Lire app uses too much RAM and does not sync correctly to iCLoud. #apple
2025-06-14: SIMD-friendly algorithms for substring searching.
2025-06-14: Setup an OpenBSD VM on macOS Using UTM. #apple #unix
2025-06-14: Why doesn’t Rust care more about compiler performance?. #rust
2025-06-18:

The distinction some make between prototyping and software development has vexed me my whole career. I have never seen a version 1.0 of a system that was acceptable. I find that every software group messes with the software up until the very time it’s deployed or delivered. — The Art of Lisp & Writing

2025-06-18: TIL about lstr, a fast and minimalist directory tree viewer, with interactive mode.
2025-06-23: /me is listening to “Shanghaï Gesture” by The Names
2025-06-25:
Summertime
2025-06-25:

Software development needs more simple joy, and I’ve found that creating toy programs is a great way to remember why I started working with computers again. — Writing Toy Software Is A Joy

2025-06-25: I have the following Jupyter kernels installed on my HD:

» jupyter kernelspec list
Available kernels:
python3 /Users/chl/.local/lib/jupyter/share/jupyter/kernels/python3
clojupyter /Users/chl/Library/Jupyter/kernels/clojupyter
common-lisp /Users/chl/Library/Jupyter/kernels/common-lisp
racket /Users/chl/Library/Jupyter/kernels/racket
wolfram /Users/chl/Library/Jupyter/kernels/wolframlanguage14.1

I renamed clojupyter snapshot to something more convenient to handle when calling jupyter console. I’m using OpenJDK 24.0.1 from Homebrew (which I symlinked in macOS default $JAVA_HOME, using sudo ln -sfn /opt/homebrew/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk) and I kept getting this annoying warning:

WARNING: A restricted method in java.lang.System has been calledWARNING: java.lang.System::load has been called by org.sqlite.SQLiteJDBCLoader in an unnamed module (file:/Users/chl/Library/Jupyter/kernels/clojupyter/clojupyter-standalone.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

To get ride of the warning, simply add --enable-native-access=ALL-UNNAMED in the kernel.json init file.
2025-06-25: TIL about this little trick: Restrict sftp with Linux user namespaces. #unix
2025-06-25: Fun with uv and PEP 723: I came across many similar posts about this new way to run Python scripts via uv. I switched to uv for managing python version and dependencies in virtual environments and I think it’s been one of the best addition to the language ecosystem in a while. #python
2025-06-26: I guess my Scheming with Vim will be easier to find on the www.

Micro posting in May

Sun, 08 Jun 2025 18:00:29 +0200

2025-05-12:
What happens when you neglected your feed reader for two months or so.
2025-05-12:

SBCL hash tables have been adaptive for almost a year now, gaining some speed in common cases, and robustness in others. — Adaptive hashing

2025-05-12: I already read this a while ago but in case I forgot to post it (and today I’m too lazy to search for it in the archives), here it is : Google Has Most of My Email Because It Has All of Yours. TL;DR “They have delivered more than a third of all the email I’ve replied to every year since 2006 and more than half since 2010."
2025-05-12: Interesting read from weeks ago (yeah, I’ve been pretty suiet lately): 100 years to solve an integral.
2025-05-12: 14 Advanced Python Features. Note that this covers features introduced in Python 3.10+. #python
2025-05-12: Graphics livecoding in Common Lisp. #lisp
2025-05-13:

Apparently, vibe coding is when you explain to your AI-based editor what the heck you wanted to do, except you’re much more into debugging code written for you. WTF it is calling vibe coding? I wouldn’t mind it being called AI coding, but vibe? — Am I becoming obsolete or just older?

2025-05-13:

Other languages may share some of these features, but Lisp has them all together. It is a language that was designed to be used as a tool for thinking about problems, and that is the fun part of programming. — Why I Program in Lisp

2025-05-13: TIL about LocalSend.
2025-05-13: Two Years of Rust.
2025-05-16: On macOS, relying on ZDOTDIR=$HOME/.config/zsh helps to declutter hidden files in $HOME. I initially put it in my $HOME/.zprofile, but you will need to use .zshenv (a symlink to .zprofile also works) in order to make your $PATH available in Neovim terminal. #vim
2025-05-16: The web today.

2025-05-16: A leap year check in three instructions. In Racket:

(define (leap-year? x) (<= (bitwise-and (* x 1073750999) 3221352463) 126976))

See also this followup post by John Cook.
2025-05-16: What’s New in Neovim 0.11. I upgraded via Homebrew some weeks ago and didn’t notice any breaking changes. However, after reading this article, I tried to setup my LSP configurations via the new facilities, and it works like a charm: root_dir has to be replaced with root_markers, and the filetypes parameters is equivalent to the basename (e.g., filetype.lua) that I put in my ftplugin directory. It is much easier to manage basic autocompletion via omnifunc. I don’t use virtual text and I don’t really care about virtual lines either, but it looks great. I revamped some of my custom mappings since Neovim now ships with default meningful keymaps for goto actions. #vim
2025-05-17: /me is listening to “Gravity (Still)” by Hooverphonic
2025-05-18:

[Numpy] took away indices and replaced them with broadcasting. And broadcasting cannot fill indices’ shoes. — I don’t like NumPy

2025-05-18: P hacking — Five ways it could happen to you.
2025-05-19: Concurrency in Haskell: Fast, Simple, Correct. #haskell
2025-05-19: Differentiable Programming from Scratch: A clear and concise exposition of DP applied to computer graphics.
2025-05-20: /me is listening to “Danny” by The Durutti Column
2025-05-20: TIL about basedpyright, which provides better defaults and additional capabilities compared to pyright, like inlay hints. #python
2025-05-20: A simple search engine from scratch. Looks interesting.
2025-05-21:

I am not saying that it’s not useful at all. If what you’re doing to begin with is copying and pasting code from e.g. Stack Overflow, then sure, because Google and other search engines have become so bad, AI can often provide you with relevant answers much faster than you shifting through page after page of useless Google junk. The reason why i don’t use AI or even code completion

2025-05-21: Perfect Random Floating-Point Numbers. TL;DR generate floating-point numbers with probabilities given by drawing a real number and then rounding to floating point.
2025-05-23:
Happy birthday to you dearest …

Micro posting in March

Sun, 30 Mar 2025 18:11:26 +0200

2025-03-04: A new version of Racket (8.16) has just been released. There’s now full support for Student t distribution. Easy upgrade (pending a minor issue with Homebrew installed libzmq):

brew upgrade
raco pkg migrate 8.15
rm -rf ~/Library/Racket/8.15
cp /opt/homebrew/Cellar/zeromq/4.3.5_1/lib/libzmq.5.dylib ~/Library/Racket/8.16/lib
raco iracket install

2025-03-09: Found this very old screenshots of podcasts I was listening to on iTunes.

2025-03-09: How many busy years before the poison?

2025-03-10: /me is listening to “Blood Moon” by Concrete Blonde
2025-03-10: I just spent half an hour decluterring my $HOME dotfiles and movinf everything I could to standard XDG config, data and state directories. Looks much cleaner now.

~
.CFUserTextEncoding .cargo/ .local/ .ssh/ Library/ backup/
.DS_Store .config/ .mail/ .stack/ Movies/ cwd/
.Renviron .ctags .npm/ .zprofile Music/ tmp/
.Trash/ .cups/ .nvm/ Desktop/ Pictures/
.cabal/ .deno/ .roswell/ Documents/ Public/
.cache/ .ghcup/ .rustup/ Downloads/ Sites/

The cornerstone for running zsh properly is to point ZDOTDIR to the relevant XDG data directory.

ZDOTDIR=$HOME/.config/zsh
for file in $HOME/.config/profile.d/*.sh; do
 . "$file"
done

2025-03-11: If you’re interested in Advent of Code challenge and Lisp, go take a look at Joe Marshall series on his blog. Last blog post here, with accompagnying GitHub repo. #lisp
2025-03-11: The data are on a 1-5 scale, the mean is 4.61, and the standard deviation is 1.64… What’s so wrong about that?: Nice read.

Micro posting in February

Sun, 02 Mar 2025 17:46:28 +0100

2025-02-11:

Boring tech behaves in predictable ways. It’s a well trodden path others have evaluated, optimised, troubleshooted, and understood. Using tech that has been subjected to all those people hours of use means you’re less likely to run into edge cases, unexpected behaviour, or attributes and features that lack documentation or community knowledge. — Boring tech is mature, not old

2025-02-11: Modern Statistics with R: From wrangling and exploring data to inference and predictive modelling. #rstats
2025-02-18: Almost two months with new MacBook and I’m back to my old habits using Tmux and Neomutt all day. Apple Mail is fine but I manage my e-mails 3 times faster with neomutt. Likewise, Apple Terminal has been enhanced over the years but nothing beats Tmux when it comes to manage long-running sessions, multiple panes, and paste buffers. That being said I’m happy everything is kept in sync and I can switch to whichever software to suit my mood. #apple
2025-02-18: Lisp as a System Service, or how to use UNIX domain sockets for managing SBCL on a remote machine. #lisp
2025-02-18: Visual explanations of mathematics: Oldies but goodies. #math
2025-02-24:

(…) it’s a good idea to first understand how the system was designed to be used before decrying it as ill-conceived or making it something it isn’t using third-party window managers, app switchers, and so on. — macOS Tips

2025-02-24: More macOS Tips & Tricks. #apple
2025-02-24: 0xProto, with its subtle ligatures, looks great. (via Leah Neukirchen)

February in review

Tue, 25 Feb 2025 20:31:04 +0100

Ragnar Groot Koerkamp is posting interesting and detailed overview of common algorithms in bioinformatics, specifically minimizers and pairwise alignments, as well as Rust benchmarks, e.g. on the one billion row Challenge.¹
Sliding window technique from a functional point of view, by Brent Yorgey.
A Tetris implementation in PostScript, because why not.
Lisp is not a realization of the Lambda Calculus. I came across this blog post after reading Lambda Calculus and Lisp, part 1. A good overview of lambda’s from an historical perspective as far as I can tell. Other stuff read almost at the same time: Lisp without cons cells.
Do You Really Want to be Doing This When You’re 50?. Even if I know the weblog is dead now, I can’t help coming back to it from time to time, or when it pops up on Hacker News.
These years in Common Lisp: 2023-2024 in review: the annual review by Vincent Vindarel.
A year of uv: pros, cons, and should you migrate: I’m a very happy user too, it helps installing and managing with virtual environments and pip recent versions of Python on quite old systems (e.g. Ubuntu 18.04). See “A good Python project manager, therefore, should have the following properties”, points 2, 3 and 4.
Simulate a Derived Distribution: I came across this page while looking for builtin way to simulate random draws from a mixture of heavy-tailed Gaussian distribution in Mathematica. The term “derived” sounds great, but confusing. Just a sidenote.
Visualizing all books of the world in ISBN-Space. Lot of interesting technical stuff going on.
The Tensor Cookbook: Not as much visual pictures I’d expected and strange $\TeX$ notation (but congrats for the TikZ or whatever technique used in the PDF handout).

♪ Memory Cassette • Last One Awake

See also Bogdan Popa’s take in Racket. ↩︎

Micro posting in January

Thu, 30 Jan 2025 20:07:23 +0100

2025-01-03: Neomutt tip: To escape space in folder name (e.g., iCloud Sent or Deleted Messages), you can use either \ when defining your inboxes in your muttrc or ^V for your bindings. E.g.:

mailboxes =icloud/Drafts =icloud/Sent\ Messages =icloud/Deleted\ Messages =icloud/Archive
macro index,pager gs "<change-folder>+icloud/Sent^V Messages<return>" "go to sent messages"

2025-01-11: After today’s quick testing, it looks like Wolfram Jupyter kernel is working as expected under Quarto (see this blog post for a previous attempt with Stata). The Racket kernel works as well (minor caveat, you’ll probably have to symlink your Homebrewed libzmq).
2025-01-12:

I can’t use Emacs, so it’s got to be usable in Vim. — Why I Chose Common Lisp

2025-01-12: I’ve been listening to some music on Apple Music app for one and half a hour and the battery still tops at its max. It’s incredible the level of micro-optimization that must be behind all this. Likewise, I can work unplugged for all day long, I will have a fair amount of time left for the day after. #apple
2025-01-15: /me is listening to “Lands End” by Siouxie and the Banshees
2025-01-15: /me is listening to “Other Side of Love” by Sixth June
2025-01-15:

I would not accept this Dock placement blasphemy from anyone else, but from Thomson, well, OK. (The correct location is on the right. Left, we’ll let you argue for. But definitely not on the bottom.) — James Thomson on the origins of the macOS Dock

2025-01-15: I migrated a bunch of stuff to keep Neomutt and Apple Mail in sync, and I still use mu to manage complex search in my maildir. Here are the results of the last mu index which accounts for the integration of listservers and digests directly into proper iCloud folders, in no more than 5s:

» mu index
indexing maildir /Users/chl/.mail -> store /Users/chl/.cache/mu/xapian
| indexing messages; checked: 14443; updated/new: 3103; cleaned-up: 4144

2025-01-15: Related to the previous micro-post, see Proper Python Project Structure 2024 by the same author, which argues against the use of requirements.txt in favor of Poetry. I used Poetry for a work project but I’m now betting on uv. See also Uv has a killer feature you should know about. #python
2025-01-15: FFmpeg By Example is a documentation website to showcase all the unique and different ways to use FFmpeg.
2025-01-15: How to C in 2016. See also some rejoinder like Some notes C in 2016 and A critique of “How to C in 2016”. #clang
2025-01-22:

Discord, overwhelmingly. Every IRC community I used to be a part of even a year ago has made the jump. It’s just more convenient for all concerned- the only holdouts are people clinging to their CLIs and people objecting for ideological reasons. — LocutusOfBorges on HN

2025-01-22:

I don’t understand the desire to shove everything into a web browser. Other than the fact that it is how tech startups extract money from venture capitalists, spy on your user base, and lock out interoperability I don’t see why a web browser is a better place to implement most things and yet, here we are. — Discord in my IRC client

2025-01-24: /me is listening to “Beware of Darkness” by Concrete Blonde
2025-01-24:

The lie-to-children here is that we pretend the do block is magical and that when it executes, it also executes side effects of functions called in it. This mental model will take the beginner a long way, but at some point, one will want to break free of it. That is when Haskell starts to really shine as a procedural language. — Haskell: A Great Procedural Language

2025-01-24: I was surprised to find R in this list of Common Unix Programs. #unix
2025-01-24: So be it, everything is an expression. #wolfram
2025-01-24: What are the worst (commonly adopted) ideas/principles in statistics?. So long CV, but I’m glad to see that the site can still generate very interesting discussions.
2025-01-25:
Listening to old music that changed my musical world back then…

2025-01-29:

Less is more. Avoid grand designs. Keep the codebase as small as possible. — The ten project management commandments

2025-01-29:

To put it briefly, human experience is characterized by Dasein (a technical term for Heidegger that can be translated literally as “being-there” or “being-here”), which tends to overcome spatial distance through meaningful action or symbolic understanding, to bring even the world close to itself, a tendency that Heidegger calls de-severance. With various modern technologies, however, this tendency of Dasein is made more complicated. — A Phenomenology of Spotify and Vinyl

2025-01-29: RHO: A Common Lisp library implementing R ideas. #lisp

Introduction to Statistics with Mathematica

Wed, 29 Jan 2025 20:43:59 +0100

I just finished reading the Notebook version of Koppler’s Introduction to Statistics with Mathematica. If you’re new to Mathematica and/or statistics, chapters 3 to 5 will teach you the basic building blocks and data structures useful for statistical analysis. Even if you are familiar with lists and nested lists, Datasets provide useful abstraction for representing statistical data and they come with great utilities, e.g. GroupBy.

This is an introductory course on descriptive and inferential statistical. There’s no linear algebra, no digressions on mathematical or computational statistics but it does the job at explaining basic concepts to newcomers, in particular the sampling distribution of a test statistic, standard error and confidence intervals, and how to interpret null hypothesis testing. I yet have to explore how far Mathematica goes with Bayesian, non-parametric and robust statistics.

Regarding the basic material itself, you will learn about List comprehension using Table and to deal with lists and nested lists, which can be flattened or viewed in a matrix form. Everything is an expression and the basic object of interaction is a list, which must bring back fond memories for Lisp regulars. I would say it is even more striking than with R. As said above, Dataset extends the capabilities offered by nested lists to associations, which is the term used to denote a specific hash table which links elements of nested lists with distinct keys.

You can view a Dataset as is in a notebook – it’s pretty well displayed and useable, btw, and you can import rectangular datasets (e.g., tab-delimited or CSV data files) into a Dataset using SemanticImport rather than the omnipotent Import function. One thing that I struggled with when I started using Mathematica two years ago as a replacement to Stata was how to alternate between the Dataset and List views. It turns out that something like Normal[dataset[All, "column"]], where column is the variable of interest, will turn a Dataset column to a regular list. Another thing I really like is that you can address any value in a Dataset like you would do in R, with All and ;; as secret keywords you must retain. Like in R, you can address variable (column) by name or index, which was not easy in Stata (where almost every entry is a variable).

I like the fact there’s an implicit apply function for dataset, e.g. dataset[Mean, "column"]. Should you want the whole split-apply-combine strategy, combine Select and GroupBy.

Mathematica already provides a lot of builtin functions for data visualization (with great defaults!) and summary statistics. There’s a lot more to see on Wolfram Function Repository. For casual statistical tests, you will need the HypothesisTesting and ANOVA package, but they are already available in the base installation of Mathematica.

♪ Tindersticks • Drunk Tank

Common Lisp tooling in 2025

Wed, 22 Jan 2025 21:00:57 +0100

Transitioning to macOS after 4 years of Linux happened to be smoother than I anticipated. Most of the tools I relied on previously were already right there in the OS (e.g. Terminal or Safari), and the rest is easily available via Homebrew. That being said, I decided to keep the bare minimum of what I need to get my work done, and to focus on PLs I want to explore more practically this year: Common Lisp, Racket, Mathematica and Haskell.

Regarding Racket and Haskell, everything is already configured with appropriate linters and fixers in my Neovim config. The LSP do the job I expect from them, that is they provide hover information, go to definition and references, and warning/error appear as expected in the Quickfix. Code actions are available in the case of Haskell.

I used to use various strategies for dealing with Lisp in the past. The best solution is probably to rely on slimv, but I wanted something simpler, especially given that I can fire a REPL myself and interact with it without relying on an external plugin.¹

The idea is that all functionality available in any editor or plugin is also available in the “vanilla” REPL. — Developing Common Lisp using Vim with tmux or ConEMU

I use msotly the same apparoach as Daniel Jay Haskin except that I have my own implementation of “send line or selection to the REPL”, a custom keywordprg to query the Hyperspec online and display it in w3m, an omnicomplete function taken from visper, and a folding function grabbed from Steve Losh’s dot files. In addition to Dan’s compiler, the following two keybindings take care of most of the rest:

nnoremap <buffer> gs :10 split term://ros -Q -- %<cr>:startinsert<cr>
nnoremap <buffer> <leader>! :sp term://ros run --eval '(ql:quickload :swank)' --eval '(swank:create-server :dont-close t)' \| SendHere<cr><C-w><C-w>

As can be seen I use Roswell to launch SBCL and to manage Quicklisp packages. Ultimately, I would like to have an automated way of generating binary executables or Docker images, but for the time being I’m mostly interested in scripting at large. I encountered some problems when installing the GSLL and common-lisp-jupyter; there appears to be some issues with the current CFFI bindings, but this patch fixed the issue. I was able to install magicl without any issue, as well as various high-level utilities, like serapeum. No luck with cl-ana, or any other packages from Gary Hollis since they all depend on hdf5 and FFi which grovel skates endlessly. This may solve the issue but I’m not willing to try all that (either downgrading hdf5 or using a staled branch of cl-ana and a CFII fork).

I mentioned common-lisp-jupyter since I tried to use Quarto with CL Jupyter kernel, and it worked like a charm. Same for the Racket kernel, by the way.

♪ Broadcast • Long Was the Year

I’m using a pretty slick Neovim config with only 5 plugins, including 2 opt plugins, all managed via the builtin plugin manager. ↩︎

lost+found 2024

Wed, 01 Jan 2025 20:51:15 +0100

Customizing Mpv (2024-08-21)

Random links to elaborate on how I built my own Mpv config.

On trend tests (2024-01-25)

A few weeks ago I discussed the Cochran-Mantel-Haenszel test. Another widely used test to assess linear trend in two-way tables is the Mantel-Haenszel trend test. There’s a very interesting discussion about trend tests on Stata FAQs, by William Sribney from StataCorp.

Approximating sine (2024-01-25)

A few weeks ago I came across this blog post: Approximating sin(x).

It functions very similarly to the taylor polynomial, except the constants have been replaced to achieve greather accuracy, (…)

Except this is not based on a Taylor expansion, but it rather relies on Chebyshev polynomials.

In Polynomial Approximations to Elementary Functions

Elementary Functions and Approximate Computing, see also Computing Machine-Efficient Polynomial Approximations

Remez algorithm, see Dandelion: Certified Approximations of Elementary Functions

Generating all or next permutation (2024-01-12)

In a previous post, I mention a way to suffle labels of a two-group dataset in order to carry out a permutation test. Usually, it’s more efficient to randomize the highest level unit of an experimental design, i.e. rather than shuffling observations use the group level.

A recursive solution to the problem of generating all permutation of a given list or array is as follows:

If the length of Permutation equals the length of Array
 Permutation Found.
Else
 For each position ‘p’ from 0 till the length of Array
 If element at position ‘p’ has been taken
 Continue to check for next position.
 Mark position ‘p’ as Taken.
 Append the element at position ‘p’ to the Permutation.
 Generate_Permutation ( Permutation, Array, Positions )
 Remove the element at position ‘p’ from the Permutation.
 Mark position ‘p’ as Available.

other algorithms
heap
domain-specific applications https://rdrr.io/cran/permute/
Sedgewick and Wayne’s algorithm https://introcs.cs.princeton.edu/java/23recursion/Permutations.java.html
https://en.wikipedia.org/wiki/Steinhaus%E2%80%93Johnson%E2%80%93Trotter_algorithm
base case
Racket
cl-permutation
Rosetta code
numpy and itertools.permutations(), https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.permutation.html#numpy.random.Generator.permutation
next permutation https://www.nayuki.io/page/next-lexicographical-permutation-algorithm
stream, generator

Phillip Good (2004) Permutation, Parametric, and Bootstrap Tests of Hypotheses, New York: Springer-Verlag

♪ Sixth June • Night Before

Micro posting in December

Sat, 28 Dec 2024 15:13:03 +0100

2024-12-04: /me is listening to “I Can’t Be” by Sally Dige
2024-12-04: Musing with three types of backup systems on my old Macbook: Arq, Borg and Kopia. I was able to mount every backup folder correctly. Now I just need to put the years end-to-end!
2024-12-21: I got a new MacBook Pro. It’s gorgeous!

2024-12-21: Tip of the day: To rename a bunch of photos on your HD according to the original date, the following works well: exiftool '-filename<CreateDate' -d %Y-%m-%d-%H%M%S%%-c.%%le -ext jpg -r ..
2024-12-26: /me is listening to “Flesh Dance” by Gothic Tropic
2024-12-27: I switched my default mail provider to iCloud, instead of this domain name. Back in time, I guess. This way I can manage my emails using Neomutt or Apple Mail (via IMAP), depending on my mood, and everything keeps sync’ed. I needed to convert my archives folder from Maildir to Mbox but Neomutt failed me when I used the following command:

neomutt -f ~/.mail/archives -e 'set mbox_type=mbox; set confirmcreate=no; set delete=no; push "T.*<enter>;s/Users/chl/tmp/archives.mbox<enter><quit>"'

I believe this doesn’t handle multiple aliases for sender correctly. However, this Python module worked like a charm and I was able to get 10k messages with correct threading in no time at all.

December in review

Mon, 23 Dec 2024 19:07:58 +0100

Dividing unsigned 8-bit numbers: Interesting technical articles with lot of benchmarks on SIMD-based floating point (long) division. The takeaway message is that AVX-512 implementation performs best over all other competitors.
Besides lot of stuff to read from his tech blog, I particularly like Daniel Doubrovkine’s art gallery, e.g. the Large Sanguine Drawings.
Should You Ditch Spark for DuckDb or Polars?: I’ve used DuckDB in the past, and I was impressed by its performance and how easy it was to install extensions. Spark, uh, it was a long time ago and I no longer have any calculations that require this type of distributed calculation. I’m sticking with the datatable module whenever I can, but I don’t that much stuff in Python these days. From what I read, the author suggests that Polars offers very good performance when it comes to Small Result Aggregation but overall DuckDB is a winner in most benchmarks. This is not surprising, after all, since Polars primarily targets data structures that are best represented as data frames.
Plotting Prime Numbers: Not something new to me since I already came across a post like this but this was refreshing.
Sandboxing on macOS: And to think I was reading this kind of article when my Linux laptop hadn’t yet crashed! Anyway, I was just curious to understood how it all works under the hood. It is noteworthy that OpenBSD (pledge and unveil) and Qubes OS (domains and app qubes) offer equivalent gatekeeping strategies.
Derivative AT a Discontinuity: Or how to differentiate the Heaviside function using actual functions rather than distributions.
Using Pandoc and Typst to Produce PDFs: I always enjoy learning how folks think about their workflow, and the tools they use. I learned about Typst one or two years ago maybe, but I never gave it a try. I’m stuck with my arcaned Markown or Org -> $\TeX$ workflows, usually processed via Emacs¹ or Pandoc.
How Much Memory Do You Need in 2024 to Run 1 Million Concurrent Tasks?: A comparison of coroutine vs. multiple threads (older post).
Minecraft Enchantment Ordering Tool: Because why not.
A joke in approximating numbers raised to irrational powers: I let you dive into the “monster” function ;-)

I was really excited to also find a document that I had created in 1991 or so listing the games and “creator codes” that I had used in these games. Creator codes were like a very crude “bundle identifier” back in the pre-OS-X era of the Macintosh. They were supposed to be unique from all other application creator codes. The document is interesting to me today because in it I had cataloged the projects that I had worked on up to that point. — Move Fast and Abandon Things

Text based email workflow on macOS: Neomutt with a Mu backend for finding contact and fast indexing/searching facilities is great, trust me.

♪ Arcade Fire • Vampire/Forest Fire

Okay, I gave up on Emacs, but I still have dozens of Makefile that rely on it. ↩︎

Hello macOS again

Sun, 22 Dec 2024 20:11:47 +0100

I thought I would write a blog post a month ago after my laptop blew up, then when I spent a week reconfiguring my old MacBook 12". I gave up, it was certainly better to let things go at that time. Now I got a new MacBook, the top of the line, and it’s refreshing. First, I never had a laptop as powerful as this one. Still, it’s a little annoying since I lost a lot of personal stuff.

Ground 0

I came back at home after my day’s work, and when I plugged the power supply back onto my laptop (Dell Latitude 7310) the screen remain blank, I mean nothing happened. It occured in the past, so I tried to hard reboot the OS to no avail. I then dismantled the computer and realized that the power supply unit was out of order. I removed the SSD but could never mount it for reading. After a few days, I had to face the fact that there was nothing left to save and that I could go back to my July general backup. Sadly, I just finished consolidating my music library with Beets and I was going to backup everything right away. That’s life, I guess. I also lost 1 year or so of pictures taken with my ageing iPhone, a collection of desktop background I made recently, dozens of shell scripts I wrote during the quiet Summer days to automate things here and there, as well as my Newboat archives ans 4 months worth of emails – which I don’t really care about but still… Since everything I do at work is synced on Github or local servers, nothing was lost on this side.

Round 1

I came back to my beloved MacBook 12". I haven’t used it in years, except to backup my text messages. It was hardly usable at first, and I had to reinstall all the stuff I removed since 2018. I reinstalled Homebrew, and then I realized that there was no longer any bottle for the software I was asking for, which means I needed to recompile everything from scratch. That’s fine, I already did this for a living in the past. It took me a week, though. The hell of dependencies on top of a slow processor, if you see what I mean. Finally, I made it and I got a working workstation which helped me to continue my activities at work. Two things came to my mind after the fact: I was glad to get my MacBook back with its retina display, and listening to my playlists on this device was much better compared to the Dell laptop. I don’t know how I put up with that one for so long. Anyway, that was funny as I happened to recompile Neomutt and Cmus, with all their dependencies, from scratch. I gave up reagrding Newsboat as it required so much dependencies. Of interest, everything Rust-related worked with no problem at all, even on Mojave which is now 6 years old and deprecated all over the place, including Homebrew.

Round 2

My benevolent colleagues helped me get a new computer when I was ready to buy a macbook myself. Instead of a MacBook Air, I ended up with a MacBook Pro M4 14". It’s a beast without further ado. I fired up Homebrew and installed a bunch of utilities, in addition to rustup, ghcup, uv, and roswell for which I used the provided install shell scripts. I now let Homebrew manage whole applications (aka “Cask”) for me, which means I do not install things by myself: either they are on the AppStore (pre paid or newly released), or as Homebrew formula or casks. My workflow hasn’t change. I still use Neovim as my daily driver, Neomutt for managing my emails, and Newsboat for RSS. I just replaced ncmcpp with Apple Music as I find it more convenient. As I once said,

These days, I’m happy with a few tools that allow me to do 90% of my work in an efficient manner.

Well, it’s time to go back to the future for now.

♪ The Other Two • Movin’ On

Micro posting in November

Mon, 02 Dec 2024 21:24:37 +0100

2024-11-11: /me is listening to “Love Hides” by The Doors
2024-11-11: Apparently, they will soon get a new home!

2024-11-11: I’m pretty happy with new management system for my audio file. Thanks to Beets I now have access to the whole power of command line utilities. Here’s a brief overview of the top genres in my library:

» beet summarize | head -n 12
genre | count
---------------------- | -----
Rock | 3045
Post-Punk | 2754
Jazz | 2509
Alternative | 1212
Electronic | 948
Indie Rock | 838
Pop | 817
Classic Rock | 679
Alternative Rock | 497
Indie Pop | 424

How many total tracks having a genre tag, excluding tracks from playlists which contain, following poor design choice, duplicate records?

» beet summarize ^context:playlist | head -n 12 | cut -f2 -d\| | tail -n +3 | paste -sd+ | bc
10603

And so on… Wanna looks at the tlistening stats for the current song?

» beet ls -f '$artist ($album): $title ($length) [$genre, $play_count, $last_played]' $(mpc current | sed s/-//)
Diana Krall (The Girl in the Other Room): Narrow Daylight (3:31) [Contemporary Jazz, 12, 2024-11-11 20:38:09]

2024-11-11: The Beauty of $\LaTeX$: Some of the crap that has kept me from using MS Word for years. Now I’m too old and remain a happy $\TeX$er for the days ahead. Fun thing: we did the same MSc (which was called a DEA at that time in France) one year apart!
2024-11-25: I’ve spent a few more or less pleasant days, long ones in any case, reconfiguring my old 12-inch macbook. Guess what about my Dell laptop and my fresh configuration of Ubuntu 24.04 LTS: the power supply blew up and the SSD along with it. Now everything seems workable (at the very best, for a 6-year old OS). More to come in a blog post later.
2024-11-26: I’m not in the mood of writing a full blog post to summarize my recent comeback to OS X. It’s still OS X on my machine, but I know it evolved as something else since 2018. I don’t know if I’m ready to buy a new Macbook with the accompagnying OS. I liked and I still like Mojave so much. Yet it was pretty painful to resurrect this laptop: Homebrew has no “bottles” for oldy software, which means I had to recompile everything from scratch for 60% of the software I reinstalled. I got into trouble with GPG, GitHub https auth, zsh misbehaving on OS X for no reason, minor annoyances with the Linux -> OS X switch, and so on. Finally, I’m close to the end of the migration process. It took me 5 full days! And I had to somehow relearn macOS shorcuts. Muscle memory helped in most cases, but still… So long, Apple #apple
2024-11-26: TIL that TERM=xtermm on Darwin behaves like TERM=xterm-mono on Linux. That was helpful to force Neomutt or Cmus to use mono color by default. Moreover, if you use TERM=vt100 for Neomutt, it behaves more smoothly when fetching new mail with mbsync. #apple
2024-11-28: Currently rebuilding my iTunes library. Back in 2016 it wasn’t the time to switch to Apple Music and TV. I’m on an unsupported laptop and OS right now, but it feels so good. Yet, congrats to Apple because I can retrieve all my data in just two or three clicks. #apple
2024-11-29: With that damn machine crash last week, I’m in the process of retagging my music library (after re-downloading some albums from YT for the second time), and soon everything should be ready to join my iTunes library, neglected since 2016. The last title I bought from Apple was the Glenn Gould album, BTW.

Back to Beets

Sun, 10 Nov 2024 20:36:11 +0100

It’s a been a few weeks with ncmpcpp as my daily driver to the mpd daemon. So far so good, it’s lightweight and it allows you to make very precise settings. Once I learned how to remove all the fancy colors (this is simply colors_enabled = no), I think I got a very decent music player and I no longer miss Cmus since it covers the same set of features plus many additional options (tag editor, display information on tracks or artists, fetching lyrics, among others).

Now I wanted to cleanup my database, and soon after I started using mpd I configured Beets to import and auto-tag my whole music directory. Other authors already described their workflow, mostly as a way to bypass Apple Music (aka iTunes). Last time I used Beets I was left with a mixed impression. This time I took some time to think about it. First I did not import everything (18k tracks) at once and I disabled the auto-tagging and moving option. I simply gathered all my albums in a dedicated folder, and likewise I kept separate folders of playlists in another parent directory. In the end I only have two directories (collection and playlists which hold all my music). Incoming music is stored in a temporary folder which is ignored by mpd (thanks to an .mpdignore file). Playlists were imported as is (they are supposed to be well crafted since they dated back from my time with Apple and iTunes). Albums were also imported as is, and I tagged them later in sort of a batch mode, i.e. albums I ripped from my own CDs or albums I fetched from the interwebz. I assumed that albums downloaded from Bandcamp were already well annotated. It took me a week or so to get a clean beet update.

[2025-01-12]
See also Enjoying music curation again to learn more about others’ config (I learned about the scrub plugin, for instance.

MusicBrainz has accurate metadata for well over 95% of the albums that I’ve passed through beet import, meaning that normalizing and checking everything is as simple as pressing “Enter” a handful of times.

I learned a few things along the way. First, it’s good to add Discogs as a source for auto-tagging in addition to MusicBrainz and Bandcamp. In case you already have cover art, disable auto-fetching album art otherwise you’re likely to end up with a cover.jpg and cover.1.jpg (on Linux) file for each album. I had to cleanup the SQLite database manually. Take your time to review the suggested options when album matches aren’t very high (< 50%); sometimes it’s good time to look for a Discogs ID rather than trust the low confidence matches (especially for special or deluxe edition, particular rare LP or collaborative work). Several pass of beet update may be necessary in order to get a cleaned database. A final note: you may notice that audio tags seem to not be updated right after running Beets (e.g., if you use id3 like me) but Beets relies on ID3v2.4. I was disappointed for a long seeing the correct annotation in ncmpcpp but not when querying audio file properties using id3info.

I gathered a fair amount of live, EPs and LPs from the web archives a few months ago. Most of the records are of acceptable quality (usually 128 Kbps) bur for live events don’t except more than bad broadcast FM recordings! We all miss Napster, right? However, despite my aversion to Google products, I realized that YouTube Music is quite a decent option to complete missing tracks from albums you like or to fetch whole albums. Things to keep in mind is that oftentimes album are not complete or need you to be logged in to download specific tracks. Finally, all tracks need to be retagged (e.g., default track number is 63 for all tracks related to the same album if I trust Beets). I use yt-dlp with the following settings (as a Bash script) to fetch music from YT:

for i in "$@"; do
 yt-dlp -f bestaudio \
 -x --audio-format mp3 \
 --audio-quality 320k \
 --embed-thumbnail \
 --add-metadata \
 --postprocessor-args "-id3v2_version 3" \
 --output "$HOME/Media/Music/tmp/%(playlist_title)s/%(playlist_index)s - %(title)s.%(ext)s" \
 "$i"
done

[2025-01-12]
Slightly updated yt-dlp config follows: (we barely need to require 320k when quality is around 256k or less on YT.)

# $HOME/.config/yt-dlp/config
-o '/Users/chl/Music/downloads/%(album_artist)s/%(album)s/%(playlist_index)s - %(title)s.%(ext)s'
--no-mtime
-f 'ba*'
-x --audio-format mp3 --audio-quality 0
--embed-thumbnail
--embed-metadata
--parse-metadata '%(album_artist,channel,creator,artist|Unknown)s:%(album_artist)s'

Then put your URLs in a dedicated text file and just call Bash:

while read url; do yt-dlp "$url"; done <youtube

I then import YT audio tracks with Beets to get automagically tagging and renaming. It is also able to automagically fetch cover art, or you may disable it and download an image file yourself. I must admit YT music is amazing. I was able to find quite old OSTs like the one for The Crow with Brandon Lee, as well as Delux edition for some album I own.

Even after you get a clean output from beet update you may still want to manually curate some artifacts like incorrect dates (original date are usually fine, but dates are often wrong – for instance, because they are for a deluxe or remix) or missing genres. I used the lastgenre plugin to get a more consistent classification (from lastfm), and I manually edited the 1000 tracks that were still missing genre information. I still have to fix the issue with improper tagging of date. Note that beet summarize and beet list are very handy to catch incorrect dates and find relevant tracks.

Along the way, I removed the mscout service I set up earlier since I’m now using the mpdstats plugin. It will be more convenient to store the play counts in the Beets database itself rather than mpd sticker databasE. Initially I started writing a shell script to retrieve my old listening statistics (which I first converted from Cmus to mscout) and integrate them in the Beets database, until I realized that Beets completely changed hundred of pathnames during importation. I’m too lazy to write a script relying on a hash for each track, so I will update the play count manually (this is already done for all my playlists, but this was the easy part!).

♪ Blue Cranes • I Forgive You

Micro posting in October

Fri, 01 Nov 2024 17:36:01 +0100

2024-10-03:

In fact, you rarely have anything to do with the operating system on a PC desktop or laptop, you mostly only deal with desktop applications - and they can suck equally bad regardless of operating system. — There is only one reason why Microsoft Windows is the dominating operating system on the PC desktop

2024-10-03:

The nice thing about boringness (so constrained) is that the capabilities of these things are well understood. But more importantly, their failure modes are well understood. — Choose Boring Technology

2024-10-03: I believe the Haskell LSP already provides that, but in case you don’t use it here are two blog posts that explain how to integrate ghcid into your (Neo)vim workflow: How I integrate ghcid with vim/neovim, Haskell dev workflow with ghcid and neovim. #haskell
2024-10-03: 100 Exercises To Learn Rust. #rust
2024-10-03: Computing the Normal Distribution Function. #python
2024-10-03: Everything I Know About The Fast Inverse Square Root Algorithm. Dealing with floating point math from Quake 3 to the present day.
2024-10-03: Failing at Combinatorics with Haskell. #haskell
2024-10-03: ♪ Mini Trees · Carrying On
2024-10-21: /me is listening to “Back In Black” by AC/DC
2024-10-21: I guess all is working now ;-) I’ve converted my automated script that shows current track on the micro-blog (for casual IRC users), I’ll just need to figure out how to manage tagging and playlist management under ncmpcpp.
2024-10-21: Trying out mpd in combination with ncmpcpp as a replacement for my long-time friendly TUI Cmus. I"m still in the process of discovering the functionalities. Everything seems okay on the mpd side – I even got MPRIS support after reading the Arch wiki (this Go solution didn’t work for me). I’m just looking for a direct way to listen to my music library without resorting on virtual playlist in ncmpcpp.
2024-10-22: /me is listening to “Amor Fati” by Washed Out
2024-10-25: /me is listening to “Man of Anatomy” by Tom Hickox
2024-10-27: I was going to spent some times trying to output the sound of my built-in speakers to my old Apple TV which is connected via a DAC to my wi-fi system, until I realized it is as simple as loading Pipewire RAOP module, pactl load-module module-raop-discover. Then, launch pavucontrol and switch to your Airplay device.

The Airplay system is now listed in the list of available devices. We can even make it the default output audio channel:

Now, since Pipewire supports multiple streaming output devices, much like mpd, I just need to configure the audio source in their config files to get everything right for the next time.
2024-10-29: Ncmpcpp with no color enabled. Much better for my eyes!

2024-10-30: /me is listening to “Darkening of the Light” by Concrete Blonde

♪ XXX • XXX

mpd and ncmpcpp

Tue, 22 Oct 2024 10:55:14 +0200

I recently gave mpd a try for my music setup. I’ve been a long time user of Cmus which served me well. Before upgrading to Ubuntu 24.04, I used to use a fork which allow to show cover art in the notification menu bar (via MPRIS and D-BUS).¹ I’ve tried several other alternative players, which all support displaying cover art by the way. Audacious and Gapless (formerly, G4Music) are really great with their minimalistic yet efficient UIs, but I guess I just want to stick to TUIs for the moment. Enter ncmpcpp which serves as a frontend to a running mpd server.

Clearly, mpd is a different beast. While I always appreciated the simplicity and usability of Cmus, and its speed as well, I must say that using a daemon to interact with my music library is something new to me. For instance, when I reboot my machine this morning, I was surprised to hear the song I started listening yesterday evening before I shutdown the machine. There’s an option whihc I did not set properly, restore_paused "yes", meaning mpd started playing its queue as soon as it was up and running. The good thing is that it is easy to manage the mpd daemon using a user unit via systemctl. MPd indexed my whole library (> 15k tracks) in a very decent amount of time. I tagged it using beats a long time ago, and I added proper tags manually since then, as well as many cover art, but I should definitively do this again, and clean up the duplicates thare spread across full album or disc sets and individual tracks in separate playlists.

I use mpc to interact with mpd for low-level operations (toggle play/pause via system hotkey, display stats, etc.). It is a command-line application which It would be great if I could integrate the 3 years of listening data gathered through Cmus someday. For the moment, it’s just the beginning, and the stats are pretty dumb. And I still don’t know if I will keep the same library structure which consists basically in a bunch of folder with artist/album and another set of folders with separate playlists, which seems silly after all although it was quite handy with Cmus.²

» mpc stats
Artists: 3410
Albums: 4513
Songs: 18532

Play Time: 0 days, 1:38:32
Uptime: 0 days, 6:50:44
DB Updated: Mon Oct 21 18:02:28 2024
DB Play Time: 55 days, 15:51:08

It is also possible to manage a “sticker” database, which is composed of added features specific to songs, albums or playlists, like play counts, ratings, comments and the like. This is specific to mpd and not all clients support it. However, I can imagine that populating a playCount field with data from Cmus could be interesting, in addition to an hourly shell script gathering information from mpg log files for current events.

[2024-10-24]
I successfully imported all my play counts from Cmus in the mpd sticker database. First, I started recording new listening statistics with mscout. It works great, and it is simple to setup as a user unit to be managed by systemd. This program has options to get statistics from individual songs as well as playlists. One minor annoyance was that it fired up a new notification for every song, but this can easily be disabled in the source code by commenting out the relevant section of code. It can also export or import statistics from or to the sticker database. Therefore I hacked one of psyomn’s gists to extract relevant information from Cmus cache (full pathname and play counts for all the songs in my library) and reformat them according to the format expected by mscout. It worked perfectly fine!

The first time I launched ncmpcpp, I was a bit lost. With the default config, you end up on an empty window, which stands for the current playlist. Here, a playlist is just a set of tracks you selected to play in a row, and not a pre-recorded list of tracks in an m3u file. Like in Cmus, there are different windows or views that you can navigate using the {1-8,=} keys (if you compiled ncmpcpp with all available options). The browser (2) and media library (4) are the ones you want to explore first: the browser summarize your media library as seen on you hard drive, while the media library is much like the first view in Cmus, a tree view of your artist/album collection. At first, I used the 4th view to select albums to play. Once you click Enter, the whole album is added to the current playlist (1) and the first track starts playing. It was cumbersome as I keep adding different albums along the way, and the playlist kept growing. Fortunately, there’s the C shortcut which allows to clean the current playlist. Then I learned that we can add tracks directly from the browser view (2) by pressing Space (instead of Enter). Finally, I learned that we can just populate $HOME/.config/mpd/playlists with m3u files, and I was good to go. I now feel at home since I got the same experience as in Cmus. Easy peasy.

In addition, things you get for free (no plugin, no additional configuration) with ncmpcpp: a playlist and tag editor, an FFT-based spectrum visualizer among other displays), the ability to stream live music from radio stations, information fetched from last.fm for artists or albums, cover art and lyrics support, dozens of keybindings to toggle cross-fading, replay gain, forward or backward seek, etc. Additionally, mpd provides logging facilities which allow to track your listening experience along the day – nothing hardcoded in a binary database like is the case for Cmus and its track counts.

♪ Class Actress • Weekend

You can use mpDris2 which provides similar facilities for mpd. It also comes with a user unit for systemd. ↩︎
Indeed, you just need to add a whole folder as a playlist (instead of manually adding track one after the other) and you’re done. ↩︎

Micro posting in September

Mon, 30 Sep 2024 20:39:33 +0200

2024-09-01: It looks like 0x0.st is up and running again. I believe it was down for a few months and I had to resort to https://transfer.sh/.
2024-09-02: TIL about a nice replacement for tldr: tealdeer.
2024-09-04: ♪ Antony & The Johnsons · You Are My Sister(Special dedicace to K.)
2024-09-06:
Pesto tart with goat cheese and pine nuts. It wasn’t too bad a job after all.
2024-09-06:

It’s both delightful and surreal to see that Valgrind is still in wide use today. — Twenty years of Valgrind

2024-09-06: A small table of interesting manpages.
2024-09-06: Old post but still an interesting personal roadmap with Vim. #vim
2024-09-06: Curiosities in Vinyl: Nice read. Looks like the author has a really comfy setup. I’ll keep the references handy.
2024-09-06: From Julia to Rust. See also Where to Start with Rust. I’ve been wanting to learn both of those languages but I’ve never really found the time for it, or perhaps the motivation to embark on new adventures. One day, maybe one day.
2024-09-11: LSP: the good, the bad, and the ugly.
2024-09-11: Plotting data in the terminal with gnuplot.
2024-09-11: Zsh Tricks to Blow your Mind.
2024-09-11: ♪ The Cure · A Night Like ThisRecorded live at the Roskilde Festival 2019.
2024-09-16:

All mainstream, general purpose programming languages are (basically) Turing-complete, and therefore any programme you can write in one you can, in fact, write in another. There is a computational equivalence between them. The main differences are instead in the expressiveness of the languages, the guardrails they give you, and their performance characteristics (although this is possibly more of a runtime/compiler implementation question). — Why Haskell?

2024-09-16:

Stick with the mainstream & boring unless a competing alternative that is so much simpler and/or more powerful, and that has an acceptable learning curve, so that it will justify leaving the flock, exists. Best resume I read in a while.

2024-09-16: If you need an easy way to convert MS docx to a PDF from the command line, don’t forget that lowriter (from Libre Office) has option to convert any document on the fly.
2024-09-17:

Accept multiple ways that users can ask for help and respond in kind. — Shell Script Best Practices

2024-09-17:

If we do not understand both the data and the models completely, it becomes very difficult to spot problems in the software we use to work on them: unexpected behaviour arising from software bugs may be mistaken for a peculiarity in either of them. It is then crucial that we minimise the chances of this happening by applying all the best engineering practices we have at our disposal. — The Pragmatic Programmer for Machine Learning.

2024-09-17: One of my favorite post-punk band from the 80’s. Big Ups: Bauhaus’ Kevin Haskins Dompe and David J Haskins Pick Their Bandcamp Favorites.
2024-09-17: Some Stata tutorials on survival analysis that are worth a read. #stata
2024-09-17: Dissecting the GZIP format. Very interesting read for those interested in compressing techniques. #clang
2024-09-17: Plotting data in the terminal with gnuplot. Still one of the most underated plotting tool for quick one-liner.
2024-09-17: Python’s multiprocessing performance problem. Lot of interesting tips and tricks for those using the multiprocessing module. #python
2024-09-24: ♪ Romain Dubois · Ardoise
2024-09-25:

If there is ONE THING the Unix world needs, it is for bash/ksh/sh to stop diverging further by permitting STUPID INPUT that cannot plausibly work in all other shells. We are in a post-Postel world. — OpenBSD now enforcing no invalid NUL characters in shell scripts

2024-09-26: ♪ Siouxsie And The Banshees · Arabian Knights
2024-09-27: Hot off the kitchen!

TUIs best of

Sun, 01 Sep 2024 21:08:19 +0200

Here’s a list of TUIs I use on my OS. They are ranked according to daily usage, from top to bottom. Since I spent my time in a Terminal managed by Tmux, with almost inevitably cmus and neomutt running in separate windows,¹ they are ranked first. I do not mention sed, awk, fzf, rg, and all the good stuff from GNU coreutils, nor zsh itself.

tmux for it is the best multiplexer available and it is highly customizable.
neovim because it has a decent terminal and good defaults.
cmus for everything that’s related to music stored on my HD.
mpv for all things related to video and music (when I’m not using cmus).
neomutt because, well, there’s a lot of emails going on when you’re employed and I also like answering questions about statistics or this site in general.
newsboat to read everything I like from the internet, i.e. people who love to share good stuff.
irssi to read and occasionally chat with others (channels: #lisp, #commonlisp, #scheme, #clojure, #chicken).
tig to browser the history of my Git repos.
rga to augment ripgrep with PDF support.
csvtk and csvlens when working with Excel or CSV files.
neofetch to quickly share my config.
onefetch when I want to get a basic summary of a Git repository.
wtwitch to check streamers and their VODs; also ytfzf to check video or music available on YT without all the fuzz of the YT web API – these tools rely on mpv of course.
asciinema to share terminal input/output when I do not want to perform live recording.
a bunch of Git addons saved in $HOME/bin folder that allow me, e.g., to update all remote repos, get various stats, rewrite author and/or history and so on.
various shell scripts to convert movie to jpeg or gif, and pdf to png or tikz to pdf, interact with tarball, etc.
some utilities for interacting with tmux or cmus, as well as neomutt, and a script named radio that triggers mpv with online radios.
I also have several handy alias in my zsh config.

Here’s a raw list of binaries I installed over the time. I won’t show my $HOME/bin folder, whcih I purged recently from old X11-only or defunct shell scripts. I still have a dozens of those lying around.

» ls .local/bin
akku@ csvtk* gen-hie* marksman* pylyzer* ruff* streamlink@ viu*
alen* deno* haskell-debug-adapter* matterbridge* py-spy* rust-analyzer* stylish-haskell* wtwitch*
ark* difft* hdc* mdbook* qlot@ scc* stylua* xlispstat*
bandcamp-dl@ duckdb* hoogle* mdcat* qr* seqkit* swish-lint@ xstata@
bb* dut* ihaskell* nvimpager* qsv* seqtk* tdf* xstata-mp@
bibclean* elm* joker* onefetch* reader* shellharden* texlab* ya@
cabal-fmt* fast-tags* lein* ormolu* rfc* sixcat* theme.sh* yazi@
clojure-lsp* floskell* lsix* pandoc-crossref* rga* stack-clean-old* ueberzug* ytfzf*
csslint* fourmolu* ltex-ls@ pdfcpu* rga-fzf* stan* ueberzugpp* zaread*
csvlens* fzf* lua-language-server* poetry@ rga-preproc* statweave.jar vale*

Apart from LSPs, I like to keep in this XDG folder stuff installed from GitHub and alike, or binaries that are too old in Ubuntu official repositories (e.g., fzf). I still need to write an install.sh shell script to download and/or update binaries from latest GitHub release. I removed everything installed with pip after I upgraded to Ubuntu 24.04 LTS and I’m now symlinking binaries from package installed in separate virtual environments in my $HOME/.Local/lib folder. I also keep binaries like matterbridge for memories. Many are still installed for testing purpose, e.g. marksman. I know this is a lot and for many of those binaries, I must admit I don’t use them often. Some binaries are also simple shell scripts for old X sessions (i3 or Regolith) and are not relevant anymore since I’m on Wayland.

♪ A Certain Ratio • The Fox

I used to launch each one in separate sessions, but I now use mostly the same session for media and news consumption, along regular off-hand tasks, and I create dedicated session for projects I’m working on. ↩︎

Micro posting in August

Sat, 31 Aug 2024 21:02:31 +0200

2024-08-01:

Creating and managing a package seemed much harder than I expected. — An unbiased evaluation of environment management and packaging tools

… to put it very mildly… This, however, does constitue a very good overview of what’s available in the Python ecocystem as of this writing. #python
2024-08-01: ♪ The Doors · The Crystal Ship
2024-08-02: And so, one year later, we got a few dozens of additional functions and a brand new name.

2024-08-06: I wanted to get rid of raw <img> tags and replace then with <figure>, available as a shortcode in Hugo. It turned out I had a lot of replacements to perform:

» rg -nc "\!\[.*\]\(" content/post | cut -d: -f2 | paste -sd+ | bc
688

Here’s a classical one-liner using Sed:

sed "s/^\!\[.*\](\(.*\))/{{< figure src=\"\1\" >}}/ content/post/*.md

I took the opportunity to add a small colored insert to each figure using the following CSS code:

figure {box-shadow: 0 0 0 .5em #f3f3ed;margin: 2em 4em;}

2024-08-06: TIL that `\%V` can be used to replace only in a visual selection (very handy for rectangular selection with ``). `#vim`
2024-08-13: Flat is a good thing but at some point I wish there could be a way to avoid unecessary duplication of core frameworks. I mean, below you'll find that I need 4 versions of the Qt goodies that I don't even use on my system.
![img](/img/2024-08-13-21-39-01.png)This all results in several Gig of wasted space, IMO. But this probaby is the way to go, as far as containerization is concerned.

» du -sh /var/lib/flatpak
11G /var/lib/flatpak

2024-08-13: [Someone’s Been Messing With My Subnormals!](https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html): one can only appreciate the obstinacy and quality of the data munging using Python and shell scripting!
2024-08-14: Yet another round of computer cleansing today. I also removed VirtualBox since I can certainly manage everything with qemu, and lot of unused Gnome garbage that I installed for testing purpose. Ironically, I also uninstalled [Gapless](micro/2024-07-25-09-44-52/) since I always returned to Cmus with which I am so comfortable and which fits my TUI workflow so perfectly. `#unix`
2024-08-14: ♪ Tindersticks · Raindrops (John Peel 1993)
2024-08-15: Just when I spent about half an hour compiling swayimg with all its dependencies, wondering how long it would take to compile the latest version of zathura and zathura-mupdf, it just came up to my mind that Ubuntu 24.04 LTS was released a few months ago. Time to upgrade the whole system I guess. `#vim`
2024-08-16:

Hopefully other email clients can get to state where they too can do all of this. — Patch Workflow With Mutt 2019

2024-08-16: Configuration, scripts and tips for using mpv as an image viewer. See also this GH issue for extended discussion.
2024-08-16: ♪ Tindersticks · Factory Girls
2024-08-16: ♪ Tindersticks · Raindrops (John Peel 1993)
2024-08-19: Looks interesting: Render and include figures in Pandoc documents using your plotting toolkit of choice .
2024-08-19: Efficient rolling statistics. #haskell
2024-08-19: ♪ Timber Timbre · Moment
2024-08-20:

kitty has to run on what I’m remoting from. tmux can work on what I’m remoting into. And that makes all the difference. — tmux is worse is better

2024-08-21:
2024-08-21: Churn and Weight: Or how to get interesting statistics for your Git projects.
2024-08-22: Racket parallel build in action.

2024-08-22: ♪ Wire · The Boiling Boy
2024-08-24:

IRC has historically been one of the most important chat programs in my life. — My IRC client runs on KubernetesPoor or lazy solution on my side: irrsi runs on a uni server, and I rsync the logs on my laptop when I’m in. I also concatenate all channel logs into monthly logs so that it’s easier to read what’s going on. This is why I may be slow to respond to /msg sometimes.

2024-08-24:

Putting Xs into squares is all the rage with statisticians. We should test that they can use bitstring macros for that. — CVS: cvs.openbsd.org: src

2024-08-24: Cleaning up the mess accumulated in my ~/.local/bin and updating binary releases here and there. I wish I was not so lazy and wrote a quick install.sh script once for all.
2024-08-24: Some really nice Gnome extensions are landing on GitHub, see openbar and forge. See also Exploring the Default Tiling Windows Feature in Ubuntu 24.04 (and Enhancing it). I still have Tactile around, but I rarely used it these days, unless I’m mirroring my desktop on a larger screen. #unix
2024-08-24: Sometimes I miss my time with macOS Mojave.

2024-08-24: TIL about theme.sh which is amazing: no dependency, self-contained, and it can be used online without setting up anything in your bashrc or zshrc config. You can not only preview colorscheme using Fzf, but you can also import your own colorscheme (it will be added in theme.sh itself. See also Foot FAQ to implement dynamic color changes. #unix
2024-08-24: Yet another TUI app that I recently added to my toolbet, with the hope that it will improve over time: tdf. I heavily rely on Zathura for things related to PDF (espacially for bidirectional sync when using Texlab), altough I came to appreciate Evince with time (it can open DVI and PS file, by the way). With async rendering and hot reloading, I can imagine that tdf could be a great add-on for previewing $\TeX$ compilation results.
2024-08-24: What scientists must know about hardware to write fast code: Great read, even for those not familiar with Julia.
2024-08-24: ♪ Wire · Outdoor Miner
2024-08-26: Installing wxpython on Ubuntu just takes 25 min. on an Intel i7-10610U (8 cores) @ 4.900GHz (389 Mo, virtual environment included). BTW, there’s no wheel for GTK4. #python
2024-08-31: This photo reminds me of the time when I was studying computer vision. Back in the time, we were happy with newly released LCD displays, although we got some CRTs lurking around. Now I can display images in my terminal (foot + sixel).

2024-08-31: Commit Mono is an anonymous and neutral programming typeface focused on creating a better reading experience.

August in review

Wed, 21 Aug 2024 14:47:12 +0200

Here are some recent bookmarks that are worth a mention somehow.

Good Python Software: This post rang a bell as I was also in the middle of transitioning all my previous Python workflows to Poetry or rye. See also Rye and uv: August is Harvest Season for Python Packaging.
I recently learned that there’s still a place for Javascript-free utilities in the world of YT and Twitch, see wtwitch and ytfzf. Sadly, the latter will probably go in quiet mode. Everything works smoothly with mpv, no subscription or OAUTH token required.¹ See also Youtube Without Youtube; Watching Videos on the Command Line.
Francis Bach has made his draft book, Learning Theory from First Principles, available for free on his website. I don’t have time to read it right now, but it now sits in my inbox.
I’m still experimenting a bit with Jasp. It’s pretty impressive whet they achieved with R as a backend. Here’s the list of packages they incorporated into Jasp.
Examples logo MetaPost: I totally forgot about this site and it was funny to browse some of the samples. If you’re interested in $\LaTeX$ and MP, be sure to check Christian Borgelt website where he hosts a bunch of nice looking slides and MP samples. And yes, he’s the author of the Apriori and Eclat algorithms (among others).
I came across the Neovim N Λ N O Theme, which is inpired by Nicolas Rougier’s Emacs theme. Originally, it was this GH repo which led me to search for a Nano theme.
I’m trying out a few Gnome extensions for a few days, in particular tophat. It looks good so far.
I just noticed that Tom MacWright now uses a micro-blogging system. Looks nice.

♪ Wire • I Am the Fly

You will need to install streamlink – even if mpv handles live streams perfectly well. The best option for Ubuntu so far is to create a virtual environment and pip install streamlink. The binary will be available in the bin directory and you can symlink to it from any directory in your $PATH. ↩︎

Plain tex'ing again

Sun, 18 Aug 2024 21:38:06 +0200

Lately I’ve been interested in plain $\TeX$ again. I briefly talked about my own customization in Neovim with vimtex in a preceding post but since then I stopped using vimtex because reasons.¹

I want to be able to write in English, with all the typos and grammatical errors you know me capable of, and in French with easy input of accented characters. I started writing in $\TeX$ a long time ago and I remember the days where I had to type \'e to get an accented é. Then we had latin1, ucs, babel with french/francais, etc. Finally I switched to Context for several years, and came back to $\LaTeX$ because of Sweave and knitr. There used to be a Context filter to process inline code chunk but it was not on par with Sweave, Statweave or Pweave. Anyway, these were old days but plain $\TeX$ is still a thing, right?

I offered a solution for accented letters via luatex last time. It’s okay, we just need to find an appropriate OTF font (there are plenty of them in /usr/share/fonts/opentype and you can probably install more from your package manager), and run luatex. Of course, this won’t solve the problem of French spacing or hyphenation. You’ll get a nicely formatted PDF output, and above all it comes with synctex support which means you’ll get forward and reverse synchronization between your text editor and your PDF viewer. If you’re using Vim or Neovim, set your compiler as luatex, as defined below, and you’re good to go.

-- compiler/luatex.vim
if exists("current_compiler") | finish | endif
let current_compiler = "luatex"

let s:cpo_save = &cpo
set cpo&vim

silent CompilerSet makeprg=luatex\ --interaction=\"nonstopmode\"\ --file-line-error\ --synctex=15\ %
CompilerSet errorformat=%f:%l:\ %m

let &cpo = s:cpo_save
unlet s:cpo_save

Yet there’s another option to alleviate the need for obscure typing of accented letters, namely xetex. As far as I know it cannot be called from latexmk (well, latexmk is for $\LaTeX$ backend after all), and it doesn’t support synctex. I used to use the $Xe\LaTeX$ backend a lot because after Context this was the easiest solution to get extra fonts incoporated in our PDFs. Using xetex as compiler also means that we need to handle bibtex (or makeindex) and the whole compilation toolchain. Usually, I would use texi2dvi for plain $\TeX$ (the texi2* suite is really handy and I relied on it a lot when I was using Sweave or knitr for my statistical reports) but it doesn’t support xetex either.

A third option to use xetex is arara and it works right out of the box. If you are curious about it all works, the rules that arara relies on are located under /usr/share/texlive/texmf-dist/scripts/arara/rules. If we add the following directives at the top of our document, we’ll get xetex + bibtex and a nicely formatted PDF.

% arara: xetex: {synctex: yes}
% arara: bibtex: { options: [ '-terse' ] }
% arara: xetex: {synctex: yes}
% arara: xetex: {synctex: yes}
\ifx\eplain\undefined \input eplain \fi

\font\bodyfont="Fontin"
\bodyfont

\parindent 0pt

Bienvenue aux caractères accentués français grâce à $\TeX$.

Here's a test for eplain: \cite[note]{knuth-1984-texbook}.

\beginsection{References.}\par
\bibliography{references}
\bibliographystyle{plain}

You can use any font you like, e.g. fc-list : family file | grep -i fontin.

Here’s the output on my machine:

~/tmp 1m12s
» arara plain.tex
 __ _ _ __ __ _ _ __ __ _
 / _` | '__/ _` | '__/ _` |
| (_| | | | (_| | | | (_| |
 \__,_|_| \__,_|_| \__,_|

Processing "plain.tex" (size: 3.3 kB, last modified: 2024-08-18
20:05:35), please wait.

(XeTeX) XeTeX engine .................................... SUCCESS
(BibTeX) The BibTeX reference management software ....... SUCCESS
(XeTeX) XeTeX engine .................................... SUCCESS
(XeTeX) XeTeX engine .................................... SUCCESS

Total: 1.744 seconds

If it looks a bit tedious to add so many directives at the top of our document, we can rely on a global or local configuration file. For instance, this allows to pass a default preamble when calling arara on a $\TeX$ document. The exact structure of the configuration file may not be obvious when reading the manual, but after some trials I got something working as expected. Put the following in a file named $HOME/.araraconfig.yaml:

!config
defaultPreamble: xetex-mp
preambles:
 xetex-mp: |
 % arara: xetex: {options: ["-synctex=15"]}
 % arara: bibtex: {options: ["-terse"]}
 % arara: xetex: {options: ["-synctex=15"]}
 % arara: xetex: {options: ["-synctex=15"]}

You may notice that I expanded the synctex option to benefit from all optimizations, especially better compression support. You can add as many compilation directives as you want, and of course define other preambles which may then be called with the --preamble option. In this case, setting a default one ensures that we can simply call arara ourdoc.tex and that all the above directives will be proceeded.

You can plug arara as your makeprg in Vim or Neovim but if you want to get useful information in case something went wrng, you’re better off adding --verbose to arara and maybe -interaction=nonstopmode to the xetex processor. Finally, you can add many more options to xetex itself, e.g. -etex.

Bonus point: You can use whatever tool you like to cleanup the artifacts generated by arara, e.g. latexmk -C, but there’s a specific rule for that in arara as well so it’s up to you:

% arara: latexmk: { clean: partial }
% arara: clean: { extensions: [ aux, log ] }

I perfer keeping the log files around in case something went wrong during the compilation, but at least you know.

♪ Warsaw • As You Said

This really is a great plugin, and it’s probably better than Texlab or other competitors that I never tried. But to be honest I don’t really need all of these goodies. I have my own way of fetching bibliographic entries via fzf, latexmk takes care of cleaning up the mess after a PDF has been produced. I wrote a quite complete interface to Texlab in Lua (i.e., similar to what you’ll get from nvim-lspconfig), which provides me with omni-completion and goto definition/reference. That’s really all what I need. ↩︎

Ubuntu 24.04: The sequel

Fri, 16 Aug 2024 17:37:19 +0200

Quickly jotted post-install steps of my previous post. It was a joy, really.

Node

I know there are various way to install the full-stack on Ubuntu, either via apt, dedicated PPAs, or nvm. Previously I was using an imported deb repository. Ubuntu is known to be somewhat lagging a little behind the latest versions, so this was the best chance to get a decent nodejs with the 22.04 LTS at some point. I’m not particularly interested in managing multiple versions of Node since I barely do any serious stuff in JS for web dev.

Currently, the official Ubuntu repository offers nodejs/noble 18.19.1+. That’s okay, but if you try to install npm, unless you’ll end up with a giant list of packages! Installing the combo nodejs + npm, with nodejs-doc (because we like to read the doc, right?) results in a mess:

0 mis à jour, 394 nouvellement installés, 0 à enlever et 2 non mis à jour.
Il est nécessaire de prendre 28,0 Mo dans les archives.
Après cette opération, 169 Mo d'espace disque supplémentaires seront utilisés.

Note that you would get almost nothing by asking apt for --no-install-recommends (about forty packages less at best).

Let’s try with the NodeSource repo instead, with the LTS release. Notice that several blog posts suggest https://deb.nodesource.com as a primary repo, but this is not from the NodeJS organization, AFAICT. In retrospect, this is probably what I installed last time:¹

nodejs/nodistro,now 20.12.2-1nodesource1 amd64 [installé]

Anyway, there are prebuilt binaries available on https://nodejs.org, as well as a suggestion to use nvm. I finally choose the later. Everything went fine except that I lost my custom lib directory (so-called “prefix” in Node parlance). I’ll check back later!

» nvm install 20
Downloading and installing node v20.16.0...
Downloading https://nodejs.org/dist/v20.16.0/node-v20.16.0-linux-x64.tar.xz...
################################################################################# 100.0%
Computing checksum with sha256sum
Checksums matched!
Your user’s .npmrc file (${HOME}/.npmrc)
has a `globalconfig` and/or a `prefix` setting, which are incompatible with nvm.
Run `nvm use --delete-prefix v20.16.0` to unset it.

» nvm use --delete-prefix v20.16.0
Now using node v20.16.0 (npm v10.8.1)

» node -v
v20.16.0

» npm -v
10.8.1

Python

Canonical apparently wants to use Python more and more for the back office. It is now harder to pip install a Python package. If you try to install Poetry using their online shell script, you will be redirected to an error message about installing packages outside a virtual environment. Same goes with pip in your $HOME directory:

» pip install --user poetry
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
 python3-xyz, where xyz is the package you are trying to
 install.

 If you wish to install a non-Debian-packaged Python package,
 create a virtual environment using python3 -m venv path/to/venv.
 Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
 sure you have python3-full installed.

 If you wish to install a non-Debian packaged Python application,
 it may be easiest to use pipx install xyz, which will manage a
 virtual environment for you. Make sure you have pipx installed.

 See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

I’m not against banning system-wide installations, on the contrary. But it seems to me that user install don’t really harm. I personally use virtual environments for every projects, but keep installing in my $HOME directory packages that I use regularly when firing up an IPython shell, e.g. numpy, scipy, statsmodels, plotnine, biopython, networkx. I don’t want to override the official policy in this case, and I note that I could install most of those package from Ubuntu repos as well, meaning I may even benefit from Ubuntu package management itself, which reminds me of this:

I manage my Python packages in the only way which I think is sane: installing them from my Linux distribution’s package manager. I maintain a few dozen Python packages for Alpine Linux myself. It’s from this perspective that, throughout all of this turmoil in Python’s packaging world, I have found myself feeling especially put out. — Python: Please stop screwing over Linux distros

Quick fact checking: There are a lot of R (thanks to r2u) and Python packages (this includes documentation packages) already available on Ubuntu 24.04!²

» apt search r-cran- | wc -l
3428

» apt search python3- | wc -l
13580

I’m waiting to see how I’ll proceed, but frankly I think I’ll just install the deb packages I use frequently and create a sandbox for the rest.

In the meantime, I just deleted my Python 3.10 library and everything that was installed under $HOME/.local/bin with pip (rg -l "\#\!/usr/bin/python3" | xargs rm).

R

After reinstalling R from the official Ubuntu source list, I got R 4.3. I was running v4.4.0 but it is not a big deal. I had a total of 498 packages (2.4 Go) in my personal library. Rather than keeping stuff compiled with another version, I took the opportunity to shorten the list of packages I will have installed by default, and use renv to manage additional packages on a per-project basis. This is in line with the approach usually taken in other PLs I use.

I selected the following list of “essential” packages, meaning I will probably use two or three of them each time I fire up R for a quick statistical investigation, excluding LSP and editor stuff, i.e. languageserver, lintr and styler: car, coin, cowplot, data.table, directlabels, ggplot2, glmnet, haven, Hmisc, languageserver, lintr, lme4, mclust, multcomp, patchwork, plotly, remotes, reshape2, renv, readxl, rms, skimr, styler, tinytest, vegan. Skimr is surely the only exotic piece of a package in this list, and it adds little to the bill after all.

This whole set of packages is going to install a bunch of additional software, but not that much compared to what testthat alone would require:³

‘bit’, ‘rappdirs’, ‘sys’, ‘Deriv’, ‘modelr’, ‘microbenchmark’, ‘bit64’, ‘cachem’, ‘memoise’, ‘sass’, ‘ps’, ‘pkgbuild’, ‘askpass’, ‘broom’, ‘numDeriv’, ‘doBy’, ‘MatrixModels’, ‘farver’, ‘labeling’, ‘munsell’, ‘fansi’, ‘pkgconfig’, ‘iterators’, ‘clipr’, ‘crayon’, ‘vroom’, ‘tzdb’, ‘checkmate’, ‘rstudioapi’, ‘fastmap’, ‘bslib’, ‘evaluate’, ‘fontawesome’, ‘jquerylib’, ‘tinytex’, ‘xfun’, ‘yaml’, ‘highr’, ‘processx’, ‘brew’, ‘commonmark’, ‘desc’, ‘pkgload’, ‘remotes’, ‘zoo’, ‘curl’, ‘mime’, ‘openssl’, ‘generics’, ‘later’, ‘rematch’, ‘prettyunits’, ‘utf8’, ‘R.methodsS3’, ‘R.oo’, ‘R.utils’, ‘carData’, ‘abind’, ‘pbkrtest’, ‘quantreg’, ‘scales’, ‘libcoin’, ‘matrixStats’, ‘modeltools’, ‘mvtnorm’, ‘gtable’, ‘rlang’, ‘quadprog’, ‘cli’, ‘glue’, ‘isoband’, ‘lifecycle’, ‘tibble’, ‘vctrs’, ‘withr’, ‘foreach’, ‘shape’, ‘Rcpp’, ‘RcppEigen’, ‘forcats’, ‘hms’, ‘readr’, ‘tidyselect’, ‘cpp11’, ‘gridExtra’, ‘htmlTable’, ‘viridis’, ‘htmltools’, ‘base64enc’, ‘colorspace’, ‘rmarkdown’, ‘knitr’, ‘Formula’, ‘callr’, ‘collections’, ‘fs’, ‘jsonlite’, ‘R6’, ‘roxygen2’, ‘stringi’, ‘xml2’, ‘xmlparsedata’, ‘backports’, ‘cyclocomp’, ‘digest’, ‘rex’, ‘minqa’, ‘nloptr’, ‘TH.data’, ‘sandwich’, ‘httr’, ‘magrittr’, ‘viridisLite’, ‘htmlwidgets’, ‘tidyr’, ‘RColorBrewer’, ‘dplyr’, ‘lazyeval’, ‘crosstalk’, ‘purrr’, ‘promises’, ‘plyr’, ‘stringr’, ‘cellranger’, ‘progress’, ‘SparseM’, ‘polspline’, ‘pillar’, ‘repr’, ‘R.cache’, ‘rprojroot’, ‘permute’

Results: 157 packages, 408 Mo.

In summary, I just installed and deleted a total of xxx packages (where xxx > 500) on various places on my laptop, from various sources on the internet, using various package manager (having almost zero reverse or orphaned dependencies management,⁴ unless you use your distro own package manager). Everything is HUUUUGE.

♪ Tindersticks • My Sister

I’ve long been using a simple Makefile to keep track of my packages list:

all:
 @echo "\033[0;32m[+] Staging local changes\033[0m"
 (apt list --installed > packages) >/dev/null 2>&1
 git add .
 git diff-index --quiet HEAD || git commit -m "Update packages"

↩︎

Sadly, no plotnine! ↩︎
Installing testthat on a fresh and empty libPath means a mare addition of packages (35 Mo) already : ‘utf8’, ‘crayon’, ‘pillar’, ‘pkgconfig’, ‘vctrs’, ‘glue’, ‘fs’, ‘pkgbuild’, ‘rprojroot’, ‘diffobj’, ‘fansi’, ‘rematch2’, ‘tibble’, ‘brio’, ‘callr’, ‘cli’, ‘desc’, ‘digest’, ‘evaluate’, ‘jsonlite’, ‘lifecycle’, ‘magrittr’, ‘pkgload’, ‘praise’, ‘processx’, ‘ps’, ‘R6’, ‘rlang’, ‘waldo’, ‘withr’. Remember the tinyverse? ↩︎
While r2u is a great initiative, I keep the habits of compiling everything from source, at least on my machine. And please note that using bspm to get native binary via the r2u apt repo doesn’t work with renv, rig (and probably pak). ↩︎

Ubuntu 24.04

Thu, 15 Aug 2024 20:35:19 +0200

I just installed and configured Ubuntu 24.04 LTS. This is the second dist upgrade I apply on the same laptop. Everything went well but this time I wanted to polish things a bit, especially remove old artifcats from multiple upgrade while keeping all my customizations. By the way, the extensions I were using (Tactile and Simply Workspaces are now useless with the new features introduced in Gnome desktop).

I won’t comment about the new Desktop, WM or app UI, there’s already a good deal of information on the web. It’s pretty slick as far as I can tell (e.g., Nautilus or the desktop menu bar). However, I should note that Ubuntu mono font now looks quite crappy in a Terminal. Since I downloaded the Nerd font equivalent a while ago, I switched to that temporarily (I know nostalgics can get the “legacy” font from a package). What follows is is merely an installation log.

After updating all packages installed on my laptop, I ran the classical do-upgrade-release -d at the shell prompt. This didn’t work. First, there appeared to be some problem with custom PPAs. I started to remove them one by one then forgot a leading asterisk which removed them altogether. A good start to say the least… Afterwards, there still was a pending issue with postgresql-12 (which may date from the 20.04 release?). Finally, the dist upgrade went fine, although some packages like texlive highlighted minor errors while some other packages were forcibly removed.

Upon rebooting the system, my plan was to remove packages tagged jammy (obs-studio), remove packages installed from custom PPA, check orphaned packages if any and record packages with residual config files for further investigation. It took me a few more minutes.

I deleted old Linux kernels (headers and images), totalling 640 Mo on my HD! I used a shell script slighty adpated from SO.

» dpkg --list | grep linux-headers
ii linux-headers-5.15.0-118 5.15.0-118.128
ii linux-headers-5.15.0-118-generic 5.15.0-118.128
ii linux-headers-5.19.0-50-generic 5.19.0-50.50
ii linux-headers-6.5.0-45-generic 6.5.0-45.45\~22.04.1
ii linux-headers-6.8.0-40 6.8.0-40.40
ii linux-headers-6.8.0-40-generic 6.8.0-40.40
ii linux-headers-generic 6.8.0-40.40
ii linux-headers-generic-hwe-22.04 6.8.0-40.40\~22.04.3

Next, I wanted to get ride of resiudal config files from uninstalled packages. For whatever reason (e.g., packages flagged as manually installed), even all packages were supposed to be “purged” and “autoremoved” from the system, there remained dozens of such files.

I listed all relevant packages as follows:

dpkg --get-selections | grep 'deinstall$' | cut -f1

I then tried to pipe this list to xargs sudo apt --yes purge but I got stuck in the middle of a dialog box asking whether I really wanted to delete everything from Postgresql 12. Shortly after C-c’ing and killing the whole process, dpkg was no longer working. I reconfigured dpkg and then all packages were flagged as purge and not deinstall this time. I finally was able to cleanup everythign and checked that I got the same number of installed packages as highlighted by neofetch:

sudo dpkg --configure -a
dpkg --get-selections | grep 'purge$' | cut -f1
sudo apt clean && sudo apt autoclean && sudo apt autoremove
dpkg --get-selections | wc -l

I also cleaned up /usr/share/applications and $HOME/.local/share/applications for unused desktop entries, my /opt directory full of application for testing purposes, and a bunch of unused software.

I still need to filter out broken executables installed by Python in $HOME/.local/bin (e.g., streamlink), install nodejs, npm and yarn, and handle software that were installed via custom PPAs (e.g., Racket). Also, I would need to review log files in /var/log/dist-upgrade to see if I missed something.

» neofetch
OS: Ubuntu 24.04 LTS x86_64
Host: Latitude 7310
Kernel: 6.8.0-40-generic
Packages: 3623 (dpkg), 17 (flatpak)
Shell: zsh 5.9
Terminal: alacritty
CPU: Intel i7-10610U (8) @ 4.900GHz

Sidenote

Alacritty 0.13 is available via apt. However I encountered some issue with the mouse pointer that disappeared from sight when above window, so that I reinstalled version 0.14-dev from GitHub that I compiled last week.
There’s a new version for Cmus available in Ubuntu repos, still without album art support; I reinstalled my old fork, messed up with various GH clones (including one which used to work, and then I deleted everything. I’ll see how to bring album art back in my notification popup if I really miss that.
Newsboat not available in official repos, don’t know why, but there’s a deb package for 23.10 and apparently a snap package for the current relese. Hey, I don’t want to use Snap, I removed it from my laptop two releases ago already!

[2024-08-17]
As advised on Linux Mint Forums, you can just install the latest available deb package (23.10 Mantic), and it works right out of the box.

Now got NeoMutt 20231103, which is a 4 yr.bump (latest version in 22.04 dated back to 2019 if I remember correctly). Got some issues with my color definition but this was easily fixed.
Tmux 3.4 works as expected.
swayimg no --gallery option with current version from apt, while the version I compiled from the master branch on GitHub works fine.
Zathura is working fine with poppler as a PDF backend; should install mupdf backend but got stuck with dependencies (it would require mupdf > 1.19, and when I checkout the latest tag which accepts mupdf I got another conflict with a now defunct library). For the moment, I will use Foliate for Epub. And I realized that using a single app for all publication formats isn’t very realistic. After all, Evince is able to open PS and DVI files, while Zathura needs a plugin for Postscript and can’t open DVI file.

Oh, I forgot to say that Minecraft is still working ;-)

♪ Tindersticks • See my Girls

The unquantified self #32

Tue, 06 Aug 2024 20:44:43 +0200

Yet another one of these posts to reflect on recent moments in my life. I just finisehd watching Season 2 of the House of the Dragons. It was great and I am now just awaiting the third Season. I wouldn’t say it’s on the same level as Game of Thrones, but maybe it’s that there’s less of a profusion of characters, which makes House of the Dragons more intimate and closer to a classic series or mini-series. I watched other mini-series previously but only The Queen’s Gambit and Ripley got my attention.

This site has been pretty quiet for the past 3 months, partly because of the busy workload, but also because of a lack of motivation. I took the opportunity to read a lot of books and draw whatever came to mind, or sometimes to mind. An excerpt follows.

I still want it darker as he said.

Anyway, the good thing is that I’m on holidays, for a good two weeks yet. I needed this to disconnect from everything, to recompose myself in a way. I’m afraid it’s not going to get any better – just turned 50, still alive.

I finally decided it was time to update the base colors of the site. Everything remains the same except for the background color of the body and note. I acknowledged source of inspiration in the Changelog. I also added rounded corner to code blocks, but I still don’t want to provide syntax highlighting. I like it when it’s just black & white, with bold statements, like in my text editor. I’m becoming increasingly allergic to garish colors and uniform white. Speaking of the text editor (Neovim), I reworked my config once again, as described in this post, but there was more to come later on. I ended up with an even more dry setup, based on 6 plugins but a lot of hand crafted VimL functions and settings.

~/.config/nvim
» tree -L 3 pack
pack
└── plugins
 ├── opt
 │   ├── neogen
 │   └── vim-table-mode
 └── start
 ├── fzf-lua
 ├── nvim-treesitter
 └── nvim-treesitter-textobjects

9 directories, 0 files

That being said, if it hadn’t been for treesitter and the terminal facilities, I could have started from scratch with vim 9, ALE and fzf.vim.

There are so many drafts waiting for me that I’d better get some rest…

aliquote master?
» hugo list drafts
content/post/trend-tests.md
content/post/approximating-sine.md
content/post/generating-permutation.md
content/post/party-more-party.md
content/post/debugging-gdb-cutter.md
content/post/ols-in-c.md
content/post/git-for-tracking-changes.md
content/post/suffix-tree-in-scheme.md
content/post/bridge-sampling.md
content/post/how-many-bootstrap-replicates.md
content/post/islp.md
content/post/bootstrap-bca.md
content/post/racket-sqlite-api.md

♪ The Gun Club • Watermelon Man

Micro posting in July

Wed, 31 Jul 2024 21:53:08 +0200

2024-07-07: ♪ Archive · Sleep
2024-07-15: Lot of useful tips and tricks for Zsh users. I particularly like the hash -d approach for directory bookmarks. #unix
2024-07-15: ♪ Editors · Formaldehyde
2024-07-25: From time to time, I make a few infidelities with cmus and use gapless (formerly g4music). It’s a great piece of software which I initially compiled from scratch until I decided to rely on the Flathub version. I once generate m3u playlists for relevant folders in my global library and they come up nicely in the playlist view. See my review here.
2024-07-27: I try to use all the features of fzf-lua, and it’s quite easy to reproduce a Git time machine with the bcommit command. If you’re not, you may want to give a shot to git-time-lapse: It does the job pretty well with zero dependencies (e.g. Telescope, plenary), and it can be installed as a simple optional plugin. #vim

Writing plain TeX using vimtex

Wed, 24 Jul 2024 21:02:57 +0200

I very much like vimtex, even if I’m not writing as much $\TeX$ stuff as I used to do in the past. It works very well with standard Latex documents that I still have on my HD. I get live compiling and forward/reverse search with the Zathura PDF viewer for free, and I can also toggle a pretty TOC along the way. I no longer use Context (maybe I should) but I know it is supposed to work similarly. As I said in a previous post, luatex is great for it allows us to write “plain $\TeX$” (for whatever definition of plain we agree on) using custom fonts and forget about accented letters for non-US writers. Since this last post, I haven’t write a single line of $\TeX$. However, since I was digging into my Neovim config for $\TeX$ stuff, I wanted to know how Vimtex handles the plaintex filetype.

And so, it all started with me trying to replicate the toy example mentioned above. It failed early. Okay, let’s forget I missed to add a final \bye statement. Moving on, I soon realized that vimtex is way more friendly with Latex than plain old tex. Since I already have my own $HOME/.latexmkrc, I never bothered checking how vimtex proceeds in this case. It looks like it has its own setting for running latexmk. Fair enough, but I couldn’t find my way to tell it to use an option like -pdlatex=luatex. Hence I tried to use a different compiler, namely arara. It allows to define the compiler and its options directly in the tex file, which is fine since I would like to avoid adding .latexmkrc or .nvimrc local config files to a simple project like this one. I also needed to define a Tex root directive to please vimtex, otherwise it complains with a “failed mainfile detection” – again because it primarily targets Latex documents.

So far, I have the following in my after/ftplugin/plaintex.vim file:

setlocal autoindent expandtab tabstop=2 shiftwidth=2
set spelllang=en_gb

let g:vimtex_format_enabled = 0
let g:vimtex_compiler_method = "arara"

nmap <buffer> gs :VimtexReload<cr> \| :VimtexCompile<cr>
nmap <buffer> g= :%!fmt -w80<cr>
nmap <buffer> <silent> go :VimtexTocToggle<cr>
nmap <buffer> <silent> gO :lua require("fzf-lua").btags()<cr>
nmap <buffer> <leader># :-1 read ~/.config/nvim/templates/plaintex<cr>

I encountered some problem when trying to refomat using Vim’s gq but gw works fine, so that may just be a conflict somewhere in my config. Note that I overwrite global settings for vimtex, but I assume that I may rarely have to switch from plain $\TeX$ to $\LaTeX$ during the same writing session. Finally, I don’t really why I need to reload vimtex in order to apply the compilation directive that are defined in the header, but since I may forget about this later I chose to reload systematically before recompiling. Here’s the template I defined for my $\TeX$ documents:

% vim: set ft=plaintex:
% arara: luatex: {synctex: yes}
%! TeX root = test.tex
\ifdefined\directlua
 \input luaotfload.sty
\fi
\font\tenrm="Alegreya" at 10pt
\tenrm

\parindent 0pt

\bye

I will probably update the above template when I will really write something more elaborated than a toy example.

♪ Claire Welles • Waiting In The Sun

Develop good habits from the start

Wed, 10 Jul 2024 20:42:46 +0200

Last month I came across a blog entry posted on HN: A Bunch of Programming Advice I’d Give To Myself 15 Years Ago. I couldn’t help thinking that some of the advices discussed here are good ones even for managing a statistical project:

Spending time sharpening the axe is almost always worth it
If you can’t easily explain why something is difficult, then it’s incidental complexity, which is probably worth addressing
Don’t underestimate the value of digging into history to investigate some bugs
Make debugging easier

All the above apply equally well to statistical analysis. Even if should select the best statistical package for the task at hand (while running a t-test is a common task that most software offer nowadays, analyzing survey data is not something to be taken lightly). As someone who value good software and the command-line (not much because I like typing commands but because I can record them for later use or review) I’ve almost always relied on R, Stata and a few dedicated software (e.g., jags, stan, xgboost). Writing code also means you should value your text editor (Emacs first, then Vim, then Neovim in my case). About the second point, I faced it both when teaching medium-to-hard techniques to graduate students or during corporate training, and by presenting my own analysis results, most of the time resulting from multivariate analysis. I was using Git mostly for mono repos so that I could track changes along project timeline (my changes, and their – i.e., the client or colleagues – changes as well). As for debugging, especially making it easier to detect errors or misbehaving programs, I believe it comes with experience and good practices.

Other rule-of-thumbs I adopted over time:

Make a plot for every p-value you report.
We have computer to do the hard work, but it’s always good to know how dof are derived, for instance. It helps checking if we are working with the correct data, but see below.
Be careful with variable encoding and missing values.
Be careful with default values of your preferred statistical package for certain routines, they may differ from other well acknowledged software (continuity correction, population vs. sample dof, etc.).
Reproducibility is a thing, but you should first be able to reproduce your own results (think about the 2-year cycle or so before publishing a paper; you may be asked to re-run the whole analysis to please one picky reviewer, who might be right by the way).

To sum up, developing good habits from the start is always worth it. I can’t count the number of times I’ve been asked to produce a result with slightly updated data 6 or 12 months after the end of a project: having a solid, well-tested script has made things easier every time.

♪ Passenger • Wicked Man’s Rest

Stata and Quarto

Tue, 09 Jul 2024 20:48:03 +0200

I’m a bit late with Quarto, which I only started to use a few months ago. I’ve heard good news about this new rendering engine for literate coding on Frank Harrell’s blog, but since I rarely write RMarkdown reports these days, I tested the command-line program and then just forgot about it.

However, a few days ago I came across a Neovim plugin which offers all the goodies you might expect to render and/or preview Quarto documents from Neovim directly. I think I found it when browsing about use cases for molten-nvim that I finally kept in my toolbox (I’m rather thrifty in this respect since I only have 8 plugins in my pack folder – no plugin manager, no nvim-lspconfig!). I don’t really care about the plugin itself since I manage to write a one-liner compiler for Vim:

silent CompilerSet makeprg=quarto\ render\ %\ --to\ $*

I don’t really care about messing up with the erroformat since quarto will stop running after the first error.

And for the remaining stuff, I just added set ft=markdown and runtime! ftplugin/markdown.vim to my after/ftplugin/quarto.vim config. That’s all folks!

Anyway, great news for Quarto users using Stata with version below 17: you can declare stata_kernel as a custom kernel rather than the default nbstata. Since I only owe a license for Stata 13 MP, I don’t know what’s new in the Stata world other than what I read on the www, but it works perfectly well, including graphical output. This was the most cumbersome thing I happened to deal with when I was writing my Stata Starter Kit using org-mode some years ago. Since I’m using Stata 13, I needed to export graphics as encapsulated PS file, and then convert them to PNG files. Of course this was automated via a Makefile, but when you find that a version number (13 < 14) prevents you from exporting your figures as PNG or SVG files, it really sucks. Now it’s all over, and I can just write Stata code and let Quarto produce a well-formatted PDF or HTML file using Pandoc. MRE follows.¹

---
title: Hello Stata
author: chl
jupyter:
kernelspec:
display_name: Stata
language: Stata
name: stata
format:
html:
toc: true
html-math-method: mathjax
highlight-style: monochrome
embed-resources: true
anchor-sections: true
citations-hover: false
footnotes-hover: false
code-copy: true
---
Trying out to run some Stata code:
```{stata}
sysuse auto
su mpg
hist mpg
```
That's it!

♪ Bauhaus • Double Dare

You’ll need to install stata_kernel first. ↩︎

Neovim 0.10

Wed, 03 Jul 2024 20:11:23 +0200

I did it again: I cleaned up unused or defunct stuff in my Neovim config. I have one excuse, though: Since I upgraded to version 0.10, I needed to account for API breaking changes but also to accommodate all the goodies, especially regarding LSP & Co.

Neovim gained a builtin commenting system, probably inherited from mini.nvim. Since I removed the plugin I was using, I’m glad this is now shipped as a default. The default theme (on the right below) has been redesigned, and it looks great (except for the diff view), but I much prefer the quiet theme (on the left) which removes almost everything and yields a very slick UI, IMO. See :help dev_theme for more info on the design of the new colorscheme.

[2024-07-06]
If I ever change my mind, I could still rely on the default theme when I’m working on my local computer and switch to the quiet theme for remote sessions. Something like that, maybe:[^1]

vim.api.nvim_command([[
 augroup updatecolorscheme
 autocmd colorscheme * :hi normal guibg=NONE ctermbg=NONE |
 hi! link QuickFixLine Visual
 augroup END
]])

if vim.env.XDG_SESSION_TYPE == "wayland" then
 vim.cmd.colorscheme("default")
 local fzf_theme = "light"
else
 vim.cmd.colorscheme("quiet")
 local fzf_theme = "bw"
end

Regarding LSP, omnifunc and tagfunc are now automatically set up when attaching an LSP. There are now default mapping to navigate diagnostics and to pop up the diagnostic window. It is also easier to determine the root folder for a project (see vim.fs.root). Inlay hints have landed in the 0.10 release (it works great with rust-analyzer and clangd), which means we can show them on demand using a simple mapping:

if client.supports_method('textDocument/inlayHint') then
 vim.keymap.set("n", "zI", "<cmd>lua vim.lsp.inlay_hint.enable(not vim.lsp.inlay_hint.is_enabled())<cr>", {})
end

I found a way to get ride of nvim-lightbulb using the following snippet:

function code_action_listener()
 local bufnr = vim.api.nvim_get_current_buf()
 local context = { diagnostics = vim.lsp.diagnostic.get_line_diagnostics() }
 local params = vim.lsp.util.make_range_params()
 local ns_id = vim.api.nvim_create_namespace('code_action')
 local r,c = unpack(vim.api.nvim_win_get_cursor(0))
 local curr = 1
 params.context = context
 local line = params.range.start.line
 vim.lsp.buf_request(bufnr, "textDocument/codeAction", params, function(_, actions, _)
 if not actions or type(actions) ~= "table" or vim.tbl_isempty(actions) then
 vim.api.nvim_buf_del_extmark(bufnr, ns_id, curr)
 else
 vim.api.nvim_buf_del_extmark(bufnr, ns_id, curr)
 vim.api.nvim_buf_set_extmark(bufnr, ns_id, line, -1,
 {id = curr, virt_text = {{"░", "DiagnosticInfo"}}, virt_text_pos = "eol"})
 end
 end)
end

if client.server_capabilities.codeActionProvider then
 vim.api.nvim_create_autocmd({"CursorHold", "CursorHoldI"}, {
 group = vim.api.nvim_create_augroup("code_action_sign", { clear = true }),
 callback = function()
 code_action_listener()
 end,
 })

 vim.keymap.set("n", "z=", "<cmd>lua require('fzf-lua').lsp_code_actions({ previewer = 'codeaction' })<cr>")
end

This is all great news from people who like to keep their editor free of any IDE fancy things. There’s an interesting thread on HN regarding all those goodies.

As an aside, following my recent post, I briefly tried to get used to Conjure but I couldn’t find my way with the log buffer. The thing that I will probably miss is the ability to set marks and reevaluate them at will. But to be honest, I reverted back to molten-nvim which provides the best compromise between inline evaluation and my full REPL driven (with a terminal opened in a split) workflow. And I like the idea that I can plot almost anything in R, Stata or Python (with plotnine) and get a nice PDF almost instantaneously.

In passing, I also reworked my whole after/ftplugin/markdown.vim settings. Treesitter now handles most the stuff I implemented myself, especially navigating between section, folding, or block selection. I still need this stuff for org.vim since I removed the orgmode plugin so nothing’s lost. I added one package, vim-table-mode, which I used to use occasionally in the past.

♪ ANOHNI and the Johnsons • Epilepsy is Dancing

[^1]: Further improvement to the quiet theme would be to remove diagnostic colors altogether:

```lua
hi DiagnosticUnderlineError guisp=Black |
hi DiagnosticUnderlineWarn guisp=Black |
hi DiagnosticUnderlineHint guisp=Black |
hi DiagnosticUnderlineInfo guisp=Black |
hi! link DiagnosticFloatingError NormalFloat |
hi! link DiagnosticFloatingWarn NormalFloat |
hi! link DiagnosticFloatingHint NormalFloat |
hi! link DiagnosticFloatingInfo NormalFloat |
hi! link DiagnosticFloatingOk NormalFloat
```

Micro posting in June

Sun, 30 Jun 2024 13:37:33 +0200

2024-06-20:

2024-06-20:

A good understanding of Vim’s various lists is a massive productivity boost — it’s taken me many years of Vim use to truly appreciate this. — Vim’s useful lists.

2024-06-20: TIL that Racket comes with a migration utility for packages installed using a preceding release: raco pkg migrate.
2024-06-20: ♪ New Order · Truth
2024-06-21:

“Vim experts” don’t prefer buffers over tabs: they use buffers as the file proxies they are and tab pages as the workspaces they are. Buffers and tab pages have different purposes so preferring one to the other makes no sense whatsoever. — Why do Vim experts prefer buffers over tabs?

2024-06-25: After 4 years I think fzf-lua remains one of my favorite Neovim plugin. That being said, it is also one of the most astonishing GitHub project where the author is so active that all issues are resolved in due time while he is also providing useful tips all time. Last find is how live grep could potentially handle current word (C-r C-w doesn’t work when using Fzf terminal). #vim
2024-06-27: I finally got ride of most of the (Neo)vim packages I was only using on an occasional basis, which led me to the following setup: (I barely use Rust, but I happen to read some Rust code from time to time.)

» tree -L 3 pack
pack
└── plugins
 ├── opt
 │   ├── haskell-tools.nvim
 │   ├── molten-nvim
 │   └── rustaceanvim
 └── start
 ├── fzf-lua
 ├── nvim-treesitter
 ├── nvim-treesitter-textobjects
 └── vimtex

Also, I deleted my custom colorscheme, even if I was just using zenbones for the past few months, and I switched to the builtin quiet colorscheme, which is great (pending some minor changes which I implemented in an autocommand). Good-bye orgmode as well, especially since I don’t use the agenda and capture stuff. #vim
2024-06-27: ♪ Northside · Tour de World

ArXiving on February 2024

Thu, 27 Jun 2024 12:24:09 +0200

Old reads…

Looking Back at Postgres (https://arxiv.org/abs/1901.01973)

I rarely use Postgres for my personal hobbies (I usually find that SQLite provides me with convenient cost-effectiveness tradeoff for pet projects), but I always appreciated it for long term projects as well as its extensions. This paper is a recollection of the early days of Postgres in the 80s.

Postgres was Stonebraker’s second system, and it was certainly chock full of features and ideas. Yet the system succeeded in prototyping many of the ideas, while delivering a software infrastructure that carried a number of the ideas to a successful conclusion. This was not an accident—at base, Postgres was designed for extensibility, and that design was sound. With extensibility as an architectural core, it is possible to be creative and stop worrying so much about discipline: you can try many extensions and let the strong succeed.

Prediction of causal genes at GWAS loci with pleiotropic gene regulatory effects using sets of correlated instrumental variables (https://arxiv.org/abs/2401.06261)

I wrote about Mendelian randomization a while ago. This paper is about its multivariate extension in the context of causal inference using instrumental variables in a study of genome-wide significant GWAS loci. When the instrumental set condition is satisfied, direct causal effects of all exposures in the model on the outcome variable of interest can be identified.

Asymptotic Online FWER Control for Dependent Test Statistics (https://arxiv.org/abs/2401.09559)

If you ever applied Bonferroni correction (or any stepwise procedure) for multiple comparison, you know that we assume that tests are independent, and in the case of Bonferonni it may be way too much conservative. This is even more complicated in the case of online testing. An alternative approach is proposed in this article, namely an adaptive version of the alpha-spending algorithm used, e.g., in interim analyses, ¹ where we update the p-values depending on whether they are believed to have arisen under the null hypothesis or the alternative.

A Computationally Efficient Approach to False Discovery Rate Control and Power Maximisation via Randomisation and Mirror Statistic (https://arxiv.org/abs/2401.12697) is yet another paper on FDR rather than FWER control. See also Gridsemble: Selective Ensembling for False Discovery Rates for related to estimating local false discovery rates in large-scale multiple hypothesis testing.

Likelihood-ratio inference on differences in quantiles (https://arxiv.org/pdf/2401.10233)

I barely used quantile regression in my previous life (aka when I was working as a full-time statistician), except in a few cases when dealing with growth curves and newborn data. I came across this paper because I’ve been a long time follower of Evan Miller’s posts. He’s actually discussing how to implement a two-sample difference-in-quantile hypothesis test and how to construct confidence intervals based on a likelihood ratio test.

On the visualisation of the correlation matrix (https://arxiv.org/abs/2401.12730)

I like biplot graphical displays, despite their subtleties regarding how rows and/or columns of a correlation or distance matrix are adjusted. In this paper, the author suugests “a weighted alternating least squares algorithm is used, with either a single scalar ad- justment, or a column-only adjustment with symmetric factorization; these choices form a compromise between the numerical accuracy of the approximation and the comprehensibility of the obtained correlation biplots.”

Are the Signs of Factor Loadings Arbitrary in Confirmatory Factor Analysis? Problems and Solutions was also mentioned in my daily arxiv digest. As I am slowly forgetting 15 years of previous work in psychometrics, I find it particularly hard to come back from time to time to such papers. Anyway, I’m happy to see that research continues in this field, especially for old questions like those. Finally, I came across this related work on PCA: Estimating the construct validity of Principal Components Analysis.

Pretraining and the Lasso (https://arxiv.org/abs/2401.12911)

A potentially interesting paper I should read.

In this paper, we ask the question “Can pretraining help the lasso?”. We develop a framework for the lasso in which an overall model is fit to a large set of data, and then fine-tuned to a specific task on a smaller dataset. This latter dataset can be a subset of the original dataset, but does not need to be. We find that this framework has a wide variety of applications, including stratified models, multinomial targets, multi-response models, conditional average treatment estimation and even gradient boosting.
In the stratified model setting, the pretrained lasso pipeline estimates the coefficients common to all groups at the first stage, and then group-specific coefficients at the second “fine-tuning” stage. We show that under appropriate assumptions, the support recovery rate of the common coefficients is superior to that of the usual lasso trained only on individual groups. This separate identification of common and individual coefficients can also be useful for scientific understanding.

♪ Keith Jarrett Trio • Sun Prayer

DeMets, D.L. and Lan, K.K. Interim analysis: the alpha spending function approach. Statistics in Medicine 13(13-14): 1341-1352, 1994. See also Introduction to Conducting Interim Analyses Using Alpha Spending. ↩︎

Cochran-Mantel-Haenszel test

Fri, 21 Jun 2024 20:53:23 +0200

Yesterday I was looking for some implementation of the Cochran-Mantel-Haenszel test in Python. Inasmuch as I like Python for web dev and string processing, I always find it surprising that people prefer to stay in Python rather than use dedicated statistical packages. Anyway, here’s how I went to benchmark the CMH package against R and Stata.

First of all, I started looking at whta’s available in the statsmodels package. I remain impressed by the progress developers have made over the last ten years in terms of the quality, usability and diversity of the procedures available. However, while there’s an homogeneity test for stratified tables (along with the Breslow–Day test), it is limited to 2x2 tables.¹ Soon after I found the CMH package whihc looks like what was I was looking for since it handles extensions of Cochran–Mantel–Haenszel statistics to K-way tables with more than two categories.

Here’s a little benchmark taken from the R on-line help for stats::mantelhaen.test in R:

> Satisfaction <-
 as.table(array(c(1, 2, 0, 0, 3, 3, 1, 2,
 11, 17, 8, 4, 2, 3, 5, 2,
 1, 0, 0, 0, 1, 3, 0, 1,
 2, 5, 7, 9, 1, 1, 3, 6),
 dim = c(4, 4, 2),
 dimnames =
 list(Income =
 c("<5000", "5000-15000",
 "15000-25000", ">25000"),
 "Job Satisfaction" =
 c("V_D", "L_S", "M_S", "V_S"),
 Gender = c("Female", "Male"))))
> ftable(. ~ Gender + Income, Satisfaction)
 Job Satisfaction V_D L_S M_S V_S
Gender Income
Female <5000 1 3 11 2
 5000-15000 2 3 17 3
 15000-25000 0 1 8 5
 >25000 0 2 4 2
Male <5000 1 1 2 1
 5000-15000 0 3 5 1
 15000-25000 0 0 7 3
 >25000 0 1 9 6
> mantelhaen.test(Satisfaction)
 Cochran-Mantel-Haenszel test

data: Satisfaction
Cochran-Mantel-Haenszel M^2 = 10.2, df = 9, p-value = 0.3345

I exported the above dataset as a dataframe with one row per observation as follows:

> dd <- as.data.frame(Satisfaction)
> write.csv(dd[rep(row.names(dd), dd$Freq), 1:3], file = "satisfaction.csv", row.names = FALSE)

I got the same result in Stata:

. import delimited /home/chl/tmp/satisfaction.csv, varnames(1)

. foreach v of varlist income-gender {
 2. encode `v', gen(`v'_)
 3. drop `v'
 4. }

. rename *_ *

. emh income jobsatisfaction, strata(gender) general

Extended Mantel-Haenszel (Cochran-Mantel-Haenszel) Stratified Test of Association

General Association Statistic:
Q (9) = 10.2001, P = 0.3345
Transformation: Table Scores (Untransformed Data)

Then, in Python, it is as simple as the following snippet:

import pandas as pd
from cmh import CMH

df = pd.read_csv("satisfaction.csv")
r = CMH(df, 'Income', 'Job.Satisfaction', stratifier='Gender')
r


 Cochran-Mantel-Haenszel Chi2 test

"Income" x "Job.Satisfaction", stratified by "Gender"

Cochran-Mantel-Haenszel M^2 = 12.29047, dof = 9, p-value = 0.1974

And so it begins. The test statistics and their respective p-values do not match. Querying the online help from IPython suggests there’s an adjustment option, with no indication of what it means actually. I thought it would maybe related to some kind of continuity correction² and I went right away check the source code on GitHub. Sadly, there was not adjustment parameter in the GH code, although there’s one in my local package (installed via pip):

# -%<---------------
 # Create stratified contingency tables
 for k in range(K):
 cat = df[stratifier].cat.categories[k]

 subset = df.loc[df[stratifier] == cat, [var, outcome]]
 xs = pd.crosstab(subset[outcome], subset[var], dropna=False)
 contingency_tables[:, :, k] = xs + adjustment
# ->%---------------

Since the author claims his code is based on Agresti (Categorical Data Analysis, 2002), I should have checked the textbook since it was long time since I didn’t open it, and compare the code with the suggested algorithm. But then I came to other interesting problems, and I think I’ll get back to this later.

♪ New Order • Blue Monday

It’s worth noting that they apply a continuity correction by default. ↩︎
If I’m not mistaken, Fisher-Yates correction only applies in the case of 2x2 tables. ↩︎

Scheming in Vim (con't)

Mon, 17 Jun 2024 21:21:32 +0200

Scheme, language server & Co.

Mon, 03 Jun 2024 14:32:17 +0200

So far, my setup for Scheme was pretty simple, and it mostly targeted Chicken Scheme. Recently, I wanted to use Chez Scheme a bit more, and I was looking for a better UX in Neovim. I’ve tried to configure several REPL-like features for Scheme, including Jupyter kernels, swank or language servers. Only the later is really of importance to me since I already get REPL features from slimv, albeit limited to MIT Scheme.

chicken-lsp-server: easy to install but doesn’t quite work
r7rs-swank: I wasn’t able to get the Chicken or Chez swank servers to run (moreover, for Chicken, you have to install additional libraries, which are not compatible with Thunderchez naming convention for, e.g., SRFIs)
scheme-langserver: didn’t work (I grabbed the run executable released on GitHub, but I also tried to compile all the stuff myself, which led me to try to install chez-exe, after which I gave up).
swish-lint: it works smoothly although there’s no hover information

I had to compile and install Chez Scheme v10 in order to compile swish and get ride of minor error message in the log file for swish-lint. But it didn’t help getting hover information to work with Chez. Note that compiling csv10 is just a matter of configure && make and you’re done in a few minutes with a working install; only make test takes a lot of time.

I am currently trying out molten-nvim (without image support and other fancy settings) to evaluate code snippets for which I have associated Jupyter kernels. This mostly mimic my old Magma setup that I abandoned in the meantime. It remains quite non-obtrusive, and I can still fire up a terminal with the appropriate shell (mostly R, Python and Stata) and get my poor-man REPL features (i.e., send current line or selection to the other split window) to work right away. Anyway, here’s the list of kernels I have on my HD:

aliquote master? 1h21m9s +
» jupyter kernelspec list
Available kernels:
 clojupyter-0.4.325 /home/chl/.local/share/jupyter/kernels/clojupyter-0.4.325
 common-lisp /home/chl/.local/share/jupyter/kernels/common-lisp
 ir /home/chl/.local/share/jupyter/kernels/ir
 python3 /home/chl/.local/share/jupyter/kernels/python3
 racket /home/chl/.local/share/jupyter/kernels/racket
 stata /home/chl/.local/share/jupyter/kernels/stata
 wolframlanguage14 /home/chl/.local/share/jupyter/kernels/wolframlanguage14

And here are the settings I use in my init.lua file:

vim.g.molten_auto_open_output = false
vim.g.molten_auto_image_popup = true
-- vim.g.molten_output_win_style = "minimal"
-- vim.g.molten_use_border_highlights = true
-- vim.keymap.set("n", "<localleader>ss", ":MoltenEvaluateOperator<CR>", { silent = true })
vim.keymap.set("n", "ss", ":MoltenEvaluateLine<CR>", { silent = true })
vim.keymap.set("v", "ss", ":<C-u>MoltenEvaluateVisual<CR>gv", { silent = true })

And here is an example of Racket code evaluated using Molten (left) or in a terminal directly (right):

There’s not much support for Scheme languages in the existing Jupyter kernels, except for MIT Scheme, ¹ but there’s one for Common Lisp which actually works. However, I should say that in this case slimv is way better. Slimv also supports Scheme and Clojure, although Racket support is pretty outdated. As for replacing MIT Scheme with Chez Scheme, no luck here.² As I said above, r7rs-swank which looks promising isn’t working for me.

The following screenshots show slimv and swish-lint working in combination on a Scheme file, providing a REPL and a handy shortcut to evaluate a defun, as well as auto-completion and linting.

♪ Grateful Dead • Loser

I haven’t tried Calysto Scheme, though. ↩︎
Also, note that by default Chez Scheme is installed as scheme under /usr/bin, and slimv expects MIT Scheme to be available under the same executable name. Using default settings work perfectly well since newer versions of MIT Scheme bundle a swank server. However I do not want to update alternatives for Scheme on Ubuntu once again, so I ended up overwriting the default Scheme implementation in my slimv copy. ↩︎

Micro posting in April

Tue, 30 Apr 2024 21:56:33 +0200

2024-04-29: I just tried musikcube as an alternative to cmus. The UI is really great, but I fear I will miss some cmus features, like having different views for albums and playlists, filters, and so on.
2024-04-29: How I stopped worrying and loved Makefiles.
2024-04-29: The history of the ROC curve: Interesting read which highlights how it is worth double checking Wikipedia with historical records.

Micro posting in March

Thu, 04 Apr 2024 09:18:35 +0200

2024-03-18: Writing Python like it’s Rust. Or how to make use of Python types in a meaningful way.
2024-03-25: I have almost two years of backup data for my home directory, thanks to kopia. It works great, it runs smoothly and doesn’t harm the RAM or disk, and it wakes up nicely when the laptop returns from idle.

chl@aliquote:/home/chl
2022-10-31 21:00:37 CET k105eb1435351bfdfba85874cd06ebf75 14.9 GB drwxr-xr-x files:226411 dirs:45856 (monthly-18)
2022-11-30 22:00:39 CET k01dd605c67ca579cbec33fb5eb386648 15.5 GB drwxr-xr-x files:232391 dirs:46879 (monthly-17)
2022-12-31 23:00:38 CET k5a01672b76af59461061c3fe25003785 15.7 GB drwxr-xr-x files:234220 dirs:47516 (monthly-16,annual-3)
2023-01-31 21:00:13 CET kec7a24dd874b20376f160f1fe3e0acd8 15.4 GB drwxr-xr-x files:226071 dirs:45741 (monthly-15)
2023-02-28 21:00:08 CET kc707921ee53be6aa7ff2be4da99387c4 14.6 GB drwxr-xr-x files:228097 dirs:46041 (monthly-14)
--%<-----------------
2024-03-25 08:12:51 CET k9cb5afcc7ce355e2b64c8f980ef6bb24 14.1 GB drwxr-xr-x files:218701 dirs:35849 (hourly-11)
2024-03-25 09:00:14 CET kdec17d1a1ed35833b0eed730c491ed87 14.1 GB drwxr-xr-x files:218716 dirs:35849 (latest-10,hourly-10)
2024-03-25 10:00:35 CET ka52066cb5588c88a7d259140e5c02683 14.1 GB drwxr-xr-x files:218702 dirs:35849 (latest-9,hourly-9)
2024-03-25 11:00:35 CET kd96a4c841b76c75e4b5b569594f9e87b 14.1 GB drwxr-xr-x files:218704 dirs:35849 (latest-8,hourly-8)
2024-03-25 12:00:35 CET k0ec385fbd40ce68a14250da3e4e72ac9 14.1 GB drwxr-xr-x files:218710 dirs:35849 (latest-7,hourly-7)
2024-03-25 13:00:35 CET k635fb4cf4fd0e3f4c45c58b7a14ed690 14.1 GB drwxr-xr-x files:218730 dirs:35850 (latest-6,hourly-6)
2024-03-25 14:00:35 CET k722ecc32f98028829e258d2582548281 14.1 GB drwxr-xr-x files:218741 dirs:35851 (latest-5,hourly-5)
2024-03-25 15:00:35 CET k87f64132d71b14eccbc7f5bb362b53aa 14.1 GB drwxr-xr-x files:218732 dirs:35851 (latest-4,hourly-4)
2024-03-25 18:07:22 CET k4af30387d39daf1dd1b3002dffde3543 14.1 GB drwxr-xr-x files:218733 dirs:35851 (latest-3,hourly-3)
2024-03-25 19:00:35 CET k66f7a97a2b50611e047524da6cc0e640 14.1 GB drwxr-xr-x files:218745 dirs:35851 (latest-2,hourly-2)
2024-03-25 20:26:20 CET kbd44c2958916a894deab100da949f302 14.1 GB drwxr-xr-x files:218745 dirs:35851 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)

2024-03-25: ♪ Flock of Dimes · Given/Electric Life
2024-03-26: "When the underlying measurements are normally distributed, you can "convert" a range measurement into an estimate for the standard deviation" using the [D2 constant](https://blogs.sas.com/content/iml/2024/02/21/range-d2.html). As always, Rick Wicklin wrote a very interesting article for those involved in QC or dealing with small samples. See also [On using flexible distributions to fit data](https://blogs.sas.com/content/iml/2024/02/26/fit-flexible-distribution.html) for another interesting story.
2024-03-26: Lasagna were tasty last week!

2024-03-26: Some estimators provide reasonable estimates even when there are correlations among observations, but estimatesof uncertainties of estimates can be badly affected by correlations. --- [What Does a Statistical Method Assume?](https://fharrell.com/post/assume/).

Micro posting in January

Tue, 30 Jan 2024 08:03:15 +0100

2024-01-08: I still hold 14 draft posts in my inbox, but for now let’s start this fresh new year with some good vibes.

A Handful Of Memories de Adam Wakeman
2024-01-08: It looks like this site renders properly on sixel-aware terminal with w3m (w3m -sixel -o auto_image=TRUE https://aliquote.org):

2024-01-11: A simplified boxplot, with (much) fewer Stata commands than the one proposed in this old post:

sysuse auto
graph box mpg, over(foreign) box(1, color(white)) ///
 medtype(marker) medmarker(mcolor(black) mlwidth(0)) ///
 cwhiskers alsize(0) lines(lpattern(solid) lwidth(medium)) ///
 ylabel(, nogrid) yscale(noline) ///
 title("{bf}Box Plot", pos(11) size(2.75)) ///
 subtitle("City Mileage over number of cylinders", pos(11) size(2))
graph export /home/chl/tmp/better-bxp.eps

2024-01-15:

No, really, I’m astonished at how much software is basically useless without an internet connection. Net is no longer something additional to your daily tasks, it is essential for your daily tasks. — We’ve become overdependent on the net

2024-01-15:

What strikes me about my personal experience with LLMs is that I have learned precisely when to use them and when their use would only slow me down. I have also learned that LLMs are a bit like Wikipedia and all the video courses scattered on YouTube: they help those with the will, ability, and discipline, but they are of marginal benefit to those who have fallen behind. I fear that at least initially, they will only benefit those who already have an advantage. — LLMs and Programming in the first days of 2024

2024-01-15: Wanna get a plain text list of your Pocket bookmarks? Go to getpocket.com/export, then apply the following snippet on the HTML result file:

cat YOURFILE.html | grep -o '<a .*href=.*>' | sed -e 's/<a /\n<a /g' | sed -e 's/<a .*href=['"'"'"]//' -e 's/["'"'"'].*$//' -e '/^$/ d' > pocket.txt

2024-01-15: ♪ Jelly Cleaver · Black Line
2024-01-16:

When optimising code, never guess, always measure. — When Optimising Code Measure

2024-01-16: Lots of tips and tricks for Mathematica aficionados.
2024-01-16: TIL about Asahi Linux. Looks cool, although Ubuntu LTS already runs fine on my Mac, but it’s an old one, not one of the newest cool kids made in Silicon.
2024-01-16:

2024-01-22: ♪ Kendra Morris · Dry
2024-01-24:

This difficulty is often not due to math - because of the aforementioned frameworks machine learning implementations do not require intense mathematics. An aspect of this difficulty involves building an intuition for what tool should be leveraged to solve a problem. This requires being aware of available algorithms and models and the trade-offs and constraints of each one. — Why is machine learning ‘hard’?

2024-01-24: I recently switch from MC 1.19.4 with Forge and Optifine to MC 1.20.4 with Iris and Sodium. Now I got a lot more mods available with less frictions overall. And it looks like we can now play MC on a laptop with texture packs and shaders all time long.It’s probably time to stop building on 1.19.4 I guess, and just start a new survival journey.

2024-01-24: TIL about Faircamp which looks amazing.
2024-01-24: Time Machine like Backups on OpenBSD.
2024-01-25: ♪ Passenger · Summer Rain

Managing Scheme libraries

Thu, 18 Jan 2024 20:42:11 +0100

A quick note on the packaging system for some of the Scheme implementations I use. It is rather incomplete at this point, I will eventually update it later.

Recently I’ve been looking at Chez Scheme documentation and various places on the intertubes to learn how to create or manage Scheme libraries. I’m used to Racket and Chicken, which both ship with a great package manager, something that is currently lacking in some of Scheme distros, especially Chez Scheme. Unlike Quicklisp, you don’t end up with every version of a single package once you update your package list, although I understand why it may be interesting to keep a copy of older packages, especially when they are linked to a working repository.

With Chicken, we usually add repositories to the exported variable CHICKEN_REPOSITORY_PATH. We can also check where system-wide is installed with chicken-install -repository. User-compiled modules usually go under $HOME/.cache/chicken-install, but I always install modules as root. Be sure to check the man page for chicken-install in any case. Racket has its own conventions to manage modules (see also Beautiful Racket); likewise, there are well-defined steps to create your own package and make it available from Racket REPL or programs. There are some package templates for Racket, but see Tutorial: Creating a Package.

As regards Chez Scheme, it seems that the best way to define a local repository for your package is via an exported variable:

[ -d "$HOME/.local/lib/scheme" ] && export CHEZSCHEMELIBDIRS="/home/chl/.local/lib/scheme:"

We can check that everything is all right as follows:

» chezscheme
Chez Scheme Version 9.5.4
Copyright 1984-2020 Cisco Systems, Inc.

> (library-directories)
(("/home/chl/.local/lib/scheme"
 .
 "/home/chl/.local/lib/scheme")
 ("." . ".")
 ("/usr/share/r6rs" . "/usr/share/r6rs"))
>

I also heard of Akku, after reading one of Travis Hinkelman’s posts, I even installed on my HD but I never used it.

♪ Joy Divsion • A Means to an End

Book review: Mathematica (2)

Sun, 14 Jan 2024 19:53:37 +0100

As a sequel of this previous post, here are two other books on Mathematica that I read this year.

An Engineer’s Guide to Mathematica (Magrad, 2014)

This book is organized in two independent parts. Part 1 summarizes Mathematica syntax. It covers basic math expressions, but also strings and quantities, which were introduced as Quantity in Mathematica 9. The author then describes how to manipulate lists and arrays, and how to define pure functions or custom modules. Symbolic and numerical mathematical applications are discussed in chapters 4 and 5, while the last two chapters deal with static and interactive graphics.

Part 2 is about engineering applications, but I was only interested in chapter 9 which deals with statistics, specifically probability distribution – and to estimate best-fitting distribution parameters, and linear and non-linear regression. Surprisingly, there’s even a section on factorial ANOVA

Except for the first chapters, explanations are rather scarce and the author usually just shows sample instructions. It’s up to you to adapt the commands and options to your own problem. Good thing is that the author provides many summary tables of commands and/or options. Also, several exercises are proposed at the end of each chapter, which I find a great way to practice all the packed information. Solutions are available on request.

Essentials of Programming in Mathematica (Wellin, 2016)

This book also features introductory chapters which I found more interesting since the author explains the how and why of each command. This is quite normal after all since the emphasis of this book is on programming rather than using dedicated functions for specific applications. I said this book is full of explanation and it also applies to small programs developed along the way, e.g., the sieve of Eratosthenes in chapter 6.

Once your code is written, your job as a programmer is not done. You must test your code under both “regular” and “abnormal” conditions, that is, using what you think is typical input and also testing the code when given pathological or atypical input. You should include tests for correctness and efficiency.

The chapter on visualization is also quite interesting since it describes some not so usual displays (dot plots, path plots). The last two chapters are especially useful if you’re a serious Mathematica developer, which I’m certainly not, and I found it particularly interesting as a complement to this already brilliant list of Mathematica resources. At the moment, this is certainly the book I would recommend to learn the intrinsics of Mathematica and to develop efficiently with Mathematica notebooks (there’s no mention of Wolframscript).

♪ The Jesus and Mary Chain • Down On Me

How many permutations

Thu, 11 Jan 2024 11:53:15 +0100

Usually, we would determine the number of permutations required to carry out a Monte Carlo permutation test based on the precision we wish to attain for the p-value. In fact, Stata provides a confidence interval for the estimated p-value when you use the permute procedure. This confidence interval is based on a binomial proportion (this is what we are estimating after all). With the example discussed in a previous post, converted as a tabular dataset, here is a sample run from Stata 13:

. insheet using how-many-permutations.csv
. gen i=_n
. reshape long v, i(i)
. permute _j r(mu_1), reps(10000): ttest v, by(_j)
// -%----
Monte Carlo permutation results Number of obs = 20

 command: ttest v, by(_j)
 _pm_1: r(mu_1)
 permute var: _j

------------------------------------------------------------------------------
T | T(obs) c n p=c/n SE(p) [95% Conf. Interval]
-------------+----------------------------------------------------------------
 _pm_1 | 14.48 5229 10000 0.5229 0.0050 .513055 .5327317
------------------------------------------------------------------------------
Note: Confidence interval is with respect to p=c/n.
Note: c = #{|T| >= |T(obs)|}

Note that we get a p-value close to the one estimated using Lisp or R. It is also possible to carry out an exact permutation test using the enumerate option. Philip Good provides illustrations with various statistical software in chapter 3 of one of his textbook.¹

The approximate 95% CI for a proportion estimated from a sample of size $n$ is

$$ \hat p \pm 2\sqrt{\frac{\hat p(1-\hat p}{n}}, $$

which can be approximated as $\hat p \pm \sqrt{\hat p/n}$ for small $\hat p$. As an example, to get a precision of two decimal places for a proportion of 0.01, we would need $n > 1600$ rearrangements ($0.01 > 4\sqrt{0.01/n}$, then solve for $n$). This is probably why the rule of thumb is to use 1000 to 10000 permutations, when not “use the maximum number of permutations”. Of course, this only applies to Monte Carlo re-rerandomization, since in the case of exact permutation tests you get the “exact” p-value you get, which is tied to your sample size. E.g., with five observations you only have 32 possibilities and 1 extreme observation out of 32 means a p-value of 0.03125.²

And here is yet another way to carry out a permutation test, although it is suboptimal (in many ways). This time, we will simply shuffle the whole sequence of observation and take the first half part of the sequence. Again, we will rely on a Monte Carlo approach and only carry out a limited number of random draws from the original sample. Last but not least, we will replace Scheme with Clojure.

As before, we will consider the sum of the observations in one group (the original, then the randomized one) as our test statistic. Other solutions such as taking the difference in means or the detrended Student t statistic are also possible.

(def xs [12.9 13.5 12.8 15.6 17.2 19.2 12.6 15.3 14.4 11.3])
(def ys [12.7 13.6 12.0 15.2 16.8 20.0 12.0 15.9 16.0 11.1])
(def xy (concat xs ys))
(def stat0 (apply + xs))
(def k 1000)
(defn stat [x] (apply + (take 10 (shuffle x))))
(def pdist (repeatedly k (partial stat xy)))
(def events (count (filter (fn [x] (> x stat0)) pdist)))
(double (/ events k))
;; => 0.525

Run time is quite decent in this case, even for large k. Now let’s check the distribution of the permuted test statistic? Here’s one with 10000 iterations (p=0.5156) and the venerable Gnuplot backend.³

set terminal png size 600, 300
set output "~/tmp/pdist.png"
set xrange [130:165]
set yrange [0:7]
unset key
unset ytics
set arrow 1 from 144.8,0 to 144.8,0.8 lc "orange" nohead
set label "Original test statistic" at 145,0.75 tc "orange"
plot "pdist.dat" u 1:(1/100.) s kdens bandwidth 1 lw 2

Overall, I was pleasantly surprised by how easy it was to perform this task in Clojure by using builtin stuff only.

♪ Happy Mondays • Olive Oil

Phillip Good (2005) Resampling Methods: A Practical Guide to Data Analysis, Boston: Birkhäuser ↩︎
Another approach to devise the number of Monte Cralo experiments to run would be to consider the worst situation $p=0.5$, in which case we find that for $n=10000$ and $n=100000$ we get a standard deviation of 0.005 and 0.0016. Hence with $n=100000$, the most pessimistic 95% confidence interval on the $p$-value has width 0.0031 = 0.31%. See Olivier Thas, Comparing Distribution (Springer, 2010), § 7.1.2.3. ↩︎
Not sure how to properly export the lazy sequence into a text file, so I’ll be using (spit "/home/chl/tmp/pdist.dat" (vec pdist)) but I need to remove the enclosing bracket afterwards and replace space with carriage return to please Gnuplot. The final file is available here. ↩︎

Probabilistic approximations of Pi

Wed, 10 Jan 2024 13:07:10 +0100

In his Mathematical Recreations and Essays, W.W. Rouse Ball mentions two ways to estimate the value of of $\pi$ to a few digit places “using probability.” I would say this is more of a frequentist estimation problem than a purely probabilistic method since it involves collecting experimental data.

The first approach considers a number of equidistant parallel straight lines lying on a plane, distance apart $a$; if we drop a stick of length $l < a$ on to the plane, then the probability that it will fall so as to lie across one of the lines is $2l / \pi a$. This means we can estimate this quantity, and therefore $\pi$, by repeating this experiment a large number of times. For instance, $\pi = 3.1419$ was found with 1120 trials by Captain Fox in 1864.¹

The second approach relies on prime numbers: If two numbers are written down at random, the probability that they will be prime to each other (i.e., co-prime) is $6/\pi^2$. Here is a little simulation in Mathematica that considers the first 250 positive integers to mimic the author’s report of an experiment in which each of 50 students wrote down 5 pairs of numbers at random, where it was found that 154 of the pairs were found to consist of co-prime numbers:

In[14]:= Total[Boole[Array[CoprimeQ, {250, 250}]], 2]/250^2 // N
Out[14]= 0.608752
In[17]:= Sqrt[6/%]
Out[17]= 3.12094

Assuming you have enough RAM, we would get closer with a larger sample size:

In[18]:= Total[Boole[Array[CoprimeQ, {10000, 10000}]], 2]/10000^2 // N
Out[18]= 0.60795
In[19]:= Sqrt[6/%]
Out[19]= 3.14153

Finally, here is a plot of random co-prime numbers in the range $[3;100]$, as suggested in the help desk:

In[15]:= randomCoprime[x_Integer] :=
 RandomChoice[Pick[Range[x], CoprimeQ[x, Range[x]]]];
ListPlot[{#, randomCoprime[#]} & /@ RandomInteger[{3, 100}, 500]]

The above function could be used to draw random integer instead of the first consecutive integers used in the previous simulation.

♪ Happy Apple • Ella by Night

Messenger of Mathematics, Cambridge, 1873, vol. ii, pp. 113, 114. ↩︎

lost + found 2023

Mon, 08 Jan 2024 16:19:24 +0100

Party More Party (2023-12-12)

But to my understanding rforest and fastcforest do not produce binary splits following the same procedure. While rforest chooses the feature and the value of the feature that minimizes the MSE in the resulting child nodes, fastcforest instead splits based on a statistical test to reject the null hypothesis of independence between the dependent variable and any of the features. Is that correct?

Bridge sampling (2023-09-18)

R packages:

How Many Bootstrap Replicates (2023-09-04)

The primary difference is that while bootstrap analyses typically seek to quantify the sampling distribution of some statistic computed from the data, permutation analyses typically seek to quantify the null distribution. That is, they seek to break whatever structure might be preset in a dataset, and quantify the kinds of patterns one expects to see by chance.

Maarten Buis on Stata list, https://www.statalist.org/forums/forum/general-stata-discussion/general/293546-bootstrap-number-of-replications

Stata’s bssize

Andrews, D. W. K. and M. Buchinsky. 2000. A three-step method for choosing the number of bootstrap repetitions. Econometrica 68: 213-251.
Poi, B. P. 2004. From the helpdesk: Some bootstrapping techniques. Stata Journal 4: 312-328

Racket and its sqlite API (2023-02-18)

Everyone has its bindings for splite these days because, well, sqlite is great. There are two main packages for Racket that I know about: db and sqlite-table.

♪ Bauhaus • Bela Lugosi’s Dead

Algorithms for statistical computing

Mon, 08 Jan 2024 12:44:33 +0100

Here are very short reviews of books I have read or reread over the years, in no particular order.

Foundations of Statistical Algorithms (Weihs, Mersmann & Ligges, CRC Press 2014): I read this book three or four years ago, and I found it interesting although quite heavy on math. It includes a chapter on verification which is close to an experimental design of testing, as the authors said. Apart from one or two papers about Stata or SAS own verification routines and Numerical Issues in Statistical Computing for the Social Scientist, by Altman, Gill and McDonald (Wiley, 2004) that I mentioned in this post, there are few books dedicated to this topic, AFAICT. Here, the authors focus on Linear Least Squares Solvers and the use of condition numbers. Interestingly, Partial Least Squares is mentioned on several occasions, both from an algorithmic perspective (chapter 5) and when dealing with model selection in multivariate regression settings. Convex problems like coordinate descent and gradient-based or Quasi-Newton methods are discussed at length in chapter 4.
Algorithms for Data Science (Steele, Chandler & Reddy, Springer 2016): This is more a textbook about data analysis per se rather than a summary of specialized algorithms for statistical computing, so I was a bit disappointed after a few chapters.
Numerical Methods of Statistics (Monahan, 2nd ed. Cambridge University Press 2011): I studied most of R code (and I did some Fortran as well!) that comes along this textbook, and I learned a lot from the extended discussion and exercises. I long considered it the Numerical Recipes (in R) of statistics.
Elements of Statistical Computing (Thisted, Chapman and Hall 1988): One of the first textbook I used to teach statistical computing using R. Although it is a bit dated, it remains very relevant for most tasks at hand, and it features several algorithms that help sharpen our understanding of common implementation of, e.g., least-squares, bracketing methods in non-linear models, QR and SVD decomposition, or density estimators and smoothers.
Computational Statistics (Givens & Hoeting, Wiley 2nd ed. 2013): This is the book I read after the one written by Thisted. I seem to remember coming across it after seeing one of Chris Fonnesbeck or Radford Neale’s (or both, maybe) course outline. It features many additional topics: markov chains, genetic algorithms and more generally dynamic programming, a full chapter on the bootstrap, and several sections on multivariate methods. Overall, it is more geared toward optimization and bayesian inference. R code is available for this 2nd edition.
Computational Statistics (Gentle, Springer 2009): Christian Robert already wrote a review of this textbook. I have read several book written by Gentle, but this one remains my reference when both numerical computing and computational statistics are of interest.

♪ The John Butler Trio • Life Ain’t What It Seems

Micro posting in December

Sat, 30 Dec 2023 09:50:15 +0100

2023-12-06: A good explanation of how Shazam work in practice, and an implementation in Python.
2023-12-06: iMessage, explained: When you learn what com.apple.madrid does on your machine, and all the other goodies Apple put into their APNs.
2023-12-11: ♪ Jelly Cleaver · Black Line
2023-12-12:

To choose an implementation is to choose a tribe. Since Scheme is so minimal, you begin to rely on extensions that are only present in your implementation, and so through code you bind yourself to a world of code, people, and practice, loosely bound to the rest of the Scheme world through a fictional first-person-plural. This is OK! Going deep into a relationship with an implementation is the only way to do great work. The looser ties to the rest of the Scheme world in the form of the standards, the literature, the IRC channel, and the mailing lists provide refreshing conversation among fellow travellers, not marching orders for a phalanx. — an opinionated guide to scheme implementations

2023-12-13: Just when I was about to try out boosting and bagging in Stata (rforest and boost), I realized that most programs now require Stata 15 or newer. Even Lasso penalization has been integrated in the core modeling facilities starting with Stata 18. At least, I can still run lassopack to perform L1, L2 regularization or a combination thereof. They use the same convention as in glmnet, where the parameter $\alpha$ stands for the elasticnet mixing parameter ($\alpha=0$ corresponds to the ridge penalty while $\alpha=1$ is the lasso penalty). The authors compared Stata builtin and R regularized estimation, which is great: Replication of glmnet and StataCorp’s lasso. See also An introduction to the lasso in Stata on the Stata blog. #stata
2023-12-13: Micro-posting goes well as far as I can tell.
2023-12-18: TIL that you can use xeyes to check if an app is running under Wayland or XWayland.

Permutation test in Lisp

Fri, 22 Dec 2023 09:26:56 +0100

I drafted this post in March, along Wilcoxon test in Lisp, then forgot about it. How about using a permutation test? Since I ran out of time with implementing a proper version in Common Lisp, and get bored of Hollander & Wolfe’s dataset, let’s look at this quick one-liner in R.

(define xs '(12.9 13.5 12.8 15.6 17.2 19.2 12.6 15.3 14.4 11.3))
(define ys '(12.7 13.6 12.0 15.2 16.8 20.0 12.0 15.9 16.0 11.1))

From there on, we need a way to generate all pairwise differences for permuted sequences. Although it is tempting to implement a procedure to permute a list of numbers,¹ and take the first half of the sequence as the first (resampled) sample, we can equivalently generate all combinations of k elements chosen among n ones.² My own implementation in R is shown in another post. Note that I use the unstandardized difference in means, not the test statistic typically computed from a t-test or its non-parametric alternative.

Now, let’s do this in Scheme:

(define (combn k xs)
 (cond ((= k 0) '(()))
 ((null? xs) '())
 (else (append (map (lambda (x) (cons (car xs) x))
 (combn (- k 1) (cdr xs)))
 (combn k (cdr xs))))))

We could mimic the R code above by computing the difference in means between each combination and its complement (i.e. the element of xs or ys not already present in the combination), but let’s simplify the problem a bit: the two samples are of equal size, hence the sum provides the same amount of information as the mean. Furthermore, we only need to compute one of the two sums since the other one can be deduced from the grand sum. Hence the sum of the permuted sequence is a sufficient statistic. Let’s test the above code by computing the sum of each sub-lists formed using 9 elements taken in xy, which results from the concatenation of x and ys:

(define xy (append xs ys))
(map (lambda (x) (apply + x)) (combn (length xs) xy))
;; => (144.8 146.2 147.1 ...

The original value is (apply + xs), and it will be our reference to compute the proportion of values that are at least more extreme on the distribution of permuted test statistics. The answer only requires to modify the previous code a little bit:

(define pdist (combn (length xs) xy))
(define stat (apply + xs))
(define pos (map (lambda (x) (cond ((> (apply + x) stat) 1) (else 0))) pdist))
(exact->inexact (/ (foldr + 0 pos) (length pos)))
;; => 0.514332416809197

(apply + pos) yields a stack overflow in this case, hence the use of right folding. Using a similar approach, I got 0.5210061 from R.

♪ Alina Bzhezhinska & Hip Harp Collective • Meditation

Regarding permutation in Scheme, I would recommend using algorithm from Shmuel Zaks, A new algorithm for generation of permutations (Technical Report 220, Technion-Israel Institute of Technology, 1981), from Programming Praxis prelude (originally written in Scheme). It works well for small n, but not that well when n=20 (because of heap size limit). Should you rather prefer using Lisp, you certainly want to use flet, zerop, and nthcdr in place of local define, zero? and list-tail (and maybe some other substitutes that I have overlooked). ↩︎
See also Combinations from a Set in Chapter 1 of Simply Scheme. ↩︎

Using R from Mathematica

Thu, 21 Dec 2023 18:10:31 +0100

Just a quick note for myself, since I don’t use that often but I may forget it once again (like I did before). The R connector, RLink, was introduced in Mathematica v9, if I remember correctly, it was sometimes a pain in the ass to use it properly, but now things seem to go smoothly once you get the right setup, although you may get some occasional crashed which require that you perform the install step again (this is just the first instruction below).

Working environment: Ubuntu 22.04 LTS, R 4.3.2, Mathematica 13.3.

First, we need to ensure that RLink is actually working (you’ll likely need to install rJava from R, which is a required step to get a working Java link):¹

In[59]:= InstallR["RHomeLocation" -> "/usr/lib/R"]

If everything went fine, you can check which version of R was linked to the active session:

In[60]:= REvaluate["R.version.string"]
Out[60]= {"R version 4.3.2 (2023-10-31)"}

Then, in order to pass data structure from Mathematica to R, we need to use RSet and pass both the Mathematica object and an internal name that will be recognized when subsequently evaluating R code.² We can then use REvaluate or type > at the prompt to get an evaluation cell targeting Julia, Python, Octave, R, and many other DSLs.

Here is a working example where we use bootstrap resampling and distribution fitting using Mathematica builtins and R’s venerable fitdistr from the MASS package.

In[53]:= data={7.05,2.49,16.09,21.3,19.21,14.88,16.41,0.19,9.71,10.18,8.81,
 2.93,18.11,22.2,9.79,14.74,4.54,16.76,8.46,24.9};
 Skewness[data]
Out[54]= -0.0333154
In[42]:= skews=Table[Skewness[RandomChoice[data,Length[data]]],{1000}];
In[44]:= StandardDeviation[skews]
Out[44]= 0.333235

The histogram for the sampling distribution of the skewness (1000 bootstrap samples) is shown on the left, together with the best fitting gaussian PDF:

The density estimator was obtained as follows:

dist=EstimatedDistribution[skews, NormalDistribution[mu,sigma]];
Show[Histogram[skews, Automatic, "PDF"], Plot[PDF[dist, x], {x, -2, 2}, PlotStyle->Orange]]

We can also use FindDistributionParameters to find the MLE assuming a gaussian distribution; moreover, we could rely on bootstrap to perform the same estimation.

In[67]:= FindDistributionParameters[skews,NormalDistribution[\[Alpha],\[Beta]]]
Out[67]= {\[Alpha]->-0.0460534,\[Beta]->0.333068}

Now let’s estimate the parameters of the normal distribution from the data using R:

In[73]:= RSet["skews", skews];
In[74]:= fit = MASS::fitdistr(skews, "normal")
Out[74]= RObject[{RObject[{-0.0460534,0.333068},RAttributes[names:>{mean,sd}]],RObject[{0.0105325,0.00744763},RAttributes[names:>{mean,sd}]],RObject[{{0.000110934,0.},{0.,0.0000554671}},RAttributes[dimnames:>{{mean,sd},{mean,sd}}]],{1000},{-319.53}},RAttributes[names:>{estimate,sd,vcov,n,loglik},class:>{fitdistr}]]
In[78]:= sd(skews)
Out[78]= {0.333235}

There are some convenient functions to get a prettier display of the results, but let’s note that with R the mean and standard deviation are estimated as -0.0460534 and 0.333068, which comparable to the values estimated by Mathematica. Since Rlink is systematically crashing with ggplot2, I will resort on lattice to display an histogram of the data together with a density plot.

In[110]:= REvaluate["library(lattice)"]
Out[110]= {lattice,stats,graphics,grDevices,utils,datasets,methods,base}
In[112]:= print(histogram(~ skews, data = data.frame(skews),
 type = "density",
 panel=function(x, ...) {
 panel.histogram(x, col = "lightblue", ...)
 panel.mathdensity(dmath=dnorm, args=list(mean=-0.0460534, sd=0.333068),
 col = "black", lwd = 3, ...)
 }))

The result is shown in the right panel of the above figure.

♪ Antony & The Johnsons • Bird Guhl

See this old post to learn more about the installation process. ↩︎
There were more convoluted ways to get the job done apparently. ↩︎

The unquantified self #31

Mon, 18 Dec 2023 20:34:48 +0100

Public holidays are coming, four days left. I’m listening to my music with Cmus, and I’m still happy with my full blown TUI environment. That being said, I installed G4Music from Flathub since I was too lazy to recompile it myself, like I did before. I came to eventually appreciate having graphical app running as containers on Ubuntu. Something like Fedora Silverblue, but without Fedora, and not even blue. Next time I install a Linux distro like Ubuntu, I will try to use only Flatpack apps for everything graphical, and compile from scratch the latest version of TUI apps I use the most (neovim, tmux, cmus, irssi, newsboat), for some of them were already seriously outdated at the time I installed them from Ubuntu official depositories. For instance, my neomutt is v20211029.

I made some drawings lately. Ink or charcoal, depending on my mood. I often shoot them using my iPhone, to keep a digital copy and because I often like them better this way. Yesterday, I arranged some of them in a booklet, which will make for a quick and dirty gift for a friend of mine, although I know she will like it.

I’m past my schedule for posting on this blog, partly because I was quite busy at work, and also busy at home cooking or playing Minecraft. We’ve made a lot of progress: most of the buildings are now fitted out, terraforming has gained momentum in the past weeks and we’re still fine-tuning the details in many places of the map. I don’t plan to build other big stuff, except the castle/monastery and 2 or 3 supplementary farmhouses. I will probably make a screencast to showcase everything I built during the last 5 months.

Finally, I’m quite happy with how reading went for the second half of the year since I read about 60 books (out of 85). This does not include technical books. It keeps me in line with previous years, and above all it allows me to disconnect a little. The same applies to drawing and writing, of course.

I was finally able to repair my vinyl turntable after buying a stylus. It took me a while and I had to wait with internet radio (ABC Lounge is quite good actually), my MP3s and bandcamp in the meantime.

There Is No Other... de Isobel Campbell

♪ Alan Pasqua • A Sleeping Child

Stata: A retrospective

Tue, 05 Dec 2023 20:46:57 +0100

I’m afraid I haven’t made much progress on my Stata Starter Kit since I started the original draft. Partly because I’m busy with other interests, partly because I don’t use Stata that much these days (and my paid licence is for Stata 13 MP), and because I give a lot of projects along the way, almost surely as they use to say in mathematical statistics. My bad! So I thought I would write down some notes I took when studying the history of Stata, a few years ago. I warmly recommend reading the proper retrospectives published in the Stata Journal, see below (there are other interesting special volumes, IIRC). Interestingly, I’ve done most of my professional statistical analyses using statistical packages that provide scripting facilities (and not menu-driven interaction), mostly R (90%) and Stata (10%). While I used Stata for real work (I mean, from data cleansing/management to statistical modelling) late, I always kept an eye on data analysis using Stata and I learned a lot from the great collection of books published by Stata Press.

Stata is almost 40 years old. The latest stable release (v18) was released on April 25, 2023. Stata is written in C, but users can access the source code of commands/programs using viewsource. Pre-compiled code cannot be accessed, though. The article Statistical software certification describes the process of certification adopted by StataCorp. Thirty Years with Stata: A Retrospective is a good read, as well as volume 5(1) of the Stata Journal which celebrated the 20 years of Stata. Needless to say, Stata comes with extensive documentation. There are 23 manuals bundled with Stata v13, totalling 11365 pages.¹

Of course there are the 41 commands that every Stata user should know (as of Stata v13), but there are also those 42 commands, originally found in Stata v1.0:

append dir infile plot spool beep do input query summarize by drop label
regress tabulate capture erase list rename test confirm exit macro
replace type convert expand merge run use correlate format modify save
count generate more set describe help outfile sort

As can be inferred, the above commands are mostly concerned with data management for rectangular datasets and regression.

The first version of Stata was a regression package and really nothing more than that. It did a little bit in the way of calculations, and it did some summary statistics, but it was all built around a regression engine. It was written over a one-year period by me initially and by Sean Becketti, who helped me later. I wrote the C code; Sean Becketti helped me a lot with the design. I would say that half of the design is mine and half the design is Sean’s in terms of what the user actually saw. A number of things became available just at that time when we started this project, and it was those things that actually caused the project to start. The first C compiler was available for the PC. — William Gould (A conversation with William Gould, in Thirty Years with Stata: A Retrospective, Enrique Pinzon (ed.), Stata Press 2015)

The plotting system has been entirely reworked starting with Stata v2.0 but you can still get good ol’ looking inline plot, e.g.:

. sysuse auto
(1978 Automobile Data)

. plot mpg weight

 41 +
 | *
 |
 |
 M |
 i | **
 l | *
 e |
 a | *
 g | * *
 e | * *
 | * * *
 ( | ** * *
 m | * * * * **
 p | ** * **
 g | * * * * ** * *
 ) | ** ***
 | * * * * ** * ****
 | * * * *
 | * * ** *** *
 12 + * *
 +----------------------------------------------------------------+
 1760 Weight (lbs.) 4840

Most commands are still available, except modify which has been superseded by replace, while spool, beep and convert have simply been deleted.

I started rewritting part of the above Stata system in Scheme,² and my hope is to provide a Racket #lang stata one day. One day…

♪ Nirvana • Breed

On Linux you can run, e.g., for i in /usr/local/stata/docs/*.pdf; \ do pdfinfo "$i" | grep "^Pages:"; done | awk '{s+=$2} END {print s}' ↩︎
I always considered Stata as primarily column-based even if it’s written in C and not in Fortran (at least, it’s way easier and way more common to use the if than the in statement), which is a perfect fit for list/vector processing. Sean Becketti talks about how useful such a “data rectangle” was back at the time in Thirty Years with Stata: A Retrospective. ↩︎

ArXiving on November 2023

Thu, 30 Nov 2023 12:24:09 +0200

The 4+1 Model of Data Science (https://arxiv.org/abs/2311.07631)

I’m still not a big fan of the term data science (as a discipline), and I much prefer statistician or data engineer, but let’s forget about that for a moment. In this article, Rafael Alvarado offers to extend the consensual view according to which data science lies at the intersection between three main domains of expertise: computer science and technology, mathematics and statistics, and domain knowledge, which I found even more confusing. Specifically, he considers the following four main domains: value, design, systems, and analytics, to which a fifth domain may be added, namely practice. In a related article, his review led him to conclude that the three driving factors of all the buzz around the sexiest job the 21st century are “(1) the long-standing opposition between data analysts and data miners that continues to animate the field, (2) an established definition of the term as the practice of managing and processing scientific data that has been occluded by recent usage, and (3) the phenomenon of data impedance.” Much ado about nothing in my opinion, unless you consider that the volume and nature of data being processed these days implies a major change in terminology. But is it really that important if it’s just a question of terminology? Because the job remains fundamentally the same, after all.

Resampling-free bootstrap inference for quantiles (https://arxiv.org/abs/2202.10992)

A paper I read last year, thanks to Evan Miller, but go read his own blog post which covers pretty all the math. While the authors originally proposed a Poisson model for bootstrapping differences in quantiles, whereby each observation in the original sample is included in each bootstrap sample according to a Poisson distribution with parameter 1, Evan Miller suggested an approximation which involves a Bessel function.

Frame to frame interpolation for high-dimensional data visualisation using the woylier package (https://arxiv.org/abs/2311.08181)

A new paper by Dianne Cook, whose contributions I have always greatly appreciated since I discovered ggobi (the website is still alive apparently). The “grand tour” (and other derivatives) is a method that allows to view data in more than three dimensions using linear projection, essentially by using rotations of a lower-dimensional projection in high-dimensional space. I’ve only seen this implemented efficiently in ggobi, but there may have been other attempts. Frames are stacked together with small interpolation to create a smooth motion. However, projection are usually rotationally invariant (when using geodesic interpolation algorithm) which may not be a desirable property when, e.g., we are looking for non-linear association in a high-dimensional dataset. The present package works along the tourr package.

Visualizing adverse events in clinical trials using correspondence analysis with R-package visae (https://arxiv.org/abs/2101.03454)

Although I’m no longer involved in medical statistics, I keep an eye open on data visualization and applied methodology in the field. This article drew my attention a few weeks ago because it relies on multivariate projection methods, in this case correspondance analysis and biplots. There’s an associated GitHub repository where you will find code (shiny and ggplot) and illustrations. The advantage of biplots as summary displays is that they help highlighting complex relationship (or, in some cases, interactions). For instance, in the associated BMC paper the authors published alongside this preprint, they used biplots to highlight discrepancies between single agent treatments and their combinations with oxaliplatin at all levels of AE classes, which explain most of the variability among treatments.

Identifiable and interpretable nonparametric factor analysis (https://arxiv.org/abs/2311.08254)

Contrary to Gaussian models or deep neural networks, which show poor performance in terms of identifiability, interpretability, and reproducibility, the authors propose to use the “NIFTY framework, which parsimoniously transforms uniform latent variables using one-dimensional nonlinear mappings and then applies a linear generative model.” Briefly stated, such an approach allows to work with a standard linear latent factor model, which is easier to interpret compared to the aforementioned models, which amounts to a Bayesian independent component analysis under certain conditions.

Generating Signed Permutations by Twisting Two-Sided Ribbons (https://arxiv.org/abs/2311.06974)

An interesting review of algorithms available to generate permutations, and specifically signed permutations which are permutations of $[n]$ objects with a $\pm$ sign. In the case of permutations, there are $n!$ solutions, while for signed permutations there are $2^n\cdot n!$ solutions. Not something I am really interested in at the moment, but I’ll keep it in mynd in case I need it. There are a lot of nice illustrations, BTW.

♪ Mini Trees • Cracks in the Pavement

Micro posting in November

Wed, 29 Nov 2023 09:19:35 +0100

2023-11-02: I generated a docset for Purescript with pursuit.docset. However, I’m skeptical about its added value. I’m using Dash docset in Neovim via vim-dasht, and I noticed that oftentimes the online help for simple query like filter or partition are pretty vague. E.g., in the case of filter the very first hits are all about _filter :: forall r a. Lens' { ... } a, while none of the first matches for partition yields the same result as a simple query on Pursuit (filter :: forall a. (a -> Boolean) -> Array a -> Array a).
2023-11-06:

Almost 50, I remain philosophical because I spent about the first ~45 of them living without Youtube. Did that well enough. — https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

2023-11-06: ♪ sean thornton · edith says
2023-11-07: If you get such a garbage when launching Firefox (official deb package, not the crappy snap one) from the command line or as an external browser for Newsboat, try force disabling accessibility as suggested [here](https://bugzilla.mozilla.org/show_bug.cgi?id=1851240).

» firefox

~
»
[Parent 25471, Main Thread] WARNING: invalid (NULL) pointer instance: 'glib warning', file /build/firefox-LOPdVD/firefox-119.0+build2/toolkit/xre/nsSigHandlers.cpp:167
(firefox:25471): GLib-GObject-WARNING **: 11:46:47.027: invalid (NULL) pointer instance
[Parent 25471, Main Thread] WARNING: g_signal_emit_by_name: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed: 'glib warning', file /build/firefox-LOPdVD/firefox-119.0+build2/toolkit/xre/nsSigHandlers.cpp:167
(firefox:25471): GLib-GObject-CRITICAL **: 11:46:47.027: g_signal_emit_by_name: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
[Parent 25471, Main Thread] WARNING: invalid (NULL) pointer instance: 'glib warning', file /build/firefox-LOPdVD/firefox-119.0+build2/toolkit/xre/nsSigHandlers.cpp:167
(firefox:25471): GLib-GObject-WARNING **: 11:46:47.027: invalid (NULL) pointer instance
[Parent 25471, Main Thread] WARNING: g_signal_emit_by_name: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed: 'glib warning', file /build/firefox-LOPdVD/firefox-119.0+build2/toolkit/xre/nsSigHandlers.cpp:167
(firefox:25471): GLib-GObject-CRITICAL **: 11:46:47.027: g_signal_emit_by_name: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed

2023-11-07: Scheme from Scratch by Peter Michaux: intro, and full list of posts. #scheme
2023-11-08:

2023-11-08: Editing Lisp dialect in Neovim like a pro: beside Vim’s builtin (e.g., vib to select inside brackets), paredit.vim (e.g., [[ go to top-level form), treesitter and nvim-treesitter-textobjects lets you navigate around your code easily, and incremental selection is a breeze. I originally used vv (init the selection), va (aka increase or <C-a>) and vx (aka decrease or <C-x>), but now I just use <tab> that other folks often use as an alias for %. #vim

incremental_selection = {
 enable = true,
 keymaps = {
 init_selection = "<tab>",
 node_incremental = "<tab>",
 node_decremental = "<S-tab>",
 },
}

2023-11-08: Learning (Some) Haskell by Building Tetris. #haskell
2023-11-09: Bit Twiddling Hacks.
2023-11-09: Profiling in Haskell for a 10x Speedup: Great read. #haskell
2023-11-10: ♪ Marianne Faithfull · Broken English (12" Remix)
2023-11-13:

During the Internet’s adolescence, words (and thus blogs) were easier to come by, because for the most part, almost all of the interaction we had online was using devices designed for interaction via text. With the shift to phones as a more popular interaction device, the ease of pouring thoughts into a textual medium has decreased. — Blogging Shift

2023-11-13:

Recursive data suggest recursive solutions. This is the key insight you need for understanding recursion. Not only does it make sense once you think about it, it also demonstrates why most other approaches to teaching recursion are essentially incorrect. — How Not to Teach Recursion

2023-11-13:

pytest supports running Python unittest-based tests out of the box. — How to use unittest-based tests with pytest

This is great since it allows me to use the same compiler (pytest) in Neovim, independent of the testing framework.

2023-11-13: Still decent timing…

aliquote master*
» make deploy
[+] Building public files in /home/chl/Sites/aliquote/public
hugo --gc --minify
Start building sites …
hugo v0.92.2+extended linux/amd64 BuildDate=2023-01-31T11:11:57Z VendorInfo=ubuntu:0.92.2-1ubuntu0.1

 | EN
-------------------+-------
 Pages | 4705
 Paginator pages | 381
 Non-page files | 1
 Static files | 1318
 Processed images | 0
 Aliases | 1
 Sitemaps | 1
 Cleaned | 0

Total in 10170 ms

2023-11-13: A Data-Centric Introduction to Computing
2023-11-15:

Human optimism leads us to believe that we can easily know where a program spends most of its time. — Four Kinds of Optimisation

2023-11-15: ♪ Millie Turner · January
2023-11-16:

A computer game taking the shape of instant messaging software. The inherently paradoxical goal of the game requires no small amount of strategy, wit and cunning: The main objective is gaining as large a following of people as possible while simultaneously getting as many people as possible banned from the game. — Explaining the Internet in 2020

2023-11-16:

There are a lot of small websites on the Internet: Interesting websites, beautiful websites, unique websites. Unfortunately they are incredibly hard to find. You cannot find them on Google or Reddit, and while you can stumble onto them with my search engine, it is not in a very directed fashion. — The Small Website Discoverability Crisis

2023-11-16: Environment variables for Wayland hackers.Note to myself:

$MOZ_ENABLE_WAYLAND
Enables Wayland for Mozilla products (notably Firefox) if set to 1. Firefox uses GTK so setting GTK_BACKEND=wayland will force it to run on Wayland regardless of if MOZ_ENABLE_WAYLAND. However, it will default to X on GDK_BACKEND=wayland,x11 unless MOZ_ENABLE_WAYLAND=1 is set.

2023-11-21: TIL that Spotify originally relied on Fisher-Yates algorithm to shuffle their playlists.
2023-11-21: The qeML package: “Quick and easy” wrappers for machine learning. Norm Matloff also published a new book. #rstats
2023-11-21: ♪ New Order · Sub-Culture
2023-11-22:

My general advice for maximizing the battery is only use your battery as much as you need, i.e. charge it as often as possible, and as little as possible. If you need a rule of thumb, stay within 30%-80%. Wouldn’t charging a lithium battery to 80% only defeat the purpose of putting the battery into longer use in the first place?

2023-11-22: Poor man “fd(find)” command:

#!/bin/sh
find . -iname "*$1*" 2>/dev/null

2023-11-22: A Very Subtle Bug, or how to re-enable SIGPIPE in the subprocess child. #python

How I manage my startup shell scripts

Tue, 28 Nov 2023 20:48:45 +0100

I have been using a custom setup to organize my init shell scripts for more more than one year long now, and I believe it’s pretty robust both for local and remote use. The idea came after I read a nice blog post about how best to manage a set of dotfiles (at the time of this writing I can’t find the blog but I will update this post if I came across it again).

The only things you really need to know, beside writing proper shell scripts, is what a login vs. interactive shell is and which init files are loaded in each cases. Briefly, an interactive shell runs in your preferred terminal emulator and this is where you usually type your commands, while a non-interactive process involves some kind of I/O operations which the user can’t interact with. A login shell is used whenever you log in on your system (after the X or Wayland server is started, this is probably the one at the top of all child processes), while a non-login shell is everything else that is started in userland after you logged in, including external process launched by separate programs without user’s action.

My .profile simply reads:

# $HOME/.profile
# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
# umask 022

for file in "$HOME"/.config/profile.d/*.sh; do
 . "$file"
done

I use zsh as my shell interpreter, so here’s my .zprofile:

[ -r "$HOME"/.profile ] && source "$HOME"/.profile >/dev/null 2>&1

[[ $- =~ i ]] && [ -r "$HOME"/.zshrc ] && source "$HOME"/.zshrc

The .bash_profile is identical, so that I can rely on the exact same config to load everything I need independent of the shell I’m using, or more precisely the machine I interact since I do a lot of remote work through ssh where I generally only use Bash and BW colorscheme. In sum, the above scheme allows to manage login shells by sourcing a common set of relevant init files. The .config/profile.d directory follows Bash scripts organization for environment variables (/etc/profile) and application-specific startup files (/etc/profile.d). I currently have three main files there:

» tree .config/profile.d
.config/profile.d
├── 00-path.sh
├── 10-editor.sh
└── 20-shell.sh

The main file is of course 00-path.sh where I set up all relevant paths for libraries and executables on my HD. Since I use Ubuntu on other machines too, this applies equally well to remote server. Here is an excerpt from this file:

# vim: noai:ts=2:

# Fix API server issue with Podman
# https://github.com/containers/podman/issues/13468#issuecomment-1062764637
export DOCKER_HOST="unix:$XDG_RUNTIME_DIR/podman/podman.sock"

# Scheme
[ -d "$HOME/.local/lib/scheme" ] && export CHEZSCHEMELIBDIRS="/home/chl/.local/lib/scheme:"

# Racket
[ -d "/usr/local/racket" ] && PATH="/usr/local/racket/bin:$PATH"

# Node
if [ -d "$HOME/.local/lib/npm" ]; then
 NPM_STORE="$HOME/.local/lib/npm"
 PATH="$NPM_STORE/bin:$PATH"
 MANPATH="${MANPATH-$(manpath)}:$NPM_STORE/share/man"
fi

# global and user's local systems
[ -d "/opt/bin" ] && PATH="$PATH":/opt/bin
[ -d "$HOME/bin" ] && PATH="$HOME/bin:$PATH"
[ -d "$HOME/.local/bin" ] && PATH="$HOME/.local/bin:$PATH"

Nothing’s really exceptional except that I use guarded statements for portability across machines. Same idea for 20-shell.sh, where I setup default config and directories for various applications:

# vim: noai:ts=2:

# Rg
if command -v rg > /dev/null 2>&1; then
 export RIPGREP_CONFIG_PATH="$HOME/.config/ripgreprc"
fi

# Bibtex (for Org and FZF)
# NOTE: Bibtex file hardcoded
if command -v bibtex-ls > /dev/null 2>&1; then
 export BIBINPUTS="$HOME/Documents/notes/:$BIBINPUTS"
 export FZF_BIBTEX_CACHEDIR="$HOME/.cache/bibtex-ls"
 export FZF_BIBTEX_SOURCES="$HOME/Documents/notes/references.bib"
fi

# News
export NNTPSERVER="news.eternal-september.org"

# Python
export VIRTUAL_ENV_DISABLE_PROMPT=1
# export PYTHONDONTWRITEBYTECODE=1

The startup sequence when I log on my computer for the first time is then:

.zprofile -> .profile -> .config/profile.d/* -> .zshrc -> .config/zsh/*

From .zshrc I source various other shell scripts specific to Zsh (e.g., completion) or common with Bash (alias and functions). When I’m on ssh or on virtual console, I disable some extra stuff like Starship prompt or colors.

if [ $XDG_SESSION_TYPE = "tty" ]; then
 unset LS_COLORS
 export TERM=xterm-mono
 export NO_COLOR=1
 alias ll="ls -alF"
 PROMPT="%~ %(!.#.%%) "
else
 eval "$(starship init zsh)"
fi

TRAPEXIT() {
 if [[ ! -o login ]]; then
 . ~/.zlogout
 fi
}

The .zlogout file takes care of cleaning the mess introduced by some programs, who insists on creating various directories here and there.

That’s it!

♪ Nick Cave and the Bad Seeds • Wonderful Life

Revisiting Random Forests

Wed, 22 Nov 2023 10:40:52 +0100

I’ve been a heavy user of Random Forests in the past (both for applied work and theoretical foundations of ensemble learning and bagging techniques). I very much like Leo Breiman’s work, and I warmly recommend you to take a look at his WALD lectures if you haven’t already done so. I’ve lost track of all the packages I tried back at the time (mostly R and C), and recently I came across Norm Matloff’s new package, qeML, which provides standard R code to perform ML workflow in seamingless fashion (much like caret did once upon a time, but it looks like folks are using pipe-oriented tidy stuff these days).

We could use the Boston Housing dataset from Breiman & Cutler, available as boshouse.txt.¹ It features 13 numerical variables for a total of 506 observations. However, it’s missing the outcome (median value of owner-occupied homes). Let’s grab the data from the mlbench package. Here’s how I would usually do using R, without splitting the dataset into a training and test sample:

library(randomForest)
data(BostonHousing, package = "mlbench")
## Codebook
## --------
## CRIM: Per capita crime rate by town
## ZN: Proportion of residential land zoned for lots over 25,000 sq. ft
## INDUS: Proportion of non-retail business acres per town
## CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
## NOX: Nitric oxide concentration (parts per 10 million)
## RM: Average number of rooms per dwelling
## AGE: Proportion of owner-occupied units built prior to 1940
## DIS: Weighted distances to five Boston employment centers
## RAD: Index of accessibility to radial highways
## TAX: Full-value property tax rate per $10,000
## PTRATIO: Pupil-teacher ratio by town
## B: 1000(Bk — 0.63)², where Bk is the proportion of [people of African American descent] by town
## LSTAT: Percentage of lower status of the population
## MEDV: Median value of owner-occupied homes in $1000s (*outcome*)
set.seed(101)
m <- randomForest(medv ~ ., data = BostonHousing, importance=TRUE)
print(m)
Metrics::mae(BostonHousing$medv, predict(m))

I got 88.18% of explained variance with a residual mean square error of 9.98. The mean absolute error is estimated as 2.10. Note that using default settings, this function considered 500 trees, and we didn’t bother optimizing the mtry parameter (or other subtleties, like nodesize and sampsize). Okay, this is just looking at raw results, and we could do way better for reporting purpose, see, e.g., this tutorial. Now, with qeML, we can perform the above steps using a one-liner:

library(qeML)
m <- qeRF(BostonHousing, "medv")
m$trainAcc
# replicMeans(100,"qeRF(BostonHousing, 'medv')$testAcc")

The accuracy on the training set (to keep close to the previous analysis which did not consider an holdout sample) again relies on the averaged prediction for the outcome (mean absolute prediction error) is now 1.17. The last expression will take longer since it runs the RF algorithm 100 times, but you’ll get more a reliable estimate of prediction accuracy on a test sample. There are some interesting vignettes on CRAN if you want to learn more about prediction using ML methods, feature selection, class imbalances, etc.

Last but not least, did you known Random Forests can be fitted directly in Common Lisp? We will first need to convert the R dataset to a plain text file (write.table(format(transform(BostonHousing, chas = as.numeric(chas) - 1), digits = 2, nsmall = 1, scientific = FALSE), file = "boshous.txt", sep = "\t", col.names = FALSE, row.names = FALSE, quote = FALSE)), load it into SBCL REPL and convert it to a Lisp 2D array. I spent a lot of time debugging a type error in Lisp because I didn’t notice that the chas variable was stored as a factor (with levels {1,2}) which get exported as {0,1}, and not {0.0,1.0} despite the call to format above.

(ql:quickload "cl-csv")
(ql:quickload "parse-number")
(ql:quickload "cl-slice")
(ql:quickload "cl-random-forest")

(defun numlist (lst)
 (mapcar #'parse-number:parse-number lst))

(defun as-array (lst)
 (make-array (list (length lst)
 (length (car lst)))
 :element-type 'single-float
 :initial-contents lst))

(defparameter *data* (as-array (cl-csv:read-csv #P"boshous.txt" :separator #\Tab :map-fn #'numlist)))
(defparameter *X* (cl-slice:slice *data* t (cons 0 13)))
(defparameter *Y* (cl-slice:slice *data* t 13))

(defparameter *rforest*
 (cl-random-forest:make-regression-forest *X* *Y*
 :n-tree 500 :bagging-ratio 0.6
 :max-depth 5 :min-region-samples 5 :n-trial 10))

(test-regression-forest *rforest* *test* *test-target*)

It’s pretty fast and it can even be parallelized using the lparallel library. I wasted enough time with a simple typing error for today, so in a future post, I will try to extend the previous code in order to compute the RMSE using a holdout sample and to plot the fitted values with Gnuplot.

That was quick, but there are many other RF models that you can try from the qeML package. Finally, let’s pay attention to Breiman’s own advice:

RF is an example of a tool that is useful in doing analyses of scientific data.
But the cleverest algorithms are no substitute for human intelligence and knowledge of the data in the problem.
Take the output of random forests not as absolute truth, but as smart computer generated guesses that may be helpful in leading to a deeper understanding of the problem.

♪ Nick Cave and the Bad Seeds • Bring it on

The augmented version of this dataset is also available from the mlbench package, and the original and augmented datasets can be downloaded directly from Statlib. It is no longer available in the de facto standard ML library for Python, though:

>>> from sklearn.datasets import load_boston

ImportError:
`load_boston` has been removed from scikit-learn since version 1.2.

The Boston housing prices dataset has an ethical problem: as
investigated in [1], the authors of this dataset engineered a
non-invertible variable "B" assuming that racial self-segregation had a
positive impact on house prices [2]. Furthermore the goal of the
research that led to the creation of this dataset was to study the
impact of air quality but it did not give adequate demonstration of the
validity of this assumption.

The scikit-learn maintainers therefore strongly discourage the use of
this dataset unless the purpose of the code is to study and educate
about ethical issues in data science and machine learning.

↩︎

A quick glance at the distribution of Pi digits

Mon, 20 Nov 2023 22:20:03 +0100

To follow up on my previous post, here is a quick investigation of the distribution of the digits of $\pi$, using a chi-squared test. Whether $\pi$ is a normal number is out of scope for this toy example. Don Knuth also discussed the use of Pearson $\chi^2$ test for studying random data in Vol. 2 of the TAOCP, Seminumerical Algorithms.

Let’s count how many times each digits (0..9) appears in the first 1000 decimal places of $\pi$, as computed from the preceding post: (Note that we need to discard the very first digit in this case.)

(defun times (lst elt)
 (count elt lst :test #'equal))

(defun explode (num)
 (map 'list #'digit-char-p (prin1-to-string num)))

(defparameter *data* (cdr (explode 31415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989)))

(dotimes (n 10)
 (print n)
 (prin1 (times *data* n)))

;; 0 93
;; 1 116
;; 2 103
;; 3 102
;; 4 93
;; 5 97
;; 6 94
;; 7 95
;; 8 101
;; 9 106
;; nil

This doesn’t seem too incompatible with a hypothesis of equidistribution in which case we would expect a value of 100 for each digit. Let’s run a $\chi^2$ test, which compute the sum of the squared difference between observed and expected counts, standardized by the expected counts:

(defun int-to-float (xs)
 (mapcar (lambda (x) (coerce x 'single-float)) xs))

(defparameter *xs* '(93 116 103 102 93 97 94 95 101 106))
(defparameter *expected* (/ (list-length *data*) (list-length *xs*)))
(defparameter *dof* (- (list-length *xs*) 1))
(defparameter *statistic* (apply #'+ (mapcar (lambda (x) (/ (expt (- x *expected*) 2) *expected*)) (int-to-float *xs*))))
;; => 4.7400002

To get the associated p-value, I will use Larry Hunter’s legacy package, augmented by Jeff Shrager, which can be found, e.g., on GitHub.

(require "asdf")
(asdf:load-asd "/home/chl/Documents/misc/lisp/lhstats/lhstats.asd")
(asdf:load-system :lhstats)
(- 1 (nth-value 0 (statistics::gamma-incomplete (* 0.5 *dof*) (* 0.5 *statistic*))))
;; => 0.8563586405247742d0

And just when I finished drafting this post, I found this related study published by Rick Wicklin on the SAS blog: Analyzing the first 10 million digits of pi: Randomness within structure.

Researchers have run dozens of statistical tests for randomness on the digits of pi. They all reach the same conclusion. Statistically speaking, the digits of pi seems to be the realization of a process that spits out digits uniformly at random.
Nevertheless, mathematicians have not yet been able to prove that the digits of pi are random. One of the leading researchers in the quest commented that if they are random then you can find in the sequence (appropriately converted into letters) the “entire works of Shakespeare” or any other message that you can imagine (Bailey and Borwein, 2013).

♪ New Order • Blue Monday

Computing Pi again

Wed, 15 Nov 2023 21:40:38 +0100

Some time ago, I wrote about computing Pi using dedicated approximating formulae and continued fractions. TIL that the Corman Lisp distribution has an example of computing $\Pi$ to any number of decimal places. The code (MIT license) is shown below:

(defun pi-atan (k n)
 (do* ((a 0)
 (w (* n k))
 (k2 (* k k))
 (i -1))
 ((= w 0) a)
 (setq w (truncate w k2))
 (incf i 2)
 (incf a (truncate w i))
 (setq w (truncate w k2))
 (incf i 2)
 (decf a (truncate w i))))

(defun calc-pi (digits)
 (let* ((n digits)
 (m (+ n 3))
 (tenpower (expt 10 m)))
 (values
 (truncate
 (-
 (+ (pi-atan 18 (* tenpower 48))
 (pi-atan 57 (* tenpower 32)))
 (pi-atan 239 (* tenpower 20)))
 1000))))

It works, and it’s kinda fast for $n < 10^5$!

CL-USER(4): (calc-pi 1000)

3141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117067982148086513282306647
0938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652
7120190914564856692346034861045432664821339360726024914127372458700660631558817488152092096282925409171536436789259036001
1330530548820466521384146951941511609433057270365759591953092186117381932611793105118548074462379962749567351885752724891
2279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005
6812714526356082778577134275778960917363717872146844090122495343014654958537105079227968925892354201995611212902196086403
4418159813629774771309960518707211349999998372978049951059731732816096318595024459455346908302642522308253344685035261931
1881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268
066130019278766111959092164201989

As you may already have guessed, it relies on the use of arctan, which we covered in the aforementioned post.

The following expansion is used: $\sum_{n=0}^{\infty}\frac{(-1)^nx^{2n+1}}{2n+1}$ for $-1<x<1$, and we look for combination of arctan such that $\text{arctan}(1)=\pi/4$. In other words, the formula can be rewritten as:

$$ \pi = 4\sum_{n=0}^{\infty}\frac{(-1)^nx^{2n+1}}{2n+1}\left( \text{something} \right). $$

Smaller “something” means faster convergence.¹

There are some magic numbers in the above code: 18, 57, etc. Where do they come from? There are many formulae, but this one is from Gauss:

$$ \pi = \sum_{n=0}^{\infty}\frac{(-1)^n}{k}\left( 48\left(\frac{1}{18}\right)^k + 32\left(\frac{1}{57}\right)^k - 20\left(\frac{1}{239}\right)^k\right), $$

with $k=2n+1$. Other formulae are available on this French blog post.

Note that in the GitHub repository, you will also find some sieves for prime numbers.

♪ Monodrama • Sarabande

Machin’s formula ($\pi/4 = 4\text{arctan}(1/5) - \text{arctan}(1/239)$) is probably the easiest to use as it amounts to $\pi = \sum_{n=0}^{\infty}\frac{(-1)^n}{2n+1}\left( 4\left(\frac{1}{5}\right)^{2n+1} - \left(\frac{1}{239}\right)^{2n+1}\right)$. ↩︎

Trying out cl-lsp

Wed, 08 Nov 2023 10:51:39 +0100

I tried using roswell as my daily Common Lisp (CL) driver a few years ago, and then got back to good old SBCL for quick scripting. Note that if you rely on Ubuntu apt repos, sbcl is likely outdated (SBCL 2.1.11.debian at the time I write this post). Roswell looks great since it allows us to install multiple flavors and/or versions of CL programs (e.g., sbcl, ccl), but also to use a simple shebang for CL scripts, like the one below:

#!/bin/sh
#|-*- mode:lisp -*-|#
#|
exec ros -Q -- "$0" "$@"
|#

(with-output-to-string (*standard-output*)
 (asdf:load-system :hyperspec))

(defun main (cword &rest argv)
 (declare (ignore argv))
 (format t "~D~%" (hyperspec:lookup (read-from-string cword))))

This allows to fetch the Hyperspec documentation for, e.g., symbol at point if you use it as a keywordprg in (Neo)vim.

Furthermore, there was an implementation of a CL LSP that apparently requires roswell, plus additional dependencies that I could not manage to install with quicklisp alone. Thus I gave up. Today, I reinstalled roswell and started again to compile the whole system to see if I could get cl-lsp working this time.

First, the install instructions are not really clear or probably just incomplete on GitHub. We are told to run ros install lem-project/lem cxxxr/cl-lsp, but this does not quite work. I keep getting a fatal error, even after installing lem-project/micros and lem-project/lem-mailbox as suggested in issue #29:

[1/3] System 'lem' found. Loading the system..
While evaluating the form starting at line 1, column 0
 of #P"/home/chl/.roswell/local-projects/lem-project/lem/lem.asd":Aborted during step [1/3].
Unhandled LOAD-SYSTEM-DEFINITION-ERROR in thread #<SB-THREAD:THREAD tid=180321 "main thread" RUNNING {10013E0003}>: Error while trying to load definition for system lem from pathname /home/chl/.roswell/local-projects/lem-project/lem/lem.asd: Couldn't execute "qlot": No such file or directory

So, qlot is missing, even after installing qlot or fukamachi/qlot. I went to https://qlot.tech/ directly and install it in my home directory, symlinked $HOME/.qlot/bin/qlot to a directory that is already reachable in my $PATH, and this solved all the problems mentioned above. You likely need to add roswell bin directory to your path, e.g. [ -d "$HOME/.roswell" ] && PATH="$HOME/.roswell/bin:$PATH". I did not bother to check if we really need micros and lem-mailbox. So now, in addition to my local quicklisp folder, I have an additional one which is managed by roswell, and additional stuff from qlot in the lem project itself:¹

~/.roswell
» tree -L 1
.
├── archives
├── bin
├── config
├── env
├── impls
├── lisp
├── local-projects
├── src
└── tmp

8 directories, 1 file

~/.roswell
» cd local-projects/lem-project/lem

lem main
» pwd
/home/chl/.roswell/local-projects/lem-project/lem

lem main
» ls .qlot
asdf.lisp bin/ dists/ local-init/ local-projects/ quicklisp/ setup.lisp source-registry.conf tmp/

lem main
» du -sh .qlot
172M .qlot

For testing purpose, I set up Neovim very basically using a filetype-specific init file (ftplugin/lips.lua):

vim.lsp.start({
 name = "cl_lsp",
 cmd = { "cl-lsp" },
 root_dir = vim.fs.dirname(
 vim.fs.find({ "." }, { upward = true })[1]
 ),
})

Here are the server functionnalities obtained from vim.lsp.get_active_clients()[1].server_capabilities:

{
 completionProvider = {
 completionItem = {
 labelDetailsSupport = true
 },
 resolveProvider = true,
 triggerCharacters = { ":" }
 },
 definitionProvider = {},
 documentFormattingProvider = vim.empty_dict(),
 documentHighlightProvider = vim.empty_dict(),
 documentOnTypeFormattingProvider = {
 firstTriggerCharacter = "(",
 moreTriggerCharacter = { ")" }
 },
 documentRangeFormattingProvider = vim.empty_dict(),
 documentSymbolProvider = vim.empty_dict(),
 executeCommandProvider = {
 commands = { "cl-lsp.interrupt", "cl-lsp.eval-range", "cl-lsp.eval-last-expression" }
 },
 hoverProvider = {},
 referencesProvider = vim.empty_dict(),
 signatureHelpProvider = {
 retriggerCharacters = {},
 triggerCharacters = { " " }
 },
 textDocumentSync = {
 change = 2,
 openClose = true,
 save = true
 }
}

Mandatory screenshot:

Formatting does not seem to work correctly (see #26), while hover information is okay. No luck with goto actions (definition, reference), nor with omnicompletion (C-X C-O). I don’t know if it is able to report diagnostics. It doesn’t even produce a log file that we could inspect for diagnostic purpose. Also, sbcl is consuming a significant amount of CPU in the background, at least enough to trigger the fans on my laptop. The process is detached correctly upon quitting Neovim, though. This looked like a really cool project, although slimv or vlime are probably better shaped for interactive Lisp development.

I’m back with my original config: hyperspec via dasht for documentation, and paredit.vim. And I removed all the intermediate junk files right away. That was a fun exercise anyway!

♪ Sad Lovers & Giants • Like Thieves

I understand that this is useful for project-based management of quicklisp libraries. ↩︎

The unquantified self #30

Thu, 02 Nov 2023 20:38:12 +0200

Fresh new month here, November. Not much to say, so this post will be short.

I’ve been fiddling with my TUI config (eh, mostly Neovim) for some days and I feel better, or sort of. At least, it’s now a purely homemade zero useless packages or package manager config. See here and there for more details.

Along the way, I’ve been writing and sketching a bit (charcoal straight away) but it’s all very disjointed.

I watched some nice mini-series recently: 1899 and Bodies. After several series of moderate quality and interest, this was refreshing indeed. I also decided I couldn’t wait for my son to be ready to watch the latest season of The Walking Dead, so I am currently watching it in the evening. I remember when he introduced me to this series: after the first episode I was like “Meh, it will not be possible to go through all those episodes this year or the year after.” Finally, I liked it but I was probably biased since I liked watching it together with my son. Anyway, the show must go on.

Finally, after 3 months off, I’m back with my little city in Minecraft. Thanks to WorldEdit mod, I was able to transfer some of my previous builds onto the new map.

Side note: I can’t believe I’m already at the 30th selfish post.

♪ Shaka Ponk • Palabra Mi Amor

Micro posting in October

Mon, 30 Oct 2023 07:56:59 +0100

2023-10-02: I’d gone to buy some charcoal fixative but there was none in the bookshop. Hold my beer, I came away with these books:

2023-10-02: TIL about paps, as a replacement for a2ps for printing to a PS file (this is useful to print Neomutt email when there’s no default printer on your machine). h/t jimifiki

set print_command="paps --font='Monospace 10' > /home/chl/tmp/mail.ps"

2023-10-03: Software used to be less demanding at one time.

Cc: haskell-cafe <haskell-cafe@haskell.org>
Date: Tue, 3 Oct 2023 06:46:02 +0200
Subject: Re: [Haskell-cafe] two HLS processes with same namemac
OS needs just 1-2GiB of wired RAM, don’t worry…

2023-10-04: ♪ Arlo Parks · Black Dog
2023-10-08: Out of curiosity, I plugged my DAC (the little box above the Denon amp in this post) onto my computer and, guess what, it just worked!

2023-10-08:

Stuff I made recently.
2023-10-09: Exploratory Data Analysis for Humanities Data, and a load of Awk and Python scripts on the Literature as Data homepage.
2023-10-09: What is in a modern code editor? Auto-formatters and terminal emulators are really a plus, indeed.
2023-10-12:

Linux is borrowing unused memory for disk caching. This makes it look like you are low on memory, but you are not! Everything is fine! — Linux ate all my RAM

2023-10-12:

Picking when to assert is much like picking when to comment, except more explicit and more effective. — How to assert for fun and profit

2023-10-12: There was a time we were able to wait the end of film or series credits to learn about the song titles and things like that. Thanks to Netflix, this is no longer possible, you will switch to the next ep. before you have time to reach the end of (all due) credits. Old times good times.
2023-10-16:

There is a phenomenon in society which I find quite bizarre. Upon our entry to this mortal coil, we are endowed with self-awareness, agency, and free will. Each of the 8 billion members of this human race represents a unique person, a unique worldview, and a unique agency. Yet, many of us have the same fundamental goals and strive to live the same life. — Going off-script

2023-10-16: The discrete logarithm. #math
2023-10-18:

Over the last decade or so, Apple has been hard at work in simplifying the user interfaces that power its myriad platforms. I’ve welcomed most of that work, but it’s hard to deny that we’ve all lost some things along the way. — The History of Cover Flow

2023-10-18: Interesting read on enumerating trees. #haskell
2023-10-19:

2023-10-19: Vim is great for cleaning a text file using search and replace, but sometimes Unix tools are even better: if you want to remove consecutive blank lines, just call cat -s on your text file.
2023-10-19: ♪ Foo Fighters · Shame Shame
2023-10-23:

But just think back. In the past say ten years, what changes and innovations have had a real and true life changing impact on you? I am sure you can think of something, but I am also pretty sure that nothing has happened on a monthly or weekly basis that was so important that you absolutely had to know about it at that very moment. — Do you feel overwhelmed trying to stay up to date with the latest tech news?

2023-10-23:

But let’s just put it this way (and here I’m being charitable): in all my years of using Word, this happened more than once. In all the time I’ve used text editors, it’s never happened, even once. — Why (and How) I Wrote My Academic Book in Plain Text

2023-10-23:

The most important thing to learn correctly about Vim is to understand that you talk to Vim when you edit text. Vim has a grammar. The next important thing is to learn how to use Vim macros. Vim - I hate to love you

2023-10-23: No more plugin manager, yet a fast startup time. #vim

2023-10-23: OpenBSD Handbook. #unix
2023-10-24: If you are using vimtex and are interested in using Evince for forward/reverse synchronization, be sure to check sved which at least provides reverse sync in Neovim. Otherwise, Zathura works as expected. #vim
2023-10-24: TIL that you can clear your Tmux paste buffers by tagging everything (C-t) then deleting it (D).
2023-10-24: Ergonomic mappings for code formatting in Vim: Very nice tutorial on code formatting and how to write VimL functions that exploit gq. #vim
2023-10-24: Pytest is really great: even simple assert statements get processed, no need to write complex fixtures or test suite. This means you can write your scripts as usual and just import pytest. To avoid contaminating your current directory with __pycache__ and .pytest_cache directories, set the exported variable PYTHONDONTWRITEBYTECODE=1 and call pytest with -p no:cacheprovider. You can set a simple compiler and errorformat for (Neo)vim by following this blog post. Personally, I use makeprg=pytest\ --tb=short\ -vv\ --no-header\ -p\ no:cacheprovider\ $* instead. #python #vim
2023-10-26: No more nvim-lspconfig means 15 ms less time spent in starting up! #vim

2023-10-27: Everything you ever wanted to know on how to execute a command in multiple files in Vim. I mostly use argo and cdo. #vim
2023-10-27: TIL that there are some interesting C programming tips on Neovim wiki. #vim #clang
2023-10-27: Interpretable Machine Learning: A Guide for Making Black Box Models Explainable.
2023-10-28:

This document is a sort of quine. If you select the whole page in your browser and paste it into a file, this text shows up encoded in a weird Markdown variant embedded in a string literal inside a Racket lisp script. Evaluating the file with racket -f $FILE evaluates the lisp, converting the weird Markdown and producing the same HTML document that your browser received. — https://cgad.ski/

2023-10-28:

Vim is not an IDE. It can be customized to look like one, but the more you try to make it to look like an IDE, the less it behaves like vim (fast and lightweight). If you need a full-fledged IDE features, use a real IDE. — Why I Use Vim

2023-10-28: I keep getting this kind of weird stuff on my pretty old iPhone. Internet was less complicated back in the time, especially when Apple was not tying up software to hardware to such an extent.

2023-10-28: Ashby’s book “Introduction to Cybernetics”: One of the very first English textbook I bought when I was a PhD student involved in some kind of Neurosciences back at the time.
2023-10-28: Compiling Chicken Scheme scripts. #scheme
2023-10-28: ♪ Fay Claassen · Almost Blue

Neovim and LSP made easy

Fri, 27 Oct 2023 18:02:14 +0200

Continuing the rewrite of my Neovim config described in this post, I decided to get ride of nvim-lspconfig. Indeed, since version 0.8 Neovim offers some convenient functions and autocommands to start an LSP server with predefined mappings which can be made aware of available server capabilities. This is well explained by Mathias Fußenegger. I came across his post some months ago and forgot about it. And then I found a similar discussion on another blog post. So why not remove nvim-lspconfig and define LSP settings in my ftplugin directory. This way, I get all vimL stuff in after/ftplugin, with appropriate guards for nvim and executable that I use for linting and formatting, and all Lua stuff in my init.lua and ftplugin.

This solved many problems at the same time. First, nvim-lspconfig doesn’t let me loop over a series of server so easily. For instance, the following loop must account for specific servers that need additional settings (e.g., clangd to get background indexing, etc.) or slight modification from the default config (e.g., racket_langserver which is defined for racket and scheme filetype – we really don’t want the second one):

local servers = {
 ...
}

for _, lsp in ipairs(servers) do
 nvim_lsp[lsp].setup({
 on_attach = on_attach(),
 capabilities = capabilities,
 })
end

Second, there are some edge cases, which I think are related to the definition of the project root directory for LSP that really need a root_dir (e.g., clojure_lsp), meaning that the server will start if I open a file with the associated filetype directly but not after a simple vim . from the root directory. On the other hand, nvim-lspconfig provides some interesting functions to define the root directory of a project (pending the minor issue just mentioned) or to query the server state, and to restart it if required.

So, how does the in-built facilities of Neovim solve my concerns? I can now isolate the logic of writing specific settings for each LSP in separate files that live in ftplugin, while defining common guarded features relevant to all LSPs. Here is a simple example in the case of Python (I use pyright in this case):

-- ftplugin/python.lua
local config = {
 name = "pyright",
 cmd = { "pyright-langserver", "--stdio" },
 root_dir = vim.fs.dirname(
 vim.fs.find({ "setup.py", "pyproject.toml", "setup.cfg", "requirements.txt", ".git" }, { upward = true })[1]
 ),
}

vim.lsp.start(config, {
 reuse_client = function(client, conf)
 return (
 client.name == conf.name
 and (
 client.config.root_dir == conf.root_dir
 or (conf.root_dir == nil and vim.startswith(api.nvim_buf_get_name(0), "/usr/lib/python"))
 )
 )
 end,
})

local augroup = vim.api.nvim_create_augroup("PythonFormatting", {})
vim.api.nvim_clear_autocmds({ group = augroup, buffer = bufnr })
vim.api.nvim_create_autocmd({ "BufWritePre" }, {
 group = augroup,
 buffer = bufnr,
 callback = function()
 vim.api.nvim_command("call FormatFile()")
 end,
})

The last block allow to complement pyright with autoformatting thanks to a custom VimL function, inspired from this post, that checks the value of formatprg associated with that filetype. This is defined in after/ftplugin, which further allows me to use gq whenever I want.

" after/ftplugin/python.vim
setlocal formatprg=python3\ -m\ macchiato

[2024-08-18]
Heiker Curiel has several blog posts that describe how to setup LSP in Neovim without relying on nvim-lspconfig, in particular Can we manage neovim’s LSP client without plugins? and A guide on Neovim’s LSP client.

Then I defined mappings and an autocommand in my init.lua file:

vim.api.nvim_create_autocmd("LspAttach", {
 callback = function(args)
 local client = vim.lsp.get_client_by_id(args.data.client_id)
 local augroup = vim.api.nvim_create_augroup("LspFormatting", {})
 vim.keymap.set("i", "<C-k>", vim.lsp.buf.signature_help, { buffer = args.buf })
 vim.keymap.set("n", "K", vim.lsp.buf.hover, { buffer = args.buf })
 vim.keymap.set("n", "gd", "<cmd>lua require('fzf-lua').lsp_definitions({ jump_to_single_result = true })<cr>")
 vim.keymap.set("n", "gr", "<cmd>lua require('fzf-lua').lsp_references()<cr>")
 if client.server_capabilities.codeActionProvider then
 vim.keymap.set("n", "z=", "<cmd>lua require('fzf-lua').lsp_code_actions()<cr>")
 end
 if client.server_capabilities.renameProvider then
 vim.keymap.set("n", "zr", vim.lsp.buf.rename, { buffer = args.buf })
 end
 if client.server_capabilities.documentSymbolProvider then
 vim.keymap.set("n", "go", "<cmd>lua require('fzf-lua').lsp_document_symbols()<cr>")
 end
 if client.server_capabilities.workspaceSymbolProvider then
 vim.keymap.set("n", "gO", "<cmd>lua require('fzf-lua').lsp_workspace_symbols()<cr>")
 end
 if client.server_capabilities.codeLensProvider then
 vim.api.nvim_create_autocmd({ "CursorMoved " }, {
 callback = function()
 vim.lsp.codelens.refresh()
 end,
 })
 vim.keymap.set("n", "z!", vim.lsp.codelens.run, { buffer = args.buf, silent = true })
 end
 if client.supports_method("textDocument/formatting") then
 -- vim.api.nvim_buf_create_user_command(bufnr, 'Format', function() vim.lsp.buf.format() end)
 vim.keymap.set("n", "g=", vim.lsp.buf.format, { buffer = args.buf })
 vim.api.nvim_clear_autocmds({ group = augroup, buffer = bufnr })
 vim.api.nvim_create_autocmd({ "BufWritePre " }, {
 group = augroup,
 buffer = bufnr,
 callback = function()
 vim.lsp.buf.format({ async = false })
 end,
 })
 end
 end,
})

Note that I use fzf-lua as a fuzzy picker for many things in this case.

Needless to say, my Neovim starts even faster now.

♪ Editors • Like Treasure

A minimalistic Lua-based Neovim config

Mon, 23 Oct 2023 20:15:38 +0200

It hasn’t taking long before I decided to trash my new Lua config. I originally wanted to give lazy.nvim a try since packer has been discontinued. I was not able to reduce my startup time further, and this wasn’t in par with my old packer-based config. Independent of this, I felt quite unhappy with the added layer of complexity provided by mason.nvim and satellite plugins. More to the point, some LSPs, linters (for nvim-lint) or formatters (for conform.nvim) were not available in Mason registry, meaning that I add to configure them outside of Mason handlers, in addition to installing them myself. That makes these plugins rather limited in their usefulness, IMHO. So, let’s rewrite it from scratch!

First, I’ve stuck to the aim of working on a single configuration file, rather than splitting it up into different files in the dedicated lua directory. Also, I decided to stop using a plugin manager since Vim and Neovim both ship with a decent plugin system. What’s more, it will help me not to update my plugins every two days or so. And I get the full Git log as well. In a few words, I ended up with the following set of plugins:

~/.config/nvim
» tree -L 3 pack
pack
└── plugins
 ├── opt
 │   ├── neogen
 │   ├── nvim-colorizer
 │   ├── vim-dasht
 │   └── vim-sexp
 └── start
 ├── Comment
 ├── fzf-lua
 ├── nvim-lspconfig
 ├── nvim-treesitter
 ├── nvim-treesitter-textobjects
 ├── orgmode
 └── vimtex

14 directories, 0 files

As can be seen, I put orgmode and vimtex in the “start” folder, which means they are loaded when Neovim is fired up. My own benchmarks suggest this has no noticeable effect on startup time. With only fzf-lua, I’m at 25 ms of startup time, while the full config increases it by about 20-25 ms. That’s cheap. Note that I don’t use a lot of plugins, I have a lot of custom settings living in my after/ftplugin directory, plus additional commands and autocommands.

I think about it this way: you really want a clever fuzzy picker, and I already Fzf to manage my shell history, browsing directory, or managing my bibliographic entries in the shell. By the way, I upgraded my very outdated version of Fzf that I got via the official apt repository. Now I’m running version 0.43.0 (d3311d9) and I can use git_bcommits and git_commits again (it stopped working in recent months, which I bissected to this commit). Besides, the availability of default LSP settings as well as treesitter “text objects” is often an underrated value: you get both linting and formatting for free (Neovim builtins and Fzf can take care of the rest for you), as well as semantic and/or structural editing capabilities. When no fixer is available, yuo can always rely on formatprg or autocommands.

I use vim-dasht to complement the LSP hover provider, or simply as a documentation handler when there’s no LSP in use (e.g., Scheme, Lisp); it also proved to be useful with Python for which pyright provides nothing more than function signature in most cases. I use neogen to generate Python and C’s docstrings. I’m too lazy to think about it (or even remember how to format Google and Numpy docstrings), so this comes handy in this particular case. As said above, I can manage formatting and linting myself, and Neovim has excellent support for completion via C-x C-o, C-x C-n, C-x C-f, and friends. The rest is already builtin, e.g., want a tree explorer, try :15Lex; need some Git info, FzfLua get you covered. Everything that is not related to text editing per se is probably better handled in your terminal by a dedicated application.

♪ Dead Can Dance • Carnival of Light

More useful Vim tips

Wed, 18 Oct 2023 21:36:37 +0200

It’s been 3 years that I am using Neovim as my daily driver. I learned a lot of things in the mean time. The only things left are macros and registers, which I don’t really use, except registers for basic stuff. Here is a list of things that come to my mind when I think about Vim (not the modal editor, rather the builtins):

Be lazy when you load plugins: I’ve been using packer for two years and it was all good, I could get a startup time below 50 ms with a dozen of packages. Now that I use lazy, I got comparable improvements in startup time with even more packages.
Don’t forget builtin commands: There are a lot of useful commands already available in Vim, and many shortcuts that are worth learning, e.g., for manipulating windows and tabs or the z and g prefixed commands.
Search and replace using regex are powerful tools: Use capture groups eagerly, like in :%s/$[0-9]*$.*/\1/; also, don’t forget you can save search results in a file, e.g. g/pattern/.w! >> out.txt, if you need to process them afterwards.
Prefer a low contrast monochromatic colorscheme to avoid Christmas tree-like effects; treesitter should take care of highlighting the right stuff, and it will be helpful to perform structural editing.

[2023-11-21]
See this excellent discussion on Overcolorization: “Syntax highlighting is a tool, and highlighting important parts of the code makes programming less convoluted. Pretty colors are nice to have, but if the point is only to make the code look fancy, there’s no real value for what we’re trying to do in the end - write better code. Let’s help each other at least at that.”

Use templates for your one-liner: You can define a set of autocommands or a simple mapping to insert a file right into your buffer; in the latter case, :-1 read allows to place a template file above the insertion point, in contrast to the default behavior of :read.
formatprg and equalprog are invaluable if you don’t use or don’t have a relevant LSP at hand. Vim is quite good at indenting, so you’re more likely to deal with formatprg. For instance, to format bibliographic entries in a Bibtex file, you could define setlocal formatprg=bibclean\ -align-equals\ -brace-protect\ -delete-empty-values\ -fix-accents\ -quiet in your after/ftplugin/bib.vim, and then gqip the current entry.
You usually want to put your custom settings in after/ftplugin rather than plugin. If possible, keep it orthogonal to your init file so that you can use a minimal init file that is shareable across machines while your after folder get you load of enhancements to the core config.
Read the documentation (and learn to navigate inside) rather than spending hours browsing the internet to learn about the latest cool plugins (which is only compatible with the next-to-last version of Neovim); that is, write small functions or commands that do 90% of what you expect, rather than plugging in yet another plugin. Most of what you will read from past Vim wikis still apply as of today in Neovim, so don’t reinvent the wheel.
We rarely need to <C-z> these days, thanks to the builtin terminal emulator.

[2023-10-25]
See also At least one Vim trick you might not know and Vim’s useful lists for useful tips.

♪ Yasmin Levy • Mano Suave

On the use of triplicate in biology

Mon, 16 Oct 2023 20:05:57 +0200

In behavioral quantitative measurement, we are used to repetition within and between condition, that is a given subject is exposed to all conditions, or a subset thereof, in a complete or incomplete factorial design, and for each condition a certain number of repetitions of the same measurement are done. This is the so-called repeated measure design. In biology, researchers rather talk about technical and biological replicates. They are not the same: technical replicates are readings collected from the same statistical unit, while biological replicates are independent statistical units. Usually, both type of replication are carried out. Note that this is different from repeated measurements since biological replicates which involve different samples are exposed to different experimental conditions. But technically, technical replicates are repeated measures within a subject $\times$ condition treatment.

Next, there are two questions that always puzzled me: (1) technical replication is often performed using triplicates; (2) mixed-effect models are rarely used in practice, rather technical replicates are averaged and the analysis is performed using statistical tests for independent group (ANOVA, t-test, etc.).

I don’t think there’s any statistical justification for the use of $k=3$ technical replicates rather than, say, $k=10$. Three observations are enough to gauge the precision of the measurement, and discard an outlying value if necessary, but larger sample sizes for biological replicates (PDF) are better especially when we want to get a more precise averaged or pooled estimate and/or we are dealing with small effect sizes. It is common in RNA-Seq experiments to consider only 3 biological replicates, although it would be considered as the absolute minimum.¹

My small Dr. DuckDuckGo literature review yielded the following articles or tech notes:

In To triplicate or not to triplicate?, the authors used mixed-effect models to assess the contribution of the between variance to the total variance and they showed that (it can be shown that the intraclass correlation coefficient $\rho < 1$ ($\rho = 0$ only in the case where the within sample units are independent), when the numbr of replicates $m \geq 2$, the width of the 95% CI for the average of the $m$ within sample units is reduced by a factor $\sqrt{\hat\rho + (1 - \hat\rho)/m}$. For $m=1$ the maximum reduction of the CI width equals $1 - \sqrt{1/m}$ when $\rho = 0$. With triplicates, we can achieve about 20% reduction for $\rho = 0.5$.

Cell lines remain problematic anyway: Imagine the case where a cell line is split into different colonies which are then left to grow either as control (untreated) vs. experimental (treated) condition; since all cells were extracted from the same tube. In this case, these are not biological replicates. See also Cell culture experiments, technical or biological replicates?.

♪ Joy Division • Transmission

Schurch, N. J. et al. 2016. How many biological replicates are needed in an RNA-seq experiment and which differential expression tool should you use? RNA 22(6): 839-851. The authors show that with only three biological replicates, “nine of the 11 tools evaluated found only 20%–40% of the significantly differentially expressed (SDE) genes identified with the full set of 42 clean replicates,” and that 20 replciates are reuquired to achieve >85% accuracy. ↩︎

ArXiving on October 2023

Fri, 13 Oct 2023 12:24:09 +0200

Fisher’s Pioneering work on Discriminant Analysis and its Impact on AI (https://arxiv.org/abs/2309.04774v1)

It’s a little bit funny to see the name of Ronald Fisher associated to AI in the title of scientific paper, especially since AI now means LLMs and other brilliant foundations for the future millennium. Anyway, the iris dataset is used to illustrate the use of a discriminant function (as the ratio of squared mean and variance) to demonstrate whether “when we use the linear compound of the four measurements most appropriate for discriminating three such species, the mean value for I. versicolor takes an intermediate value and, if so, whether it differs twice as much from I. setosa as from I. virginica, as might be expected, if the effect of genes are simply additive, in a hybrid between a diploid and a tetraploid species.” ¹ Hence, a genetic hypothesis, which can be exploited to find the optimal linear combination of population means, $\mu_1 -3\mu2 + 2\mu_3 = 0$ (respectively, setosa, versicola and virginica), was at the heart of Fisher’s seminal paper, like many of his other contributions. Sadly, the iris dataset is mostly used in the context of unsupervised classification (k-means, for instance), as if people forgot that it relies at its heart on a well-defined hypothesis (much like people forgot or still ignore that Student (Gosset’s nom de plume) developed his test while working in a brewery). It seems the point of the authors is the notion of a discriminating rule, which, when applied to AI problems, amounts to a supervised approach that aims to find a “rule to separate distinct populations, and then to use that rule (or a different classification scheme) to allocate a new observation to the given populations.”

A compendium of data sources for data science, machine learning, and artificial intelligence (https://arxiv.org/abs/2309.05682)

This article summarizes a long list of data repositories spanning a dozen or so domains (including satellite data or sports, for instance). I will keep it on hand just in case I need some dataset one day.

Demystifying Statistical Matching Algorithms for Big Data (https://arxiv.org/abs/2309.05859)

Propensity scoring is widely used in the absence of a controlled experiment, e.g. in observational studies, where as the authors say matching is used to pair treated and control units with similar value of potential confounders (aka, the propensity score). Usually, matching is done with replacement, that is a control is paired to several treated units (1-to-n matching, with replacement). There are other estimators (e.g., inverse probability weighting or doubly robust methods), though. Oversampling and replacement strategies may alter the balance between bias and variance. In the case of matching without replacement, greedy approaches are used; in Stata, there’s a command calipmatch which does the job for you, while in R the /de facto/ packages are probably Matching and MatchIt. In this paper, the authors review alternative approaches, namely optimal statistical matching (linear balanced and unbalanced assignment problem, maximum cardinality matching, and minimal cost matching).

Subgroup detection in linear growth curve models with generalized linear mixed model (GLMM) trees (https://arxiv.org/abs/2309.05862)

Here is another paper by Achim Zeilis on regression trees. This time, it deals with “extended” GLMMs for identifying subgroups in growth curve modeling. It’s been a long time since I first used partykit for tree-structured regression and classification models, and this was in a benchmark with classical CART, logic regression, and ensemble methods (random forests and boosting); this didn’t end up in a paper, but it was a fun exercise. Since then, a lot of work has been put forward into this framework and I’m not really up to date. Unlike other methods for partitioning latent growth curves, this modeling framework set up a mixed approach: “GLMM trees do not fit a full parametric model in each of the subgroups defined by the terminal nodes of the tree. Instead, fixed-effects parameters are estimated locally, using the observations within a terminal node, and the random-effects parameters are estimated globally, using all observations.”

A Change-Point Approach to Estimating the Proportion of False Null Hypotheses in Multiple Testing (https://arxiv.org/abs/2309.10017)

I have used FDR correction for multiple comparisons here and there, as an alternative to sequential methods like Holm’s approach (which is the default in R’s p.adjust, btw). The authors offer an alternative data-driven method to the classical Storey’s method (i.e., a family of true null proportion of false null hypotheses estimators) for estimating the FDR. In this case, the sorted p-values are approximated using a piecewise linear function with one change point (inversion of slope), as illustrated below. This is somewhat akin to the knee/elbow method or scree plots for determining the number of principal components or factors to retain in PCA or FA, which is discussed in section C.1.

A New Bootstrap Goodness-of-Fit Test for Normal Linear Regression Models (https://arxiv.org/abs/2309.10614)

Yet another article discussing the benefit of using bootstrap to assess model goodness-of-fit. Rather than visual inspection of so-called residual plots (residuals vs. fitted values), especially in multivariate regression, or relying on either the White or Breusch-Pagan test. The test presented here allows to test all assumptions of normal linear regression:

For test level $\alpha$, candidate normal linear model $\mathcal{M}$, bootstrap iterations $B$, sample size $n$, and a null hypothesis that $\mathcal{M}$ is adequately specified:

Resample, with replacement, outcomes with covariates to generate a bootstrap sample of size $n$.
Fit model $\mathcal{M}$ to this bootstrap sample, and with this fitted model, calculate $\text{Var}[\text{GOF}]$and record this statistic.
Repeat steps 1-2 $B$ times to generate an empirical bootstrap distribution for $\text{Var}[\text{GOF}]$.
Construct a $100\times (1−\alpha)$% bootstrap confidence interval for $\text{Var}[\text{GOF}]$.
If this interval does not contain $2n$, reject the null hypothesis at the $\alpha$ level. If it does contain $2n$, the null hypothesis was not rejected and model $\mathcal{M}$ does not appear to exhibit lack-of-fit.

Unveiling Challenges in Mendelian Randomization for Gene-Environment Interaction (https://arxiv.org/abs/2309.12152)

As a sequel to this 10-years old post, here is a paper that uses genetic variants as instrumental variables (2-stage predictor substitution and the 2-stage residual inclusion) to estimate causal effects in GWAS on lifestyle and environmental factors.

♪ Public Image Ltd • Rise

Fisher, R. A. (1936) The use of multiple measurements in taxonomic problems. Ann. Eugen., 7, 179–188. ↩︎

Neovim revamp

Thu, 12 Oct 2023 11:34:11 +0200

Recently, I switched to lazy.nvim to manage my Neovim plugins. I’m thus writing yet another post on Vim with a fresh new configuration. Reasons are multiple but quite simple: I wanted a more minimalistic set of config files (ideally one or two Lua files) allowing me to keep my after/ftplugin directory as optional in case I need to transfer my config to other machines; packer looks like it will no longer be in use (it is already archived), like is the case for null-ls.nvim; I still want blazing fast startup time (before moving to that new config, I was well under 30 ms, as measured by hyperfine or vim-startuptime).

TL;DR I got a twofold increase in startup time, but I only need to manage a single lua file now.

~/.config/nvim 16m18» hyperfine "nvim --headless +qa" --warmup 5
Benchmark 1: nvim --headless +qa
 Time (mean ± σ): 67.4 ms ± 1.0 ms [User: 52.3 ms, System: 15.0 ms]
 Range (min … max): 65.9 ms … 70.4 ms 44 runs

Yes, I’m back to the good old era of the 500+ LOC vimrc. In my case, I now have a single init.lua file to manage my plugins and general settings, nicely organized and presented thanks to native (and efficient) Vim folding. I still have various directories lying around, though. Here is the big picture:

~/.config/nvim
» tree -L 1
.
├── after
├── compiler
├── ftdetect
├── init.lua
├── lazy-lock.json
├── spell
└── templates

5 directories, 2 files

I discarded part of a forked version of mini.nvim that I was using for completion, as well as several hand-made helper functions. Meanwhile I also cleanup stuff dispatched in several Lua files and put stuff in either my after directory, or directly into my init.lua file. Overall, nothing really changed, except that I’m now relying on dedicated packages to manage the colorscheme and statusline of Neovim stable release (v0.9.2 at the time of this writing). Breaking changes are inevitable with “over-the-beta” software, which is why I stopped using the dev version of Neovim in recent months. And since I don’t have the time nor inclination to read the changelog every two weeks, or after every new stable release, I let package maintainers do the right stuff for me. I’ve been using a bare bone custom monochromatic theme for three years now, but as I said in a previous post, the zenbones plugin is quite good and I very much like the zenbones and zenwritten themes on light background. They also render nice on a dark background by the way. Lualine is okay for most part and it fulfills all my requirements.

The biggest issue for me was to replace null-ls¹ since it provided me with a bunch of useful linters and fixers but it came out it was not too difficult to get rid of it since conform.nvim and nvim-lint came to the rescue shortly. I am aware it can be done with autocommands as explained on SO or simple mappings (which is what I already have in some of my after/ftplugin files, but these two packages allow to manage linters and formatters in a more comfortable way.

Startup time is a bit longer (60-70 ms instead of 30 ms with my previous config), but I use a different set of plugins so it’s hard to compare. The most notable changes are the addition of auto-installers, like mason.nvim (with mason-lspconfig.nvim and mason-tool-installer.nvim, which works alongside nvim-lspconfig, and linting/formatting packages cited above). I already installed all LSPs, linters and formatters I need using pip, npm, curl or whatever, but the big advantage of this approach is that everything is self-contained so that I can transfer my config files to any other computer and get everything installed right away. Mason-based tools will install everything (using a registry list in the case of Mason itself) within a single click or config line.² It also means you end up with a huge $HOME/local/share/nvim directory (3.3 Go with 9 LSPs, compared to less than 180 Mo with my previous config). This is not a big deal if you have a large HD, but if you make regular backup using Kopia, you may want to exclude that directory from your hourly or daily backups. As an example, consider the following listing, in the middle of my configuration: some LSPs, formatters and linters take up some space, really:³

$ .local/share/nvim/mason/packages
» du -sh ./* | sort -h
2,8M ./shfmt
4,0M ./purescript-language-server
14M ./stylua
15M ./shellcheck
19M ./texlab
20M ./docformatter
20M ./isort
27M ./black
28M ./pyright
28M ./sqlfmt
29M ./clojure-lsp
41M ./typescript-language-server
44M ./rust-analyzer
105M ./r-languageserver
124M ./clangd
2,8G ./haskell-language-server

[2023-10-19]
Some after-thoughts: chktex is not available to install via mason-installer while racket-languageserver must be set up manually (it cannot be configured with mason-lsp). This means you still have to add extra stuff unless you submit a patch to the maintainers. You can define an auto-format command with conform, which may render some of your own function call or autocommands useless in your after/ftplugin folder. This means we must be careful with synchronizing all filetype-specific settings. I would prefer that everything could be centralized within conform setup function.

I tried nvim-cmp as a replacement of my personal fork of mini.nvim completion module (this was before the mini.nvim project introduced submodules), but I removed it after a few days. I never really like autocomplete and popup windows after every two keystrokes, and I’m probably fine with Vim’s omni complete (C-x C-o), when I need it.

And then comes nvim-orgmode, which is so great compared to my previous half hand-made configuration. But I will talk a bit more bout this in a separate post.

♪ Bauhaus • Exquisite Corpse

By the end of writing this post, I learned it has been forked as none-ls.nvim. ↩︎
Compared to nvimlsp-config, Mason uses a different naming convention, so be careful. ↩︎
Of note, the R language server took a lot of time to install, compared to other LSPs. At this point, I’m thinking of removing it since I don’t really need it except for the keywordprg (on-line help), which I can manage by my side. ↩︎

Rule of three and the Chebyshev inequality

Mon, 02 Oct 2023 14:11:06 +0200

In medical statistics, especially studies on adverse events, there’s a well-known rule of three (different from other rules of three) which states that for binary outcomes, if no event has been observed on $n$ individuals a 95% confidence interval is constructed as $[0; 3/n]$. We can derive a confidence interval for the opposite (100% of events) by symmetry ($[1−3/n,1]$). Hence the name, because the chance of such an extreme event is at most three in $n$. The derivation is given in, e.g., If Nothing Goes Wrong, Is Everything All Right? Interpreting Zero Numerators.

There’s another handy (sort of) rule of thumb. If you remember what you learned from your intro course in statistics and probability density functions, you probably know that you can expect 95% of your data to lie within 2 standard deviations around the mean, and that there’s close to 3‰ chance that data lies above 3 standard deviation from either side of the mean. This is sometimes referred to as the 68–95–99.7 rule. Chebyshev’s inequality is generally used to justify the fact that a certain fraction of values can be no more than a certain distance from the mean. Contrary to the rule discussed previously, it specifically states that: $$ \Pr(\mid X - \mu \mid \geq k\sigma) \leq \frac{1}{k^2}. $$ It applies to any continuous distribution that has a mean and a variance. When $k=2$, for example, 75% of the observations are found to lie in the range $\mu\pm 2\sigma$.

Now, assuming a unimodal distribution with mean $\mu$ and finite variance $\sigma^2$, for any $\lambda > \sqrt{8/3} \approx 1.633$, $$ \Pr(\mid X - \mu \mid \geq \lambda\sigma) \leq \frac{4}{9\lambda^2}. $$ This is known as the Vysochanskiï-Petunin inequality. When $\lambda = 3$, the upper probability bound equals $4/81 \approx 0.05$.¹ To my knowledge, applications of this inequality mainly are in control charts or financial statistics, but there may be some other interesting applications.

[2024-02-07]
See also How likely is a random variable to be far from its center?.

♪ Antony & The Johnsons • Cripple and the Starfish

Chebyshev’s inequality would yield about twice that value. ↩︎

Micro posting in September

Thu, 28 Sep 2023 08:24:22 +0200

2023-09-03:
Carole Zalberg, Tes ombres sur les talons (Grasset).
2023-09-04: 13 ways to look at table joins. #database
2023-09-04: An Intuition for Logarithms. #math
2023-09-04: Doppler looks like iTunes reinvented. (via Tom Macwright)
2023-09-04: binary search trees: A thorough treatment of binary search trees enriched by an interesting comment thread.
2023-09-05:

Curl supports a crazy number of protocols, from HTTP, FTP and TELNET to IMAP, LDAP and GOPHER. It runs on 92 operating systems and has over 20 billion installations worldwide. — Mastering curl: interactive text guide

2023-09-06: TIL about CIEL, an an Extended Lisp that looks really cool. It includes a bunch of useful libraries (Alexandria, Trivia, arrow macros, etc.) and provide an IPython-like REPL experience (shell pass-through, helper commands, documentation and lisp-critic). #lisp
2023-09-06: ♪ Angel Olsen · All The Good Times
2023-09-07: Visual explanations of core machine learning concepts: Very nice and interactive illustrations. #statistics
2023-09-11: ♪ Bentley · Left Right Goodnight
2023-09-12:
A little wrapper around the terminal version of What-The-Format.
2023-09-12:

One of the guiding lights of the UNIX utilities or software tools has been the deeply-felt conviction that text should be stored in as simple, as general a format as possible so that any program can easily process it. This idea (it seems to have been present from the beginning) has had the widest impact possible on UNIX in all its varieties. However, there has been a regrettable tendency to move away from it in recent times, especially among commercial software developers. — UNIX Evolution: 1975-1984

2023-09-12:

The point is, there’s no reason why this bug — and it is a bug! — should have stuck around for so long. If your terminal emulator gets told to make text blue, and instead it makes it unreadable, then you can’t say that’s not a problem with the terminal emulator. — That Annoying Shade of Blue

2023-09-12: A slightly updated version of all my emails as discussed in a post in 2014. Note that I’ve reduced my mailbox to 20% of its original size since I wrote the blog post, and I keep archiving very few new messages. Also I resubscribed to a bunch of mailing-lists in 2022, which I keep in separate Maildirs but they are included in this chart. I’m still a happy user of mu.
Updated code (because mu-find now uses the system locale to format dates):

$ export LC_ALL=C
$ mu find --sortfield=date --fields="d" date:17y..1d | \awk -F " " '{print $5 "-" $2}' | \uniq -c | awk '{print $2 " " $1}' > 1.dat

library(ggplot2)
library(scales)
library(zoo)
Sys.setlocale("LC_ALL","C")
d <- read.table("1.dat", header = FALSE)
d$V1 <- paste(d$V1, "15", sep = "-")
d$V1 <- as.Date(as.POSIXct(d$V1, format = "%Y-%b-%d", tz = "GMT"))
names(d) <- c("Time", "Emails")
p <- ggplot(data = d, aes(x = Time, y = Emails)) +
 geom_line(color=grey(.5)) +
 geom_line(aes(y=rollmean(Emails, 12, na.pad=TRUE)), linewidth = 1.5) +
 scale_x_date(breaks = "1 year", minor_breaks = "2 months",labels = date_format("%Y")) +
 xlab("") +
 theme_bw()
ggsave("allmyemails.png", p, width = 12, height = 5)

2023-09-13:

In the end, the main thought here is the following: build using simple proven techniques even if you’re using a technology like Haskell. For anything extra (like dependent types, formal verification, etc.) you might want to think twice and thrice. And still remain uncertain. — The types got you

2023-09-13: OMG, now they call it Drive… #apple

2023-09-13: Common Lisp Nova Spec: A better (?) help center than the CLHS. #lisp
2023-09-13: Yet Another Rust Resource (YARR!): If you ever wanted to quickly get some insights about Rust, it offers a nice overview of the language.
2023-09-15: Category Theory Illustrated “is a primer in category theory and other mathematical theories that is made to be really accessible to people with no prior exposure to the subject, without being dumbed down, by utilizing visual explanations."
2023-09-18: Another nice article by Brian Hayes, on AI and the end of programming.
2023-09-18: It appears that Neomutt can be quite slow when filtering/searching emails using body keyword or patterns and the “limit” (l) command. It seems that Neomutt needs to open all emails beforehand, which makes sense but slow down any search operation. Now, as suggested in #3273, we can speed up things a lot by disabling the auto_view text/html feature. My benchmarks confirm those of the poster, with around 13K messages in my Archives. I don’t know how to disable auto_view temporarily, but since I can use mu-find to search my email database, with almost the same syntax, I do’nt have to worry much: results are almost instantaneous!

macro index \Cf "<shell-escape>mutt-mu " "mu find"
macro index gf "<change-folder-readonly>~/.mu/results<enter>" "mu find results"

Here, mutt-mu is a short shell script for mu find --clearlinks --format=links --linksdir=~/.mu/results "$1".
2023-09-19: Lovely: pastel.

2023-09-20:

(Python) types are very broad hints and they are sometimes lies. — Compiling typed Python

2023-09-20:

The idea is to take a train somewhere (unspecified) for four hours, have lunch, then take another train home. The train would be my 90mph office with scenery for those four hours in each direction, and I’d get to enjoy lunch somewhere new. I’d do a full days work with the view out the window constantly changing. — The 90mph office

2023-09-20: Trying out texlab since yesterday. So far it has been working smoothly alongside vimtex and null-ls (to format bibtex files). Last year I played with Tectonic, which looks great, especially if you don’t want to manage a full installation of the $\TeX$ system. Both seem to play well together. #vim #tex
2023-09-20: Exploring the Top Neovim Distributions: LazyVim, LunarVim, AstroVim, and NVChad — Which One Reigns Supreme?: either you get limited capabilities or you are blowing your RAM, right? #vim
2023-09-21: TIL about Harlequin which looks really great. #database
2023-09-21: Thanks to notify-send.sh I can now update the mail notification from Neomutt in Ubuntu notification center since it allows to replace an existing notification. This should be available in more recent versions of notify-send (Ubuntu ≥ 22.10).

set new_mail_command="notify-send.sh --replace-file=/tmp/muttnotification -a neomutt -u low -i neomutt 'New Email' '%n new messages, %u unread.' &"

Beforehand I tried notify-send.py which didn’t quite work.

2023-09-24:

2023-09-24: Building LISP, in under 6 C files (876 LOC). #lisp
2023-09-24: Sidenotes In Web Design: I didn’t know there were so many JS and non-JS based framework for HTML sidenotes.
2023-09-24: ♪ P!nk · Glitter In the Air
2023-09-25: Measured: Typing latency of Zutty (compared to others): Sadly no stats for foot. Given the existing benchmarks, I would expect it to be in lowest range of typing latency as well but this would need some perf tests. #unix
2023-09-25:

Weekend cookery.
2023-09-25: ♪ Portishead · The Rip
2023-09-28:

It was the precursor of what was later known as the Gifi System, which is a large reorganization of descriptive multivariate analysis techniques in such a way that they apply both to numerical and non-numerical data. That covers regression and principal components analysis, factor analysis, all those techniques in one common framework and ultimately in a series of computer programs as well. My dissertation was basically the first programmatic statement of that program. The Gifi project itself ran until about 1990, I left in 1987, but there was still some ends to wrap up and the Data Theory group in Leiden that still existed at the time, wrapped up those loose ends. They produced a book, the final version of the Gifi book. — Interview of Jan de Leeuw by Lisa Wijsen

I was surprised to learn that he’s still publishing peer-reviewed papers and I was also very impressed to see that he’s still hard at work on working papers and books. h/t

Null zone vs. error bars

Wed, 27 Sep 2023 07:04:35 +0200

Maarten Boers wrote a paper entitled Null bar and null zone are usually better than the error bar to compare group means in graphs. Frank Harrell also mentioned null bars in one of his technical articles on his website if my memory serves me right, but I can’t get my hands on it again. Null bars depict the range within which the means of the groups fall if the null hypothesis cannot be rejected at the selected $\alpha$-level. Considering a two-group comparison, it reflects $\bar X \pm \tfrac{1}{4} \text{95% CI}$ where $\bar X$ is the the grand mean. Hence, the range covered by the null zone amounts to half the CI for the difference in means.

For one I always preferred displaying confidence intervals to standard error or standard deviation, even in the case of ANOVA design,¹ with the exception of standard error of measurement in psychometrics. The null bar is not that much different except it is centered on the grand mean and, for large sample, it is equivalent to $\bar X \pm \text{SE}_\text{d}$. I recently tried to circumvent the problem of frequentist inference using Bayesian approaches, which are common in molecular phylogenetics, even if Bayesian credible intervals are not stricly equivalent to frequentist confidence intervals.²

I found no R package to plot null zone like in Figure 3 of the aforementioned paper. Here is a quick and dirty R script that mimics null zone for a line graph using data that I discussed in an old handout of mine for the analysis longitudinal data. First, let’s load and plot the data.

library(Hmisc)
library(ggplot2)
library(hrbrthemes)
library(directlabels)
library(data.table)

options(digits = 6, show.signif.stars = FALSE)
hrbrthemes::import_roboto_condensed()
theme_set(theme_ipsum(base_size = 11))

sepsis <- foreign::read.dta("sepsis.dta")
sepsis$treat <- as.factor(sepsis$treat)
sepsis$id <- factor(sepsis$id)
sepsis.long <- melt(data.table(sepsis), measure.vars = 8:14, id.vars = 1:2)
celsius2fahr <- function(x) (x-32) / 1.8
sepsis.long$value <- celsius2fahr(sepsis.long$value)

d <- with(sepsis.long, summarize(value, llist(treat, variable), smean.cl.normal))

p <- ggplot(data = d, aes(x = variable, y = value, shape = treat, color = treat)) +
 geom_point(size = 2) +
 geom_line(aes(group = treat)) +
 geom_errorbar(aes(ymin=Lower, ymax=Upper), width=.1) +
 scale_color_manual("", values = c("steelblue", "orange")) +
 scale_x_discrete(labels = seq(0, 6*2, by = 2)) +
 guides(color = "none", shape = "none") +
 geom_dl(aes(label = treat), method = list("smart.grid", cex = .8)) +
 labs(x = "Durée de suivi (heure)", y = "Température (°C)")
p

And the code that computes the null zone:

null.zone <- function(data, conf.int = 0.95) {
 x <- data[[1]]
 y <- data[[2]]
 stat <- t.test(x, y, conf.level = conf.int, var.equal = TRUE)
 m <- mean(c(x, y), na.rm = TRUE)
 se <- as.numeric(stat$stderr)
 df <- as.numeric(stat$parameter)
 ci <- m + c(-1, 1) * qt((1 + conf.int)/2, df)/2 * se
 c(Mean = m, Lower = ci[1], Upper = ci[2])
}

s <- split(sepsis.long, by = "variable")
r <- data.frame(lapply(s, function(x) null.zone(with(x, split(value, treat)))))
r
# temp0 temp1 temp2 temp3 temp4 temp5 temp6
# Mean 38.0149 37.6909 37.5240 37.4363 37.3514 37.2299 37.2808
# Lower 37.9111 37.5864 37.4240 37.3379 37.2502 37.1326 37.1817
# Upper 38.1187 37.7954 37.6241 37.5347 37.4527 37.3273 37.3799

First call to split uses data.table, second one is base R.

dr <- transpose(r)
names(dr) <- rownames(r)
dr$x <- seq(1, nrow(dr))
p <- p + geom_ribbon(data = dr, aes(x = x, ymin = Lower, ymax = Upper), inherit.aes = FALSE, fill = grey(.3), alpha = 0.2)
p

The shaded area above corresponds to the area in which the two means at each time point should lie if the null hypothesis cannot be rejected at a two-sided 5% level. Unprotected t-test assuming equal variance at each time point confirm that the two groups only differ on the first occasion:

round(as.numeric(lapply(s, function(x) t.test(value ~ treat, x, var.equal = TRUE)$p.value)), 3)
# [1] 0.499 0.001 0.000 0.000 0.000 0.000 0.000

As an alternative, one may look at the nullabor R package, which provides functions to support quantifying the significance of structure seen in plots of data.

♪ Queens of the Stone Age • I never Came

W. D. Dupont (2009) Statistical Modeling for Biomedical Researchers, Cambridge University Press. ↩︎
See, e.g., Pragmatism should Not be a Substitute for Statistical Literacy, a Commentary on Albers, Kiers, and Van Ravenzwaaij (2018). ↩︎

Confidence intervals and coverage

Tue, 26 Sep 2023 08:48:37 +0200

A blog post I found in my draft folder which I put here, without further ado.

According to the frequentist definition of a confidence interval, the 95% lower and upper bounds can be interpreted as follows: for 100 independent samples, we expect 95 of the CIs to include the true parameter. While I was working on another post, I wrote a quick Python script to estimate how well confidence intervals for a mean or a proportion cover the true parameter value. Since the arithmetic mean applies equally to continuous or binary outcomes (in the latter case, we get the proportion as desired), the code is quite simple. Also note that I use a normal approximation, which may not be that suitable with very small samples. Needless to say, I ended up drafting a little script by the end of August and forgot to highlight misbehaving CIs (i.e., the 5 not so in a row)….

Here is my own implementation, which assumes raw data (i.e., a series of observations for each sample):

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

def twosamp_ci(xs, ys, level=0.95):
 alpha = 1 - level
 xs, ys = np.asarray(xs), np.asarray(ys)
 nx, ny = xs.shape[0], ys.shape[0]
 px, py = np.mean(xs), np.mean(ys)
 diff = px - py
 if np.unique(xs).shape[0] <= 2:
 se = np.sqrt(px * (1-px)/nx + py * (1-py)/ny)
 else:
 sx = np.std(xs) * np.sqrt(nx/(nx-1))
 sy = np.std(ys) * np.sqrt(ny/(ny-1))
 se = np.sqrt(sx**2/nx + sy**2/ny)

 qz = norm.ppf(1 - alpha / 2)
 lower = diff - qz * se
 upper = diff + qz * se

 return lower, upper

n = 100
iter = 100
cis = {}
for i in range(iter):
 xs = np.random.normal(loc=11, scale=1.5, size=n)
 ys = np.random.normal(loc=11, scale=1.5, size=n)
 cis[i] = twosamp_ci(xs, ys)

y = cis.keys()
x = list(zip(*list(cis.values())))
plt.hlines(y, x[0], x[1])

It could be rewritten without even using Numpy since the math and statistics modules include the stuff we need.

♪ Agnes Obel • Familiar

Foot terminal for Wayland

Mon, 25 Sep 2023 20:14:06 +0200

I’ve been playing out of the blue with the foot terminal emulator after several months in Gnome terminal. It’s incredibly fast compared to the latter, of course. You’ll get noticeable improvement in terms of response and typing latency when listing directory using exa or running sudo apt update && apt list --upgradable, for instance. Typing in Vim goes even smoothly. I know I can achieve the same results with Kitty, but foot itself is advertised as a fast and minimalist terminal emulator for Wayland. I’m on Wayland, and I like trying things out, especially if this improves my workflow.

I’ve used Alacritty for one year and finally came back to Kitty. The latest additions (e.g., switching themes or update notifications) were life changing and made it close to the bare bone terminal emulator available system-wide on Ubuntu (aka, Gnome terminal). You’ll get many additional capabilities, though: image support, pipes to external programs (also known as kitten), good URL support, multiplexing for free, and so on. Overall, with foot there’s not much difference compared to Kitty except you’ll get less additional capabilities. However, image support (via sixel, see this excellent overview) and handling URL is quite good. Search capabilities in the scrollback are great too, and it even works under Tmux: Try typing Ctrl+Shift+R and you’ll get a search box in the bottom-right of your terminal. Note that the version of foot available in Ubuntu 22.04 LTS stock apt packages is 1.11, and to align with recent development you’ll need to add the following to your $HOME/.config/foot/foot.ini config file:

[key-bindings]
show-urls-launch=Control+Shift+o

You may also want to disable the extra space on the right of the terminal windwo by disabling the scrollback marker. Otherwise, you’ll get strange looking horizontal menu/status bar (e.g., tmux, vim, newsboat). This is easily achieved as follows:

[scrollback]
indicator-position=none

If you miss the Ubuntu mono font from Gnome terminal, just add font=Ubuntu Mono:size=8 in the [main] section.

Unfortunately, Neovim terminal does not support sixel for displaying image, only chafa and viu, which means you cannot benefit from foot’s support when using fzf-lua for previewing images in the fuzzy file finder.

Quick post, but if you are looking for cheap alternative to Gnome terminal with better input latency give foot a shot.

♪ Portishead • Mourning Air

Some interesting R packages

Tue, 19 Sep 2023 09:23:08 +0200

It’s interesting to see how few R code I write these days, compared to my early professional carrier as a medical statistician. I could have some opportunities to do biostatistical stuff but most of the time I am never asked to and my job resolves around genomics these days. That’s fine by me, I still have some opportunities to learn new things, especially in NGS and bioinformatics. At work I mostly use Python and Bash, because they fulfill their role for string processing and task orchestration. When I’m not involved in programming, I’m using more or less esoteric bioinformatic software, for which my rough estimate of their lifetime is about 2 years after having been described in a peer-reviewed paper. Anyway, I still do some statistical stuff, mostly for fun at home, and most of the time R comes to the rescue, even if I also use Stata and Mathematica.

Here are some R packages I discovered or reinstalled in 2023:

First, I learned that there’s a dedicated webiste for ggplot extensions. There you will find ggstance (horizontal boxplots, for instance), ggalt (useful for spline fitting) or ggradar (radar or spider charts, which I implemented in base R 14 years ago!). There is many cool stuff that been developed around ggplot, which is great.
gtsummary as a quick replacement for gt when one to summarize a data frame. It produces an HTML page by design. However, it supports several backends, including gt, to output PNG (requires the webshot2 package, and one of chrome-based browser), DOCX or TEX (as a longtable) files. Quick one-liner:

library(gt)
library(gtsummary)
tbl_summary(iris) |> as_gt() |> gtsave(filename = "/tmp/out.tex")

janitor to manage your dirty dataset. There’s no magic incantation, though, so I believe it does not prevent the user to closely inspect, and not only eye-balling, the data.
parameters for managing and displaying model estimates; not sure I would really need this, but I’ll keep it in mind.
emmeans and margins, although I’m used to the effects and rms packages.
dbplyr and multidplyr: I’m still not versed into the tidyverse, but I must admit that the dbplyr package offers the best of the two worlds, a robust query language on top of an efficient database system. As such, instead of using the data.table package, we can imagine using Arrow to read and/or write large data files, and duckdb to query the collected data using their efficient streaming algorithms, but see some examples of use in this tutorial. I’d like to benchmark this approach with a full-blown H2O solution. I was quite surprised by the performance and ease of use of the Python datatable package from the H20 team.

There are many other packages that I once installed but removed soon after using them. There are also lot of packages that I am not aware of. In fact, I stopped my technological watch years ago, and never really came back, so that I’m mostly comfortable with what I’ve learned long time ago. By the way, it was already cool. Curiously, while I was using the Bioconductor project and its packages 13 years ago for GWAS analysis, I no longer use them, except for RNASeq or metagenomics-related stuff.

♪ Benoît Delbecq • Anamorphoses

Helix looks great indeed

Mon, 18 Sep 2023 20:22:20 +0200

I already talked about the Helix editor. I try to use it on a daily basis from time to time. It works perfectly on virtual consoles in Ubuntu or via ssh if you don’t need an X server. I now have some consequent muscle memory anchored in my Neovim key bindings, and I often make some mistake when working with Helix. I believe I don’t use it often enough to really enjoy its capabilities and the editing model underlying this Kakoune/Vim hydra. I know there are some support for Vim keymapping, but I try to work with builtins.

What I really like:

It works out of the box. Just install the binary (from a tarball or Flathub), any additional LSP stuff you need (e.g., lldb-vscode, pyright, black), and check that it has been detected with hx --health. Then you’re probably good to go. There’s barely anything to configure (unless you use pyright LSP for Python).
There are a few fuzzy pickers builtin, and they are great: LSP finder and gotos, file picker, etc. Everything comes with a popup menu “à la which-key” which helps finding the right binding in case you forgot it. Otherwise, the popup menu is rather unintrusive if you are quick enough for any two key mapping.
Since it is a single app with everything bundled, it will work across different machines the same way. Even with no configuration (see above), you’ll get a very capable text editor.
Git support is acceptable, at least there’s a Git gutter and goto next/prev change. Bonus, you get a visual selection for free in the later cases.
Tree sitter support is excellent, and unlike Neovim you don’t need to install extra plugins (which don’t exist anyway, at the time of this writing).
Native DAP support is nice, especially when you see how picky it is to configure everything in Neovim with nvim-dap (the plugin is great, it’s just that it is a lot of extra configuration for lldb-vscode, debugpy, etc.). I’m eagerly awaiting full support for Python.
So many good defaults, e.g., start in a directory with hx . and you get a file picker.
I don’t use builtin colorschemes but they look gorgeous as advertised. Here is my boring colorless theme, by the way:

What I miss:

No built-in terminal. I use it a lot, be it for simply launching a program alongside or for running REPL or test suites.
Muscle memory, as I said. I wish there were a better way to add support for, e.g., $, 0 or _. Switching from Neovim to Helix too often means wandering through the vagaries of keyboard shortcuts to navigate text.
cw and yiw. Argh, you need to make a selection first. Muscle memory again…
Filetype-based text soft/hard wraping, reflowing, and things like that, although I am aware of the hard-wraping (:reflow) capabilities. I would really like soft-wraping and I miss Vim’s gqip for hard-wraping.
I miss Vim’s visual blocks which I find especially useful to add comment at the start of several lines or remove extra whitespace, etc. I know we can do the same with multiple selections, but it’s, well, different.
No spell checke, but this may change soon.
I still need to find my way through the use of pipes instead of Vim’s r! although it is a minor issue.

♪ Patricia Brennan • Improvisation VI

JASP for Bayesian statistics

Fri, 15 Sep 2023 18:05:41 +0200

I installed JASP recently as I wanted to see how it evolves since the last time I used it (several years ago). Unfortunately, there’s no package, tarball or AppImage so you need to install it from Flathub, using flatpak. With regard to my last post, this also was an occasion to try out flatpak. Overall, it works quite well, it doesn’t update itself or its packages de manière impromptue, and it is simple to use. Of course, you end up with giant applications since it bundles every dependencies it needs. For instance, JASP is as large as 2.6 Go once installed in /var/lib/flatpak. Anyway, the installation went smoothly, and I came to appreciate the flatpak way of installing desktop app. I might even replace all my AppImage’s with those from Flathub.¹

Back to Bayesian statistics. Several years ago I mentioned a Bayesian alternative to the traditional Student t-test, but the post found its way into the draft folder forever. Rather than writing and submitting to Stan or JAGS a model file, here is what I would do in R. First, using the brms package, we can mimic a t-test assuming unequal variance as follows:

> library(brms)
> m <- brm(bf(len ~ supp, sigma ~ supp), data = ToothGrowth, family = student, cores = 4)
--8<------ snip --
> summary(m)
 Family: student
 Links: mu = identity; sigma = log; nu = identity
Formula: len ~ supp
 sigma ~ supp
 Data: ToothGrowth (Number of observations: 60)
 Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
 total post-warmup draws = 4000

Population-Level Effects:
 Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 20.89 1.27 18.41 23.45 1.00 5225 2737
sigma_Intercept 1.87 0.14 1.59 2.15 1.00 4015 2762
suppVC -4.01 2.00 -7.84 -0.13 1.00 4398 2992
sigma_suppVC 0.22 0.20 -0.16 0.61 1.00 4197 2944

Family Specific Parameters:
 Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
nu 25.77 14.91 6.55 62.61 1.00 4616 2776

Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
>

The tidybayes package is a handy companion if you work with brms objetcs. We can compare the above results with what would be obtained from the now archived BEST package (latest version is from 2021):²

> library(BEST)
> grp <- with(ToothGrowth, split(len, supp))
> BESTmcmc(grp[[1]], grp[[2]], parallel = TRUE)
Waiting for parallel processing to complete...done.
MCMC fit results for BEST analysis:
100002 simulations saved.
 mean sd median HDIlo HDIup Rhat n.eff
mu1 20.797 1.2951 20.798 18.254 23.372 1 61131
mu2 16.889 1.6041 16.890 13.796 20.098 1 62000
nu 42.396 31.2307 34.208 3.113 104.074 1 24364
sigma1 6.765 0.9835 6.666 4.953 8.723 1 47897
sigma2 8.449 1.2427 8.314 6.178 10.928 1 43607

'HDIlo' and 'HDIup' are the limits of a 95% HDI credible interval.
'Rhat' is the potential scale reduction factor (at convergence, Rhat=1).
'n.eff' is a crude measure of effective sample size.

This takes a much shorter amount of time. Kruschke wrote a paper about this approach, ³ which is implemented by the above function call. From the on-line help:

The function uses a t-distribution to model each sample, and generates vectors of random draws from the posterior distribution of the center (mu) and spread or scale (sigma) of the distribution, as well as a measure of normality (nu). The procedure uses a Bayesian MCMC process implemented in JAGS ⁴.

Kruschke also discussed two-group comparisons in section 16.3 of the second edition of his famous textbook.⁵ As noted by the author, “When using t distributions for robust estimation, we could also estimate the normality of each group separately. But because there usually are relatively few outliers, we will use a single normality parameter to describe both groups, so that the estimate of the normality is more stably estimated.”

Now, it’s time to try out JASP. Here’s what we get after specifying an informed prior (Student t distribution with default scale factor of 0.707 and df=1) instead of a scaled Cauchy distribution:

This module relies on the BayesFactor package, where the ttestBF implements work made by Morey and coll.⁶ JASP also offers a Robust T-Tests module which provides Bayesian model-averaged t-test, as implemented in the RoBTT package. See also A conceptual introduction to Bayesian Model Averaging for an introduction. This involves longer computation time, but with default parameters we obtain the following results:

There are various options that I did not try, but overall I find that this is a great way to get started with Bayesian statistics for beginner students. No need to learn complex R code or to install various stuff. However, we need to carefully check what packages are used under the hood, what priors are used, and what statistics are reported (e.g., Bayes Factor, posterior estimates, etc.).

♪ Anna Weber • Idiom IV

I just noticed that the latest release of the Helix editor is available on Flahub as well. ↩︎
If you are using R in your terminal, R CMD INSTALL from a shell is your best friend. Note that you will also need to install the HDINterval. ↩︎
Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General 142(2):573-603. Please note that the author reported modes rather than means. ↩︎
Plummer, M. (2003). JAGS: A Program for Analysis of Bayesian Graphical Models Using Gibbs Sampling, Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC 2003), March 20-22, Vienna, Austria. ↩︎
Kruschke, J. K. (2015) Doing Bayesian Data Analysis (2nd ed.), Academic Press. ↩︎
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t-tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237 ↩︎

ArXiving on September 2023

Wed, 06 Sep 2023 12:19:00 +0100

Unidimensionality in Rasch Models: Efficient Item Selection and Hierarchical Clustering Methods Based on Marginal Estimates (https://arxiv.org/abs/2309.00553)

Since I am no longer involved in psychometrics or this kind of stuff (even on Cross Validated), I rarely read this kind of papers nowadays. Selecting misfitting items on a scale has a long history, especially for Rasch modellers. This paper exposes a sort of clustering method that allows to flag item, based on the variance of a mixing distribution, that do not belong with a set of items sharing a commin trait.

Optimal Scaling transformations to model non-linear relations in GLMs with ordered and unordered predictors (https://arxiv.org/abs/2309.00419)

Still a memory of my previous job where I used to use optimal scaling a lot, but in the context of projection methods in multivariate exploratory data analysis. Here, Meulman and collaborators consider optimal scaling as an alterantive framework to GAM to allow non linearity in discrete predictors. In particular, this alleviares the need for dummy coded variable levels, via quantization with monotonicity constraints, which facilitates the interpretation of the resulting output. There will probably an R package available at some point. See also ROS Regression: Integrating Regularization with Optimal Scaling Regression.

An introduction to graph theory (https://arxiv.org/abs/2308.04512)

A complete course on graph theory, including network flows, for graduate students. It is rather extensive (422 pp.) and there are a lot of illustrations.

Permutation testing in high-dimensional linear models: an empirical investigation (https://arxiv.org/abs/2001.01466)

This is an extension to the permutation testing framework to the case where the number of predictors exceeds the sample size. In classical settings ($p < n$), the startegy amounts to compute the test statistics under random permutation of the residuals.¹ When $n \ll p$, however, regularization methos like the elastic net appoach msut be used. The authors recall that for minimizing prediction error, ridge regression is often preferrable to Lasso, principal components regression, variable subset selection and partial least squares. Moreover, ridge regression is close to the Freedman-Lane approach, which is based on semi-partial correlations. The authors finally suggest to use double residualization, which is inspired by the Kennedy method, which residualizes both $Y$ and $X$ and proceeds to permute the $Y$-residuals, ² but in this paper the authors replace the least squares regression by ridge regression.

Tutorial: a priori estimation of sample size, effect size, and statistical power for cluster analysis, latent class analysis, and multivariate mixture models (https://arxiv.org/abs/2309.00866)

This is a short review of power analysis for k-means, Ward agglomerative hierarchical clustering, c-means fuzzy clustering, latent class analysis, latent profile analysis, and Gaussian mixture modelling. Results based on simulated datasets are summarized in Table 1 of the paper, reproduced below.

Supervised dimensionality reduction for multiple imputation by chained equations (https://arxiv.org/abs/2309.01608)

In this paper, the authors discuss the use of principal component, principal covariates, and partial least squares regression, instead of unsupervised PCA, with MICE. Results show that supervised appraoches perform better, and that supervised principal component regression has smaller bias and better confidence interval coverage for a wider range of retained components, independent of the number of latent variables.

♪ Bauhaus • Stigmata Martyr

Freedman, D. and Lane, D. A nonstochastic interpretation of reported significance levels. Journal of Business & Economic Statistics, 1(4):292–298, 1983. ↩︎
Kennedy, F. E. Randomization tests in econometrics. Journal of Business & Economic Statistics, 13 (1):85–94, 1995. ↩︎

Why I like Ubuntu, eventually

Mon, 04 Sep 2023 15:34:30 +0200

A few years ago I switched to Linux after having been a fervent Mac user for over 13 years. I somewhat lost interest in the Apple ecosystem on my Macbook, and I was mostly using TUI applications anyway. I never upgrade past Mojave, and I’ve been following the up and down of each new release afterwards. My phone is an iPhone 6 which can’t be upgraded anymore. I don’t plan to buy a new one: too expensive, too large. I only bought the iPhone 13 mini for my son for his birthday, at a reduced price. Anyway, I wish the iPhone 3 was still a thing. I don’t really like the idea that OS upgrade are so much tighted to the device. And the sandboxing policy, which I always found at an angle with what OpenBSD offers, for example. Well, I already wrote my thoughts about all of that, apparently.

Switching to Ubuntu was like going back in time since I was a Linux user during my PhD, and I used a lot of distros: Madrake (1 yr), Red Hat (1 yr), Slackware (6 mo) and finally Debian (2 or 3 yr). I didn’t really miss anything when I started using Ubuntu as I was comfortable with the command line, and all the applications I used on the Mac mainly ran on the command line. I also installed OpenBSD on another laptop, but no other Linux boxes. Note that I did not try any other mainline or fancy Linux distros, so in this post I am only speaking of Ubuntu.

Recently, I came across a nice blog post by Alan Pope: Why I use Ubuntu. Among the key points, I like that it works and that it offers a relatively secure and consistent UI. Compared to Debian, it is way easier to bring back to life an old Macbook pro with retina display and specific wireless driver. I may try Fedora at some point, at least to remind me of my RPMs days. I should not that there are things that I don’t really like, but snap is probably one of the ugly thing that comes to my mind. You will even find outdated versions of deb packages (e.g., the tree program). I already purged everything related to snap on my machine, but I keep seeing applications that are only available via snap or flatpak (e.g., the statistical packages Jasp or Jamovi) along with Docker sometimes. At this point, I don’t really know if flatpak is a better option or not. Maybe I should give it a try. Anyway, here’s a good thread on how to uninstall snap and/or install flatpak instead: How to completely remove snap and replace with flatpak?.

I’ve been using Ubuntu now for eighteen years. I know (roughly) how it works. I am familiar with the release cadence and set my expectations accordingly. I can manage to upgrade from one release to another without the sky falling in. When something does go wrong, I’m able to fix it or find out how to. I rarely ’nuke and pave’ a system, even if it seems beyond repair.

I am probably a lucky guy but my Ubuntu personal machine never crashed. I had occasional problems on one of the two servers I manage at work, but this was easy to fix. I have a very spartan use of graphical applications, and I mostly work in a terminal. I discussed this extensively in previous posts. What I really like is the package manager: compared to Homebrew on my Macbook, everything is in order, perfectly in sync with upstream versions, without any dependency issues (most of the time). When you remove a program, the OS takes care of properly (most of the time) removing its dependencies as well.

I’ve tried various setups in the past (i3 or Regolith on x11), pseudo-tiling WM (Tactile extension for Gnome) and then settled upon the default settings, pending minor customizations (e.g., removing the Dock, enhancing the menu bar), but see How to get by without using a tiling WM. Tmux is really all what I need to manage my terminal sessions, and you know, it’s just plain text.

Software should be well designed because good design avoids many problems during its lifetime. Also, because good design can offer much additional gain. Indeed, much effort should be spent on good design to make software more valuable. The Unix Philosophy pro- vides a way to design software well. It offers guidelines to achieve good quality and high gain for the effort spent. — Why the Unix Philosophy still matters

♪ Adam Wakeman • The Island

Micro posting in August

Thu, 31 Aug 2023 09:22:17 +0200

2023-08-21:
Home sweet home.
2023-08-21: GNU Parallel, where have you been all my life?: Yet another nice example of why parallel is such a great program. I learned to use it a while ago, after having read Jearoen Janssens’ wonderful book, Data Science at the Command Line, in its first edition. BTW, the 2nd edition is available online for free.
2023-08-21: Non-Euclidean Geometry Online: a Guide to Resources: As the title suggests, a bunch of online resources to learn more about elliptical and hyperbolic geometries.
2023-08-22:

This post is a detailed discussion into user profiles, their directories, and how they are—to put it bluntly—in total disarray on Windows and Linux (I haven’t used a Mac in ages, but I assume the situation is very similar there, too). Applications treat the user profile as a dumping ground, and any user with a reasonably wide list of installed software will find their user profile very difficult to traverse after some time in use. There are platform conventions and attempts to standardise things on more open-source platforms, but a lot of developers resolutely refuse to change the behaviour of their software for a variety of reasons (some less valid than others). — $HOME, Not So Sweet $HOME

2023-08-22: TIL about lei: Using lei, b4, and mutt to do kernel development.
2023-08-23:

If you cannot solve the proposed problem try to solve first some related problem. Could you imagine a more accessible related problem? A more general problem? A more special problem? An analogous problem? Could you solve a part of the problem? — G. Polya, How to Solve ItWhich is probably why we face so many times the XY problem on forum, QAs sites, and mailing lists.

2023-08-23: The missing C++ smart pointer.
2023-08-23:

Sneak preview of summer’s latest culinary preparations.
2023-08-28:

Also, whenever someone tells me that Arch never breaks, “You just have to read the blog”. Uh, okay. No thanks. I’m not reading all the latest news about any other device I allow updates to. I’m not starting that nonsense in my fifties. — Why I use Ubuntu

2023-08-28:

Now, I’ll be quick to admit that you probably shouldn’t spend a huge amount of time trying to install something from source. For many packages the best case scenario is a complicated process requiring> specific build tools and dependencies with their own complications. The worst case scenario is poor or lacking documentation and no idea how to even get started. — Try installing from source firstI did install from scratch almost any app/program on my Mac 15 years ago. At that time, Mac systems were transitioning from 32 to 64 bits, and it was kind of a mess for some programs. Then I got bored and I choose to rely on Homebrew. Then Homebrew got sort of broken on my Macbook, and I was back at the magic incantation configure && make && make install.

2023-08-28: As a sequel of one of my last benchmarking post, here’s a rough estimate of Polars vs. Datatable performance when reading a 34 Mb file (NYC Flights Dataset, available from avrious sources; see e.g., this post):

import time
import datatable
import polars

tic = time.time()
flights = polars.read_csv("nycflights.csv", null_values="NA")
toc = time.time()
print(f"Elapsed time: {(toc-tic) * 10**3:.2f} ms")
## => Elapsed time: 54.61 ms

tic = time.time()
flights = datatable.fread("nycflights.csv")
toc = time.time()
print(f"Elapsed time: {(toc-tic) * 10**3:.2f} ms")
## => Elapsed time: 57.60 ms

Polars also offers a lazy CSV reader using scan_csv, which is way faster (1.22 ms). #python
2023-08-28: A Modern C Development Environment: A complete step by step tutorial to develop using Docker and GitHub workflow. Note that clang-tidy is available in ALE for Vim. If you use Neovim, the the clangd LSP already uses clang-tidy, or you can just add --clang-tidy to clangd parameters. Extra options can also be added in a .clang-tidy file. See also clang --help-hidden.

2023-08-28: Email Authentication: A Developer’s Guide: I got my emails treated as spam by Gmail when using my www provider (OVH), which is probably why. And this doesn’t happen when I send emails via may mac.com account. Note that Google only started rejecting my principal domain name a few months ago, or maybe last year.
2023-08-28: Identity Beyond Usernames. For random UUIDs, see also Fixed Bits of Version 4 UUID.
2023-08-29: My thoughts on Haskell in 2020. tl;dr Stick to Simple Haskell. #haskell
2023-08-29: Understanding Automatic Differentiation in 30 lines of Python: A simplified yet effective illustration of AD in the context of neural network training. #python
2023-08-29: Why does all() return True if the iterable is empty?. #python
2023-08-29:

Mood of the day.
2023-08-30: Until now when reading my feeds using Newsboat, I was simply filtering unread feeds, sorted by count using a dedicated keybinding (bind-key S rev-sort, which allows me to simply press Su to get the desired output). Now, I learned about l which filters unread feeds. It is quite handy since it acts as a filter and the list of unread feeds empties itself as you read articles.
2023-08-30: ♪ Benoît Delbecq · Anamorphoses
2023-08-31:

Nushell isn’t exactly a shell, at least not in the traditional Unix sense of the word. Nushell is trying to answer the question: “what if we asked more of our shells?” — The case for Nushell

With recent support for data frames, it becomes an interesting challenger for tiny on-the-fly data wrangling scripts. Beware that it relies on polars-arrow which (most of the time) require a recent stable Rust. Also, if you encounter an E0034 (“multiple applicable items in scope”) when compiling polars-related dependencies, try adding --locked to cargo install. When you’re done, enjoy the power of data frames right into your shell!With my last timing for processing a 30 Mb CSV file, loading the file in Nu shell is a breeze:

~/tmp> timeit {dfr open ./nycflights.csv}
70ms 682µs 766ns
~/tmp> timeit {open ./nycflights.csv}
1sec 194ms 477µs 961ns

First instruction uses dfr open while the second instruction relies on builtin open command.

2023-08-31: Among many of the text-based UIs advantages, I found that the ability to copy-paste almost anything using Vim shortcuts via tmux or the shift+ctrl combo of my terminal emulator is one of my preferred. Here is an excerpt from my long-running (server side) Irssi session, as a tribute to Vim’s creator:

23:57 -A_Dragon(A_D@libera/staff/dragon)- [Global Notice] The Matrix bridge is experiencing continued stability and privacy issues; we have requested it be shut down until they are resolved. Matrix have agreedto shut down the bridge no later than 1400UTC on Saturday. Feel free to ask further questions in #libera-matrix
---- Day changed to 05 août 202317:35 WALLOP kline: Bram Moolenaar finally :wq after many years contributing to free software, his relentless hard work making vim arguably the #1 editor in the world. RIP Bram, and thank you for your work.https://groups.google.com/g/vim_announce/c/tWahca9zkt4
---- Day changed to 06 août 2023

2023-08-31: TIL that GitHub renders warning or note blocks with a special formatting in Markdown document, e.g. here.

Bayesian power analysis and confidence intervals

Wed, 30 Aug 2023 09:33:34 +0200

We are often tasked with computing the number of subjects needed to achieve a certain statistical power, especially in biomedical sciences. Various formulae are available online or in dedicated textbooks.¹ Here is a simple illustration with a two-sample proportion: Let’s say we want to compare the efficacy of a new treatment (N) with that of the reference treatment (R) in a clinical trial. The outcome is the percentage of complete remission after one week. We know that this percentage equals 50% with R, and we expect 80% of recovery with N.² To achieve 95% statistical power, how many patients do we need to enroll?

First, let’s say we can only enroll 50 patients per group. Then, using standard formulae and considering a 5% type I error, $\Phi = \frac{1.107 - 0.785}{\sqrt{0.25 \times 2/50}} = 3.22$, which means we will end up with 90% power.³ Now, to achieve 95% statistical power, $\Phi = 3.61$, hence $n = \frac{2 \times 0.25 \times 3.61^2}{(1.107 - 0.785)^2} = 63$ subjects per group. Dealing with proportions usually involves the arcsin transformation, which can be applied to raw data or directly to estimated proportions. This is discussed in Ryan¹, pp. 116-117. Here is what I get with Stata 13 for a two-sample proportions likelihood-ratio test: (Note that we get N=63 when using a chi-squared test)

. power twoproportions 0.5 0.8, test(lrchi2) power(0.95)

Performing iteration ...

Estimated sample sizes for a two-sample proportions test
Likelihood-ratio test
Ho: p2 = p1 versus Ha: p2 != p1

Study parameters:

 alpha = 0.0500
 power = 0.9500
 delta = 0.3000 (difference)
 p1 = 0.5000
 p2 = 0.8000

Estimated sample sizes:

 N = 130
 N per group = 65

The same principle applies to the case of the arithmetic mean and a two-sample Student t-test, of course. Note that in the case of small proportion (here, the difference between $P_N$ and $P_R$, the number of subjects required to achieve a certain power $1-\beta$ will grow up quickly as $\beta$ decreases: to detect a 5% difference in proportion, with $P_R=0.5$, using 1000 subjects per arm will give you only 61% power; for a 2.5% difference with the same number of subjects, don’t expect to get more than 20% power.

Bayesian approaches⁴ usually yield smaller sample sizes compared to frequentist approaches, and most of the time no closed-form solutions do exist which means numerical computation must be undergone. I found an interesting review of the pros and cons of those approaches in a recent article of The American Statistician: A Review of Bayesian Perspectives on Sample Size Derivation for Confirmatory Trials. And here is a blog post that discusses a simulation-based approach for power analysis using posterior distributions: Sample size determination in the context of Bayesian analysis.

Likewise, computing confidence intervals most of the time relies on the use of a normal approximation in the case of two-sample proportions, and it is covered in most textbooks in the case of two-sample experiments. Basically, we would compute a 95% CI for the difference between two proportions as follows: $(P_N–P_R) \pm \Phi^{-1}(\alpha/2)\sqrt{P_N(1-P_N)/n_N + P_R(1-P_R)/n_R}$, where $\Phi$ is the cumulative distribution function for the standard normal distribution (with mean 0 and variance 1), and $\alpha$ is the predefined type I error rate. Typically, $\alpha=0.05$, and $\Phi^{-1}(0.025)=-1.96$ and $\Phi^{-1}(0.975)=1.96$.

Now, we can take a different approach and rely on credible intervals, as illustrated in Murphy’s updated edition of his famous ML textbook,⁵ which I am currently reading. The problem is stated as follows: Let $X\sim\mathcal{N}(\mu, \sigma^2=4)$ where $\mu$ is unknown but has prior $\mu\sim\mathcal{N}(\mu_0,\sigma_0^2=9)$. The posterior after seeing $n$ samples is $\mu_n\sim\mathcal{N}(\mu_n,\sigma_n^2)$. How big does $n$ have to be to ensure $P(l\leq\mu_n\leq u\mid D) \geq 0.95$, where $(l,u)$ is an interval centered on $\mu_n$ of width 1, and $D$ is the data?

Here is the full derivation in the case of a two-sample problem with a continuous outcome. As stated, we want an interval that verifies $P(l\leq \mu_n \leq u\mid D) \geq 0.95$, where as usual:

$$ \begin{equation} \begin{aligned} l & = \mu_n + \Phi^{-1}(\alpha/2)\sigma_n = \mu - 1.96\sigma_n \cr u & = \mu_n + \Phi^{-1}(1-\alpha/2)\sigma_n = \mu + 1.96\sigma_n \end{aligned} \end{equation} $$

We need to find $n$ such that $u - l = 1$, that is $2(1.96)\sigma_n = 1$, or $\sigma_n^2 = \frac{1}{4(1.96)^2}$ with $\sigma_n^2 = \frac{\sigma^2\sigma_0^2}{n\sigma_0^2+\sigma^2}$. Hence we have

$$ \begin{equation} \begin{aligned} n\sigma_0^2 + \sigma^2 & = (\sigma^2\sigma_0^2)4(1.96)^2 \cr n & = \frac{\sigma^2(\sigma_0^24(1.96)^2-1}{\sigma_0^2} \cr & = \frac{4(9\times (1.96)^2-1}{9} = 61.0212 \end{aligned} \end{equation} $$

We would need at least $n\geq 62$ subjects per group.

♪ Jean Carn, Adrian Younge & Ali Shaheed Muhammad • Black Rainbows

Thomas P. Ryan. Sample Size and Determination and Power. Wiley, 2013. ↩︎ ↩︎
Bouvenot & Vray. Essais Cliniques. Théorie, Pratique et Critique (4ème éd.). Lavoisier, 2006 (p. 248). ↩︎
An other interesting question would be to ask what is the minimal expected recovery percentage we would get with 50 patients per group at 80% power. The derivation is as follows: $\Pi_N - \Pi_R = 2.80\sqrt{(2 \times 0.25)/50} = 0.28$, so that $\Pi_N = \Pi_R + 0.28 = 0.785 + 0.28 = 1.065$. In other words, $P_N = 76%$. ↩︎
I’m not talking about Bayesian updating whereby we sample size is increased during the course of a study by using the Bayes factor to quantify the degree of support for a hypothesis agaisnt the alternative given the observed data. ↩︎
Probabilistic Machine Learning: An introduction. See all available textbooks here. ↩︎

August in review (2)

Mon, 21 Aug 2023 13:41:36 +0200

As a sequel of one omy last posts, here are some more bookmarks that were awaiting some kind of a short review.

Infrequently asked questions in comp.lang.c: I once thought about writing a similar thing for common questions on Cross Validated, then gave up since it became clear that it would require way much more time that I could possibly allow for this task.
WTF is going on with R7RS Large?: Besides a short but nice summary on how Scheme implementations evolved overtime, this post tries to explain what happened behind the scenes regarding the future of R⁷RS Large which should be entirely SRFI-based, contrary to previous revisions of the Scheme language.
A Guide to Undefined Behavior in C and C++, Part 1: A 3-part series on undefined behavior (resulting from erroneous operations like accessing a null pointer or dividing by zero) in C, as the title suggests.
Probabilistic Machine Learning: Advanced Topics: I enjoyed reading the very first book written by Kevin Murphy, Machine Learning: A Probabilistic Perspective. Then I discovered that he wrote two other books totalling more than 2000 pp. of fresh material.
Here is new version of Modern CSV if you happen to process raw CSV files in your terminal.
How to Recalculate a Spreadsheet: Lot of useful tips and informations regaridng the management of live spreadsheets.
Web reading list: I like when bloggers provide a list of the blog they follow themselves, or directly an RSS or Atom feed. Check out his website too, it is beautifully organized and laid out.
Decimal to fraction: Another great post by Dr. Drang on continued fractions and their convergents (i.e., rational approximations to an irrational number represented as an infinte continued fraction).

Code is not literature and we are not readers. Rather, interesting pieces of code are specimens and we are naturalists. So instead of trying to pick out a piece of code and reading it and then discussing it like a bunch of Comp Lit. grad students, I think a better model is for one of us to play the role of a 19th century naturalist returning from a trip to some exotic island to present to the local scientific society a discussion of the crazy beetles they found: “Look at the antenna on this monster! They look incredibly ungainly but the male of the species can use these to kill small frogs in whose carcass the females lay their eggs.” — Code is not literature

The Vi Lovers Home Page … this makes a great starting point. Agree. See also this HN thread. And RIP Bram Moolenaar.
Symmetry in Chaos: Beautiful illustrations of various attractors.

Why Static Languages Suffer From Complexity: A sound review of statically and dynamically typed PLs, where the statics level is where all linguistic machinery is being performed at compile-time, while the dynamics level is where code is being executed at run-time, according to the author.
Clojure’s deadly sin: Interesting discussion of Clojure’s lazyness (as a feature).
The Reluctant Sysadmin’s Guide to Securing a Linux Server.
I’m betting on HTML: “What I mean is that people are not typically motivated to adopt new social media platforms for reasons that may not be entirely clear. I’m not condemning these efforts - I believe there’s a future there, and I’m watching as the development progresses. However, I believe we’re already sitting on a tried and tested solution: HTML.”
It’s 2023, so of course I’m learning Common Lisp: TL;DR CL looks great for interactive development, including debugging.
What every IT person needs to know about OpenBSD. That says it all!

♪ Estrada Real • Chico Pinheiro

August in review

Mon, 21 Aug 2023 09:53:01 +0200

Here is a not so short list of things I bookmarked during the summer.

Blowing up my compile times for dubious benefits: An interesting discussion (and application in Rust) on optimizing move generators in chess engines.
Why Modality and Chording?: Probably one of the best discussion I’ve seen on Vim modal editing vs. Emacs chording.
Do you know how much your computer can do in a second?: Nicely packaged and very instructive.
Elements of Data Science: A new textbook by Allen Downey on data science, with applications in Python. What I like with Downey’s books is what he said upfront: “When I use a term, I try to define it immediately, and when I use a programming feature, I try to explain it.”
deta: Functional Database Mapping: It provides a handy way of mapping data between database tables and Racket structs. I will need to try this at some point.
Structures in C: From Basics to Memory Alignment: For those interested in C programming.
A Functional Introduction To Computer Science: This is a two-part textbook on functional programming with Racket, with lot of exercises.

When I first started teaching using Racket in first year, I was inspired to change my research focus to functional programming languages. Consequently, when I set out to design the two-course sequence whose materials I adapted for FICS, I had a wealth of recent reading to draw on. Among the sources I used, besides the first edition of HtDP, were the early editions of “Programming Languages: Application and Interpretation” by Shriram Krishnamurthi, the second edition of “Essentials of Programming Languages” by Friedman, Wand, and Haynes, “LISP In Small Pieces” by Christian Quienniec, “The Scheme Programming Language” by Kent Dybvig, “ML For The Working Programmer” by Larry Paulson, and the papers of Chris Okasaki. There are others I’m forgetting because I’m writing this away from my office bookshelves.

Extending a Language — Writing Powerful Macros in Scheme: The author uses Chez Scheme (and Emacs) to illustrate the definition and use of syntax objects and syntax-case macros. For another in-depth account of Scheme macros, see The Adventures of a Pythonista in Schemeland, and Alexis King’s posts on that very specific topic.
Is there an open source implementation of Mathematica-the-language?: Some useful links are to be found in this thread and associated comments.
CPU Utilization is Wrong: In which we learn that CPU utilization is a misleading metric, most of the time.
Division of integers by constants: Even if the focus is on embedded systems, the two recipes discussed in this post are highly instructive (TL;DR division by a constant is equivalent to multiplication by the reciprocal of that constant).
How Tail Call Optimization Works: A short illustration of recursive tail-call in C.
{n} times faster than C - part one: On how to overoptimize a simple C loop.
Clojure Plays Mario: In which tha uthor describes an AI that could complete all of the levels in the original Super Mario Bros for the NES (except for the Bowser levels that have mazes).
Web Browser Engineering: An online book on designing a small bu efficient web browser in Python.
An Introduction to Statistical Learning: I never realized that there was also a Python version of the ISLR textbook. I guess we should call it ISLP then.

Even if the ideas you’re sharing with the world aren’t original you can enrich them with your personal view. As Bill Thurson said 2 all of us have clear understanding of a few things and murky concepts of many more. There is no way to run out of ideas in need of clarification. — Nobody cares about your blog

♪ Aaron Goldberg • Poinciana

The unquantified self #29

Thu, 10 Aug 2023 10:07:43 +0200

A whole month without writing anything. What a salutary change of scenery after all! I must say I was getting a bit fed up with the computer and I really needed some holiday away from it all. Done. Back to the blog, whatever that means these days.

I have a draft which dates back from mid July, and a long list of bookmarks which are awaiting for my review. Other than that, nothing’s really new. I took some days off, played Minecraft for many long hours, helped a friend of mine renovating his flat, done a lot of tidying up and renovating in my own flat, and here we are.

Work in progress

I watched a lot of not-so-great TV series recently, but I dicovered a nice internet radio (ABC Lounge). I also have a long list of books from the Actes Sud collection to read or reread, but see below. I haven’t read a lot this year, which is somewhat surprising since I’m used to read almost 100 books a year.

As can be seen in the above picture (right), the gaming station for my son is ready as well.

♪ Phidel • The Stag vs the Hare

Micro posting in July

Sun, 30 Jul 2023 11:56:26 +0200

2023-07-06:

Data-oriented design takes its cues from the data which is seen or expected. Instead of planning for all eventualities, or planning to make things adaptable, there is a preference for using the most probable input to direct the choice of algorithm. Instead of planning to be extendable, it plans to be simple and replaceable, and get the job done. Extendable can be added later, with the safety net of unit tests to ensure it remains working as it did while it was simple. Luckily, there is a way to make your data layout extendable without requiring much thought, by utilising techniques developed many years ago for working with databases. — Data-Oriented Design

2023-07-06:

Usually I send read-only documents to clients. They don’t know or care what program created the> PDF I sent them. The fact that they cannot edit my reports is a feature, not a bug: if I’m going> to sign off on something, I need to be sure that it doesn’t include any changes that someone else> made that I’m unaware of. Convert LaTeX to Microsoft WordI used to do the same for years when I was consulting. Main issue was tabular output, which can be beautifully rendered in $\LaTeX$, but not so in plain MS Word.

Regression and DFBETAS

Tue, 11 Jul 2023 20:55:58 +0100

In a previous post, we discussed how influence measures can provide some insights on data quality or model fit. There are various cutoff values that are used to interpret each of these quantities, e.g., $\text{Cook’s d} > 4/(n-k-1)$, $\text{DFFITS} > 2\sqrt{p/n}$, $\text{DFBETAS} > 2/\sqrt{n}$. The latter emphasizes regression coefficients themselves.

Considering a linear predictor $\beta_0 + \sum_{j=1}^k\beta_jx$, the DFBETA value for the $j$th coefficient for the $i$th observation is $\hat\beta_j - \hat\beta_j^{(-i)}$, where $\hat\beta_j^{(-i)}$ is the $j$th coefficient of the regression estimated without the $i$th observation. It’s mostly the jacknife approach to validating a regression model by discarding one observation at a time. From a computational perspective, it is not necessary to compute all $n$ full model since we can rely on linear algebra to relate OLS fit for all $n$ observations to that of any based on $n-1$ observations.¹ Often times, absolute values of DFBETAs are considered instead of signed values, and it is also common to standardize those values since their scale depend on that of the variable of interest. Standardization is done by dividing DFBETAs by the corresponding standard deviation of the $i$th-deleted regression model, and statistical packages often refer to them as DFBETAS instead of DFBETA (note the extra S).

The threshold or reference value, $2/\sqrt{n}$, can be seen as a size-adjusted influence effect accounting for the standard error estimate of the corresponding coefficient. In effect, it amounts to highlight the same theoretical number of influnetial points regardless of sample size, much like when using standardized residuals and referring their distribution to a Student $t$ or $\mathcal{N}$ distribution.

Detecting Influential Points in Regression with DFBETA(S) provides an illustration using simulated datasets of varying size in R. There are some other great articles on the StatLab Archives from the University of Virginia. I am familiar with resistant or robust regression methods (the aforementioned blog post shows how to use DFBETAS to overcome the effect of influential points on a regression line). Here is some old R code illustrating the use of robust regression, as described in Fox & Weisberg on a subset of the Ascombe dataset:

options(hrbrthemes.loadfont=TRUE)
library(MASS)
library(ggplot2)
library(gridExtra)
library(hrbrthemes)
library(directlabels)

set.seed(101)
theme_set(theme_ipsum(base_size = 11))
clr6 = ipsum_pal()(6)

p = ggplot(data = anscombe, aes(x3, y3)) +
 geom_point() +
 geom_smooth(method = "lm", se = FALSE, col = clr6[1]) +
 geom_smooth(method = "rlm", col = clr6[2]) +
 labs(x = NULL, y = NULL) +
 annotate(geom = "text", x = 13.5, y = 10.5, label = "LM", size = 2.8, col = clr6[1]) +
 annotate(geom = "text", x = 13.5, y = 8, label = "RLM", size = 2.8, col = clr6[2])

ggsave("~/fig-lrm-anscombe.png", p3, w = 6, h = 4, dpi = 300)

I just learned about Theil-Sen Regression. Instead of minimizing squared residuals, the slope is calculated by taking the median of the slopes between each pair of points in the data, which basically amounts to compute every possible $(y_j - y_i)/(x_j - x_i)$ if $x_i \neq x_j$.² It follows that the OLS estimate of a simple regression slope can be conceptualized as a weighted average of pairwise slopes, with weights $(x_j - x_i)^2$. Go read the rest of the blog post if you are interested in robust regression.

♪ Grateful Dead • Viola Lee Blues

Belsley, D. A., Kuh, E., and Welsch, R. E. (1980). Regression diagnostics: Identifying influential data and sources of collinearity. John Wiley & Sons. ↩︎
Young, D. S. (2017). Handbook of regression methods. Chapman and Hall/CRC. ↩︎

Data frames for Racket

Mon, 10 Jul 2023 14:02:08 +0200

Lately, I’ve been playing with another implementation of data frames in Racket: tabular-asa. The data-frame package by Alex Harsányi is the first one I used (one or two years ago) and I found it comfortable to use except it doesn’t handle categorical variables, but see my preceding review.¹ This one doesn’t either, but it handles missing values, like the data-frame package, and grouping or aggregating functions are a bit more intuitive to use (no need to rely on an external index) and closer in spirit to the R or Python ecosystem.

Here is a toy example inspired by Travis Hinkleman’s recent post about his own implementation of data frame for Chez Scheme. Note that the NYC flights dataset is a 29 Mo file, so it stands for a relatively honest benchmark for Racket’s data processing capabilities. In what follows I am mostly interested in the time it takes to load such a dataset.

First, let’s try the default CSV reader. Hereafter I’ll be using a poor-man solution for timing the whole process of loading the dataset in Racket REPL (I’m using Racket 8.6 [cs]). As a result, the data will not be available outside the scope of this block. You will need to define the variable properly to access it later.

(require csv-reading)
(require net/url)

(define file
 (get-pure-port (string->url "https://www.travishinkelman.com/data/nycflights.tsv")))

(time
 (define data (csv->list file))
 (void))
; => cpu time: 4366 real time: 9121 gc time: 410

Not very convenient since we get a list of lists that would require some post-processing, but it’s fast enough. Here is a quick benchmark of the data-frame package for the sake of comparison. Note that I converted the TSV to a proper CSV format since this package doesn’t handle other separator.

(require data-frame)

(define file "/home/chl/tmp/nycflights.csv")

(time
 (define data (df-read/csv file #:headers? #t))
 (void))
; => cpu time: 3708 real time: 3708 gc time: 759

Since it doesn’t handle categorical variables or dates (meaning they are stored as raw strings) some procedures like df-describe are pretty useless for variables like carrier, dest, or time_hour, and there’s not much you can do unless you recode then using numeric codes, which is what I ended up doing once. Anyway, there’s a handy tutorial on the author’s website if you want to learn about this package.

Now, here is what we get with tabular-asa:

(require tabular-asa)

(time
 (define data (call-with-input-file "nycflights.csv" table-read/csv))
 (void))
; => cpu time: 7135 real time: 7142 gc time: 1271

It looks like it takes a bit longer to process the whole file, but that’s okay. We’ll get plenty of useful procedures to inspect and munge this dataset.

(define subset (table-cut data '(distance dep_delay dest)))
(display-table subset)
; => distance dep_delay dest
; 0 1400 2 IAH
; 1 1416 4 IAH
; 2 1089 2 MIA
; 3 1576 -1 BQN
; 4 762 -6 ATL
; .. ... ... ...
; 336771 213 #f DCA
; 336772 198 #f SYR
; 336773 764 #f BNA
; 336774 419 #f CLE
; 336775 431 #f RDU
;
; [336776 rows x 3 cols]
(table-shape (table-distinct subset '(dest)))
; => 105
 3
(define grouped-data (group-count (table-groupby (table-cut subset '(dest dep_delay)) '(dest))))
grouped-data
; => dest dep_delay
; 0 ABQ 254
; 1 ACK 265
; 2 ALB 419
; 3 ANC 8
; 4 ATL 16898
; .. ... ...
; 100 TPA 7407
; 101 TUL 299
; 102 TVC 96
; 103 TYS 579
; 104 XNA 1011
;
; [105 rows x 2 cols]

It appears this package provides a load of useful features, which may be familiar to R’s data.table users. We need to be careful since some procedures return a sequence (which you may need to convert to a stream to consume its first or rest values eventually) or a table, depending on the context. With the above grouped data table, we can easily build a kernel density estimate of departure delays with a few additional commands:

(require plot)

(define xs (for/list ([x (table-column grouped-data 'dep_delay)]) x))

(parameterize ([plot-font-face "Roboto Condensed"]
 [plot-font-size 11])
 (plot (density xs)
 #:x-label "Departure delay"
 #:y-label "Density"
 #:out-file "/tmp/plot.png"))

Of course, most of the above grouping and aggregating operations could be performed with the data-frame package. What’s interesting in this case is that author makes heavy use of lazy data structures and relies on a column-major internal representation of rectangular datasets.

♪ Glenn Gould • Goldberg Variations

I also came across uke in addition to the packages mentioned in my previous post, and the dataframe package. I will review them when time allows. ↩︎

Micro posting in June

Sat, 01 Jul 2023 09:41:19 +0200

2023-06-06:

But the language landscape has changed a lot since then, and realistically no programmer today cares about what made a language stand out 50 years ago. Clearly, the good ideas have been copied into other languages. Paul Graham even suggests this convergence towards Lisp is inevitable. I wouldn’t go so far. But this begs the question: Is there anything left? Are there any features that couldn’t be copied so easily into the various descendants of Algol? — What Are the Enduring Innovations of Lisp?

2023-06-06:

The idea is that if I found it confusing, lots of other people probably did too, even though the information might theoretically be out there on the internet somewhere. Just because there is information on the internet, it doesn’t get magically teleported into people’s brains! — Some blogging myths

2023-06-06: Reduce vs fold in Common Lisp: Interestingly, R whihc is often referred to as a Scheme-like language has Reduce as one of its higher-order function, but no fold[lr] functions in base. Modeling Data With Functional Programming in R (whole book available in 2024 maybe?) looks like an interesting take on this topic. #rstats #lisp
2023-06-12:

Your true audience (those who care) is about 1% of your subscribers. 1000 followers on Twitter?> Your audience is 10 people. 10000 email subscribers? 100 people. — Blogging Myths You Should Care About

2023-06-12: ♪ Johanna Warren · I’d Be Orange
2023-06-15:

In my view this isn’t about adhering to the XDG standard[3], it’s about getting things out of $HOME. Unix dotfiles were always a (somewhat accidental) hack[4], and over the years we’ve accumulated entirely too many of them in our $HOMEs. The XDG option isn’t particularly perfect, but it’s at least a standard approach and it achieves the goal of getting dotfiles out of $HOME. As a side effect the XDG approach makes things more legible if you look in ~/.config. — Where your program’s configuration files (‘dotfiles’) should go today

2023-06-19:

Shell programming is remarkably easy in many cases; what’s sad is that this common case (file processing) is far complicated than it needs to be. This is not a problem limited to shell; while shell is especially tricky, it is difficult to correctly process POSIX pathnames in all languages. — Filenames and Pathnames in Shell: How to do it Correctly (via The shell scripting trap)

2023-06-19: >>> import sqlean as sqlite3. #database #python
2023-06-20: Interesting Lisp-related stuff on this HN thread about Reddit. #lisp
2023-06-20: Just found out that we can get cover art with cmus thanks to this fork!
2023-06-20: Never used them, but I noticed there are now two alternative workflow to barebone Git: sapling and gitless.
2023-06-20: Functional Programming in Data Science Projects: Interesting thoughts about FP for statistical computing. Another take was once available on Brian Lee Yung Rowe’s weblog, soon to be published as Modeling Data With Functional Programming In R. Thomas Mailund also wrote some words about that approach (see Functional Data Structures in R). And of course, Mathematica with its native support for multiple programming paradigms would be a good candidate too. #statistics
2023-06-22: Desktop Linux Hardening. Lot of useful advices in this article. E.g.,

snapd (Snap) assigns a unique ID to your installation and uses it for telemetry. While this is generally not a problem, if your threat model calls for anonymity, you should avoid using Snap packages and uninstall snapd.

2023-06-22: Moving Blazingly Fast With The Core Vim Motions, but see the whole tutorial. The author makes use of sound illustrations and takes us on a tour of Vim’s core concepts, with casual or VS Code users in mind. #vim

Micro posting in May

Tue, 30 May 2023 09:17:33 +0200

2023-05-03:

Functors and monads are powerful design patterns used in Haskell. They give us two cool tricks for analyzing data. First, we can “preprocess” data after we’ve already trained a model. The model will be automatically updated to reflect the changes. Second, this whole process happens asymptotically faster than the standard method of preprocessing. In some cases, you can do it in constant time no matter how many data points you have! — Functors and monads for analyzing data

2023-05-03: Interesting blog post if you want to learn more about Chicken Scheme’s internals: CHICKEN internals: data representation . #scheme
2023-05-07: Back to Kitty, as a replacement for Alacritty which I enjoyed using the past few months. I still use Tmux, though. #unix
2023-05-07: I will probably watch Millenium again, and I thought I need to watch Noomi Rapace’s recent movies. Seven sisters (which is not listed on his filmography on Wikipedia) was so great after all. I watched Close two days ago; quite bad, actually. Bright was better, though.
2023-05-09:
Spinach, goat cheese and chorizo.
2023-05-09:

But there’ll be winners and losers – everyone agrees. If it’s good, then the productivity gains will be unevenly distributed and those with only basic abilities - in programming, writing, music will be replaced by the machines, or by someone using a machine to produce a lot more of the product. If it’s bad, the people using the AI will benefit but those at the other end of the algorithm, those subjected to AI-powered policing, healthcare, or hiring are subject to the inaccuracy, bias, or malice built into the system. — The one about AI

2023-05-09:

Config (282.0 total hours, 5.0% of all time): Time spent on actual Emacs configuration. — 916 days of Emacs

2023-05-09:

For example, you remove an attribute from a class definition. Your existing objects get (lazily) updated to reflect that change, following a rule you even have control upon. You don’t have to restart a process and then re-create your objects. The same is true for Lisp web development. You can create a new route, compile it and try it live without restarting the server. You didn’t have to restart a process. It’s all very interactive with instant feedback! — Why Lisp?

2023-05-09: Today I had to rollback nvim to a previous state: one or some of the last pushes on the nightly branch broke my LSP setup, which would no longer autostart. No luck, just when I said I never encountered a single breaking change in Neovim core in months. #vim
2023-05-15:
First peonies of the year.
2023-05-17:

Autocompletion is not an aid to make it easier to write code. It does not do that. What it does actually do is make it easier to architect code. Writing code is easy; all it involves is writing text. Any trained monkey can write code. Deciding what to write is decidedly harder. That’s the true task that programmers do. Following from that assertion, writing code and deciding what code to write (designing or architecting) are distinctly separate tasks. Therefore you should not decide what to write as you are writing code. — Why I Don’t Use Autocomplete

2023-05-17: TIL about click, a package for creating beautiful command line interfaces in a composable way with as little code as necessary. #python
2023-05-23:

At the time, however, Linux didn’t seem to have a word processor (I’m sure it actually did). What did people use to write documents? When I asked someone at the local Linux Users’ Group about this, they told me about LaTeX. And that began a very long love affair with typesetting in general and TeX in particular. I have used some variation of TeX/LaTeX to write pretty much every paper, every article, every talk, every letter, every slide, and every lecture I’ve written for the last twenty-five years. — Beautiful Documents with Groff

2023-05-23:

It basically boils down to two things - using type hints as much as possible, and upholding the good ol’ making illegal states unrepresentable principle. — Writing Python like it’s Rust

2023-05-24: Nice stuff. The evolution of a Scheme programmer. You may also like The Evolution of a Haskell Programmer. #scheme #haskell
2023-05-25:

The people who insist that the manpages are all you need will sometimes dismiss guide-type documentation as tedious to work through; they’d rather learn things from a reference, they say, because that way they can jump around in it and look for the specific bits that are relevant to them right now. And that’s fine—if they’re right that the stuff they’re skipping over isn’t relevant to them. But it also has negative practical consequences. — I Didn’t Learn Unix By Reading All The Manpages

2023-05-27:
Pretty little music player. See this review for more details.
2023-05-27:

Or maybe I’m just an old guy looking back and romanticizing the early days. I’m back to coding in C again, writing little inconsequential games to amuse myself. Or maybe there was something a little magical then. Simpler times? — Same Stop

2023-05-27: ♪ P!nk · Bad Influence
2023-05-30:
Shoot at night
2023-05-30:

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it? (Brian Kernighan) — Clever Code Considered Harmful

2023-05-30:

it also confirmed another sad aspect of being on the spectrum: you can have a very large tolerance for people having bad expectations of you because you learn at an early age that you must accommodate others in order to survive. — Autism is everywhere and in everything

G4Music

Sat, 27 May 2023 12:17:37 +0200

While browsing deskto.ps I noticed a good looking music player, Amberol. Unfortunately, it is only available as a Flatpak and it failed to compile on my machine because of a conflicting version of lib-gtk4-dev. Then, I came across G4Music, which is inspired by Amberol and comes with a bunch of additional features. Here it is in action:

Once dependencies were resolved (valac, meson, libgtk-4-dev, libadwaita-1-dev, libgstreamer1.0-dev, libgstreamer-plugins-base1.0-dev), the compilation and building stages took less than 30 seconds. This is the first time I use Meson as a build system and it looks really fast.

G4Music is presented as a low memory application, and a fast indexer of local music. Indeed G4Music found my 14k files when I first launched the application. I didn’t test the drag’n drop support from Gnome file manager. The gapless and replay gain features work great. It makes use of embedded album cover art, without caching image thumbnails which is really a nice idea. We also get notifications in Gnome menu bar for free. Tracks can be sorted by album, artist or tracks, or souffle altogether. As I am writing this post, there’s currently no play counts or option to manage custom playlists.

The UI is really slick and useable, with minimal settings that are actionable in a few motions. If you are looking for a simple music player, give it a try. Overall, I like it a lot.

[2023-12-16]
See this post by Andrey Listopadov for an extensive review of available players for Linux.

♪ New Order • 5 8 6

April in review

Tue, 09 May 2023 21:04:48 +0200

Interesting things I found while browsing the internet.

Eli Bendersky’s blog turned 20 recently. Last year it was the turn of Simon Willison’s one. So much nice stuff and things to learn on those weblogs.
Atuin might be a handy replacement for fzf when it comes to manage my shell history. From what I’ve read, there are some interesting features like fuzzy search using Skim, pattern-based deletion, activity log, custom filters and statistics, and many more.
Reading macOS Internals reminds me of my old days with macOS (I still prefer the old Mac OS naming scheme), especially the good parts since I stopped upgrading all my machines after Mojave. If you still work on macOS, go read some of those items, which were published on Ars technica, you will surely learn useful stuff and understand macOS became the OS it is nowadays.
I came across one of Rachel Kroll’s post where she shared her minimalistic setup, including interesting pieces of config for nano, her daily driver editor (but see her related post). I find it fascinating that the same config has been used for so many years.
I’ve used polars in Python, yet I did not run any serious comparison or benchmark against pandas or datatable. It would be nice to have something like that for Racket, as well as Apache Arrow (I thought I saw a related package on GitHub, but I cant’ get my hands on it anymore).
Let’s Build a Simple Database. Not the first time I came across this kind of blog posts, but I wanted to mention it here, for the record.

I’m not exaggerating when I say assertions are the greatest thing since sliced arrays, and have cut my debugging time by half.
Rule of thumb: Every time you think something is true about your data, and it’s not ensured by the type system, add an assertion that checks it.
Don’t just occasionally use them, drench your code in them. When your code so much as sneezes, they should shake off like sand after the beach. This is the way. — How To Survive Your Project’s First 100,000 Lines

John D. Cook once wrote about posits, and here is another article on this topic. Besides the cost-accuracy trade-off, the TL;DR seems to be: “Posits have an elegant design based on thinking about numbers projectively, but the lack of NaNs, the two’s complement representation, and not having signed zeros and infinities may be alien to some floating point practicioners.”
Yet another X11-based terminal emulator which I never tried, but the benchmarks look impressive: Measured: Typing latency of Zutty (compared to others).
I watched strager’s YT video on hashing and I highly recommend it, much like any of his live streams on Twitch or blog posts. Incidentally, I found this interesting post (after landing on Beautiful Branchless Binary Search) about another way of defining a hash function: Fibonacci Hashing: The Optimization that the World Forgot (or: a Better Alternative to Integer Modulo). See also Modern perfect hashing for strings.
I keep getting some of the emails I send from this domain name to Gmail users rejected by Google. I find it a pity, but go read Is Gmail killing independent email? if you want to learn more. I may be wrong, since I’m not using a self-hosted email provider, but sadly I left Gmail to not live at the expense of a single supplier and I’m left with undelivered messages on the other end.
A visual book recommender. Lovely.

♪ Michael Foster • Resin Rain

Micro posting in April

Thu, 27 Apr 2023 11:18:06 +0200

2023-04-03: Trying out J!

 comb=: [:;(,.&.><@:>:@;\.)^:(i.@>:@-~`[`(1:<@i.@{.~<:@-,2:))
 2 comb 3
0 1
0 2
1 2

Reading answer like this makes me think I should have spent more times with J when I first discovered it years ago.
2023-04-03: ♪ Avenged Sevenfold · So Far Away
2023-04-04:

The immediate future is easy to predict, because the same things that has been happening for years will continue to happen: Some of the exciting recent advances will be cancelled out by new compiler regressions due to fancy new compiler features. We users will create still deeper code stacks that racks up latency, even more so with Julia 1.9 now that latency has improved and we can “afford” to do so. — Julia’s latency: Past, present and future

2023-04-06:

If you see a count: int, what kind of guarantee do you have that the count name is actually bound to an int object at runtime? — The different uses of Python type hintsNo guarantee at all since Python’s type hints do not enforce type safety: “The Python runtime does not enforce function and variable type annotations.” However, the author makes a good point in exposing the different use cases of type hinting.

2023-04-06: TIL about ruff, for linting Python code. #python
2023-04-06: Fast reference to common SRFI. Handy bookmark. #scheme
2023-04-10: Farewell @kinkybambou. Even after I left Twitter, I have been following you irregularly to hear from you. I have no words. All of my thoughts to you, Minette, Loulou and Nono.

2023-04-11:

Writing lets me do that while also helping me avoid going around in circles. When thoughts are in my head, it’s easy for them to get jumbled up. I miss things, and I keep coming back to the same thoughts, leading to the unproductive ruminating. But writing my thoughts down stops that process. I am forced to organize my thoughts in a coherent manner and to acknowledge when they don’t make sense. Thoughts in my head are like a mixture of dirty water, while writing is like a filter. It removes the nonsense from my thinking. — Why I Blog

2023-04-11: TIL about closing a stale SSH connection.
2023-04-11: Common Lisp Tweaks. #lisp
2023-04-13: Looks nice: “I like Janet so much that I wrote an entire book about it, and put it on The Internet for free, in the hopes of attracting more Janetors to the language.” — Why Janet?
2023-04-13: Time to go back watching God’s Crooked Lines.
2023-04-13: Writing a Simple Garbage Collector in C. #clang
2023-04-15:
Back from the old town.
2023-04-19:
Today’s lunch.
2023-04-19:

In a perfect world, I imagine a little robot reading everything alongside me. He records the references and key ideas of each piece. After I write a post, he’d comb through my post sentence by sentence, linking every important phrase to its source. — Keep Stuff Linkable

2023-04-19:

You want to work with a co-worker on a piece of software? If you’re in the same room, sitting at the same table, looking at screens side-by-side, any communication between those two computers is actually happening [by sending signals] across the ocean to some data center, to some other server and then coming back to you. One: that’s crazy. But two, it’s also very slow and expensive and it doesn’t have to be that way. — Offline Is Just Online With Extreme Latency

2023-04-19: ♪ Adam Wakeman · Sunrise Over L.A.
2023-04-21:

Most of the software I write is not for sure going to become a big system. — Why Perl?

2023-04-21: TIL about another DB backend: How RocksDB works.
2023-04-21: A ChatGPT Emacs shell. Of course. What if we can’t put this right under an Emacs key chord? ;-) Nice work by the way! See also Building A ChatGPT-enhanced Python REPL.
2023-04-21: ♪ Cabaret Voltaire · Be Free
2023-04-22: At some point, I should make a chart to inspect how the tags on the main blog evolved over time. I expect that R progressively faded in after 2016, and that the rate of functional programming language-related posts increase in recent years, while readings and other technical reviews actually remain quite the same as in previous years. #this
2023-04-22: TIL about vifm. It looks like a much more improved version of lf, and I should probably give it a try some day.
2023-04-22: The Serokell and Tweag blog are really amazing if you are interested in Haskell (or Rust, or Python). Even older posts are still relevant today: Enter the matrix, Haskell style. #haskell
2023-04-22: Blade Runner 2049: +1. Although, (1) it is fair too long (163’), and (2) the end (I mean, the last hour really) leaves much to be desired. Now, I started watching The Last Kingdom, after Vikings and Vikings Valhala. I guess I will still prefer Vikings over the two other ones. #self
2023-04-25: ♪ Lady Blackbird · It’ll Never Happen Again

Book review: Mathematica

Mon, 24 Apr 2023 19:53:37 +0100

Here are my very first impressions on some of the books on Mathematica that I read recently.

Mathematica Data Analysis (Suchok, 2015)

I did not really like this book¹ but I did learn some useful trick for munging data using Mathematica. Note that when I say I did not like the book, it mostly has to do with like the structure of the book itself, and the shortcuts that the author took here and there. It is like the author assumes that the reader already know Mathematica higher-order functions or that /@ is a shorthand for Map, and there’s no logic about how data are processed. It looks like a series of recipes, and we don’t learn how to think “in Mathematica” to manage data. Furthermore, I found it lacks some structure: just after talking about image analysis (face recognition, or barcode analysis), we are (not so) deep inside time series analysis, and then this is all about classical inference and two-sample tests. I was a little bit worried when I read that for a single simulated series of values, the result table mentioned paired T or Z tests,² as shown below (left), and I just stopped reading at this point.

Mathematica Data Visualization (Saquib, 2014)

Same complaint here. Again, it assumes that the reader is already familiar with data munging in Mathematica, and the introductory second chapter only covers basic properties of lists and functions. Then, CSV data files are imported using the magic Import, and some illustrations using ListPLot are given together with information about customizing the resulting output. Chapter 3 deals with time series while chapter 4 introduces more elaborated charts (histograms, similarity maps, wordcloud, graph and chord diagrams). Together with the short discussion on high-dimensional data, these were the most interesting parts of the book. We should acknowledge the fact that Mathematica provides great defaults, especially for plotting and displaying data in a notebook. The image on the right in the above panel is an example taken from the book.

Mathematica by Example (Abell & Braselton, 2021)

This is a book I would recommend for someone who wants to learn Mathematica to do applied mathematics. This is not a book about Mathematica programming per se, but how to get your job done using the many facets of Mathematica. Chapter 2 describes baisc operations on numbers, including rationales. The authors sometimes raise subtle issues regarding Mathematica’s rules for evaluating, e.g. Input or Traditional form like Sin[Pi/6]^2 versus Sin^2[Pi/6] (the latter is interpreted as the former only in TraditionalForm). Typesetting is not so great (a mix of mono and bold fonts, image or screenshots of running notebooks, or various mention of US universities for the choice of qualitative color palette,³ etc.), but it is not hard to follow. After you finish reading this chapter, you already know how to write and simplify simple mathematical expressions, define your own function and use Plot (including the Epilog option!). After Chapter 2, you should be able to represent graphically many mathematical expression surely. Chapter 2 deals with calculus, and it comes with very nice illustrations (Fig. 3.7 reminds me of one of my earlier post about the Newton-Raphson algorithm). It introduces the Manipulate command which brings dynamic and interactive visualization in Mathematica. I learned most of this stuff using a French textbook (Mathématiques pour les sciences de l’ingénieur avec Mathematica (tome 1), by Carmasol, 2013) but it was a great refresher.

In Mathematica, lists and tables are the core elements you often end up manipulating. Chapter 4 will help learn you more about that, and I learned a lot about that too. Chpaters 5 and 6 discuss more advanced math stuff.

♪ Killing Joke • Darkness Before Dawn

This pretty applies to most of Packt’s published stuff, with few exception regarding Python and Clojure. My impression is that over time the quality of published works has constantly deteriorated, like PLoS One in the scientific field: easy come, easy go, sure, but also the easier it is to publish, the less attention is paid to the competition. ↩︎
This somewhat reminded me of SAS who insists on displaying the results of non-parametric testing as well as normmality or equality-of-variance when you explicitly ask for a Student t-test. ↩︎
Apparently, the blue-dark yellow colorscheme is from the University of California-San Diego :-) ↩︎

Using fzf-lua

Sun, 23 Apr 2023 20:29:50 +0200

I use a boring TUI low-color setup, and I have never been as happy since it allows to focus on the task at hand. Anyway, my Neovim configuration has been relatively stable for the past few months, and since Neovim 0.7-dev I haven’t encountered a single breaking change in Neovim core, which is pretty good sign. At the time of this writing, the nightly version of Neovim is 0.10.0-dev (LuaJIT 2.1.0-beta3).

Today I decided to give a try to fzf-lua as a replacement for fzf.nvim. Since it is my daily driver to navigating files, history, etc. in Neovim, it must be at least as efficient as Junegunn Choi’s wonderful plugin. It looks like it is. Everything works perfectly, with the added benefits of a more comfortable Git UI (you can stage or unstage files, and preview local changes), an online help for available shortcuts depending on the context, more utilities (just take a look at the number of functions to grep for something in your files); last but not least, it support Neovim builtin LSP and diagnostics, which I was missing from fzf.vim.

I easily updated my custom grep function, which fuzzy search project directory when on a blank line or the word under the cursor, with FzfLua grep_* utilities:

" fzf.vim version
" fzf.vim version
" function! misc#fzf_rg() abort
" let currWord = expand('<cword>')
" if len(currWord) > 0
" execute ':Rg ' . currWord
" else
" execute ':Rg'
" endif
" endfunction

" fzf.lua version
function! misc#fzf_rg() abort
 let currWord = expand('<cword>')
 if len(currWord) > 0
 execute ':FzfLua grep_cword'
 else
 execute ':FzfLua grep_project'
 endif
endfunction

Ok, not a big deal. No other configuration required, except for displaying all Fzf window as before, in a bottom split. This was easy to configure, though:

require("fzf-lua").setup({
 winopts = {
 split = "belowright 10new",
 border = "single",
 preview = {
 hidden = "hidden",
 border = "border",
 title = false,
 layout = "horizontal",
 horizontal = "right:50%",
 },
 },
})

So, I think I’ll be switching definitely in a few days if nothing bad happens in the mean time.

Finally, I really like the quote on the GH README:

Why Fzf-Lua
… and not telescope or any other vim/neovim household name?
As @junegunn himself put it, “because you can and you love fzf”.

[2023-04-24]
Well, actually it didn’t take long before I realized how magical this plugin is, and what kind of enhancements it brigs to fzf.vim. The integration with LSP and the quickfix which I use a lot is simply wonderful, and all the defaults and options were carefully crafted! The lua bindings for fzf-bibtex also work like a charm, and I added a simple helper function to open the PDF file right from the Fzf prompt:

local file_open = function(selected, opts)
 local result = vim.fn.system("bibtex-cite", selected)
 fh = "/home/chl/Documents/papers/" .. result:sub(2) .. ".pdf"
 vim.api.nvim_command("silent !zathura " .. fh .. " &")
end

This assumes that the name of the PDF file is the Bibtex key. If that’s not the case, I guess it will be harder unless the filename is available in the Bibte entry itself (and you will need to parse that entry specifically).

♪ Chico Hamilton • Goodbye Baby Blues

Parsing CSV files in Racket

Fri, 21 Apr 2023 12:21:08 +0200

This ought to be a place to post some code about reading/writing CSV files in Racket, then I realized I had a draft in my inbox regarding Racket packages to import rectangular datasets. In fine, I will simply review what I expect from such a package, and how it could be interconnected with a package for data preprocessing. I’m currently writing one for Newlisp, so maybe I can factor in the ideas and code a bit.

First off, Racket offers excellent support for reading CSV files. Since I often need to import data in CVS format, this is kind of great. As in one of my previous post, I usually rely on make-csv-reader-maker from the csv-reading package. Reader specs are generally easy to define, at least for the kind of rectangular dataset I used to process. However, I don’t use the built-in iterator to process each row one at a time (same as a yield statement in Python’s generators), and just call csv->list. This has the advantage (and inconvenient, because then we’ll need a lot of lambda’s to process the elements of the sublists) of returning a list of sublists.

Even if I tend to use a combination of apply and map a lot, as can be seen in my related posts, I should mention that most Scheme implementations come with append-map (same as CL’s mapcan), which is often used to “flatten” a list of sublists, or to apply a procedure to a list and return a new transformed list. Sometimes it may simplify our workflow. All three following expressions return the same results (a list of squared integers):¹

(map (lambda (x) (expt x x)) '(1 2 3 4 5))
(append-map (lambda (x) (list (expt x x))) '(1 2 3 4 5))
(apply append (map (lambda (x) (list (expt x x))) '(1 2 3 4 5)))

Regarding the csv-reading package itself, I discovered that there are some builtins, namely csv-for-each and csv-map, to apply some stuff on each row or records from the CSV file. As convenient as these utilities are, we also often want to work by column (think variable in statistical computing), or to select a subset of data (row by column) or to filter rows and return some specific comumns. You can think of the subset function in R, or the egen (or bysort for simple queries) procedure in Stata.

Of course, it would be great to have some additional procedures to support the use of csv-reading, keeping in mind that they should apply to other flat files as well (e.g., JSON): select (select one or more columns), subset (select one or more rows), group (create a view using row x column), aggregate (compute some statistics on a view, which is mostly the split-apply-combine strategy). Also, we would want to get rownames and colnames, like in R, such that we can refer to row or column using proper labels and see what’s left in a view or subset when printing it in the REPL. Finally, we need to handle categorical data and missing values nicely. In the latter case, we could simply use a pair of (value . weight), instead of relying on bit masking or a shadow table. This means we will need to use weighted statistics,² but it is already implemented in math/statistics for the most import ones. Additionally, it would be interesting if we could exploit an SQL backend like SQLite or DuckDB instead of working with arrays, lists or sequences.

While I would prefer to roll my own version, my previous review of the available Racket packages led me to the following candidates:

data-frame, which is the package I used for some time.
data-table, “a library for manipulating tabular data in Racket.” It has interesting features like working with remote data structures but it seems pretty limited in terms data manipulation, and the project hasn’t received any update in several years.
rebellion, “a suite of libraries for defining new struct-based types including record types and enum types, and much more.” It features a tabular data structure, where column names are stored as keywords.
sawzall, “A grammar of data manipulation for Racket, inspired by dplyr.” It is probably the closer in spirit to R’s (new) way of data munging. However, the project hasn’t received any update in the last two years. It depends on the data-frame package, and it comes with a companion package for pgraphics (graphite).

There may be other packages I am missing, of course. The data-frame package is really great except it doesn’t quite support categorical data, at least last time I tried it. I’m working on it, though, which eventually will mean we could write things like that:

(require data-frame)
(require "utils.rkt")

(define d (df-read/csv "../data/fev.csv"))
(df-describe d)
(recode (vector->list (df-select d "sex")))

I’m interested in Jack Firth’s package, though, for it provides handy data structures, including tables iterations and comprehensions. It’s just that I’m not familiar with this package, and I would need to benchamrk the record (with constant-time lookup of field values by name) and table objects. It is well documented, like the data-table package.

[2023-11-16]
TIL there’s another package which I didn’t find when I drafted this post: RacketFrames.

♪ Bentley • Been a While

There’s also filter-map, at least in Racket, which is probably a better fit in this case: (filter-map (lambda (x) (expt x x)) '(1 2 3 4 5)) (no need to construct a list explicitely). Note that filter-map may be more interesting when one wants to filter items in a row depending on a certain predicate. ↩︎
This was suggested a long time ago by Evan Miller. See 3rd paragraph in this technical review. ↩︎

ArXiving on January 2023

Wed, 19 Apr 2023 12:19:00 +0100

(I know, I know, it’s a bit late for a January review.)

Polynomial spline regression: Theory and Application (https://arxiv.org/abs/2212.14777)

The Box-Cox transformation is one of the many tools one can use to ease traditional linear modeling of data.Semi-parametric regression, whihc I never really used for real work is also an option. Flexible regression methods, as emphasized in Harrell’s RMS textbook¹ or the use of generalized additive models² are another. I’ve been doing some research on orthogonal polynomial, restricted cubic splines, and other spline fitting procedures, and then I found this paper. It introduces different frameworks (Truncated Power, B-spline, and P-Spline), and a quick application on a time-series dataset.

Parametrization Cookbook: A set of Bijective Parametrizations for using Machine Learning methods in Statistical Inference (https://arxiv.org/abs/2301.08297)

Looks interesting at first sight, but I haven’t had time to read it. Maybe you will be interested, though.

Unlike the parametrizations classically used in Machine Learning, the parametrizations introduced here are all bijective and are even diffeomorphisms, thus allowing to keep the important properties from a statistical inference point of view, first of all identifiability

An Introduction to Probabilistic Programming (https://arxiv.org/abs/1809.10756)

This is a 300-page long tutorial on probabilistic programming written by one of the authors of Anglican, which seems like a staled project at this time. Still, it is a very valuable account of probabilistic programming at large and a peculiar implementation targeting the JVM. See also these slides.

A Practical Introduction to Regression Discontinuity Designs: Extensions (https://arxiv.org/abs/2301.08958)

This is a rather long monograph on RDDs. It covers local randomization (which assumes that the assignment of units above or below the cutoff was as if random inside a given window and that in this window the potential outcomes are unrelated to the score), fuzzy RDD (which accounts for non-compliance), and discrete and multidimensional designs. There’s a preliminary monograph from the same authors that only deals with the classical RDD, as well as a practical for medical applications.

Improving Software Engineering in Biostatistics: Challenges and Opportunities (https://arxiv.org/abs/2301.11791)

Programming is ubiquitous in applied biostatistics; adopting software engineering skills will help biostatisticians do a better job.

A few months ago, I read on a blog that statisticians were very bad at programming because they lack skills in software engineering. Let me suggest two additional remarks: applied statisticians must provide results, quickly whenever possible, and when they can use agile programming they do, most of the time. This does not mean they will deliver a product of industrial grade, but if they can reproduce their own results six months later, this already is good news. I guess many software engineers never looked at statlib or established algorithms; you can use TDD, CI, regression tests and the like; if your algorithm is far from the standards, it won’t do any good.

A statistical framework for planning and analysing test-retest studies for repeatability of quantitative biomarker measurements (https://arxiv.org/abs/2301.11690)

Sometimes, I feel like I’m reading the same story every year: “biomarkers”, measurement error, practical significant difference, etc. Ioannidis warned us a long time ago.

From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions (https://arxiv.org/abs/2301.11797)

The authors hereby suggests “probabilistic top lists as a novel type of prediction in classification, which bridges the gap between single-class predictions and predictive distributions.” His conclusion is that the Brier score is a good candidate. The problem with discrimination indices is that they perform poorly when the observed prevalence of the outcome is very low (which happens pretty often in medical applications) despite highly significant variables.

Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming (https://arxiv.org/abs/2104.05372)

I learned about Dex on Darren Wilkinson’s blog. Later on, I learned about Futhark, but see A comparison of Futhark and Dex. I never used any one of these DSLs, but I warmly recommend reading what Darren Wilkinson has to say about those frameworks.

♪ Train • Drops of Jupiter

Frank E Harrell (2015) Regression Modeling Strategies, Cham: Springer International Publishing ↩︎
T. J. Hastie & R. J. Tibshirani (1990) Generalized Additive Models, Chapman & Hall/CRC; Wood, S. N. (2017). Generalized Additive Models: An Introduction with R (2nd ed). Chapman & Hall/CRC. ↩︎

The unquantified self #28

Mon, 17 Apr 2023 11:09:39 +0200

I’m still tired all day long (it looks like it spans the half-life of my beta blocker, but I’m not really sure that it’s all about that). Anyway, at least I’m on vacation for two good weeks. And that which cannot be seen does not exist.

I know I often change my mind and move from one project to another at the speed of light. I also do what I do when I feel like it. Natural disposition we’ll say. So, after spending two months drawing with ink or charcoal during the last quarter of 2022, after going back to computational statistics and applied biostatistics on this blog or on Github, here is where I spent some of my evenings laying down blocks… in Minecraft during one month or so.

Things are going well, and the village should be ready in a couple of weeks. It has been extraordinarily relaxing to get back to building on MC, after a hiatus of several years. Now, I’m not done with statistical computing. I still have to finish my ongoing project and get back to writing an efficient statistical library for Racket.

I watched two good movies recently: God’s Crooked Lines and Anon. It was some good time after a few series without much quality. Also, I listened at least twice to all of my music collection, which is about 14k MP3 files, and the vinyl player works hard when I’m at home (actually listening to Nick Cave’s B sides & Rarities a lot)!

♪ Adam Wakeman • Draa Valley

Solving OLS problem using SVD

Fri, 07 Apr 2023 09:12:39 +0200

I came across a Racket package which aims to provide basic linear regression fitting utilities to Schemers. Unfortunately, it relies on the use of normal equations to solve the linear systems, which is rarely a good idea (think about what happens if the design matrix has a very irregular shape, or even condition number). Moreover, it is not very flexible regarding data i/o, but that’s not the point of this post.

Direct methods (i.e., matrix decomposition) have been discussed beforehand, e.g. here (Lisp) and here (Racket). Here is another direct method which relies on the SVD. Considering the linear system $X\beta = y$, the idea is to decompose $X$ as $\underset{m\times m}{U}\overset{m\times n}{\Sigma}\underset{n\times n}{V^T}$, where $U$ and $V$ are unitary matrices, and $\Sigma$ is a diagonal matrix filled with 0 for rows $n+1,\dots,m$.¹ Then, $\hat\beta$ is obtained as $V^T\Sigma^{-1}U^Ty$. See The QR Algorithm Computes Eigenvalues and Singular Values to learn more about the relationship between the QR and SVD matrix decompositions.

To sum up, although solving normal equations is usually the fastest method, its squared condition number renders it less stable; QR decomposition is more stable at the expense of a larger cost, and SVD works well even in case $X$ is rank-deficient.

Here is a direct application in Racket, using the same example we used in a previous post. I’ll be using the excellent flomat package by Axel Søgaard.²

#lang racket
(require flomat csv-reading)

(define (import file)
 (csv->list ((make-csv-reader-maker
 '((separator-chars #\,)
 (quote-char . #f))) (open-input-file file))))

(define data (map (lambda (xs) (map string->number xs)) (cdr (import "/home/chl/tmp/birthwt.csv"))))
(define xs (matrix (map (lambda (xs) (list-ref xs 1)) data)))
(define ys (matrix (map (lambda (xs) (list-ref xs 9)) data)))

(define X (augment xs (flomat-ones (nrows xs) 1)))
(define-values (U S V) (svd X))
(times V (inv (diag S)) (transpose (sub U 0 0 (nrows U) 2)) ys)
;; => (flomat: ((12.429712027714643) (2655.74446970517)))

[2023-05-02]
To learn about more “traditional” Racket arrays, check the documentation or the following blog post, by Alex Harsányi: A Racket Array Tutorial.

Note that the above procedure returns a full matrix (m x m) for U, but we don’t really need the extra columns, hence the subsetting to the first two columns of $U$. For the record, we got the following results using Newlisp:

(gsl:QR-solve xs y)
;; => (2655,744469705172 12,4297120277146)

The Python version follows:

import numpy as np
import pandas

data = pandas.read_csv("/home/chl/tmp/birthwt.csv")

x = np.c_[np.ones(data.shape[0]), data["age"]]
y = data["bwt"]
U,S,V = np.linalg.svd(x, full_matrices=False)
V.T @ np.linalg.inv(np.diag(S)) @ U.T @ y
# => array([2655.74446971, 12.42971203])

♪ Emma Ruth Rundle • Blooms of Oblivion

The number of columns can also be reduced (i.e. discarding the smallest singular values). ↩︎
If you get an error regarding a missing libgfortran (likely v3), be sure to install the latest version of the sci package which includes a fix for Ubuntu 22.04. ↩︎

Micro posting in March

Wed, 29 Mar 2023 19:36:38 +0200

2023-03-08: Testing my new mapping of the month:

nnoremap <expr> <tab> tabpagenr('$') ==? 1 ? "<C-^>" : ":tabnext<cr>"

Basically, use TAB to switch to the alternate buffer when there’s no other tab, otherwise go to next tab (and cycle). #vim
2023-03-17: Peonia is waiting for its time…

2023-03-20:

If you type something that looks like a filename, you probably want to prioritize filename matches. When your search query looks like a filepath, you probably want to use an algorithm designed for filepath matching. This makes zf a “do what I mean” program. — A different approach to fuzzy finding

2023-03-20: Together with Vinfo, dpans2texi makes it easy to navigate through CL help files. Be sure to apply the following patch as well. #vim #lisp

2023-03-20: ♪ Plain White T’s · Hey There Delilah
2023-03-23: Multiple Linear Regression in SQL with Only SUM() and AVG(). For those interested in multiple regression and residualized regressors as well! #statistics
2023-03-27:

Deforming programs so they are ‘iterative’ in order that they do not run out of the stack we imagine to be so costly is ridiculous: if you have a program which is inherently recursive, let it be recursive. — The absurdity of stacks

2023-03-27: The LAPACK (Linear Algebra PACKage) online document browser is really nice. TIL there’s even a routine to perform QR factorization without pivoting to a real or complex matrix (related R package: QR).
2023-03-27: The Many Faces of an Undying Programming Language. An entertaining discussion of Lisp, Scheme(s), and other dialects. #lisp #scheme
2023-03-27: littler 0.3.18 on CRAN. It’s been a faithful and reliable alternative to Rscript for a long time now (17 years and counting). #rstats

Welford algorithm in Newlisp

Mon, 20 Mar 2023 11:54:10 +0100

A few years ago we discussed the Welford algorithm to compute the variance of a vector of numerical values without catastrophic cancellation. In addition to being numerically stable, the difference with Kahan-compensated two-pass computations for large data sets are very small. See also Numerically stable algorithm for computing the running mean for related approaches, and John D. Cook’s original post. The stability and numerical accuracy of one-pass algorithms are also discussed in several textbooks.¹

While reviewing Newlisp statistical function in the stat module, I noticed it should get it wrong in this case:

(module "stat.lsp")
(set 'x (sequence 1 5))
(stat:var x)
;; => 2.5
(stat:var (map (lambda (x) (+ 1e10 x)) x))
;; => -16384

The author uses the formula (sub (sum-sq X) (div (mul (sum X) (sum X)) (length X))) (later corrected for bias), which is the same that was discussed in the previous post.

Note that Racket has no problem with this cornercase.² Let’s implement the single-pass Welford algorithm in Newlisp, as a rough translation of the R code provided in the previous post:

(define (variance xs)
 (let ((mean 0.0)
 (variance 0.0)
 (n (length xs)))
 (dotimes (i n)
 (setq curr (nth i xs))
 (setq tmp mean)
 (setq mean (add mean (div (sub curr mean) (+ i 1))))
 (setq variance (add variance (mul (sub curr mean) (sub curr tmp)))))
 (div variance (- n 1))))

(variance (map (lambda (x) (+ 1e10 x)) x))
;; => 2.5

A more idiomatic way of writing a standard function in CL, which follows Racket’s two-pass approach, would be:

(defun mean (xs)
 (/ (reduce #'+ xs) (length xs)))

(defun variance (xs)
 (let ((mean (mean xs))
 (n (length xs))
 (/ (reduce #'+ (map 'list #'(lambda (x) (square (- mean x))) xs))
 (1- n))))

I should not that Tom MacWright gets it right too in his simple-statistics module:

const ss = require("../dist/simple-statistics.js");
ss.sampleVariance([1, 2, 3, 4, 5].map((x) => x + 1e10));

The take away message is that you should either rely on a two-pass algorithm (i.e., compute the mean first, then the squared deviations from the mean) or the Welford algorithm, especially if you are interested in running statistics.

[2023-11-23]
As I was surveying the various statistical packages lying around on GitHub, I found many buggy implementations (e.g., basic-statistics.lisp in LispUtils). However, cl-ana got it right:

(ql:quickload "cl-ana")
(defparameter *xs* '(1 2 3 4 5))
(cl-ana::variance *xs*)
;; => 5/2
(cl-ana::variance (mapcar (lambda (x) (+ 10000000 x)) *xs*))
;; => 5/2

♪ Tracy Chapman • Fast Car

See, e.g., N.J. Higham, Accuracy and Stability of Numerical Algorithms (2nd ed.), SIAM 2002. (section 1.9) ↩︎

Here’s the test case:

(require math)
(define x '(1 2 3 4 5))
(exact->inexact (variance x #:bias #t))
;; => 2.5
(exact->inexact (variance (map (lambda (x) (+ 1e10 x)) x) #:bias #t))
;; => 2.5

↩︎

Fitting an OLS model in Newlisp

Sat, 18 Mar 2023 20:55:42 +0100

Yet a new post on Newlisp for statistical computing. This time we will try to fit a simple linear regression model using the GSL facilities offered by Newlisp modules. Unfortunately, the FFI bindings are rather sparse, and only a few linear algebra functions are implemented. Since the QR decomposition is available, we will use it, as we did in an another post. Recall that the vector of estimated parameters can be computed as $R^{-1}Q^Ty$, where $Rx = \bar y$, $y$ is rotated as $\bar y = Q^Ty$ and $R$ is triangular.

Same example as before (Hosmer & Lemeshow’s birth weight dataset), where we are interested in regressing the baby weights (g) onto mother’s age (years).

(module "gsl.lsp")

(define zip (lambda (x y) (map list x y)))

(set 'data (read-file "/home/chl/tmp/birthwt.csv"))
(replace "," data " " 0)
(set 'values (map int (parse data)))
(set 'x (rest (select values (sequence 1 (length values) 10))))
(set 'y (rest (select values (sequence 9 (- (length values) 1) 10))))
(set 'xs (zip (series 1 1 (length x)) x))

(gsl:QR-solve xs y)
;; => (2655,744469705172 12,4297120277146)

Great! Compare to the built-in function:

(module "stat.lsp")

(stat:regression x y)
;; => (2655,744469705173 12,42971202771453)

(set 'yhat (stat:fit x y))
(set 'out (zip (zip x y) yhat))
(write-file "/home/chl/tmp/lbw_fit.dat" (stat:list2ascii out))

At this point, I’m afraid there’s not much we can do to go forward since the stat module does not offer a way to build a regression or ANOVA table.

$ sed -i -e 's/(//g' -e 's/)//g' -e 's/,/./' lbw_fit.dat
$ gnuplot -p -e "set terminal png size 800, 600; set output '~/tmp/lbw_fit.png'; \
 set grid; set xlabel 'Mother age (yr.)'; set ylabel 'Baby weight (g)'; \
 plot 'lbw_fit.dat' u 1:2 w p pt 6 notitle, \
 'lbw_fit.dat' u 1:3 w l lc rgb 'black' lw 2 notitle"

By the way, Gnuplot itself is able to fit a linear regression from raw data. Finally, here are Stata’s results for reference:

. import delimited /home/chl/tmp/birthwt.csv
. regress bwt age

 Source | SS df MS Number of obs = 189
-------------+------------------------------ F( 1, 187) = 1.54
 Model | 815483.22 1 815483.22 Prob > F = 0.2165
 Residual | 99154172.6 187 530236.217 R-squared = 0.0082
-------------+------------------------------ Adj R-squared = 0.0029
 Total | 99969655.8 188 531753.488 Root MSE = 728.17

------------------------------------------------------------------------------
 bwt | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
 age | 12.42971 10.02278 1.24 0.216 -7.342539 32.20196
 _cons | 2655.744 238.8571 11.12 0.000 2184.544 3126.945
------------------------------------------------------------------------------

♪ Rebound • Sebadoh

Plotting with Newlisp

Thu, 16 Mar 2023 14:57:46 +0100

I installed newlisp manually, although I now realize there’s a deb package available for Ubuntu 22.04. Anyway, I was missing two files: guiserver.lsp and guiserver.jar. The Lisp file is easy to get from the modules page, and I just copied it to the right place (/usr/local/share/newlisp) but I found no indication on how to get the Java file. So I grabbed a copy of guiserver.jar from the guiserver.tgz tarball referenced on Links and Code Contributions, and it just works. Note that I also had to install libcanberra-gtk-module along the way, in order to remove a minor warning when issuing plotting commands.

The statitstics module only contains very basic stuff, but there’s also a gsl module which gives us more power. Unfortunately, it expects GSL library in /usr/local/lib, and not /usr/lib. I symlinked libgslcblas.so and libgsl.so into /usr/local/lib, and this solved the problem.

Let’s get some data:¹

(load (append (env "NEWLISPDIR") "/modules/plot.lsp"))
(set 'data (read-file "birthwt.csv"))
(replace "," data " " 0)
(set 'values (map int (parse data)))
(set 'x (rest (select values (sequence 2 (length values) 10))))
(set 'y (rest (select values (sequence 9 (- (length values) 1) 10))))

The x and y variable defined above represent the weight of the mother and of the baby, respectively. There might be a better way to mine those values from the tabular dataset, but contrary to xlispstat, newlisp isn’t really a col-major language. Anyway, we are ready to generate a basic scatterplot:

(set 'plot:title "Birthweight dataset")
(set 'plot:sub-title "Hosmer & Lemeshow, Wiley 2000 (2nd ed.)")
(set 'plot:unit-x "Mother weight (lbs)")
(set 'plot:unit-y "Baby weight (g)")
(plot:XY x y)
(plot:export "fig-bwt-newlisp.png")

That was quite easy, but I fear there’s not much plotting functions available except plot, which produces univariate line plot. I guess we will have to resort to Gnuplot or plotutils for more capabilities. However, let that not stop us investigate into this scripting language. In future posts, I expect to show how to fit basic linear models using GSL and to perform some data mining tasks.

♪ Ray Mantilla • Martinez

The birthwt dataset is described here. ↩︎

Wilcoxon test in Lisp

Tue, 14 Mar 2023 18:33:21 +0100

As a sequel of this previous post, Mann Whitney test, let’s revisit the Wilcoxon-Mann-Whitney test for independent samples. In the case of paired samples, it amounts to a one-sample test, as is the case of Student t-test, since we can simply consider the signed difference between the two series of observations. The only difference with the parametric approach is that we will work with the rank of each observation, rather their raw value.

Unfortunately, there’s almost zero standard function for classical test of hypothesis in LispStat. Most of user-friendly statistical routines were implemented separately for the Vista or Arc statistical packages. There once was a dedicated repository for xlispstat on the UCLA server, but now there’s only the CMU library of code which contains LispStat-related routines (when their server is not down). So let us roll our own routine.

Regarding the Wilcoxon family of non-parametric tests, the subtleties are mainly about handling tied values, and using an approximate or exact distribution, which also means dealing with continuity correction. Note that Stata, SPSS, SAS, or Mathematica, as far as I can tell, do not offer continuity correction.¹

LispStat has a rank function, but unfortunately it does not compute the average rank in case of ties. Here is an improved version which return average rank for ties: (Note that rank indexing start at 1 and not 0 as in LispStat’s rank.)

(defun rank2 (xs)
 (let* ((rs (+ (rank xs) 1)))
 (dotimes (i (length xs))
 (if (> (length (which (= (nth i xs) xs))) 1)
 (setf (select rs (which (= (nth i xs) xs)))
 (repeat (mean (select rs (which (= (nth i xs) xs))))
 (length (which (= (nth i xs) xs)))))))
 rs))

(setf x '(1 2 4 3 2 5))
(rank2 x)
;; => (1 2.5 5 4 2.5 6)

There’s more than one way to report results from a Wilcoxon-Mann-Whitney, and the test statistic is usually labelled either as U (SPSS & Co.) or W (R), statistic (Python) or TestStatistic (Mathematica), or directly as a standard normal quantile ($z$, in Stata). Even the definition of the test itself (Wilcoxon rank sum or Mann-Whitney test) is not clear, as noted in the R on-line help for wilcox.test. Similar comments are made in Stata manuals, or by the author of the Haskell’s statistics package.

Hence, it may happen that only the sum of ranks is reported, or the sum of ranks minus a correction factor ($n_{\star}(n_{\star}+1)/2$, where $\star$ denotes the index for the first or second group²), which is the case of the first two naming schemes (although, technically, U is the shifted version of the test statistic), while the $z$ value refers to the standardized test statistic that follows a normal distribution.³ Assuming $n_1 < n_2$, we define $W_R$ the sum of the ranks assigned to the $n_1$ observations in the first sample. Critical values for this statistic for small $n$ can be used.⁴ When $\text{min}(n_1, n_2) > 10$, we use the following Gaussian approximation:

$$ Z = \frac{W_R-\mu_{W_R}}{\sigma_{W_R}}, $$

where $\mu_{W_R} = \frac{n_1(n+1)}{2}$ and $\sigma_{W_R} = \sqrt{\frac{n_1n_2(n+1)}{12}}$ ($n = n_1+n_2$). The formula for the variance is only valid in the case of large samples, though. A corrected estimate, based on Fisher’s randomization principles, is $\sqrt{n_1n_2s^2/n}$ with $s=\frac{1}{n-1}\sum_{i=1}^n(r_i - \bar r)^2$, which remains valid even in case of tied ranks, see the Stata [R] manual.

For this illustration, we will compute the Mann-Whitney test statistic as follows. This mostly follows the R’s way since I got lost with Stata’s formula:⁵

(defun duplicates (xs)
 ;; https://stackoverflow.com/a/65788371
 (if (endp xs)
 '()
 (let ((count (count-if #'(lambda (x) (eql x (car xs))) xs)))
 (cons count (duplicates (nthcdr count xs))))))

(defun wilcoxon-test (data)
 (let* ((n1 (length (nth 0 data)))
 (n2 (length (nth 1 data)))
 (n (+ n1 n2))
 (rs (rank2 (combine data)))
 (nties (duplicates (combine data)))
 (w nil)
 (z nil)
 (p nil))
 (setf tobs (sum (select rs (iseq n1))))
 (setf w (- tobs (/ (* n1 (1+ n1)) 2)))
 (setf mu (* n1 (/ n2 2)))
 (setf sigma (sqrt (* (* n1 (/ n2 12))
 (- (+ n1 n2 1)
 (/ (sum (mapcar #'- (^ nties 3) nties))
 (* (+ n1 n2) (+ n1 (- n2 1))))))))
 (setf z (/ (- w mu) sigma))
 (setf p (* 2 (- 1 (normal-cdf (abs z)))))
 (list w z p)))

Note that we do adjust the estimate of variance for the case of tied ranks, as does R or as described on Wikipedia, which in this case implies using a different formula for $\mu_{W_R}$ ($\frac{n_1n_2}{2}$). However, if there’s no tied values, nties contains only 1’s values and (sum (mapcar #'- (^ nties 3) nties)) = 0, which means the correction term has no effect and $\sigma = \sqrt{n_1n_2/12 \times (n_1+n_2-1)}. The rest msotly agrees with R intermediate computation.⁶

Let’s test this implementation using an example taken from R. The two series shown below summarize permeability constants of the human chorioamnion (a placental membrane) at term (x) and between 12 to 26 weeks gestational age (y). The alternative of interest is greater permeability of the human chorioamnion for the term pregnancy (Hollander & Wolfe (1973), 69f.). For simplicity, and because we only implemented a bilateral test, we will consider a two-sided p-value:

(setf x '(0.80 0.83 1.89 1.04 1.45 1.38 1.91 1.64 0.73 1.46))
(setf y '(1.15 0.88 0.90 0.74 1.21))
; (sum (first-n (length x) (rank2 (combine x y))))
(sum (select (rank2 (combine x y)) (iseq (length x))))
;; => 90
(wilcoxon-test (list x y))
;; => (35 1.224744871391589 0.22067136191984682)

Here are the results I got for the same dataset with other statistical packages, using default parameters:

Software	Test statistic	P-value
R	35	0.2544
Python	35	0.2544
Stata	1.225	0.2207
Mathematica	35	0.2446

As a final remark, the Wilcoxon-Mann-Whitney is technically a permutation test, at least in the case of not too large sample size, as we’ve seen above. This means it is valid only when all the rearrangements of the data are equally likely under the null. When the variances differ, the test is no longer a test for the difference in location, strictly speaking.

♪ Adele • All Night Parking

Stata does offer a continuity correction for the median test, though. ↩︎
Once you determined the sum of ranks in one group, you get the sum of ranks for the other group since the sum of all ranks equals $N(N+1)/2$, with $N=n_1+n_2$, so it doesn’t really matter which group we base our calculation upon (usually, we take the first or the smallest in size). ↩︎
Indrayan, A. and Malhotra, R.K., Medical Biostatistics (4th ed.), CRC Press, 2018. ↩︎
Wilcoxon, F., and Wilcox, R.A., Some Rapid Approximate Statistical Procedures, Lederle Laboratories, 1964. ↩︎

If I read the formulae correctly, this should work:

(setf mu (/ (* n1 (1+ n)) 2))
(setf ss (standard-deviation rs))
(setf sigma (sqrt (/ (* n1 n2 (* ss ss)) n)))
(setf z (/ (- tobs mu) sigma))

↩︎

We see that $W_X = 3 + 4 + 14 + 7 + 11 + 10 + 15 + 13 + 1 + 12 = 90 $. The sum of all ranks equals 120. Inspecting the relevant R code for wilcox.test,⁷ it is clear that they use the corrected statistic $W_X - n_X(n_X+1)/2$, hence $W = 90 - 10\times 11/2 = 35 $ ($10\times 11/2$ is the sum of ranks we would get for $X$ should we compute rank separately in $X$ and $Y$). ↩︎

R code from getAnywhere(wilcox.test.default):

--8<--------
r <- c(x - mu, y)
r <- rank(if (is.finite(digits.rank))
signif(r, digits.rank)
else r)
n.x <- as.double(length(x))
n.y <- as.double(length(y))
if (is.null(exact))
exact <- (n.x < 50) && (n.y < 50)
STATISTIC <- c(W = sum(r[seq_along(x)]) - n.x * (n.x +
1)/2)
-------->8--

↩︎

Generating variable names in Lispstat

Thu, 02 Mar 2023 10:04:01 +0100

For a side-project I went back into Lispstat, which was developed in the 90s, to perform basic biostatistical analysis. At that time, there was no such thing like a data frame, but only data arrays composed of numerical values thta could be imported from a space delimited text file. Usually, you “selected” the variables you were interested in one at a time. Assuming you have two series of values (i.e., two variables), you can split a response variable by a grouping factor like this:

(def geno1 (select age (which (= 1 genotype))))
(def geno2 (select age (which (= 2 genotype))))
(def geno3 (select age (which (= 3 genotype))))

I often use a loop to generate variable name with a common prefix or suffix. In R it is easily done using a for loop and assign (see this example), while in Stata it can be done in a forvalues or foreach construct. Here is an example from my Stata Starter Kit (draft):

xtile weightc = weight, nq(4)
quietly tabulate weightc, gen(weightc)
drop weightc
local i = 0
foreach v of varlist weightc1-weightc4 {
 local i = `i' + 1
 rename `v' w`i'
}
list w* in 1/5

Lispstat was built on the XLISP dialect of Lisp, which means it is mostly the same as Common Lisp except that something are missing here and there, most notably the loop macro. There are several constructs available for iterating over a range or a list, like do, dotimes and dolist. Since we cannot use dotimes to iterate over a sequence starting at a value other than zero,¹ let’s try with dolist:

(dolist (n (iseq 1 3))
 (set (intern (concatenate 'string "GENO"
 (write-to-string n)))
 (select age (which (= n genotype)))))

Dont’ forget that variables are represented internally as uppercase symbol, so even if you call your variable geno1 at the prompt or in an expression, it really is GENO1 in the environment.

It is a little more verbose than the previous expressions, but once converted to a macro, it can be reused many times.

♪ Arcade Fire • Rebellion

In Common Lisp, we can write a little macro for that purpose, but it requires using loop. ↩︎

Mono color scheme for tty

Wed, 01 Mar 2023 09:20:22 +0100

I already discussed the use of monochrome color scheme in the past. The main benefit for me has been to avoid the “christmas tree” effect¹ in my text editor (Neovim), but see On choosing your colorscheme. Such minimalist settings helps me focus on the task at hand, while while maintaining high portability depending on the capabilities of the machines I work on (macOS and OpenBSD sometimes, ssh access to computing servers most of the time).

[2024-09-02]
See also Color Conundrum by Akshay who made a good point here: “Imagine highlighting an entire book because all of it is important.”

I use Zsh as my main shell, but sometimes I also use Bash. I keep a minimal bashrc config file in case I need to use my default settings on a remote server, and I have the following settings in my .bashrc, which I mostly use on virtual consoles or when connected via ssh:

if [ "$XDG_SESSION_TYPE" = "tty" ]; then
 unset LS_COLORS
 export TERM=xterm-mono
 export NO_COLOR=1
 alias ll="ls -alF"
 alias vim='nvim -c ":set notermguicolors" -c ":colorscheme mono"'
 PS1="\W \$ "
else
 eval "$(starship init bash)"
fi

As can be seen, if we are connected via ssh or a simple tty, we’ll be disabling most of the fuzz defined elsewhere to prettify the prompt, using exa for nice listing, etc. I also use these settings for running a terminal inside Neovim, for example.

Regarding other TUIs, I have chosen color schemes that can accommodate black & white screen either by using few colors (Tmux or Irssi) or by defining mono color (Neomutt). The “mono” colorscheme for Vim is inspired by Joshua Stein own color scheme,² who also authored the NO_COLOR site. Here is my default theme (left) and the mono theme (right) when editing this post:

Overall I quite like working on a virtual console, especially since I can still listen to music running on Cmus in a regular X session, so here is a screenshot of the same editing session on a virtual console:

♪ Dave Gahan & Soulsavers • Metal Heart

Nicolas P. Rougier. On the design of text editors. arXiv:2008.06030 [cs.HC], 2020. ↩︎
Basically, I just removed the underline attribute for most text elements. ↩︎

Micro posting in February

Tue, 28 Feb 2023 09:07:58 +0100

2023-02-06: Best way to learn about recent (non-breaking) changes in ggplot2 is to rerun your old R code. Now I know that :

Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0. Please use `linewidth` instead.The `<scale>` argument of `guides()` cannot be `FALSE`. Use "none" instead as of ggplot2 3.3.4.

Other than that, I found that the theme_minimal() is as good as the hrbrthemes:theme_ipsum. Less dependencies, clear and elegant graphical output.

2023-02-07: I just made my GH biostats public, just so you know how I like playing with translating R code to other languages, without any hope of finishing this new side project anytime soon. Details in this longer blog post. Want some crazy plots? Go to the figs directory in each language and compare the output. Be aware this is highly work-in-progress, as advertised on the GH repo, and there are many bugs lying around. Anyway, no more raw Matplotlib for me, I felt in love with plotnine.
2023-02-07: ♪ Sheri Weldon · A Taste Of Honey
2023-02-08:

The GIL is a major obstacle to concurrency. For scientific computing tasks, this lack of concurrency is often a bigger issue than speed of executing Python code, since most of the processor cycles are spent in optimized CPU or GPU kernels. The GIL introduces a global bottleneck that can prevent other threads from making progress if they call any Python code. There are existing ways to enable parallelism in CPython today, but those techniques come with significant limitations (see Alternatives). — PEP 703 – Making the Global Interpreter Lock Optional in CPython

2023-02-08: I’ve been following Darren Wilkinson’s work for a long time now, and even if I’m not really interested in Scala I keep checking his blog and GitHub from time to time. He just released a course on functional programming for scientific computing, available at GH. Go check his blog to learn more: An introduction to functional programming for scalable statistical computing and machine learning.
2023-02-08: Shells are Two Things. The Python scripting example reminds me of a blog post written by Mathias Fußenegger who uses Haskell in interpreted mode as one of his scripting engine (see Why Haskell became my favorite scripting language), even if this totally unrelated to the points raised by Fernando Borretti.
2023-02-11: Made with ♡ using Stata. See this blog post for details.

2023-02-13:

Perhaps I’m on one of my periodic ‘minimalist’ adventures (which would seem to be borne out by my keyboard choice), but Emacs has recently felt a bit much to me, and made me curious about the current state of the art in Neovim. — Neovim againI’m also quite happy with my minimal setup these days. It’s been running smoothly for several months now.
Alacritty with font ligature support, Tmux, Cmus, Neovim nightly.

2023-02-14:

Dear LazyWeb: Setting up a new Mac, what are some groovy new monospace fonts for terminals and IDEs? — Modern MonoI like screenshots. See also Monospace Acid Test for a more elaborated overview of modern monospace fonts.

2023-02-16:

So basically, I pay a yearly fee to get access to their SDK? Most manufactures provide that to you for free because they really want you to build apps for their platform. — Apple doesn’t want you developing hobby apps

2023-02-16:

UNIX contains a number of features very seldom offered even by larger systems. — Unix Edition Zero.

2023-02-18: I recently came across some Racket code on Reddit that implements Fenwick trees, aka binary indexed trees (BIT). Fenwick’s article, which is not the original paper on that particular data structure, A New Data Structure for Cumulative Frequency Tables (PDF) explains all the details, but there’s also a nice tutorial, as well as some applications for competitive programming.

2023-02-20: The highest upvotes I got on Cross Validated is for a quick reply where I mentioned the Gifi “nom de plume”, which I miss a lot. Enjoy your “unfinished” retirement, Mr./Dr. Jan de Leeuw, and happy foometrics (PDF)!

If Foo is a science then Foo often has both an area Foometrics and an area Mathematical Foo. Mathematical Foo applies mathematical modeling to the Foo subject area, while Foometrics develops and studies data analysis techniques for empirical data collected in Foo. Each of the social and behavioural sciences has a form of Foometrics, although they may not all use a name in this family. — de Leeuw, 2006

2023-02-20: The thing I really appreciate under Ubuntu or OpenBSD compared to macOS is the package manager. So bad Apple decided to move away from the Un*x world. I miss the old good stuff that was lying around there. #apple #unixAlso, I deleted Apple Music on my iPhone. Thanks for the goodies. Back to listening to ripped CDs!

2023-02-20: ♪ Lou Reed · Lisa Says
2023-02-23: I successfully installed Forge and Controllable for the Minecraft (Java Edition) running on my laptop. Now I can use a PS5 controller to play instead of keyboard shortcuts. Okay, it must be 5 years since I last played, but I’m glad to find a game environment close to the PS4. Sadly, I can’t play my old PS4 maps on the Java edition, which means I need to create a new one.

2023-02-23: Kopia is ridiculously fast and over-optimized. I don’t miss Arq or Borg. I have an hourly schedule for local snapshots, and a weekly sync to an external HD, which never failed me so far.

2023-02-23: ♪ Marianne Faithfull · Crazy Love
2023-02-23: ♪ Memory Tapes · Bicycle
2023-02-27: Each time I take a look at This week in Neovim (#33 right there), I feel like there’s so much new stuff published every week that we will get another (M)ELPA for Neovim packages at some point. #vim
2023-02-27: I’ve been using part of mini.nvim last year but for now I’m just loading the completion module. I noticed there’s an increased number of modules developed since last time I checked. I warmly recommend those modules if you want something light, easy to install and to setup. #vim
2023-02-27: My IRC bouncer (BNC4FREE) has been down for one week. Time to go back running Irssi in a detached Tmux session on a local server.
2023-02-27: European Route Planning: Clear and elegant use of postgres and shell scripting.
2023-02-27: Spatial sampling with R. Lot of pretty illustrations and good old base R code. #rstats
2023-02-27: ♪ Washed Out · Amor Fati

February in review

Mon, 27 Feb 2023 08:33:14 +0100

Learn Vim (the Smart Way): Yet another tutorial on using Vim effectively. The author highlights the use of fzf as a fuzzy finder, which further motivates taking advantage of Vim inner features.

Everyone programs differently. Upon introspection, you will find that there are one or two features from your favorite editor / IDE that you use all the time. Maybe it was fuzzy-search, jump-to-definition, or quick compilation. Whatever they may be, identify them quickly and learn how to implement those in Vim (chances are Vim can probably do them too). Your editing speed will receive a huge boost.

It’s been a long time since I last read some obfuscated code, and it was always C code. This one does the job, and it is written in Haskell: Squeezing a sokoban game into 10 lines of code. See also How I would do FizzBuzz, by the author of the LSP ecosystem in Mathematica.
poste.io offers SMTP + IMAP + POP3 + Antispam + Antivirus Web administration + Web email. I know it’s a hard job, and I won’t host my own mail server, even if Gmail sucks with randomly blacklisting my own email address with error code 550 5.7.26. Anyway, in case someone is looking for a quick way to handle his/her emails from a local computer.
In Few lesser known tricks, quirks and features of C, I learned about a fun way to use named parameters in function call.
Programming Odds & Ends: There’s a lot to learn and keep in mind when using R and/or Python (sadly, there’s no real benchmark with Pandas or datatable). And now I know there’s something new called the fastverse, which includes the data.table package. As the author suggests, “you may not [always] want an additional package dependency beyond what you’re using, and may need a base R approach.”. Keeping it small and low on external dependencies is always good.
sqliteviz: Instant offline SQL-powered data visualisation in your browser. Looks great. Here’s what I came up in a few clicks with our ToothGrowth dataset from previous posts:

♪ Chip Wickham • Sais

The unquantified self #27

Thu, 23 Feb 2023 20:21:41 +0100

The last entry in this series was in October, four months ago. Not much has changed, I guess, with the exception of an exceptionally long-lasting and painful fatigue from time to time.

No more drawing, at least for the time being. I took a few days off to rest and take a break from the daily routine. I watched three movies and I only appreciated one (Jung_e), sort of. TV shows are still listed on their own page, but I must admit I haven’t found anything very exciting.

As advertised recently, I configured my Minecraft (Java Edition) installation to allow for the use of an USB PS5 controller. This was easier than I originally thought since you just need to install the latest Forge for your current version of MC and then the Controllable mod. However, my son who is more versed into PVP needed MC 1.8.9, so I looked for the corresponding version of Forge (1.8.9-forge1.8.9-11.15.1.2318-1.8.9) but I didn’t find any relevant version for the controller mod. Hopefully, there’s Joypad-Mod-1.8.9.jar which you can find here. You will likely need to ensure that the controller is activated and also reconfigure the default settings.

A screenshot lost in my files follows, which is one of my last screenshots on Last.fm. I heard about Kodi but finally I think I’m fine with cmus and mpv, and everything is going well on the music side, I still have to listen to 7-8 hours a day.

♪ Active Child • You Are All I See

Bootstrap t-test

Wed, 22 Feb 2023 20:55:34 +0100

A few years ago I ran a workshop where I showed how to carry out a simple (bilateral) permutation test in R. Assuming you have a matrix d with two columns of values, and no missing values, it reads:

s0 <- mean(d[,1]) - mean(d[,2]) ## observed statistic
x <- c(d[,1], d[,2])
idx <- combn(seq(along = 1:20), 10) ## {20 \choose 10}
f <- function(k) mean(x[k]) - mean(x[-k])
s <- apply(idx, 2, f) ## test statistic s
pobs <- sum(abs(s) >= abs(s0)) / length(s)

Using the coin package the same test is done as follows:

library(coin)
oneway_test(value ~ variable, data = reshape2::melt(d[1:2]),
 alternative = "two.sided", distribution = "exact")

I also showed how to devise a bootstrap test of hypothesis in R. This time, we need to take care of correctly centering the group means, though:

x1 <- d[,1] - mean(d[,1]) + mean(x) ## x is c(d[,1], d[,2])
x2 <- d[,2] - mean(d[,2]) + mean(x)
B <- 999 ## no. bootstrap samples
s <- numeric(B) ## vector of test statistics
for (i in 1:B) {
 x1s <- sample(x1, replace=TRUE)
 x2s <- sample(x2, replace=TRUE)
 s[i] <- mean(x1s) - mean(x2s)
}
pobs <- (1 + sum(abs(s) > abs(s0))) / (B+1) ## s0 is mean(d[,1]) - mean(d[,2])

Note that the for loop can easily be replaced with a function call in a replicate statement, of course. Here is some results with the ToothGrowth dataset we discussed earlier, which can be prepared as follows:

> data(ToothGrowth)
> d <- as.data.frame(with(ToothGrowth, split(len, supp)))
> x <- ToothGrowth$len
> s0 <- mean(d[,1]) - mean(d[,2])
--8<-- copy/paste code above ---
> pobs
[1] 0.043

Of course, we could compute confidence normal, percentile or BCA confidence intervals, with little effort. Stata does it smoothly, as usual:

. use toothgrowth, clear
. bootstrap dobs = (r(mu_1) - r(mu_2)), reps(1000) seed(101): ttest len, by(supp)
. estat bootstrap, all

Bootstrap results Number of obs = 60
 Replications = 1000

 command: ttest len, by(supp)
 diff: r(mu_1) - r(mu_2)

------------------------------------------------------------------------------
 | Observed Bootstrap
 | Coef. Bias Std. Err. [95% Conf. Interval]
-------------+----------------------------------------------------------------
 diff | 3.7 .0643175 1.8568749 .0605921 7.339408 (N)
 | -.0248661 7.282081 (P)
 | -.178198 7.126667 (BC)
------------------------------------------------------------------------------
(N) normal confidence interval
(P) percentile confidence interval
(BC) bias-corrected confidence interval

You can download the dataset here.

♪ Three Days Grace • I Hate Everything About You

Python for quick statistical computing

Tue, 21 Feb 2023 18:58:44 +0100

Python has a lot of base packages that can be used to perform most of our statistical computing tasks. There’s a good review of what’s available when you install Python in Serious Python, by Julien Danjou (No Starch Press, 2018), and often times you don’t really need extra stuff.

Here are two small illustrations of my wanderings with Python for basic statistical stuff. For a project at work, I needed a quick implementation of bootstrapping. This was just to ensure that the statistic I was computing on the full dataset was reliable enough. I was not interested in drawing confidence intervals or anything more fancy from the bootstrapped statistics. Initially, I tried using the asyncio module, but finally I ended up using multiprocessing (see Multi-blast and parallel processing for another use of this module).

Here’s my code:

import multiprocessing

B = 500
boot = []
p = multiprocessing.Pool(processes=multiprocessing.cpu_count() - 2)
tic = time.time()
data = numpy.loadtxt(f)
n = data.shape[0]
idx = numpy.random.choice(n, (B, n))
boot.append(p.map({foo}, (data[i, :] for i in idx.tolist())))
toc = time.time()
print(f">>> Elapsed time: {toc - tic:.2f}s")

Replace {foo} with the function that returns a statistic of interest.

Of course, for serious work you would need to compute 95 or 99% percentiles as well as correct for bias and skewness of the bootstrap distribution (BCA correction).

Here is another case: a toy implementation of k-fold cross-validation. I know various libraries, including the scikit-learn, offers much more elaborated solution to the design of cross-validated statistics, but this time I was in a hurry and I just needed a quick way to rely on training/test datasets. The basic idea was just to create a partition of my dataset using numpy and a reducing function to rearrange the train and test datasets. At a glance, it looks like this:

from functools import reduce
from statistics import mean

import numpy as np

x = np.random.normal(0, 1, 100)
partition = np.array_split(x, 10)

for k in range(len(partition)):
 curr = partition[k]
 other = reduce(
 lambda x, y: np.concatenate((x, y)),
 [x for i, x in enumerate(partition) if i != k],
 )
 print(f"curr ({k}) = {mean(curr)}, other = {mean(other)}")

Again, there’s room for improvement since usually we would first shuffle the data indices before partitioning the array in k sets.

♪ Cinderella • Don’t Know What You Got

Using managed functions in sqlite

Sun, 19 Feb 2023 10:09:26 +0100

Sqlite comes with a bundle of built-in functions including aggregate functions such as count(), sum(),¹ or avg(). No function to compute standard deviation, though. There are ways to compute the variance in a query directly, and its square root, but hopefully sqlite3 comes with a set of extensions that can be loaded directly into sqlite command. See also miscellaneous function.

The following assumes that you already downloaded the extension-functions.c from the link above file and compiled it as shared library. On Linux, this is as easy as:

% gcc -g -fPIC -shared extension-functions.c -o extension-functions.so

Here is a toy example:

sqlite> .load ./extension-functions
sqlite> create table data(x real not null);
sqlite> insert into data select (abs(random() % 10)) as x from generate_series(1, 20);
sqlite> select round(stdev(x), 2) as sd from data;
sd
----
2.56

Note that generate_series() is already loaded in the command-line shell.

How about writing a custom function, e.g. a percentile() function which would return the percentile rank of an observation in a series of values? There’s a two-part series on this topic, which explains the basics: Writing a Custom SQLite Function (in C). Such functions are known as Run-Time Loadable Extensions, and there’s some boilerplate C code to get started.

If you are in a hurry, though, sqlean provides a bunch of domain-specific functions.² In the case of statistical functions, you will find the median, various pre-defined percentile as well as a percentile rank function, Pearson correlation coefficient, as well as sample and population estimate of the variance or the standard deviation, and many more. Installation is quite simple since there are binaries available for each OS.

sqlite> .load ./stats
sqlite> create table data(x real not null);
sqlite> insert into data select (abs(random() % 10)) as x from generate_series(1, 20);
sqlite> select avg(x) as avg from data;
avg
----
4.55
sqlite> select percentile(x, 25) as p25, percentile(x, 50) as p50, percentile(x, 75) as p75 from data;
p25 p50 p75
--- --- ---
2.0 5.0 7.0

[2023-06-19]
Recently, sqlean was made available for Python as a dedicated package, see Python’s sqlite3 with extensions.

♪ Lauren Bush • In a Mellow Tone

There’s also a total() function which behaves a little bit differently as it returns “0.0” in case the query returns only NULL values, while sum() will return NULL like other SQL variants. ↩︎
See also SQLiteStudio which seems to provide a function editor. ↩︎

Processing large data: Duckdb vs. datatable

Fri, 17 Feb 2023 19:06:33 +0100

I read Rasmus Bååth’s last post, Three strategies to tackle Big Data in R and Python, and I thought I would run my own benchmark on duckdb with the data discussed in my review of Exploratory Desktop some years ago.¹ Moreover, I’m interested in trying out Python datatable fread function.

[2023-04-06]
See also In-process, in-memory databases for a more detailed comparison of DuckDB vs. SQLite.

Duckdb has long been a database backend I wanted to try, along with TileDB (I followed Dirk Eddelbuettel work on its R frontend, but go checkout his website for more information; I also warmly recommend his online course on Data Science Programming Methods).

Let’s benchmark loading some data then. I fetched data for year 2021 (1.6 Go), which includes 3,669,928 records according to the codebook.² For the sake of comparison, loading the same data file in sqlite took less than 1,5 minutes, with little RAM caching. Simple queries are a breeze as expected, although I forgot to enable proper timing (.timer on):

sqlite> .mode csv nat2021us
sqlite> .import nat2021us.csv nat2021us
sqlite> SELECT DISTINCT dob_mm, COUNT(*) AS count FROM nat2021us GROUP BY dob_mm ORDER BY CAST(dob_mm AS UNSIGNED) ASC;
dob_mm,count
1,277533
2,266725
3,303139
4,293630
5,301343
6,314024
7,326611
8,330740
9,326280
10,315909
11,302309
12,311685

Now using duckdb:

import duckdb
import time

tic = time.time()
con = duckdb.connect(database='duckdb_python_db')
con.execute('drop table if exists nat2021us')
con.execute("""
 create table nat2021us as
 select * from read_csv_auto("nat2021us.csv")
""")
toc = time.time()
con.close()
print(f"Elapsed time: {(toc-tic) * 10**3:.2f} ms")

It shortly raised 99% but then we have to wait for about 1 minute for the data to be available at the IPython prompt:

100% ▕████████████████████████████████████████████████████████████▏
Elapsed time: 62689.09 ms

And here is the datatable approach:

import datatable as dt

tic = time.time()
nat2021us = dt.fread("nat2021us.csv")
toc = time.time()
print(f"Elapsed time: {(toc-tic) * 10**3:.2f} ms")

I was a little bit surprised at first: it took less than 5s (!) , without much more RAM caching:

100% |██████████████████████████████████████████████████| Reading data [done]
Elapsed time: 4136.24 ms

Then, again results are computed at the speed of light:

In [4]: nat2021us[:, dt.count(), dt.by(dt.f.dob_mm)]
Out[4]:
 | dob_mm count
 | int32 int64
-- + ------ ------
 0 | 1 277533
 1 | 2 266725
 2 | 3 303139
 3 | 4 293630
 4 | 5 301343
 5 | 6 314024
 6 | 7 326611
 7 | 8 330740
 8 | 9 326280
 9 | 10 315909
10 | 11 302309
11 | 12 311685
[12 rows x 2 columns]

Clearly, the datatable package is a winner here. I’ll keep these quick results in mind when I have to load large datasets in Python.

♪ Latarnik • Marianna

See also Processing large CSV files. ↩︎
National Center for Health Statistics (2021). Data File Documentations, U.S. Natality Data Files, 2021 (machine readable data file and documentation, CD-ROM Series), National Center for Health Statistics, Hyattsville, Maryland. ↩︎

On the sweep operator in regression settings

Thu, 16 Feb 2023 08:32:57 +0100

Regression involves many linear algebra tricks. We already discussed the case of influence measure which can be computed most of the time in one pass, even in the case of ML.¹ Beside model diagnostics, several other algebraic properties are in use when dealing with the normal equations. Recall that parameters of a linear regression model can be found using iterative solution (e.g., gradient descent) or the following algebraic equation: $\hat\theta =(X^TX)^{-1}X^Ty$, where $\theta$ is the parameter of interest (a vector of regression coefficients), $X$ the design matrix and $y$ the response variable. This requires inverting a matrix but there are ways to alleviate its cost as we’ve seen in a previous post on matrix decomposition.

Under the OLS framework, two elimination methods gained popularity long ago. Since we are dealing with a system of linear equations, Doolittle method, which is based on Gaussian elimination, is one option, the other being the Gauss-Jordan (full) elimination. The latter leads to the sweep operator.² Rick Wicklin wrote two nice blog posts on this topic. Basically, the sweep operator allows to “sweep in” or “sweep out” rows of the $X^TX$ matrix, and you can even “sweep in” columns in any order, which amounts to add effects to a model.

The algorithm is discussed in one of Richard B. Darlington’s note. Julia code is available on these blog posts, while R code is provided in The Statistician’s Apprentice: An Introduction to the SWP Operator. Finally, it is worth noting that:³

The SWEEP operator can be programmed to produce generalized inverses and create, as by-products, such items as the Forward Doolittle matrix, the Cholesky decomposition matrix, the Hermite canonical form matrix, the determinant of the original matrix, Type I sums of squares, the error sum of squares, a solution to the normal equations, and the general form of estimable functions.

♪ Poppy • Flux

W.J. Shih and S. Weisberg. Assessing Influence in Multiple Linear Regression With Incomplete Data. Technometrics, 28(3): 231-239, 1986. ↩︎
John F. Monahan. Numerical Methods in Statistics. Cambridge University Press (2011). ↩︎
James H. Goodnight. A Tutorial on the SWEEP Operator. A Tutorial on the SWEEP Operator. The American Statistician, 33(3): 149-158, 1979. ↩︎

Statistical charts using Mathematica

Wed, 15 Feb 2023 08:09:39 +0100

This is yet another post on Mathematica where I am trying to document my journey through statistical analysis using this very beast. In previous posts, I show how to draw interaction plots using Python or Stata. Now it’s time to see what we can do with Mathematica.

First we will make a little digression with a very basic plot of group means with confidence intervals. The HypothesisTesting package provide the MeanCI function which returns a list of lower and upper 95% confidence limits. Let’s load the same genotype data as before, and I’ll keep using plain lists rather than associations:

d = Import["polymorphism.dta", "Dataset"];
ds = Flatten /@ List @@@ Normal @ d;

Then, we easily get our desired confidence intervals: (output is prefixed with =>)

Needs["HypothesisTesting`"]
cis = Table[{x[[1, 3]], MeanCI[x[[All, -2]]]}, {x, GatherBy[ds, #[[3]]&]}]
=> {{1., {58.1871, 71.0986}}, {2., {59.3357, 69.423}}, {3., {44.706,
 56.044}}}

The values {1, 2, 3} correspond to the genotype 1.6/1.6, 1.6/0.7, and 0.7/0.7, respectively.

This is fine, but to manage error bars in a scatterplot, it is now recommended to use ListPlot instead of ErrorListPlot (as of Mathematica v12). Moreover, rather than creating a list with two values (the lower and upper limits of the 95% CI), why not use the Around function, which works directly with ListPlot? Here is my take, considering a Gaussian distribution rather than a Student t distribution:

ErrorMarginNormal[x_, p_] := Quantile[NormalDistribution[0, 1], 1 - p/2] * StandardDeviation[x]/Sqrt[Length[x]]
MeanCINormal[x_] := Around[Mean[x], ErrorMarginNormal[x, .05]]
cis = Table[{x[[1, 3]], MeanCINormal[x[[All, 2]]]}, {x, GatherBy[ds, #[[3]]&]}];
=> {{1., Around[64.64285714285714, 5.856898699924958]}, {2., Around[
 64.37931034482759, 4.825889674452653]}, {3., Around[
 50.375, 5.212899350342959]}}

From there on, it’s just a matter of Listplotting the whole thing:

ddn = MapAt[# + RandomReal[{-.05,.05}]&, ds, {All, 3}]; (* jittering on x-axis *)
xlabs = {{1, "1.6/1.6"}, {2, "1.6/0.7"}, {3, "0.7/0.7"}};
Show[ListPlot[ddn[[All, {3, 2}]], Ticks -> {xlabs, Automatic},
 AxesLabel -> {Genotype, Age}, PlotRange -> {{0.5, 3.5}, Automatic}],
 ListPlot[cis]]

I did not find an option to add a bit of jittering on the x-axis, hence the derived data table ddn where I added random noise on the abscissae. Results is shown in the left panel below.

As an another illustration of the flexibility of Mathematica regarding statistical graphics, here is our interaction plot from the previous posts.

d = Import["../data/toothgrowth.dta", "Dataset"];
ds = Flatten /@ List @@@ Normal @ d;

avg = Table[{x[[1, 3]], Mean[x[[All, 1]]]}, {x, GatherBy[ds, #[[{2, 3}]]&]}];

ddn = MapAt[# + RandomReal[{-.05,.05}]&, ds, {All, 3}];
ddn = SplitBy[ddn, #[[2]]&];

Show[ListPlot[ddn[[All, All, {3, 1}]], AxesLabel -> {Dose, Length},
 PlotRange -> {{0.25, 2.25}, Automatic}, GridLines -> Automatic],
 ListPlot[Take[avg, 3], Joined -> True, PlotStyle -> ColorData[97][1]],
 ListPlot[Take[avg, -3], Joined -> True, PlotStyle -> ColorData[97][2]]]

The result is displayed in the right panel of the above figure.

♪ Joan As Police Woman • Holiday

Dealing with categorical variables in Mathematica

Tue, 14 Feb 2023 11:23:10 +0100

Until now, I was always defining a design matrix by hand to fit, say, a linear regression model with categorical covariates in Mathematica. If you know how to define a linear model in Mathematica, DesignMatrix works exactly the same. I suggested this approach to a Matlab user a long time ago. Then I realized that Mathematica provides NominalVariables since at least version 7. Of course, I was searching using different keywords (e.g., “categorical predictors” or “dummy coding” – the latter being used for dummy variables in a mathematical sense) all time long.

The take away message is that if you happen to work with a rectangular table where categorical predictors are stored using their labels rather than a numerical code, and you want to perform a one- or two-way ANOVA, or an analysis of covariance, then NominalVariables -> Automatic is your best friend.

As an illustrtaion, consider the following Stata dataset, polymorphism.dta,¹ which includes 3 variables (participant ID, age, genotype with three levels). The magic of Import is that it works with Stata data format too, especially when it is imported as a Dataset:

d = Import["polymorphism.dta", "Dataset"];

If you prefer the traditional Table view (right panel in the above figure), you can omit the last option, and if you really need to add a header, you can add it in the front of the list:²

labels = {"id", "age", "genotype"};
d = Prepend[Import["polymorphism.dta"], labels];

Note that you can also flatten your Dataset using ̀Flatten /@ List @@@ Normal @ d (h/t kglr. I much prefer working with lists than Dataset, but that may well be because I have had bad experiences with GroupBy and GatherBy on Datasets so far.

Back to model fitting. To change the reference category, say consider the third category as the baseline, I did not find an easier way than recoding the numerical values using an anonymous function like so:³

dd = MapAt[4 - #&, d, {All, 3}

Then we can fit a one-way ANOVA and print an ANOVA table using the following commands:

m = LinearModelFit[dd[[All, {3, 2}]], x, x, NominalVariables -> x];
Normal[m]
m["ANOVATable"]

Note that regressors come first, and the response variable is the last one (age is the second column in our case). If you have multiple explanatory variables, use a list instead, e.g. {x1, x1^2, x2}, {x1, x2}. It is also possible to let Mathematica build indicator variables for categorical variables using NominalVariables -> Automatic; it works well when categorical variables are stored as strings and not as integers in your Table.

♪ Jeff Beck Group • Situation

William D. Dupont (2009). Statistical Modeling for Biomedical Researchers (2nd ed.), Cambridge. ↩︎

You could also create a Dataset on the fly:

d = Import["polymorphism.dta"]
MakeDataset[data_List, header_List] := Dataset[AssociationThread[header, #] & /@ data]
MakeDataset[d, labels]

↩︎

Note that this is equivalent to defining a named function instead of using an anonymous function, e.g.
```
f[x_] = 4 - x
MapAt[f, d, {All, 1}]
```
↩︎

Stata plot of the week #4

Fri, 10 Feb 2023 20:40:06 +0100

This is a continuation of a previous post on building interaction plots in Stata, where I briefly mentioned the margins and marginsplot commands. As I am working on some statistical code as a side project, I need to reproduce a lot of R plots made with ggplot2.

Let’s say we want to reproduce the picture shown at the end of this post. It is a combination of jittered scatterplot with (mean) line plot superimposed. There are surely some Stata packages that are lying around the internet to do this with a magic command, but let’s do this the hard way using built-in commands only. Note that I generally set plotplain as my default color scheme in Stata.¹ So we also need to manage color at some point. But let’s get some data first.

use toothgrowth, clear

summarize

egen dosec = group(dose), label
quietly anova len dosec#supp
quietly margins dosec#supp

At this point, we can use marginsplot directly and start customizing its display options:

marginsplot, noci title("") xtitle(Dose (mg/day)) ytitle(Length (oc. unit)) ///
 addplot(scatter len dosec if supp == 1, ms(oh) jitter(5) ///
 mc(ebblue) text(20 1 "OJ", color(ebblue) size(medlarge)) ///
 xscale(r(0 4)) xlab(0(1)3) || scatter len dosec if supp == 2, ///
 ms(oh) jitter(5) mc(orange) text(10 2 "VC", color(orange) ///
 size(medlarge))) scheme(uncluttered)

In this case, we use the uncluttered color scheme (see also this review of various other schemes on the Stata blog), which gently manages the color for us (although I did not choose the right blue/orange combination in this case).

Here is a similar command built using the menu dialog options for marginsplot, which works with the plotplain scheme:

marginsplot, noci recast(line) ///
 plot1opts(lcolor(ebblue)) plot2opts(lcolor(orange)) ///
 addplot((scatter len dosec if supp == 1, mcolor(ebblue) jitter(5)) ///
 (scatter len dosec if supp == 2, mcolor(orange) jitter(5))) ///
 ytitle(Length (oc. unit)) xtitle(Dose (mg/day)) xscale(range(0.75 3.25)) ///
 title("") legend(off)

I should note taht I was not able to reorder the legend (to suppress the additional keys generated for the scatter added plots) via the menu options. Likewise, I did not bother standardizing the symbol shape as oh. Anyway, this is not hard to fix and both results are shown side by side below:

The anovaplot command works also quite well, actually:

anovaplot dosec supp, scatter(jitter(5))

♪ Rammstein • Amerika

Here is my profile.do configuration by the way:

sysdir set PLUS "~/.local/lib/ado/plus/"
sysdir set PERSONAL "~/.local/lib/ado/personal/"
sysdir set OLDPLACE "~/.local/lib/ado/"
set seed 101
set logtype text
set matastrict on
set maxvar 5000
set scrollbufsize 2000000
set tracedepth 1
// set varabbrev off
set scheme plotplain
graph set eps fontdir ~/.local/share/fonts
graph set window fontface "Roboto Condensed"
if "`c(console)'"!="console" graph set eps preview on
graph set eps logo off
graph set eps fontface "Roboto Condensed"
graph set print logo off
// set autotabgraphs on, perm
set max_memory 8g, permanently

↩︎

Biostatistical methods

Mon, 06 Feb 2023 20:30:52 +0100

I am currently working on porting a workshop I held on several occasions in the past on Biostatistical modelling into different different languages. The exact title in French is Méthodes biostatistiques, in honor of one of my first textbook on mathematical statistics (Philippe Tassi, Méthodes statistiques, 2004), although the handouts I wrote are mainly about applied and not theoretical statistics. The GitHub repository is around there.

I refactor part of the R code, and then started to write Stata and Python scripts to reproduce the R textual and graphical outputs. I intend to write more translations, at least for Clojure, Mathematica and Racket. Of course, it is somewhat limited by the available capabilities of each language, since I don’t have time nor the inclination to rewrite everything from scratch. I know I won’t be in trouble with Stata, but I doubt I will be able to equate all R’s results with any other programming languages. The rms and ggplot2 packages are damned good. Fortunately, Python has a nice package for the grammar of graphics, which I started using lately – I was really impressed that everything worked as expected for someone used to R syntax, and Stata has good facilities for faceted displays. We will see what’s doable in other languages.

My original idea with this side project was to practice again, of course (otherwise we end up forgetting the good old reflexes), but also to get an idea of what can be done from one language to another and if the native features of a language influence the way to approach the “data problem”. I learned Stata after R (and SPSS and SAS and Statistica and Lisp-Stat), and I learned a lot about how to organize or recode data to optimize data processing and to benefit from built-in features.

I had for a long time a big prejudice about Python because in my opinion if you can’t do interactive, specifically visual, exploratory analysis of the data then it’s better to turn to more adapted solutions. I still think that there’s room for improvements in the Python land for statistical computing – and I do not pretend nor am I interested in fixing the problem, but I must admit things have improved in the past few years. And as I noted above, plotnine is really a great add-on to the Python “data stack”.

Let’s see how it goes then. I will write a more detailed review of what I’ve learned from this porting experiment, and what are the pros and cons of each of those languages with respect to biostatistical modelling at large.

♪ Bauhaus • Spirit

Regression diagnostics and influence measures

Thu, 02 Feb 2023 20:55:35 +0100

In one of my answer on Cross Validated I discussed base R solution for diagnosing a regression model. Specifically, I said that “The function influence() (or its wrapper, influence.measures()) returns most of what we need for model diagnostic, including jacknifed statistics. As stated in Chambers and Hastie’s Statistical Models in S (Wadsworth & Brooks, 1992), it can be used in combination to summary.lm(). One of the example provided in the so-called “white book” (pp. 130-131) allows to compute standardized (residuals with equal variance) and studentized (the same with a different estimate for SE) residuals, DFBETAS (change in the coefficients scaled by the SE for the regression coefficients), DFFIT (change in the fitted value when observation is dropped), and DFFITS (the same, with unit variance) measures without much difficulty.”

All of this relies on linear algebra and does not require to perform matrix multiplication or inversions again and again. My impression is that few people really use influence measures (contrary to, say, variance influence factors) even if it may bring some insights into data or model quality. These are simple quantities, though. See one of my old workshop (in French) on statistical modeling (slide #13). For instance, the leverage statistic, or hat value, is defined for observation $i$ as

$$h_i = \frac{1}{n} + \frac{(x_i - \bar x)^2}{\sum_{i=1}^n (x_i - \bar x)^2} = \frac{1}{n} + \frac{(x_i - \bar x)^2}{(n-1)s_x^2}.$$

Note that there is no strict correspondence between outlying and high leverage observations. The mean leverage value is $(k+1)/n$, where $k$ is the number of parameters in the linear model. In the case of simple linear regression, $k=2$. The rule of thumb is that an observation must be carefully checked if $h_i > \frac{2(k+1)}{n}$, which is usually what dedicated statistical packages highlight when asking to print individual diagnostic data.

Fortunately, the Python statsmodels package provides OLSResults.get_influence(), which returns everything we need.

Here is a sample Python script which demonstrates how to compute Cook’s distance¹ from a multiple regression model as illustrated in R’s on-line help for influence.

Here are the (filtered) results from R:

 dfb.1_ dfb.pp15 dfb.pp75 dfb.dpi dfb.ddpi dffit cov.r cook.d hat inf
Japan 0.63987 -0.65614 -0.67390 0.14610 0.388603 0.8597 1.085 1.43e-01 0.2233
Libya 0.55074 -0.48324 -0.37974 -0.01937 -1.024477 -1.1601 2.091 2.68e-01 0.5315 *

♪ The Loyal Seas • Last of the great machines

Cook’s distance can also be defined in terms of leverage values. ↩︎

Data tables in Python

Tue, 31 Jan 2023 13:21:30 +0100

Although Pandas is a great package to manipulate rectangular data structures, I often find myself a bit lost with its syntax (the use of loc or iloc, for instance). I learned about the datatable package a while ago, then as usual forgot about it. Here is a short overview of how I generally use it for biostatistical stuff I happen to do during my free time.

Let’s get some data first. We will consider the ToothGrowth dataset from R builtin’s, which deals with the effect of vitamin C on tooth growth in guinea pigs:

The response is the length of odontoblasts (cells responsible for tooth growth) in 60 guinea pigs. Each animal received one of three dose levels of vitamin C (0.5, 1, and 2 mg/day) by one of two delivery methods, orange juice or ascorbic acid (a form of vitamin C and coded as VC).

Fortunately, it is available in Python thanks to the rdatasets package:

from rdatasets import data, descr
from plotnine import *
import datatable as dt

print(descr("ToothGrowth"))
dataset = data("ToothGrowth")
dataset.head()
## len supp dose
## 0 4.2 VC 0.5
## 1 11.5 VC 0.5
## 2 7.3 VC 0.5
## 3 5.8 VC 0.5
## 4 6.4 VC 0.5

For those familiar with the R data.table package, the syntax will be easy to grasp since it is 95% the same in the most use cases. For example, if we want to aggregate data using group means, we can proceed as follows:

d = dt.Frame(dataset)
d[:, dt.mean(dt.f.len), dt.by("supp", "dose")]
## | supp dose len
## | str32 float64 float64
## -- + ----- ------- -------
## 0 | OJ 0.5 13.23
## 1 | OJ 1 22.7
## 2 | OJ 2 26.06
## 3 | VC 0.5 7.98
## 4 | VC 1 16.77
## 5 | VC 2 26.14
## [6 rows x 3 columns]

Other variations are possible, e.g. count the number of unique or missing values, summarize the data table using mean, min or max, etc. Here are tow further examples:

d.mean()
## | len supp dose
## | float64 float64 float64
## -- + ------- ------- -------
## 0 | 18.8133 NA 1.16667
## [1 row x 3 columns]

d[:, dt.count(), dt.by("supp")]
## | supp count
## | str32 int64
## -- + ----- -----
## 0 | OJ 30
## 1 | VC 30
## [2 rows x 2 columns]

Like the R data.table package, there’s an over-optimized fread function which can handle CSV, Excel, and many more formats, while automagically detecting the appropriate type of variable. Moreover it is supposed to handle large data files.¹

The H2O.ai team discusses the basic features of the datatable package on their blog: Introducing DatatableTon – Python Datatable Tutorials & Exercises. You may also like An Overview of Python’s Datatable package.

As a sequel to my previous post on seaborn and plotnine, let’s see how we can reproduce the following plot (left panel), which was made in R:

R code is shown below:²

r <- aggregate(len ~ dose + supp, data = ToothGrowth, mean)
p <- ggplot(data = ToothGrowth, aes(x = dose, y = len, color = supp)) +
 geom_point(position = position_jitterdodge(jitter.width = .1, dodge.width = 0.25)) +
 geom_line(data = r, aes(x = dose, y = len, color = supp)) +
 scale_color_manual(values = c("steelblue", "orange")) +
 guides(color = FALSE) +
 geom_dl(aes(label = supp), method = list("smart.grid", cex = .8)) +
 labs(x = "Dose (mg/day)", y = "Length (oc. unit)")
p

And here is the Python version:

r = d[:, dt.mean(dt.f.len), dt.by("supp", "dose")]
(ggplot(d, aes(x="dose", y="len", color="supp"))
 + geom_point(position=position_jitterdodge(jitter_width=.1, dodge_width=0.25))
 + geom_line(r, aes(x="dose", y="len", color="supp"))
 + scale_color_manual(values=["steelblue", "orange"], guide=False)
 + labs(x="Dose (mg/day)", y="Length (oc. unit)")
 + theme_minimal())

Not much of a difference either way. See the final result on the right panel of the previous figure.

♪ Badflower • Ghost

It would be interesting to benchamrk datatable.fread against R data.table or pandas.read_csv, like I did in my review of Exploratory Desktop. ↩︎
Shorter version for the impatient: lattice::xyplot(len ∼ dose, data = ToothGrowth, groups = supp, type = c ("p", "a")). ↩︎

A grammar of graphics for Python

Mon, 30 Jan 2023 13:21:25 +0100

I use Matplotlib for all my plots in Python. Seaborn is becoming a really great way to use Matplotlib backend for statistical graphics. I originally thought it was only working with Pandas data frames, but it also accepts Numpy libraries as well as built-in Python types like lists and dictionaries.

Here is an example of plot from the on-line tutorial:

import seaborn
flights = seaborn.load_dataset("flights")
seaborn.relplot(data=flights, x="year", y="passengers", hue="month", kind="line")

To produce this kind of plot using Matplotlib alone, it would requires a lot more instruction as we need to group data by month and use year on the x-axis. The syntax above is mostly comparable with qplot from the ggplot2 R package.

This dataset is basically a data structure we studied in another post. To reproduce our earlier small multiples (see right panel above), we would write:¹

g = seaborn.relplot(
 data=flights, x="year", y="passengers",
 col="month", col_wrap=4, kind="line",
 linewidth=4, zorder=5, legend=False
)
for month, ax in g.axes_dict.items():
 seaborn.lineplot(
 data=flights, x="year", y="passengers", units="month",
 estimator=None, color=".7", linewidth=1, ax=ax,
 )

The seaborn.relplot command is used for facetted displays. The on-line doc says that it

provides access to several different axes-level functions that show the relationship between two variables with semantic mappings of subsets. The kind parameter selects the underlying axes-level function to use

In the above example, the same effect could be achieved using seaborn.lineplot instead. There’s a lot more commands in the API, most of them targeting statistical applications (e.g., rugplot, ecdfplot, kdeplot, strippplot, boxplot). Combined with statsmodels and (scipy](https://scipy.org/), I feel like it provides a complete statistical toolbox for newcomers to Python for statistical munging, albeit with less specialized routines compared to R or Stata.

Next to Seaborn, there’s plotnine, which started to be developed six or seven years ago. Its syntax is even closer to ggplot2, so if you’re an R user you may want to start with plotnine directly. For instance, the above plot could be produced as follows:

from plotnine import *
(ggplot(flights, aes("year", "passengers", color="factor(month)")) + geom_line())

Note that you don’t need to call pyplot’s show() in this case. The number of geoms that are currently supported make it a good competitor to other ggplot-like alternative in Python. Actually, the ggplot package looks staled; I haven’t tested lets-plot, and I don’t remember the third option I tried a few years ago but it was nothing compared to plotnine.

Our facetted small multiples would then be written as follows:

from plotnine import *
flights2 = flights.rename(columns={"month": "mmonth"})
(ggplot(flights)
 + geom_line(aes("year", "passengers", color="factor(month)"), size = 3)
 + geom_line(flights2, aes("year", "passengers", group="factor(mmonth)"), color="grey")
 + facet_wrap("~ month", nrow=3)
 + scale_color_discrete(guide=False))

[2023-05-09]
Allen Downey makes a heavy use of Matplotlib, oftentimes through custom plotting functions. See figure 3 in this blog post for a variant of small multiples that I personally like a lot.

♪ Face The Beat: Session 7 • Dance My darling – Love & Hate

There’s a slightly different illustration of small multiple time series on Seaborn gallery. ↩︎

Successful Lisp

Fri, 27 Jan 2023 10:09:25 +0100

Here are some words about Successful Lisp, by David Lamkins. I started reading it last Summer, then stopped, then get at it again before Christmas. Meanwhile, I haven’t wrote a single line of CL… A short review appeared years ago on Eli Bendersky’s website.

In Lisp, the parenthesis is the punctuation.

This book is available on one of Gilbert Strandh’s websites. It aims at describing what Lisp is, what it is good for, and how the language differs from other languages mostly derived from Algol. The above quote is just a joke, since many newcomers to Lisp-based languages seem to dislike those pesky parentheses that sit everywhere in Lisp or Scheme code blocks.¹

As stated above, this is a book about Lisp, the language, and not Lisp as used in scientific computing or AI applications. Chapter 3 provides a brief overview of essential Lisp concepts: syntax and evaluation of forms (atoms and lists) using prefix notation, special forms and macros, etc. It is organized as a set of 12 lessons, along with non-trivial illustrations sometimes (e.g., setq versus set). Most importantly, it covers data structures like vectors, arrays or structure and hash tables. Whenever I read about CL’s array, I think of one of Tamas Papp’s older post but I think CL still provides workable solution for numerical applications, as can be seen from some of Robert Smith’s projects. Chapters 4, 6, and 8 deal with local and global variable, namespace, lexical binding and the like. It is the “essentials” but I believe it is a disguised way to compare CL to other programming languages, with the exception of tail recursion and reader macros which are specific to functional languages. Chapter 5 discusses loops, and I’d learned to remember Stata’s own looping procedure thanks to the distinction between dotimes and dolist in CL. Chapters 7 and 14 is all about CLOS, the object system in CL. Chapter 9 discusses error handling; see The Common Lisp Condition System: Beyond Exception Handling with Control Flow Mechanisms for a modern approach to control flow in CL. Sequences and operations on sequences (map, filter, reduce) are described in chapters 12 and 13. Closures are discussed in chapter 15, while the infamous “equal” comparisons are summarized in chapter 17. Chpaterd 19 and 20 are about streams and macros:

Lisp macros are Lisp programs that generate other Lisp programs.

Except for chapters 32 and following, the rest of the book is merely about how to program in CL and to manage a CL project.

People often recommend Practical Common Lisp to get started with Lisp. I think this book is also worth a look. Between two Lisps also provides a good discussion of CL versus Scheme, Clojure or whatever.

♪ Artic Monkeys • R U Mine?

I do, on the contrary, like them from a purely syntactically viewpoint, at least much better than significant indentation like in Python or Haskell. If you think about it a little, there’s not much of a difference between the following two statements: print("Hello") and (pprint "Hello"). By the way, John D. Cook already mentioned this on Twitter years ago. If your text editor supports bracket matching — and you’re not forced to use Emacs — you will easily navigate between balanced parenthesis, and more generally the highly structured code that Lisp or Scheme provide. I also hope that treesitter will be a great help in the future. After this brief parenthesis, let’s return to the subject of this post. ↩︎

Micro posting in January

Thu, 26 Jan 2023 09:17:57 +0100

2023-01-04:

In the beginning, there were blogs, and they were the original social web. We built community. We found our people. We wrote personally. We wrote frequently. We self-policed, and we linked to each other so that newbies could discover new and good blogs. I want to go back there. — Bring back personal blogging (h/t Jack Baty)

2023-01-04: TIL that TeXlive now includes pdfxup as a replacement for the good old pdfnup. Time to update my oldie shell alias alias p2x1="pdfnup --nup 2x1 --landscape --suffix '2x1' --batch ".
2023-01-04: The faker’s guide to reading (x86) assembly language.
2023-01-04: Understanding N + 1 queries problem 01 Jan 2023.
2023-01-04: ♪ Kaiser Chiefs · Ruby
2023-01-17: ♪ John Scofield · Danny Boy
2023-01-18:

The family therapist Salvador Minuchin declared, “The human experience of identity has two elements: a sense of belonging and a sense of being separate.” This is as good a description of digital identity as it is of our psychological identity. A digital identity contains data that uniquely describes a person or thing but also contains information about the subject’s relationships to other entities. — Defining Digital Identity

2023-01-20: As a sequel of this old post of mine, here is the complete archive of the 467 answers I provided on Cross Validated during ten years or so. Note that this a raw (unedited) HTML, with many tiny bugs along the way, mostly related to Unicode encoding and missing images from imgur. I can probably with the UTF-8 issue by using a different processor than htmldoc. As for the latter, I believe the issue comes from Pandoc, since I retrieved a larger number of images manually using curl (tarball archive).
2023-01-20: ♪ Carpenter Brut · Maniac (feat. Yann Ligner)
2023-01-21: RIP Tweetbot. Ironically, I started my review of this app by saying that Twitter retracted their own client for OS X long ago; now they simply suspended access to 3rd party clients.
2023-01-23:

In fact, the vast majority of my editing tasks – probably as much as 98% – are performed exactly as if I was in Vim. (For the other 2% I’ve invented my own Vim-style keybinds.) Normally, I don’t notice that I’m actually not in Vim. The fact that I happen to be running the Emacs process is a mere technicality. — On Escape Meta Alt Control Shift

2023-01-23: Algorithms for Modern Hardware is a gold mine in the field of high performance computing.
2023-01-23: Elements of Data Science is “an introduction to data science for people with no programming experience. My goal is to present a small, powerful subset of Python that allows you to do real work in data science as quickly as possible.” Great work by Allen Downey as always.
2023-01-24:

Users who want to block all such external mediaanalysisd look-ups can do so using a software firewall to block outgoing connections to Apple’s servers by that process through port 443. That may well disable other macOS features. — Is Apple checking images we view in the Finder?

2023-01-24: Twenty years-old stuff that are still lying around, even though I wear the ring most of the time.

2023-01-24: Secure Ubuntu 18.04 server setup: Never tried to use an encrypted LVM volume unlockable via SSH. It looks like an interesting approach:

Applying encryption is fast when it is done upon creation: since the initial contents of the partition are ignored, they are not encrypted; only new data will be encrypted as it is written. — Ubuntu LVM Encryption

2023-01-24: ♪ Massive Attack · Inertia Creeps
2023-01-27: Listening to the last album from Tindersticks, Stars at Noon. Great artwork, as always. Now I need to buy their Past Imperfect compilation, which featured a limited edition box set comes with a double LP of Live at Glasgow City Halls, 5th October 2008.

Hash collision

Wed, 25 Jan 2023 09:17:27 +0100

It is usually assumed that the probability of getting the same hash value for two different input items is approximately $\frac{k^2}{2N}$, for $k$ random hash keys ($0 < k < N$) taking up to $N$ possible values.

In essence, it is a lot like the birthday paradox that we discussed earlier. Let us start to enumerate the case where hash keys are all unique. With $N$ possible values, after we pick the first key, there are $N-1$ remaining keys out of $N$, hence the probability $\frac{N-1}{N}$ of randomly generating two distinct integers. Then, $N-2$ keys remains, and so forth, so that in the end the probability of randomly generating $k$ integers that are all unique is $\frac{N-1}{N} \times \frac{N-2}{N} \times \cdots \times \frac{N-(k-1)}{N}$. Using Taylor expansion, this last expression can be approximated by $e^{\frac{-k(k-1)}{2N}}$, which is what we ended up considering in the Birthday’s paradox as well. The probability of a hash collision is therefore

$$1 - e^{\frac{-k(k-1)}{2N}}.$$

However, it can further be shown that for $x$ small, $1-e^{-x} \approx x$, which means that the probability expressed above can also be approximated with $\frac{k(k-1)}{2N}$. No more floating points! FInally, if $k$ is large, then $k(k-1) \approx k^2$, and we find our $\frac{k^2}{2N}$ expression from the beginning.

♪ Toggi Jonsson Quarted • Von

January in review

Sat, 21 Jan 2023 18:41:31 +0100

Modern Polars offers a side by side comparison of the Polars and Pandas libraries for working with rectangular datasets in Python. From reading the Polars documentation, I like the idea of having select or filter methods, instead of relying on Pandas default indexing facilities. We can even address rows and columns using numbers, like in R, which would make transitioning from R data frame less hard than when using Pandas. See also Replacing Pandas with Polars. A Practical Guide.
A Visual Guide to SSH Tunnels: Local and Remote Port Forwarding: This is a handy tutorial for local and remote port forwarding with great illustrations. It covers everything you need to know to use ssh tunnels confidently. “The mnemonics are “ssh -L local:remote” and “ssh -R remote:local” and it’s always the left-hand side that opens a new port.”
If you are looking for a quick but opinionated overview of Vim’s for editing code at the speed of light, Vim is touch-typing on steroids does a great job.
snakemake for doing bioinformatics - a beginner’s guide : I’m still not convinced that snakemake is the best way to manage data processing using shell commands, and I’m still relying on Bash scripts and occasionally good old Makefile. Things that really rock, though, are job caching and multi-core capabilities (based on binary knapsack optimisation).
A while ago I learned about phased packages in Ubuntu (there seems to be an increasing number of such phasing process in the 22.04 LTS, by the way), but Chris Siebenmann covers it elegantly in one of his recent blog post.
Faster virtual machines: Speeding up programming language execution: For compiler and static typing enthusiasts. See also Assembly Optimization Tips for more optimization on the dark side.
Demystifying Apache Arrow: A quick and decent summary of Apache Arrow main features for data munging.
Reflecting on Transducers, or how to get around Scheme missing a generic way to operate over different collections.
These Years in Common Lisp: 2022 in review: A thorough overview of what happened last year in the CL ecosystem. This covers various topics like the previous year (concurrency, databases, games and graphics, etc.). I like this kind of year in review since it allows me to discover a lot of interesting projects like Kons-9, for instance. If you are into gaming (coding and/or playing), you probably heard of Kandria which was released in recent months. If you want to learn more about CL, the author is currently working on a Common Lisp Video Course.
Rust for professionals is a short introduction to Rust, intended for developers that already know another language, according to its author. It is a bit like learn xx in xx hours/days/weeks, but it may help getting a taste of the language before delving in the Rust book.

♪ kim dracula • Paparazzi

Mathematica for statistics

Wed, 18 Jan 2023 21:12:13 +0100

I have long been interested in replacing R or Stata with Mathematica. I know it’s not entirely possible due to the large amount of dedicated statistical routines (especially, post-fit procedures) available in these two statistical packages, but Mathematica is fine for most use cases I encountered the last two years, i.e. fitting a linear regression to a bivariate datasets, fitting splines, computing pairwise correlation coefficient, working with contingency tables, etc.

Eralier on I wrote a quick tutorial on how to fit a simple logistic regression model with Mathematica. Anton Antonov has a nice blog which describes common tasks in statistics or machine learning, and plently of examples in his GitHub repository on R vs. Mathematica.

One of the nice feature of Mathematica is that the basic data structure for working with data is the list. Expressions like {{1, 0.8}, {2, 3.2}, ...} allows to construct a bivariate series, which can be manipulated using lot of built-in functions, and formatted accordingly in Matrix form or as a grid. Moreover, there’s now the Dataset structure, although it is not ready to be consumed by most statistical fitting routines I’m interested in. Anyway, using lists as the basis for higher-order data structures is really nice, especially when you’re already involved in the Python ecosystem. I like the following reply on HN:

Mathematica is an interesting case study for a general-purpose software package that happens to be more feature-full and functional – not just “more convenient” or “better UX” – than any open-source counterparts.
I tend to use some proprietary scientific software, but a lot of it is because academia already has already invested in codebases for, say, Stata, GAMS, Matlab and so on. But Matlab is two steps removed from raw Fortran; and what sets it apart from the many identical-syntax clones are a few narrowly-oriented toolsets aimed at some kinds of engineers.
Mathematica is the only one I buy versions for my home computer. It’s very, very good.

Same for me, except that I also have a paid license for Stata 13 MP.

Here is a toy example of fitting a linear regression model in Mathematica:

data = Table[{3 + i + RandomReal[{-3, 7}], i + RandomReal[{-2, 5}]}, {i, 1, 20}];
Dataset[Prepend[data, {"x", "y"}]]

lm = LinearModelFit[data, x, x];
lm["BestFit"]

Show[ListPlot[data], Plot[lm[x], {x, 0, 30}]]

Even if this is a contrived example, it shows how easy it is to generate an artificial dataset, which can be printed much like an R data.table or Pandas data frame, to feed built-in functions. Most of Mathematica builtin functions are self-contained, which means there are a lot of optional parameters (but default values are usually good) and function names are generally well chosen. Default graphics are also well composed, as shown below:

One day, I will probably translate my old biostatistics tutorial to Mathematica.

♪ Lars Bygdén • Fall Into The Night

Recently

Tue, 17 Jan 2023 20:41:53 +0100

Same desktop environment for over five or six months, similar window tiling across session, and same tools as well: Neovim, Irssi, Tmux, Alacritty, Zsh, Cmus, Newsboat. No more config tweaks, since almost everything work as expected. Once you get the right tools, properly configured, stick with them and use them everyday.

I’ve been pretty busy the last two weeks with a new course on bioinformatics for genomic applications. I did a biostatistics refresher too, after a long hiatus (5 years with no teaching at all). It was exhausting, though. Some stuff need to be improved for the next session, but overall I was pretty happy with how things went along the course.

I deleted my Last.fm account (I got the annual statistics I wanted to see, and I’m not interested in any tracking or recommendation system). I didn’t renewed my paid subscription to Apple TV and Music, since they decided to increase their rates (at least by 2 € each). I’m almost done with Apple products by now, but I will keep by email address as well as my MacBook which runs fine under Mojave. Mojave is v10.14 while we are now at Ventura v13. I think I never look at the new functionalities starting from Catalina, and I don’t think I missed a lot in hindsight. Newly released laptops look great, with top notch specs, even for the new MacBook Air, although there have now become unaffordable when you want to increase default RAM or HD capabilities. The same applies to iPhone, not to mention the fact that they are now only available in a format close to a tablet.

♪ La Rue Kétanou • La fiancée de l’eau

Micro posting in December

Fri, 30 Dec 2022 20:35:59 +0100

2022-12-05: The Database Cookbook For Developers. #database
2022-12-05: ♪ Neil Young · Heart Of Gold
2022-12-06:

And we see that machines can fail Statistics, just like college students. — Just How Good Is ChatGPT in Data Science?

2022-12-08:

Code can only ever be fast if it’s written with the contraints of computer hardware in mind, and idiomatic Python isn’t. — What’s great about Julia?

2022-12-08: ♪ New Order · 5 8 6 (2020 Digital Master)
2022-12-15: ♪ Mal Waldron · Mistral Breeze / Sieg Haile
2022-12-19: LosslessCut: The Swiss Army Knife of Lossless Video/Audio Editing.
2022-12-20: JSXGraph: Dynamic Mathematics with JavaScript.
2022-12-22:

Wavefunction Collapse is a very independent-minded algorithm, and needs almost no outside help or instruction. You feed it an example of the vibe you’re going for, and it figures everything else out for itself. Despite this self-sufficiency, it is surprisingly simple. It doesn’t use any neural networks, random forests, or anything else that sounds like machine learning. — The Wavefunction Collapse Algorithm explained very clearly

2022-12-22: Comprehensive Rust.
2022-12-22: How I still use Flash in 2022. I sometimes miss that time. CSS is great but Flash was really adding value to some indie websites back in the day.
2022-12-22: Reanimate: Build declarative animations with SVG and Haskell. #haskell
2022-12-22: ♪ Joy Division · Love Will Tear Us Apart
2022-12-26: APITable, an API-oriented low-code platform for building collaborative apps and better than all other Airtable open-source alternatives.
2022-12-26: Challenging algorithms and data structures every programmer should try.
2022-12-26: Introduction to Locality-Sensitive Hashing. #statistics
2022-12-26: The Boxer, the Wrestler, and the Coin Flip: A Paradox of Robust Bayesian Inference and Belief Functions (PDF, 5 pp.). #statistics
2022-12-26: ♪ Matthew E. White · Will You Love Me
2022-12-28: Happy lazy loading of Neovim plugins (actually 15 plugins, including 13 opt plugins):

» vim-startuptime -vimpath nvim

Extra options: []
Measured: 10 times
Total Average: 15.662100 msec
Total Max: 16.085000 msec
Total Min: 15.505000 msec

2022-12-28: 26 programming languages in 25 days.
2022-12-28: How to annotate literally everything.
2022-12-28: ooh.directory: If you’re looking for unusual or hard-to-find websites via Google or friends… (h/t bsag).
2022-12-28: ♪ Wolfsheim · Once In a Lifetime
2022-12-30: Have been sick during two weeks. One week later, here are my top tags on Last.fm:

Computing eigenvalues using QR decomposition

Wed, 28 Dec 2022 09:31:49 +0100

Matthew Drury made a great job explaining how eigenvalues are usually computed, via the QR decomposition. This is an iterative algorithm which is available in Numpy, numpy.linalg.qr, while in R it is directly qr. We already discussed the QR decomposition in a previous post. Basically, the idea is as follows: Let $X$ be a symmetric matrix, compute $X_k = Q_kR_k$, for $k=1,\dots,n$, and update $X_{k+1}=R_kQ_k$. The sequence of $X_n$ converges to a diagonal matrix $D$ of eigenvalues, such that the eigenvectors are the columns of $\prod_i Q_i$.

In Racket, the very first iteration would read:

(require math)
(define X0 (matrix [[12 -51 4]
 [ 6 167 -68]
 [-4 24 -41]]))
(define X (matrix+ X0 (matrix-transpose X0)))
; (define D (diagonal-matrix '(1 1 1)))
(define D (identity-matrix 3))
(define-values (Q R) (matrix-qr X))
(define D (matrix* D Q))
(define X (matrix* R Q))

Wrap the above code in a function, iterate 20 to 30 times and then the $X$ matrix should contain the eigenvalues on its diagonal.

Of note, the Householder method for computing the QR decomposition in Racket is available on Rosetta.

[2023-05-23]
For other numerical approaches to computing eigenvectors and eigenvalues, see this excellent blog post by Marc Khoury: Numerical Algorithms for Computing Eigenvectors.

♪ Clan of Xymox • A Day

December in review

Mon, 19 Dec 2022 10:33:05 +0100

A quick summary of what I’ve read lately.

I’ve been trying codon recently. I didn’t make extensive benchmarks on some of my numerical code, but it already looks like it could be really interesting to speed up some computation, or at least to have sane static type checking ahead of time.
GNU/Linux shell related internals, by Viacheslav Biriukov, provides a very nice descritption of Unix shell, pipes & Co. I came across another good website about computing: Beautiful Software, which is Christopher Alexander’s research initiative on computing and the environment.

Ultimately, we want to help human intuition to do what it does best. That means people’s interactions with computers should only help them to do what is needed, to inform them, to help them connect to the real world, and to draw their attention towards important things they’ve forgotten, and away from unimportant things they’re too focused upon.

People keep getting moving (GitHub, Twitter, Slack, etc.). Now, Gitea changed their mind and Codeberg launches Forgejo. I’m still on GitHub since I only host hobby projects there, plus a couple of work-related shared and private repositories. I was interested in Codeberg or SourceHut at some point, but finally ended up staying on GitHub. However, I should probably jutst host my own RCS system.
I still have a lot of Lisp and Scheme-related books to review (one of those reviews will probably end up on this blog by the end of the year), but in the meantime you may enjoy reading The empty list by Tim Bradshaw. It’s always a pleasure to read his articles on Lisp.
The year of Linux on the desktop? A Linux evening On a related note, the RAM used by some of the common WMs on Linux is astonishing, e.g. KDE or Gnome, compared to lightweight solutions.

The magic disappeared and our optimism has since faded. Our websites are slow and insecure; our startups are creepy and unprofitable; our president Tweets hate speech; we don’t trust our social media apps, webcams, or voting machines. And in the era of coronavirus quarantining, we’re realizing just how inadequate the Internet turned out to be as a home of Mind. — Where Did Software Go Wrong?.

The above blog post somewhat reminds me of Niki Tonsky’s Software disenchantment.

Algebraic Effects - You Can Touch This!: An interesting discussion on condition systems in CL.
It’s time for advent of code. I keep reading interesting blog posts on the subject. Here’s one: A Neat XOR Trick, to find the first W-character window in which every character is unique in O(N) running time, with no dependence on the window size.

♪ Bauhaus • Bela Lugosi’s Dead

Recently

Thu, 08 Dec 2022 09:23:53 +0100

I’ve been away from this blog for a while now, but I’ve have been busy elsewhere, mostly drawing in my sketchbooks at home, and tackling various projects at work. Recently we got a 96-core Xeon with 512 Go RAM and an NVidia GPU. I can’t wait to test what it does for intensive parallel computing.

I deleted my Twitter account some time ago. Since then, most of my technological reviews come from my RSS feeds, Hacker News, ArXiv, and some targeted DDG queries. I don’t miss Twitter, my timeline was a just a giant mess even after several rounds of curation. I left some months before the mass transfer of users to Mastodon. Most of the bright people I was following on Twitter left as well, so I don’t regret my choice. But I don’t plan to join Mastodon or the Fediverse. Recent articles that discuss the Twitter-to-Mastodon switch and that attracted my attention are: soem of jwz’s post as referred to by Tom MacWright in his [montly review][] and in which legibility comes at a price.

I’ve been working on a fun side project using Clojure, and I was pleasantly surprised by the quality of the language server. Clojure is a great langauge after all. It’s been 8 years since I promised myself to seriously invest in it and I procrastinate around missed opportunities or other more urgent projects. Same for Julia, since it appears to be much more stabilized nowadays.

I’ve been trying Helix editor again, but overall I like my Neovim configuration better. However, I should note this really is a great project and it’s good there’s some competitive solutions sitting around. Sadly, Onivim development has stopped recently. Yes, I too love editors.

[2023-05-23]
See this recent review of Helix: A Walkthrough on Helix — A Post-Modern Modal Text Editor. It’s hard to fight muscle memory, though, even if there’s some support for Vim keybindings.

♪ New Order • Leave Me Alone

Micro posting in November

Tue, 29 Nov 2022 19:35:26 +0100

2022-11-03: ♪ Melanie De Biasio · Your Freedom Is the End of Me
2022-11-04:

Before we embark on yet another defense of the language along the lines of: parentheses in Lisp are great, but only insiders can understand them. Let’s notice a second element that often confuses those who learn the language: in Lisp everything is prefixed. — Why Lisp

2022-11-04: Linux Boot Partitions and How to Set Them Up. #unix
2022-11-04: ♪ Joy Division · Transmission
2022-11-08:

It’s easy to get hung up on this. I’ve definitely felt the self-imposed pressure to only write something if it’s new, and unique, and feels like it’s never been said before. This is a mental trap that does nothing but hold you back. — What to blog about

2022-11-08: Enjoying music curation again (via O. Valadares’ Blog).
2022-11-08: Performance Musings. #unix
2022-11-08: This is not your grandfather’s Perl.
2022-11-12: ♪ The White Stripes · Truth Doesn’t Make A Noise
2022-11-14: Futhark is a small programming language designed to be compiled to efficient parallel code.
2022-11-14: Numbers in a Nutshell, an Update. #lisp
2022-11-14: Understanding async Python for the web. #python
2022-11-14: Why does zsh start so slowly?. #unix
2022-11-14: ♪ Patti Smith Group · Seven Ways of Going
2022-11-15: Beware of fast-math.
2022-11-15: ♪ Anna Smyrk · Wallace Street
2022-11-16:

Privacy is necessary for an open society in the electronic age. Privacy is not secrecy. A private matter is something one doesn’t want the whole world to know, but a secret matter is something one doesn’t want anybody to know. Privacy is the power to selectively reveal oneself to the world. — A Cypherpunk’s Manifesto

2022-11-16: Great set of lecture notes: QuantEcon Lectures.
2022-11-16: Why does calloc exist?. #clang
2022-11-16: treevis.net - A Visual Bibliography of Tree Visualization 2.0 by Hans-Jörg Schulz. #dataviz
2022-11-16: ♪ Bauhaus · In The Night
2022-11-17:

Get ready to read a sentimental post about a data structure, and don’t tell I didn’t warn you. — In defense of linked lists

2022-11-17: Blot/Painting p5js sketch. Lovely.
2022-11-17: Build Your Own Text Editor. #clang
2022-11-17: ♪ Catherine Moan · Drop It!
2022-11-22: Today’s readings: Clojure’s Transducers in Haskell, and Understanding Clojure transducers through types. #clojure #haskell
2022-11-22: Python Tips & Tricks. #python
2022-11-22: Writing Interpreters in Rust: a Guide. #rust
2022-11-23:

Using a 12-bit colour depth limits the available colours, so slight changes to hue, chroma, and luminance must be made, but these are small enough not to be noticeable. The resulting palette has evenly-spaced hues, only small variations in chroma, and smoothly increasing and decreasing luminance. — The 12-bit rainbow palette

2022-11-23: Quirks of C. #clang
2022-11-23: x86 and amd64 instruction reference.
2022-11-28:

Ranking is a farce. Apparent performance is actually attributable mostly to the system that the individual works in, not to the individual himself. — W. Edwards Deming, via Statistical process control after W. Edwards Deming

2022-11-28: Figuring out round, floor and ceil with integer division. #clang
2022-11-28: GNOME doesn’t need to be that huge. #apple #unix
2022-11-28: ♪ Anthony Joseph · Swing Praxis (Kaidi Tatham Remix)
2022-11-29:

The short version is that most of your Instagram photos are hidden from your followers unless they receive engagement in the form of likes and comments. — The Indie Web is Dead, Long Live the Indie Web

2022-11-29: So many things already built-in in Mathematica… via Using Julia from Mathematica

2022-11-29: Everything I wish I knew when learning C. #clang
2022-11-29: Introduction to Genomics for Engineers. #bioinformatics
2022-11-29: Pattern Matching and Tail Recursion in Emacs Lisp. #lisp
2022-11-29: Sane Vim defaults (from Neovim). #vim
2022-11-29: The Matrix Calculus You Need For Deep Learning.
2022-11-29: ♪ Norah Jones · To Live

What not to compute

Sun, 30 Oct 2022 09:14:07 +0100

Usually for odd powers of the sine function, we use a little trick that relies on trigonometric identities and substitution. For instance, $\int\sin^3(x)dx = \int\sin(x)\sin^2(x) = \int\sin(x)(1-\cos^2(x))$, then using the substitution $u=\cos(x)$, we get $\int -(1-u^2)du\vert_{u=\cos(x)} = \cos^3(x)/3 - \cos(x) + c$.

In Numerical Methods That (Usually) Work, Forman S. Acton included a short interlude on what not to compute. This section is intended as a way to think carefully about numerical computation with a computer based on analytic solutions and rather think whether good enough approximations can be used.

As an example, consider the expression $\int_0^{0.3} \sin^8\theta\cdot d\theta$. We have

$$ \begin{equation} \begin{aligned} \int_0^{0.3} \sin^8\theta\cdot d\theta &= \left[(-\frac{1}{8}\cos\theta)(\sin^4\theta + \frac{7}{6}\sin^2\theta + \frac{35}{24})(\sin^3\theta) + \frac{105}{384}(\theta - \sin 2\theta)\right]_0^{0.3}\cr &= (-0.119417)(0.007627 + 0.101887 + 1.458333)(0.0258085) + 0.004341\cr &= -0.0048320 + 0.0048341 = 0.0000021 \end{aligned} \end{equation} $$

Note that this formula allows to compute a small number based on the difference between two much larger numbers. However, using a crude approximation for $\sin\theta$,¹ we get

$$ \int_0^{0.3} \theta^8\cdot d\theta = \frac{1}{9}\left[\theta^9\right]_0^{0.3} = 0.00000219. $$

A second term in the series can be added if more precision is required. You may also like Approximating the Sine Function.

♪ Christian Death • Spiritual Cramp

Recall that $\sin(x) \approx x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \dots$. ↩︎

Micro posting in October

Sat, 29 Oct 2022 16:01:45 +0200

2022-10-03: Newsboat can use any external browser for displaying posts. E.g., Elinks instead of Firefox to allow for a more comfortable browsing experience while staying in a terminal:

2022-10-03: New LSP features in Neovim 0.8. #vim
2022-10-05: Mathematical recipes (early draft).
2022-10-05: Querying Postgres Tables Directly From DuckDB. #database
2022-10-05: ♪ This Mortal Coil · Song to the Siren
2022-10-06: Design and Implementation of Programming Languages. #racket
2022-10-06: How to configure neovim to properly format python code?. Simpler solution is to add this mapping to your after/ftplugin/python.vim: nmap <buffer> g= :!black --quiet %<cr>:redraw!<cr>:!isort -q %<cr>:redraw!<cr>. Not sure if we really need two redraw, but it works quite well. This could be put in an autocommand if we want to format on save, say. #vim
2022-10-06: Move a running process into a tmux session. This happened to me in the past, and I came up with a different solution, which involves the same steps but without reptyr IIRC. Now I forgot how I did…
2022-10-06: What are the odds?
2022-10-06: ♪ Bauhaus · Bela Lugosi Is Dead
2022-10-07:

If you just want the executive summary, here it is: these are definitely interesting algorithms from an arithmetic complexity theory standpoint – especially for the case of 4×4 matrices over finite fields, where (to the best of my knowledge) Strassen’s algorithm from 1969 was still the reigning champion. These algorithms are also practically relevant, meaning that not only do they have better asymptotic lower bounds than Strassen’s algorithm, they are still algorithms you might actually use in practice, unlike essentially everything else that has been written on the topic in the last 50 years: these algorithms are correct, and will in principle win over Strassen’s algorithm with large enough matrices, but that cut-off is well beyond the sizes that anyone is actually doing computations with. — On AlphaTensor’s new matrix multiplication algorithms

2022-10-07: An open-source tribute to the macOS Drift screensaver. I always enjoyed the “flame” screensaver, but that may well do the job as well! #apple
2022-10-07: Partitioning in Postgres, 2022 edition. #database
2022-10-07: ♪ Front 242 · Headhunter
2022-10-09: ♪ Dead or Alive · You Spin Me Round (Like a Record)
2022-10-10: PDF processing and analysis with open-source tools.
2022-10-10: Running GUI Linux in a virtual machine on a Mac. #apple
2022-10-10: ♪ London After Midnight · Demon
2022-10-11:

(…) code should be as evident and transparent as possible in it’s intent and declarative programing (therefore functional programming) help us achieve this goal more effectively and efficiently than imperative programing. — Why I use typed functional programming

2022-10-11: The Plain Text Project.
2022-10-17: https://www.solopianoradio.com/. A good deal of piano for everyone!
2022-10-17:

Instagram users can only write plaintext without any formatting and it even doesn’t look nice onthe phone and you have to insert an image even in case you only want to post a message and youknow what? Nobody cares. I can only talk about russian speaking segment of the platform but theamount of activity there is just crazy. People not only share cat photos there, but also buy andsell stuff, write analytical articles, do workshops and trainings without any problem. The factthat sometimes they have to literally edit the photo and put a text overlay on top of it (and itoften looks like some of the worst myspace examples) doesn’t bother anyone. And all thecommunication is solved with a restricted instagram account or a closed whatsapp group. — From the basics to complexity and back

2022-10-17:

The academic prestige of Latin and Greek underwent a steady, centuries long decline. Where do C and assembly fit on this map? We certainly aren’t in a Medieval University, where these topics were mandatory, but neither are we in a post-modern sociology class, where the classics are not only dead, but whose distant memory haunts our conscience. — Learn enough C to survive

2022-10-17: Never got into J, but what a language! Random walk in 2 lines of J.
2022-10-17: C23 is Finished: Here is What is on the Menu. #clang
2022-10-17: How Python programmers can uncontroversially approach build, dependency, and packaging tooling (+ a note on Zig). #python
2022-10-17: Typing Haskell in Haskell (PDF, 38 pp.). #haskell
2022-10-17: ♪ Clan of Xymox · Equal Ways
2022-10-18: TIL about FiraGO. Togther with Fira Math, it makes for a perfect combination of fonts for handouts and slides. Now, it’s a matter of adding the following declaration in my $\LaTeX$ preamble:

\documentclass[presentation,9pt]{beamer}
\usepackage{fontspec}
\defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}
\setmainfont[Numbers={Tabular}]{Fira Sans}
\setsansfont{Fira Sans Condensed}
\setmonofont{Fira Code}
\usepackage[mathrm=sym]{unicode-math}
\setmathfont{Fira Math}
\usepackage{lstfiracode}

And it works perfectly well using Emacs Org export facilities!
2022-10-18: OpenBSD Minimalist Desktop. #unix
2022-10-18: make a static site with find(1), grep(1), and lowdown or Markdown.pl.
2022-10-24:

Despite the fact that I spend hundreds of dollars a year and hours of work to host my own email server, Google has about half of my personal email! Last year, Google delivered 57% of the emails in my inbox that I replied to. They have delivered more than a third of all the email I’ve replied to every year since 2006 and more than half since 2010. — Google Has Most of My Email Because It Has All of Yours

2022-10-24:

I dread the day in which I will hear a loud knock on my door and one of those programmers will barge in informing me that he is going to store a piece of his furniture in the middle of my living room, if I don’t mind. — Dotfile madness

2022-10-24:

I’m definitely not in favor of fossilizing Unix, but there’s a difference between avoiding fossilization and the kind of minimal, mathematical purity that we see GNU Grep trying to impose here. Unix has long since passed the point where it had that sort of minimalism in the standard commands. Modern Unix has all sorts of duplications and flourishes that aren’t strictly necessary, and for good reasons. One of them is that it’s nicer for the actual people using Unix. — We are stuck with egrep and fgrep (unless you like beating people)

2022-10-24: Compilation database. #clang
2022-10-24: Debugging C With Cosmopolitan Libc. #clang
2022-10-24: Eggplant is a self-hosted music streaming service.
2022-10-24: ♪ Adam Wakeman · Speak
2022-10-25:

I think it all stems from Apple’s desire to simplify things for themselves, architecture-wise — Apple Silicon is quite innovative in bringing the advantages of iOS devices to Macs (performance + power efficiency). The terrible decision, in my view, has been to also want to bring the iOS look and feel to the Mac. It was unnecessary, it has broken so many tried-and-true Mac interface guidelines, and it has delivered a massive blow to the whole operating system’s identity. Just to make the Mac what, more fashionable? — My next Mac might be the last

2022-10-25: Howdie! Eglot has landed on master
2022-10-25: Overthinking Leetcode’s Two Sum with SIMD. #rust
2022-10-25: Render looks like an interesting alternative to Heroku.
2022-10-26:

I find the developer experience of Common Lisp to be superior to almost anything imaginable, and this is not an empty statement: having used all sorts of IDEs and editors for over 25 years, I have seen many. — Between two Lisps

2022-10-26:

That’s right: what’s good for the lurkers is also good for everybody else. Just like all the great practices that enable remote work also made things better for anybody working in an office (if you remember what that was like and can believe that so many people just put up with that for so long), so does favoring transparency and following up on questions in the open improve all your team dynamics. — Learning By Lurking

2022-10-26: Python 3.11 is much faster than 3.8. #python
2022-10-26: The history of documented Unix facilities. #unix
2022-10-26: Time is an illusion, Unix time doubly so…. #unix
2022-10-26: ♪ Johnny Klimek & Tom Tykwer · Neo and Trinity Theme (Johnny Klimek & Tom Tykwer Exomorph Remix) [from “The Matrix Resurrections”]
2022-10-26: ♪ Kavinsky · Nightcall
2022-10-27:

On the one hand, the hardware is great and so is the performance. On the other, getting a new Mac today means it comes with Ventura or Monterey preinstalled, which is unfortunate, and of course there is no downgrade path. — My next Mac might be the last

2022-10-27: Little shell script to listen to some free internet radios:

#!/usr/bin/env bash
# sources
jazz="https://jazzradio.ice.infomaniak.ch/jazzradio-high.mp3"
misc="https://www.lemellotron.com/stream"
piano="https://pianosolo.streamguys1.com/live"
case "$1"
 injazz) src=$jazz;;
 misc) src=$misc;;
 piano)src=$piano;;
 *)echo "Unknown streaming source...";;
esac
mpv --no-video $src

2022-10-27: TIL that sending Fzf results to the quickfix list is as simple as pressing <cr> after selecting items using <tab>. #vim
2022-10-27: Shell Script Best Practices. #unix
2022-10-27: ♪ Clan of Xymox · A Day

The unquantified self #26

Wed, 26 Oct 2022 20:03:14 +0200

October is going to an end. I have only partially recovered from the fatigue and other little worries of the last months, which are still under investigation. But, anyway, I am still around. I’ve been playing with my crayons lately. It’s been 25 years that I hadn’t made any drawing, so I was surprised to find myself spending each evening in front of a kraft paper notebook. The sixty or so drawings produced are partly the product of external sources of inspiration (Lee Zimmerman, Enki Bilal, Kandinsky, Matisse) and of more personal reflections. This is just ink, and a bit of charcoal, after all. No eraser whatsoever. Just sketches.

I’ve been watching some new TV series (Alice in Borderland, La disparue de Lørenskog – I’m too lazy to look for the English title, and Deadwind). These were all great short movies for the past evenings. I still haven’t seriously started reading except for technical books. As a consequence, the number of books I will have read by the end of the year will probably be one of the lowest figure in years. But I’m confident I’ll find some gems in the pile of books waiting on my shelves.

Lastly, I’m in the process of configuring a home server based on OpenBSD that will run on my old Macbook pro. It will be fun, eventually. I will also be managing a nice server (32 HT cores, 512 Go RAM, 832 iGPU cores) at work. Time to brush up my sysadmin skills, which are now close to zero to be honest. In the meantime, I’m an happy Linux user and I have no regret for my past macOS applications. I read some reviews of the latest OS (Ventura) and I realized that the UX keeps getting worser after each new release (see also My next Mac might be the last).

♪ Jack Johnson • Sitting, Waiting, Wishing

ArXiving on October 2022

Tue, 25 Oct 2022 10:09:25 +0100

Approximating Persistent Homology for Large Datasets (https://arxiv.org/abs/2204.09155)

At first sight, it sounded really interesting, especially since the authors used bootstrap, and perhaps with bioinformatics applications in mind:

Persistent homology (…) produces a statistical summary in the form of a persistence diagram, which captures the shape and size of the data.

Then, I realized that it’s a hard paper (for me at least). I’ll keep it under my hat for later, when I’ll have clearer ideas.

A new preferential model with homophily for recommender systems (https://arxiv.org/abs/2204.11819)

I’ve been teaching the basis of recommender systems in an engineering school for two years. I liked it, especially when discussing matrix factorization algorithms and Co. Applications speak for themselves (everybody knows Amazon or Spotify, right?), and I missed Seth Brown’s beer recommendation engine. This paper is more about social networks and preferential attachment (rich-get-richer and homophily). The authors show that the number of vertices in their K-groups preferential attachment model follows a power law, and they dscuss two applications on real web data.

Locality Sensitive Hashing for Structured Data: A Survey (https://arxiv.org/abs/2204.11209)

Locality sensitive hashing (LSH) is used to hash alike items into the same buckets with high probability, which means it can be used as a data reduction or clustering method, bypassing any supervised learning process. However, this usually does not apply to ordered items, like trees, graphs or sequences. The authors survey relevant techniques for such hierarchical LSH algorithms.

Marginal log-linear models and mediation analysis (https://arxiv.org/abs/2204.08538)

Yet another paper on log-linear analysis, this time in the context of mediation analysis (a hot topic in psychiatric and psychological studies in the last ten years when people realized that causal analysis may or should not be that easy).

Simultaneous confidence intervals for the interpretation of primary and secondary effects in factorial designs without a pre-test on interaction (https://arxiv.org/abs/2204.08336)

Here, the author discusses the use of parametric or non-parametric simultaneous confidence intervals between the levels of a primary factor for both separate and joint secondary factor levels in an unbalanced factorial design with 2 or 3 factors (where one of the factor has more than 2 levels).

A Note on Simulation-Based Inference by Matching Random Features (https://arxiv.org/abs/2111.09220)

We can, and should, do statistical inference on simulation models by adjusting the parameters in the simulation so that the values of randomly chosen functions of the simulation output match the values of those same functions calculated on the data.

It reminds me of raking or margin adjustment in survey analysis.

♪ Queens of the Stone Age • Long Slow Goodbye

Zero-plugin linting and fixing in Neovim

Mon, 10 Oct 2022 10:13:45 +0200

Like Vim, Neovim comes with a bunch of preconfigured “compilers” (i.e., general settings for makeprg and errorformat), yet it is possible to define additional compilers when you’re not relying on a language server or, as in my case, when you want to use a specific linter via makeprg. Indeed, when it does not make sense to use a build system, it might be interesting to consider a linter as an alternative for Vim’s make feature. The results of invoking :make are displayed in the quickfix window, which I particularly like. Likewise, the formatprg option allows one to define a fixer for gq‘ing on a selection. We can also define a mapping to fix the whole buffer. This way, we can have linting and fixing facilities without relying on external plugins, like null-ls, ALE or efm-langserver.

Usually, I stand by the already defined compilers, but I added two specifications, one for pytest and the other for shellcheck. The following is for pytest, and it has been proposed by Phelipe:¹

if exists('current_compiler')
 finish
endif
let current_compiler = 'pytest'

if exists(':CompilerSet') != 2 " older Vim always used :setlocal
 command -nargs=* CompilerSet setlocal <args>
endif

CompilerSet makeprg=pytest\ --tb=short\ -vv\ $*
CompilerSet errorformat=
 \%EE\ \ \ \ \ File\ \"%f\"\\,\ line\ %l,
 \%CE\ \ \ %p^,
 \%ZE\ \ \ %[%^\ ]%\\@=%m,
 \%Afile\ %f\\,\ line\ %l,
 \%+ZE\ %mnot\ found,
 \%CE\ %.%#,
 \%-G_%\\+\ ERROR%.%#\ _%\\+,
 \%A_%\\+\ %o\ _%\\+,
 \%C%f:%l:\ in\ %o,
 \%ZE\ %\\{3}%m,
 \%EImportError%.%#\'%f\'\.,
 \%C%.%#,
 \%+G%[=]%\\+\ %*\\d\ passed%.%#,
 \%-G%[%^E]%.%#,
 \%-G

function! FixColumnNumber()
 if b:current_compiler !=? 'pytest'
 return
 endif

 let qflist = getqflist()
 for i in qflist
 let i.col = i.col + 1
 endfor
 call setqflist(qflist)
endfunction

augroup FixPytestQuickFix
 au!
 autocmd QuickFixCmdPost <buffer> call FixColumnNumber()
augroup END

For shellcheck, I have the following in my compiler directory:

CompilerSet makeprg=shellcheck\ -f\ gcc
CompilerSet errorformat=%f:%l:%c:\ %trror:\ %m\ [SC%n],
 \%f:%l:%c:\ %tarning:\ %m\ [SC%n],
 \%f:%l:%c:\ %tote:\ %m\ [SC%n],
 \%-G%.%#

In both cases above, the tricky part is to manage the errorformat. Once you have defined your custom compiler, you can declare it in your filetype file in after/ftplugin and optionally defined an appropriate makeprg command. For instance, in the case of shellcheck I use the following:

compiler shellcheck
setlocal makeprg=shellcheck\ -f\ gcc\ %
nmap <buffer> <silent> g= :!shfmt -i 2 -ln posix -sr -ci -s -w %<cr>:redraw!<cr>

The last line define a mapping to format the whole buffer using shfmt. I use g= for vim.lsp.buf.format() when a language server is available too. Of course I realize that the ALE or null-ls plugins handle all of that in a more elegant and efficient way, especially since it is asynchroneous contrary to the approach presented above and you are limited to one linter unless you write a shell script to gather several commands,² but that’s my take for exploiting Neovim’s builtins.

I summarized sone of the options I have in my vimrc folder in the following table.

Filetype	LSP	Linter	Fixer
c	clangd	built-in	built-in
python	pyright	built-in	black, isort
racket	racket-langserver	built-in	built-in
haskell	hls	built-in	built-in
purescript	purescriptls	built-in	built-in
sh	not used	shellcheck	shfmt
javascript	not used	quick-lint-js	prettier

♪ Greg Abate • Sunshower

It is available on GitHub. ↩︎
I don’t really mind formatting a buffer using a shortcut, although it is possible to use an autocommand to format it on save. ↩︎

One year scrobbling

Fri, 07 Oct 2022 09:28:48 +0200

It’s been a full year of scrobbling… I registered an account on Last.fm last year. Originally, this was mostly to get recommendations and to track the music I listen to on Cmus, as an alternative to Apple Music which I no longer really use (even if they offer an online platform now). I prefer to stand by my offline records.

Anyway, the statistics speak for themselves now: these are almost 36,000 scrobbles in one year, which means between 60 and 90 hours each week, and 700-800 tracks weekly on average. I don’t have a premium account so it’s hard to figure out a detailed report of my activities. Of course, I know what I listened too, this is mostly of the post-punk and jazz genres.

Last year summary is shown below (October–December):

My music library is composed of 13,632 tracks in MP3 format, as of today, and a bunch of additional music and video grabbed from Youtube or old MP3s not properly tagged yet. It’s by far the most reliable collection I had over the last 25 years, since I spent a fait amount of time cleaning dead records and tagging all valid records using beets and MusicBrainz. I don’t mind the cover, most of the time, but I know Rhythmbox is able to fetch them automagically, so I have most of them somewhere cached on my computer.

Of course, this does not mean that I only listen to digital music. I have some good old vinyles standing right there in my home office. Best therapy ever. See you next year!

♪ Cocteau Twins • Blood Bitch

Alacritty

Thu, 06 Oct 2022 18:48:43 +0200

Three weeks ago, I managed to solve a recurring problem on my Ubuntu workstation: using reliable and portable settings for Bash, Zsh and Tmux. I don’t use Bash as an interactive shell that often but I like having it configured anyway. Since most of my aliases and functions are written for Zsh, which is the shell I use all day, I needed to make them available for Bash, as well various environment variables and PATH settings. I ended up using a common $HOME/.config/profile.d directory, easy to stow or to version control, to store all my login settings, and keep using $HOME/.config/zsh for interactive shell stuff, with appropriate links from $HOME/.bashrc. Since I use starship to define my prompt, it applies equally well to Bash or Zsh. Problem solved.

Even if I find Gnome terminal really great, I wanted to get ride of window decorations to really benefit from the Tactile shell extension. To my knowledge, only Kitty and Alacritty allow to remove window decoration. I decided to give Alacritty a try since I already know Kitty quite well but I don’t need an additional level of multiplexing since I am already happy with Tmux.¹ Those who don’t use Screen or Tmux are probably less happy with Alacritty alone, except for its responsiveness. Because it is really fast, indeed. The very first days, I was kind of annoyed by the rendering process, which is far better than any other terminal I experienced in the past. Try to autocomplete path or command in Zsh and you will understand what I mean.

The other nifty feature of Alacritty is that it has a scrollback buffer (activated with Shift+Ctrl+Space) where you get Vim keybindings at your finger tip to navigate, select and yank any portion of text. That’s even better than under Tmux, to the point that I now often switch to this mode of selection inside Tmux itself. It also looks like there’s something called hint that allow to automagically highlight some regexed part of the screen, but I didn’t get into it at the time of this writing.

Like Kitty, Alacritty allows to define your own color scheme (the structure is quite identical, in fact – just a plain text file with foreground/background colors, and all 16 terminal colors in Hex format) as well as a custom set of fonts. I switched to Fira Code (after Iosevska, JetBrains Mono and Ubuntu Mono, respectively) because I like it and it comes with SemiBold font shape, which I find less intrusive than plain Bold. There’s not italic font shape, though, but it can be overcomed by using a variant of the regular font shape, as shown below:

font:
 normal:
 family: Fira Code
 style: Regular
 bold:
 family: Fira Code
 style: SemiBold
 italic:
 family: Fira Code
 style: Medium
 size: 11.0

John Cook discussed font that come with ligatures, like Fira Code, on his blog. He seems to prefer plain text (i.e., no ligature) for code, and I tend to agree with him except that now that I mostly code in Python, Scheme or Haskell I find them convenient, especially for Haskell or Purescript. We both agree that ligatures add some values to prose documents, though:

There’s one place where I believe Fira Code would be ideal: code examples inside a prose document. In that context you care about aesthetics and you want a monospaced font.

♪ The Psychedelic Furs • Love My Way

I stopped using Kitty on Ubuntu when I realized that it will probably never add support for terminal activity as what’s available in Tmux. I had other problems I don’t remember, but they may well be related to backward search in the scrollback buffer not available unless we had an extra Python plugin. ↩︎

Speeding up Neovim

Wed, 05 Oct 2022 11:21:43 +0200

This probably is the last and heaviest iteration of my Neovim setup cleansing days. Starting with a 36-40 ms startup time and 10-12 plugins, I ended up with a 26-30 ms startup time and a total of 9 plugins, with half of them not loaded most of the time. I could possible optimize this startup time by converting my color scheme to pure Lua, but I prefer to keep it as plain old VimL since I may reuse it with Vim instead of Neovim.

There are a lot of useful (and less useful) advices on the web, which mostly amounts to relying on pure Lua-based plugins or general settings (except for autocommand), disabling unused built-in stuff, eventually disabling Shada (I didn’t), etc. Unlike others, I don’t use much plugins, so I am not concerned with plugins that can be properly lazy loaded or not. I like to have a bunch of settings in my after/ftplugin directory, since I find it easier to manage language-specific options there. It’s all written in VimL but I may consider using ftplugin directly with Lua code at some point.

Here are the latest benchmarks using medium performance of my CPUs:

» hyperfine "nvim --headless +qa" --warmup 5
Benchmark 1: nvim --headless +qa
 Time (mean ± σ): 27.6 ms ± 0.2 ms [User: 22.7 ms, System: 5.1 ms]
 Range (min … max): 27.3 ms … 28.3 ms 105 runs

I uninstalled Telescope and got my way back to Fzf. Navigating my buffers or filesystem now works like a charm, unlike Telescope which I found less responsive and reliable (oftentimes the caret was not on the right spot). I also removed null-ls to get rid of plenary dependency. That’s a bit silly, because I really like this plugin and the author/maintainer does a really great job to keep up to date with Neovim upstream deprecations or changes. Yet I don’t want to rely on extra dependencies. I have to figure out how to replace null-ls linter/fixer facilities. At the moment, I delegated fixers to formatprg option, but I though I will maybe configure the efm language server to simplify the task and get linting facilities. Finally, I moved all other plugins as opt plugins, either using filetype-based filters (vimtex, nvim-lspconfig, nvim-parinfer) or key or command-base triggers (Comment, nvim-colorizer). As for treesitter, I simply added a event = 'BufRead' conditional, which allows to gain a little when firing up Neovim in a directory or without any file as I often do to define a proper working directory. Besides, I removed unused stuff in my autoload directory, as well as useless autocommands.

Here’s the list of plugins I’m currently using (in addition to packer), as of this writing: junegunn/fzf, junegunn/fzf.vim, lervag/vimtex, gpanders/nvim-parinfer, norcalli/nvim-colorizer.lua, numToStr/Comment.nvim, nvim-treesitter/nvim-treesitter, neovim/nvim-lspconfig. I could probably get ride of nvim-lspconfig since the built-in LSP can now be configured more easily starting with Neovim v0.8. See New LSP features in Neovim 0.8.

I also reorganized my Lua files using a cleaner structure:

» tree
.
├── config
│   ├── diagnostics.lua
│   ├── lsp.lua
│   └── treesitter.lua
├── plugins.lua
├── settings
│   ├── abbrev.lua
│   ├── autocmd.lua
│   ├── command.lua
│   ├── init.lua
│   └── mappings.lua
└── utils
 ├── completion.lua
 ├── jump.lua
 ├── statusline.lua
 └── surround.lua

3 directories, 13 files

It’s also been a week that I reconfigured my whole desktop using a light color theme and a mix of tiled and floating windows, thanks to the Tactile extension and Ubuntu built-in shortcuts to move or resize windows (Alt-F7 and Alt-F8). Since I don’t move terminal window a lot, this works for me and I don’t feel like I need to install Sway.

Here is a screenshot of the two workspaces I generally use all day:

♪ New Order • Bizarre Love Triangle

Micro posting in September

Sat, 01 Oct 2022 19:01:45 +0200

2022-09-01: Shane Quartet, Stablemates.
2022-09-01: I think I still prefer Cmus and Irssi over Rhythmox and Element. Lean text interfaces forever. #unix
2022-09-01: Regular cleaning of $HOME done: +4 Go. Also removed a lot of unused stuff on /, including Discord which I no longer use.
2022-09-04: BENEE · Doesn’t Matter.
2022-09-04: Trying out Tactile as an additional Window Manager for Gnome shell. Looks great. No auto-tiling (you have to manually assign new windows to predefined tiles on a grid, but this is flexible. However, it works like a charm right (and out of the box), unlike Forge or gTile. Gnome default tiling behavior works alongside, which is great. Together with Simple terminal or Gnome terminal (preferred), this makes for a great tiling WM. Also, none of the shell extensions (Unite, undecorate, …) I tried were able to properly remove window decorations, and it seems like there always are edge cases anyway.

2022-09-06:

The WHATWG spec follows the good old browser mantra of being very liberal in what it accepts and trying to guess what the users mean and bending backwards trying to fulfill. (Even though we all know by now that Postel’s Law is the wrong way to go about this.) It means it’ll handle too many slashes, embedded white space as well as non-ASCII characters. — My URL isn’t your URL

2022-09-06: Waiting for a reply to my query on Vim Vixen GH repo. Would be great to have the same facilities as on Nyxt browser to yank both URL and title at some point.
2022-09-06: Gemini: A Modern LaTeX Poster Theme. #tex
2022-09-06: How fast are Linux pipes anyway?. #unix
2022-09-06: Persistent SSH sessions with proper scrollback. I have to try this with both ssh jump and Tmux instead of Screen, and with scrolbback buffer enabled because i like it.
2022-09-06: ♪ Placebo · Teenage Angst
2022-09-07:

I have had my eye on Julia for many years, and I now consider it ready to be my “everyday” programming language. — From Common Lisp to Julia

2022-09-07: > The story is different with a REPL: you will have played with each piece of code in isolation before running the whole program, which makes you quite confident that each of the sub-tasks is well implemented. — What makes a good REPL?
2022-09-07: Difftastic, the Fantastic Diff. #git
2022-09-07: ♪ Apoptygma Berzerk · Until The End Of The World
2022-09-07: ♪ Lords of the New Church · Russian Roulette
2022-09-08:

The historical justification for a /bin, /sbin and /lib separate from /usr no longer applies today. They were split off to have selected tools on a faster hard disk (which was small, because it was more expensive) and to contain all the tools necessary to mount the slower /usr partition. Today, a separate /usr partition already must be mounted by the initramfs during early boot, thus making the justification for a split-off moot. In addition a lot of tools in /bin and /sbin in the status quo already lost the ability to run without a pre-mounted /usr. There is no valid reason anymore to have the operating system spread over multiple hierarchies, it lost its purpose. — The Case for the /usr Merge

2022-09-08: Tip of the day: If you want to get ride of Ubuntu putting screenshots with arbitrary filenames in arbitrary folder, without resorting on the good old gnome-screenshot tool, just use:

% xdg-user-dirs-update --set PICTURES ~/Media/Pictures
% cd ~/Media/Pictures/Captu* ## NOTE: update according to your settings
% rename 's/Capture\ d’écran\ d[eu]\ //g;s/\s/-/g' *

Don’t forget to update the last instruction to reflect basename on your system, use a Makefile or shell script to automate the process. Usually, Ubuntu will create a subfolder (“Screenshots” or “Captures d’écran” or whatever) in the default PICTURES folder. What’s funny is that instead of a straight single quote, Ubuntu decided for whatever reason to use a curly quote! I used to use the following Bash script on X11, but it no longer works under Wayland:

scrot '%Y-%m-%d-%H-%M-%S.png' --thumb 300x200 -e 'mv $m $f ~/Media/Pictures/screenshots'

Here, screenhots is a soft link to the default subfolder where screenshots are saved.
2022-09-08: I Tracked Everything I Read on the Internet for a Year. It reminds me of my defunct Emacs refiled links.
2022-09-08: ♪ Antony & The Johnsons · You Are My Sister
2022-09-09: Analyzing Every Clojure Project on Github. #clojure
2022-09-09: At least one Vim trick you might not know. #vim
2022-09-09: Ramanujan’s factorial approximation. #math
2022-09-12:

I have an IDE, but I keep going back to essentially the same setup you have. I find that GUI IDE’s tend to try to be too many things at the same time. I like having nothing on the screen that isn’t directly related to what I’m doing, and I strongly dislike ribbons of things I could click on but rarely, if ever, do. — I don’t use an IDE, should I be using one?

2022-09-12: SQLite Doesn’t Use Git. #git
2022-09-12: Why you shouldn’t invoke setup.py directly. #python
2022-09-12: mvp.css: A minimalist stylesheet for HTML elements.
2022-09-12: ♪ Helen Merrill · You’ve Got a Date With the Blu
2022-09-13:

Common Lisp is a fine language with exceptional interactivity, multiple implementations providing the potential for acceptable performance, useful features like condition system, and a stable standard base. This, in theory, makes is very much suited toward data processing and analysis. Although practice paints a less favorable picture, there are still use cases where Common Lisp could be considered to be a very good pick. Also, I simply like writing Lisp code. — sirherrbatka/vellum

2022-09-13: What is a LSM Tree?
2022-09-13: mtime comparison considered harmful. #unix
2022-09-13: ♪ Killing Joke · Sun Goes Down
2022-09-16: ♪ New Order · Blue Monday (Instrumental Outtake)
2022-09-19:

Purely functional programs are easier to test, they make better and more reliable APIs, and our environments, compilers, and run-time systems take advantage of functional style. — Getting rid of set-car! and set-cdr!

2022-09-19: Lovely.

One way to solve the delays is to disable your internet connection. This is tough medicine, but if you notice these delays, try it for an hour just to verify that indeed the issue is resolved by disabling internet connectivity. — macOS 10.15: Slow by Design

2022-09-19: Algorithms for Competitive Programming. See also my review of The Competitive Programmer’s Handbook.
2022-09-19: An X11 Apologist Tries Wayland. #unix
2022-09-19: Automatic Differentiation in 38 lines of Haskell. #haskell
2022-09-19: ♪ The Only Ones · Peter and the Pets (2008 Remastered)
2022-09-25:

I don’t tap my own phone. I don’t xerox postcards before I mail them back from vacation. I don’t take a voice recorder when I go out with friends. And I don’t have a copy machine at home to duplicate hand written notes I may send. — The cost of saving sent e-mailI do like having local copies, both of my handxritten notes and of my sent e-mails. Digital nomadism, with one foot in the analog world.

2022-09-25: > I keep trying to quit vim. I keep ending up inside a terminal, inside vim, writing code. Like SQLite, vim is an island of stability over my career. While I wish IDEs were better, I am extremely thankful for tools that work and respect the effort I have taken to learn them, decade after decade. — Software I’m thankful for
f2022-09-25:

The internet is no longer a place where a relatively-small fraction of the human population go to find something different than everyday life. It is everyday life for billions of people. As long as life is megacorps and information gatekeepers, so too is the global ubiquitous internet. Mission accomplished. — Searching the Creative Internet

2022-09-25: Incrementing Vectors.
2022-09-25: Using GPT-3 to pathfind in random graphs.
2022-09-28:
Cmus revamped!
2022-09-29:

The relationship between developers and App Store reviewers is tense at best. Most people are trying to build well-designed, useful mobile apps. Apple has instituted App Store reviews to maintain a high-quality bar for apps and weed out spammy or nefarious actors, using human evaluators to test individual apps and provide direct feedback. However, malicious apps are relatively rare; arguably, Apple doesn’t do a great job filtering them out anyway. — Outdated vs. Complete

2022-09-29: TIL about Rclone, thanks to the following blog post: An Upside-Down Backup Strategy.
2022-09-29: Data types à la carte. #haskell
2022-09-29: Explained from scratch: private information retrieval using homomorphic encryption.
2022-09-29: Types at the edges in Python. #python
2022-09-29: ♪ Nadje Noordhuis · Migration
2022-09-30:

My personal productivity cycle is tied to this industry hype cycle. My schoolfriend back in 2003 was my personal “technology trigger.” From him, I rapidly learned some standard tools from Gartner’s plateau of productivity: PHP, MySQL, Apache. Then I surveyed the plateau more widely, finding Python, Postgres, and Nginx. A little further away, I had heard of Gartner’s trough of disillusionment. I sneered at the Perl, Java, CORBA. I’ll keep my Python, thanks. — The hacker hype cycle

2022-09-30: The full text of my Introduction to the Theory of Programming Languages book (second printing, 1991) is now available.
2022-09-30: Alternatives to bloatware.
2022-09-30: Homotopy Type Theory. #maths
2022-09-30: ♪ The Dresden Dolls · Coin-Operated Boy

Unified colors of TUIs

Wed, 28 Sep 2022 20:02:45 +0200

The other day I got tired of my terminal color scheme (Nord) which makes no distinction between dark and light colors, except for white and black variants, and I was looking for a prettier color scheme that could be used in a light or dark theme as well. Indeed, I used to use a light theme for all my command-line tools, including Vim (see, e.g., the screenshot at the end of this post) until I switched to the Nord palette. However, when I’m tired, or at night, I prefer a dark theme. And, well, sometimes I happened to miss my good old light theme. So I devised yet another theme for my terminal, and then for Neovim.

Here is what I finally decided to stand by for the next few weeks:

This palette was grabbed from the homepage of https://terminal.sexy, and the white/dark variants are based on the Nord palette and personal taste. I don’t know its name, though.

The other reason I designed a new color theme for the terminal is that I wanted it to be useful for Neovim too. As I said, I have two main colorscheme, which are automatically set up based on the value of background. This is done with an autocommand that looks like this:

function! misc#set_theme() abort
 if &background ==# 'dark'
 colorscheme nord
 else
 colorscheme light
 endif
endfunction

--8<--------

augroup colortheme
 autocmd!
 autocmd OptionSet background call misc#set_theme()
augroup END

The advantage is that this is all managed by Vim itself, as discussed in a related post. I have an extra apocryptic command in my init.lua file to handle possible Tmux glitches:

-- HACK: work around bad detection of background in Tmux (no OSC11 support)
-- https://github.com/neovim/neovim/issues/17070
vim.api.nvim_exec(
 [[
if $TERM ==# "tmux-256color"
 lua vim.loop.fs_write(2, "\27Ptmux;\27\27]11;?\7\27\\", -1, nil)
endif
]],
 true
)

Later, I wrote a shell script that allows me to switch my color theme on the go. Since I now use Alacritty which has a hot reload capabilitites this works quite well. Alacritty is really great, but I am better talking about Alacritty in a another post. That’s how I manage to use two colorschemes in my terminal and Neovim.

Those two colorschemes, which are based on modified versions of Nord and Vimbones from the zenbones plugin, are great for daily use, but they are not consistent one each other. And I’m too lazy to find a good fit for a common color palettes. Less is better, most of the time, so I rewrote from a scratch a single colortheme, as I did three years ago for Vim. It tooks me less than an hour to get a working theme for both color themes (light and dark): no color, except for constant and stuff I want to be aware of (warning or error message, bad spelling, etc.). Again, I take advantage of Alacritty’s handling multiple fonts to highlight keywords or comments with medium or semi-bolded typefaces. It’s all good!

♪ Morcheeba • Trigger Hippie

Currying and repeated calls of a function

Mon, 19 Sep 2022 14:06:44 +0200

In my previous post I discussed the repeated application of a function to build a list of results. This was done using a named let form first, then through currying. See what’s happening?

(import (chicken random))

(define (curry n f)
 (if (= n 0)
 (lambda (x) x)
 (lambda (x) (f ((curry (- n 1) f) x)))))

(define (square x) (* x x))
((curry 5 (lambda (x) (cons (square 2) x))) '())
;; => (4 4 4 4 4)

(define (random max) (pseudo-random-integer max))
((curry 5 (lambda (x) (cons (random 10) x))) '())
;; => (9 6 5 3 7)

Since intermediate results are not exploited in successive calls to f in the above example, this amounts to make n independent calls to f, unlike the following use of curry, where intermediate results are reused:

((curry 2 (lambda (x) (square x))) 2)
;; => 16
((curry 3 (lambda (x) (square x))) 2)
;; => 256

In the first case, we compute $2\times 2 = 4$, then $4\times 4 = 16$, while in the second case we add an extra step with $16\times 16 = 256$.

The most natural application of currying is probably to compute an iterated or cumulative sum in statistical computing. As an alternative one may consider a “reducer”, like reduce or foldl. Currying is also important in functional programming when it comes to manage function multiple arguments, since it allows to express $f(x, y)$ as $g(x)(y)$, for instance. In Python, the partial function from the functools package does just that. In Scheme, we can introduce as many lambdas as we need in a function, or use specialized macros.¹ As an example, consider a function that expect two arguments, (define (greater? x n) (> x n)) – I use such constructs in combination with filter to select observations in a sequence of numerical values; it can be converted to a one-argument function using an extra lambda, (define (greater? n) (lambda (x) (> x n))). Then instead of (greater? 3 1), we would simply write ((greater? 1) 3). The function now returns a new function that knows the value of n.

The SICP has another application of repeated call to a function in exercise 1.43:

If f is a numerical function and n is a positive integer, then we can form the nth repeated application of f, which is defined to be the function whose value at x is f(f(…(f(x))…)). For example, if f is the function x + 1, then the nth repeated application of f is the function x + n. If f is the operation of squaring a number, then the nth repeated application of f is the function that raises its argument to the 2^nth power. Write a procedure that takes as inputs a procedure that computes f and a positive integer n and returns the procedure that computes the nth repeated application of f.

Unlike the previous case, this times it’s all about composing function using an auxiliary function, compose:

(define (compose f g)
 (lambda (x)
 (f (g x))))

Now, a recursive call to a function that calls itself $n$ times, e.g. $f(x) = x^2$ which for $x=5$ and $n=4$ is $(5 \times (5 \times (5 \times 5)))$, can be written as:

(define (repeated fn n)
 (if (= n 1)
 fn
 (compose fn (repeated fn (- n 1)))))

In the above, fn isn’t called n times, instead this function returns the function fn itself $n-1$ times and the lambda from compose takes care of calling those new functions, plus an extra call via the guard which makes for a total of $n$ calls. Getting ride of the compose helper function, we simply have the following lambda in action:

(define (repeated fn n)
 (if (= n 1)
 fn
 (lambda (x) (fn ((repeated fn (- n 1)) x)))))

See the Scheme wiki for further discussion on this exercise.

As a sidenote, the following function is defined (and explained) in Simply Scheme:

(define repeated
 (let ((= =) (- -))
 (lambda (fn number)
 (if (= number 0)
 (lambda (x) x)
 (lambda (x)
 ((repeated fn (- number 1)) (fn x)))))))

♪ The Fall • Totally Wired

There’s even an SRFI under review. ↩︎

Bootstrap resampling using Scheme

Sun, 18 Sep 2022 10:34:29 +0200

In a previous post, we discussed bootstrap resampling in Lisp, using LispStat. Let’s revisit the idea of the bootstrap using Chicken Scheme.

Recall the nonparametric bootstrap algorithm:

Let $X = (x_1, x_2, \dots, x_n)$ be a sample of i.i.d observations.
Generate $B$ random samples of size $n$ by drawing with replacement from $X$. Let call those new samples ${X_1^{\star}, X_2^{\star},\dots, X_B^{\star}}$.
Consider a statistic $S(X)$, which may be any estimator, test statistic or average prediction for instance. Any aspect of the distribution of $S(X)$ may be assessed from the distribution of the bootstrap samples directly, ${S(X_1^{\star}), S(X_2^{\star}),\dots, S(X_B^{\star})}$.

The variance of the bootstrap estimate, $\mathbb{V}[S(X)]$ is then:

$$ \mathbb{\widehat V}[S(X)] = \frac{1}{B-1}\sum_{b=1}^B\left(S(X_b^{\star}) - \bar S^{\star}\right)^2. $$

where $\bar S^{\star}$ is the bootstrap sample mean, $\frac{1}{B}\sum_{b=1}^BS(X_b^{\star})$.

Basically, we need a way to (1) shuffle (with replacement) a list of indices (the observation index in a list or vector) or to shuffle the data values directly, then (2) call the same function on $B$ random lists, and finally (3) compute a sum (or any other test statistic). Below is some Chicken Scheme code.

To pick a random element from a list, the following lambda will do the job (replace pseudo-random-integer with random if you are using Racket):

(import (chicken random))

(lambda (x) (list-ref x (pseudo-random-integer (length x))))

Let’s test it:

(define xs '(1 2 3 4 5 6 7 8 9 10))

(list-ref xs (pseudo-random-integer (length xs)))
;; => 9

Of course, your result will likely be a different one, since I don’t use a fixed seed for the PRNG. To repeat the same call to the above procedure, pick say, (length xs) times we can use a simple iterator:

(define (pick x) (list-ref x (pseudo-random-integer (length x))))

(define len (length xs))

(let loop ((n 1)
 (rs '()))
 (cond ((> n len) rs)
 (else (loop (add1 n) (cons (pick xs) rs)))))
;; => (6 6 9 4 8 3 4 2 4 10)

Since we sample with replacement, it is normal to expect some duplicates since the average bootstrap sample omits 36.8% of the original sample.

Everything is now in place. We just need to add one more outer loop to build $n$-samples $k$ times.

Let’s first get some data:

(import (chicken io))

(define fs "birthwt2.csv")

(define data (call-with-input-file fs (lambda (p) (read-lines p))))

The above is quite rudimentary and return a list of strings, with the header as the first element. We don’t really need the header, and we want to keep birth weight (bwt), which is the last numerical value. To parse the data, we will split each string on a comma, using Chicken string-split,¹ and extract the bwt value. Here is how it works:

(import (chicken string))

(car (reverse (string-split (list-ref (cdr data) 0) ",")))
;; => 2523

Now, we just have to iterate over the whole list. We’ll be using map instead of for-each since we want to build a list of values along the way:

(define (value xs)
 (string->number (car (reverse (string-split xs ",")))))

(define bwt (map value (cdr data)))

And, now the core algorithm for the bootstrap (I will only compute the sum of the values, since it is a sufficient statistic in this case). Note that I converted the preceding named loop to a currying function:

(define (curry n f)
 (if (= n 0)
 (lambda (x) x)
 (lambda (x) (f ((curry (- n 1) f) x)))))

(define (sample xs)
 (if (null? xs)
 '()
 ((curry (length bwt) (lambda (xs) (cons (pick bwt) xs))) '())))

(define (boot xs b)
 (let loop ((n 1) (rs '()))
 (cond ((> n b) rs)
 (else (loop (add1 n) (cons (apply + (sample xs)) rs))))))

(boot bwt 20)
;; => (562214 564519 548880 544008 569931 587845 564206 558517 546902 555674 544962 554010 560677 559470 548108 533815 556976 548125 571216 575198)

Done.

From there on, it is not really diffcult to write a full standalone procedure for bootstraping a test statistic, especially since Chicken Scheme provides some basic summary statistic in the statistics module. In a future post, I will show how to compute a 95% confidence interval for a correlation coefficient, and compare this estimate with the asymptotic CI computed using Fisher’s z transformation.

♪ The Durutti Column • Experiment in Fifth

See also string-tokenize (SRFI-13). ↩︎

Vim Vixen

Sat, 17 Sep 2022 09:57:23 +0200

One or two years ago I tried Tridactyl to ease navigation in Firefox using familiar Vim shortcuts. It is not that I do not like Firefox’s shortcuts, but I find them limited in a certain sense. How do you navigate in a page without using a mouse? At some point, you often need to use the mouse to follow an hyperlink. Most keyboard-driven navigators provide a way to highlight all links in a web page using consecutive alphabetical letters or digrams (often times, this is triggered with the f or F shortcut, like in Vim), and to use the corresponding label to follow the link. When there are a limited number of hyperlinks on a page this happens to be quite fine. The two other aspects of managing a running web browser instance that are important to me are: how to efficiently manage my opened tabs and the current web page I ma looking at (e.g., going straight to the bottom or the top of the page), and how to search in a web page. Nyxt browser solves both problems very nicely using familiar Vim shortcuts. How about Firefox?

Here comes Vim Vixen. This is yet another keyboard-driven extension to navigate in Firefox. Contrary to Tridactyl it does not require to alter the default starting page, and it comes with the same (sane) default keybindings, which will be familiar to Vim users of course. There are some additional shortcuts that are quite handy too. For instance, you can yank the current URL to the clipboard using y, or delete a tab using d (u will reopen the last closed tab). Navigating your opened tabs is quite easy, using J and K, like is the case for opening a new page using o (open in current tab), t (open in new tab) and w (open in new window). You can also duplicate the current page (zd), and manage zoom level using the z prefix as well: zo to zoom out, zi to zoom in, and z= to reset default zoom.

Overall, I like it a lot. I should note that sometimes it loose focus and we are back with usual Firefox shortcuts. For instance, Vim Vixen search shortcut (/) becomes unavailable and we have to use Firefox’s own quick search box. There are sometimes some other glitches when dark theme are enabled on some websites: The search bar which normally sits at the bottom of the window appears bigger than it should be. This is easily solved by triggering the search shortcut once.

Of note, I installed a bunch of other extensions for Firefox, which I find quite convenient to navigate the web these days (think GDPR, cookies, trackers, etc.): uBlock Origin, Decentraleyes, and I don’t care about cookies, which changed my life really. All of them were recommended by Luke Smith, including Vim Vixen.

♪ Leonard Cohen • Traveling Light

ArXiving on August 2022

Fri, 16 Sep 2022 10:09:25 +0100

Here are a few papers that I read over the past weeks, in the CS and Stat category as usual.

Inverse Probability Weighting: the Missing Link between Survey Sampling and Evidence Estimation (https://arxiv.org/abs/2204.14121)

Inverse probability weighting is used in survey and epidemiological analysis. I only used this technique once, and it was in quick comparison with propensity score analysis, where the key idea was in both case to achieve balance between treated and control subjects in order to reduce confounding. See Chan, K. C. G., et al. Globally efficient non-parametric inference of average treatment effects by empirical balancing calibration weighting (Journal of the Royal Statistical Society: Series B 78(3), 2016) for recent work. In this article, the authors discuss the use of binning and smoothing, or, as an alternative, hierarchical modeling. Their conclusions are that those Bayesian estimators perform better (i.e., they have lower mean squared errors) compared to simple IPW estimators.

Redefining Populations of Inference for Generalizations from Small Studies (https://arxiv.org/abs/2204.14156)

This article deals with the inference population that small sample studies usually focus on, and how it can be improved to foster valid generalizations (of the population average treatment effect, or PATE). This is based on the results from a cluster randomized trials in educational assessment. The two approaches considered here are coined “quantitatively optimal” and “policy-relevant” approaches. In the former case, the analysis relies on propensity scoring with varying cutoffs or the distributions of covariates. In the latter case, the idea is to identify subgroups that may benefit from the intervention. To improve imprecision, bot approaches can be undertaken, while for bias reduction only the quantitatively optimal approaches seem relevant.

Practical considerations for specifying a super learner (https://arxiv.org/abs/2204.06139)

Along with classical parametric regression models or ensemble methods in machine learning, the super learner algorithm allows to learn a prediction function without needing to set on a fixed modeling approach: as an entire pre-specified and adaptive strategy, it allow the modeler to consider several predictive models at once. Here’s the big picture:

For the practictioner, the core decisions are to (1) specify the performance metric for the discrete super learner, (2) derive the effective sample size, (3) define the V-fold cross-validation scheme, (4) form library of candidate learners, and (5) to use discrete super learner to select the candidate with the best cross-validated predictive performance.

PaC-trees: Supporting Parallel and Compressed Purely-Functional Collections (https://arxiv.org/abs/2204.06077)

This paper presents PaC-trees, a purely-functional data structure supporting functional interfaces for sets, maps, and sequences that provides a significant reduction in space over existing approaches. A PaC-tree is a balanced binary search tree which blocks the leaves and compresses the blocks using arrays. We provide novel techniques for compressing and uncompressing the blocks which yield practical parallel functional algorithms for a broad set of operations on PaC-trees such as union, intersection, filter, reduction, and range queries which are both theoretically and practically efficient.

Correction of overfitting bias in regression models (https://arxiv.org/abs/2204.05827)

This article describes solutions to correct for the biais arising from overfitting datastes where $p/n < 1$. Besides regularization techniques, the asymptotic characteristic function of the ML estimators can be used to estimate the overfitting bias using only a small set of non-linear equations. The maths are dense, and I will probably need to reread the hard part.

♪ New Order • Video 5 8 6

Debugging in Vim

Thu, 15 Sep 2022 07:18:22 +0200

I learned recently about Termdebug, which is built-in debugger for Vim and Neovim. Since I stopped using nvim-dap and MS debug adapter protocol, I’ve been using good old GDB for C, and print statements for Python. That’s fine. In the latter case, I also drop a breakpoint() instruction sometimes, and use pdb. Apparently, we can’t use pdb with Termdebug, but gdb and rust-gdb work just fine. For Haskell, this is slightly a different problem arising from unavailable debug symbols. However, ghci works also just fine since it comes with a bunch of debugging commands.

Here are the corresponding documentation: gdb, ghci, pdb.

Near the end of the day, I’ve succesfully configured Termdebug for C and Rust, as well as pdb and ghci for Python and Haskell. I defined some wommon mappings to launch the debugger in a split window, together with other utilities to make or launch an REPL. Also, inspired by Cheat40, I wrote a quick cheatsheet for the various commands available in each debugger. They are mostly comparable, especially between gdb and pdb, but Haskell’s ghci comes with additional settings like :set -fbreak-on-exception to log steps until an exception occurs.

A sample configuration for python is shown below:

augroup whitespace
 au BufWinEnter *.py let w:m2=matchadd('WarningMsg', '\%>80v.\+', -1)
 au BufWinLeave *.py call clearmatches()
augroup END

compiler pyunit
setlocal makeprg=python3\ %
setlocal errorformat=

setlocal foldmethod=indent
setlocal foldlevel=99

nmap <buffer> gs :10 split term://python3 %<cr>:startinsert<cr>
nmap <buffer> gS :10 split term://ipython --simple-prompt<cr>:startinsert<cr>
nmap <buffer> gK :!nvim-keywordprg python <cword><cr>
nmap <buffer> <localleader>dd :10 split term://python3 -m pdb '%:p'<cr>
nmap <buffer> <silent><localleader>? :<C-u>Cheatdebug<cr>

ia <buffer> pp print(f"DEBUG:")<Left><Left>
ia <buffer> bb breakpoint()<Esc>mb

highlight link myHighlight WarningMsg
match myHighlight /breakpoint()/

Note that I use Neovim built-in LSP to get hover information on symbol at point, but I also have a custom mapping to launch pydoc3, or even Dasht.

Unfortunately, setting a breakpoint in pdb or ghci will not highlight the corresponding line in the buffer, and of course Termdebug output is more rich than those simple shells. However, note that this setup does not require any additional dependecies beside gdb and rust-gdb. No complex setup to attach to a running process or to download various adapters (this always worked fine, though; it’s just that it is far more advanced than what I discuss here, for results that are close in both approaches).

Here is a preview of a fake debugging session on a Python script:

♪ Queens of the Stone Age • Long Slow Goodbye

Quick Bash script templates

Mon, 12 Sep 2022 10:44:58 +0200

Over the years, I wrote a lot of little helper Bash scripts to automate things on macOS or Ubuntu. Most of the times it was simply a matter of putting a series of variables and commands, some echo and the like, after a shebang line. In the end, this was messy but it worked as intended.

Here are two templates that may simplify the process of writing small shell scripts. They both share the same principles: the use of “subcommand” and the display of a short help message in case argument $1 is empty (since there seems to be so much a lack of convention that I never remember if we need to pass help, --help or -help after program name to get some help).

The first template is minimalist. There’s no error checking, and it is assumed that instructions can be grouped logically into independent main options like this:

#!/usr/bin/env bash

case "$1" in
fizz)
 echo -e "\033[0;32m[+] Process first option\033[0m"
 ;;
buzz)
 echo -e "\033[0;32m[+] Do some banana\033[0m"
 ;;
help)
 echo -e "Usage: $0 {fizz|buzz}"
 ;;
*)
 echo -e "Usage: $0 {fizz|buzz}"
 ;;
esac

As can be seen, this is pretty basic stuff, and everything is handled via the first argument to the shell script (recall that $0 is the program name, $1 its first argument, etc.). Of course, feel free to add all sort of guards (set -e, set -u, set -o pipefail) at the top of your script. My default Vim template for shell scripts include most of them, and I delete unnecessary ones depending on my needs.

[2022-10-27]
See also Shell Script Best Practices to learn a lot more tips and tricks. It also features a nice Bash template.

The second template is a little more involved as it exemplifies the use of proper subcommands, as in Hugo or Git for instance. The following script is a real script that I use to update a changelog with Git revisions in specific Git repositories on my hard drive (my website and all Git projects that are in the same directory, $HOME/cwd). Here’s the gist of it:

#!/usr/bin/env bash

whos=$(basename "$0")
tmpdir="$HOME"/tmp

_help() {
 echo "Usage: $whos <subcommand> [options]"
 echo "Subcommands:"
 echo " update Update log entries"
 echo " show Show log entries"
 echo ""
 echo "For help with each subcommand run:"
 echo "$whos <subcommand> -h|--help"
 echo ""
}

_update() {
 cd "$HOME"/Sites/aliquote || return
 git log --author=chl --pretty=format:"* %as %an <%ae> - [${PWD/*\//}/%h] %s" >"$tmpdir"/home-chl-sites-aliquote.log

 for d in $(find "$HOME"/cwd -maxdepth 1 -type d ! -name "cwd"); do
 WD=${d##*/}
 if [[ -d "$HOME/cwd/$WD/.git" ]]; then
 cd "$HOME"/cwd/"$WD" || return
 git log --author=chl --pretty=format:"* %as %an <%ae> - [${WD}/%h] %s" >"$tmpdir"/home-chl-cwd-"$WD".log
 fi
 done

 cat "$tmpdir"/home-chl-*.log >"$tmpdir"/logger
 sort -k2 -r "$tmpdir"/logger >"$HOME"/Documents/logs/changelog
 rm "$tmpdir"/home-chl-*.log "$tmpdir"/logger
}

_show() {
 less "$HOME"/Documents/logs/changelog | rg "[0-9]+-[0-9]+-[0-9]+"
}

subcommand=$1
case $subcommand in
"" | "-h" | "--help")
 _help
 ;;
*)
 shift
 _${subcommand} "$@"
 if [ $? = 127 ]; then
 echo "Error: '$subcommand' is not a known subcommand." >&2
 echo " Run '$whos --help' for a list of known subcommands." >&2
 exit 1
 fi
 ;;
esac

In this case, each subcommand is implemented as its own function, and they are processed at the command line using a simple variable interpolation. If you need more flexibility to handle your subcommand, I think this is better suited than the preceding sample script. I came across a similar approach in a Gist a while ago but I forgot to bookmark it and I can’t credit the author unfortunately.

♪ Gilad Hekselman • Flower

Haskell and Vim

Sun, 11 Sep 2022 11:21:42 +0200

Between two procrastination sessions on my couch due to my recent unexplained health problems that force me to rest on bed most of the time, I spent a few hours (uh, over several days and in several times because of the general lack of energy) cleaning up the files that were lying around on my computer, writing some maintenance scripts, devising cleaner dark and light colorschemes for my terminal and Neovim, and reviewing my Neovim configuration. It now starts like a breeze, as you can see by yourself:

~ » hyperfine "nvim --headless +qa" --warmup 5
Benchmark 1: nvim --headless +qa
 Time (mean ± σ): 36.1 ms ± 0.3 ms [User: 29.1 ms, System: 7.4 ms]
 Range (min … max): 35.5 ms … 37.0 ms 78 runs


~ » hyperfine "nvim --headless +qa" --warmup 5
Benchmark 1: nvim --headless +qa
 Time (mean ± σ): 28.1 ms ± 0.6 ms [User: 22.6 ms, System: 5.6 ms]
 Range (min … max): 27.0 ms … 29.7 ms 103 runs

The above benchmarks were run with default (700-2,300 MHz) and high (up to 4,900 MHz) performance settings on my laptop. Most of the optimizations were done at the level of package management: I removed a bunch of unused stuff – the list of plugins I used went down from 15 to 10 plugins in the last two months, and moved most of the plugins to the opt directory. Lazy loading almost halved the above startup time indeed.

I even wrote a little doc file for my custom commands. Other than that, I spent a little time working on a general helper Bash script for the keywordprg utility, in order to get some help whether I’m using a Language server or not. As an example I have this little script that can be used from the command line or as a K handler in (Neo)vim:

#!/usr/bin/env bash

BROWSER=elinks

case "$1" in
stata)
 "$BROWSER" https://www.stata.com/help.cgi\?"${2}"
 ;;
python)
 pydoc3 "${2}"
 ;;
r)
 Rscript -e 'args <- commandArgs(TRUE); help(args[2], package=c("base", "stats"), help_type="text")' --args "${2}"
 ;;
lisp)
 rc=$(hyperspec_lookup.ros "$2")
 "$BROWSER" "$rc"
 ;;
help)
 echo "Usage: $0 {stata|python|r|lisp|help}"
 ;;
*)
 echo "Usage: $0 {stata|python|r|lisp|help}"
 ;;
esac

exit 0

Not very polished, but it works as intended. The Lisp case was discussed in a previous post.

Back to the matter of this post. I originally intended to write about (Neo)vim and Haskell. After some reviews, I found that most of what I expected to say is already available online. I’m a hobbyist programmer, and I don’t spend a lot of time with Haskell or Clojure. When I started to play with those languages some years ago this was always with Emacs, and I got Intero and Cider to assist me. For Clojure there seems to be something like Conjure, but I usually rely on Clojure LSP myself. Regarding Haskell, there’s some Intero for Vim too. Starter kits relying on ghcide, hlint, hoogle are also available, but I don’t like starter kits anymore.

Without further ado, here are some retrospective posts on the state of Haskell and Vim in recent years:

Vim and haskell in 2016: Stephen Diehl has a long history with Haskell, even if it is mostly outdated material now that Haskell LSP is a thing, you may want to take a look at this post and other related posts on his site.
Haskell Development with Neovim: This is mostly the old-fashioned way (several tools tied together with the help of Vim plugins).
Vim and Haskell in 2019: This is a more modern take on Haskell+Neovim, with the old way of installing HIE and its wrapper.
(neo)vim and Haskell, 2021 edition: I ended up in the same position, using the now consolidated Haskell LSP and Neovim builtin stuff (instead of Coc, as the author does). In my view, Neovim builtins for LSPs are still inferior to Coc, but I came to like the defaults: less friction, more people behind them, and it’s always evolving, mostly for the better.

♪ Galaxie 500 • Today

Algorithms for functional programming

Thu, 08 Sep 2022 19:52:36 +0200

I started reading Algorithms for Functional Programming, by John David Stone (Springer, 2018), and it’s one of my preferred technical book of this Summer. My plan is to finish reading it before the start of the new academic year, and then read Functional Algorithms, Verified!, then Purely Functional Data Structures, by Okasaki, and maybe Bird’s Pearls of Functional Algorithm Design in the next few months. Stay tuned!

In this book, the author uses Scheme to illustrate all the concepts discussed in the book. After a brief introduction to functional programming and Scheme (Chapter 1), the author starts discussing more involved concepts. I should note that even if Chapter 1 is an introductory chapter, it features material that you will not find in introductory tutorial on Scheme, like receive expressions¹ or advanced let constructs. In Chapter 2, the author discusses several add-ons for working with lists (e.g. map) and procedures (adapters, couplers, etc.), which allows to define all kind of function composition or dispatiching. Later, reducers and iterators are implemented, which for a complete distinct set of operations one usually perform using recursion alone. An example if

(define (ceiling-of-log-two bound)
 ((pipe (iterate (pipe >initial (sect >= <> bound))
 (cross double add1))
 >next)
 1 0))

I find this formulation a bit heavy, and I prefer the more classical appraoch the author suggests as an alterntive a few paragraphs later:

(define (ceiling-of-log-two bound)
 ((rec (doubler most-recent-double count)
 (if (>= most-recent-double bound)
 count
 (doubler (double most-recent-double) (add1 count))))
 1 0))

There is a nice overview of recursion using natural numbers, again using “recursion managers”. Later the author presents a tally function, simialr to what’s available in Python or Clojure

The rest of the book is organized around data structure (Chapter 3 covers sum and product types, unions, pairs and named pairs, boxes which are restrictions of pairs, and various operations that can be performed on lists, trees and bags, sets, tables, and buffers) and various kind of algorithms along with the data structures they are best tied to (sorting, graphs, and sublist search), which makes for another half of the book. I am still reading or rereading some parts, because despite everything this book remains quite dense unless you have a good Scheme and have many years of hindsight (which is not my case).

In each chapter, or even in between sections, a useful recap of Scheme procedures currently discussed is provided, with type hinting and a brief description. I found this very handy as it allows to go back to previous chapter for quick checks.

♪ Iggy Pop • Some Weird Sin

Quoting the author, in let-expressions, it is an error if any expression in a binding speciﬁcation has more than one value. Frequently, however, we want to have local names for the values of a multiple-valued expression. A third local-binding construction, the receive-expression, provides for this possibility. ↩︎

The unquantified self #25

Sun, 04 Sep 2022 12:27:15 +0200

I’m trying out a new desktop setup, see below.

Little customizations here and there: terminal transparency (10%) and refined dark theme,¹ uniformized statusline (no background color, darkgrey foreground where possible) and uniform Neovim statusline (vim.o.laststatus = 3), tactile extension for window placement (after many attempts to find a suitable way of tiling windows altogther on my screen). It feels good, especially when you are anxious while waiting for biomedical test results. Still, I stand by the idea that Gnome WM is already a good fit for how I use to use my desktop, but being able to snap window on a grid is an invaluable add-on, especially since it does not conflict with natural tiling behavior of Gnome WM; moreover, when you use focus-follow-mouse mode, it is quite easy to activate windows on the fly.

I am also back to Irssi and Cmus as my daily drivers for IRC and music. Matrix and Element are nice, but I only need the IRC bridge to be honest. Rhythmbox is also a great aplpication, but sometimes it is easier to just have Cmus running in a terminal, alongside other programs I use in Tmux. What’s funny is that I can switch to a virtual console (Ctrl+Alt+F3) and I can still listen to my current playlist.²

Other than my desktop, I spent a little more time in front of my TV, watching Secret City and Mindhunter (same main actress), Tribes of Europa (only one season, and I don’t think there will be a sequel), and lately Sandman. See my informal movie ratings. Unfortunately I haven’t been able to read much books because of small vision problems, among others. No new musical playlist at the moment, except that I got myself a present few days ago. I’m lsitening to my good old playlists all day, though.

If you follow this blog more or less regularly, you will have noticed an increased rate of posting of the main section. I’m afraid this won’t last forever, since the frequency with which I was writing blog posts is mostly due to the fact that I haven’t slept much for 4 weeks. My body is beginning to reach its limits and I will have to rest from time to time, but with a little hope I will remain faithful to my posting habits. I’m mostly diving with Scheme at the moment, and I have other posts in mind.

♪ Chase Elodia • You

For aesthetics, but also to alleviate suffering after long sessions in front of my computer. ↩︎
I do not often need a virtual console except when I want to be in full retor computing atmosphere, or like recently when I need to check that my colorscheme equally apply to $XDG_SESSION_TYPE = “tty” and TERM=xterm-mono settings. ↩︎

Summing random Uniform deviates

Sat, 03 Sep 2022 17:37:55 +0200

Chris Wellons once asked the following question before providing some code to estimate the result, which I call brute force estimate but I do it a lot too:

How many times should a random number from [0, 1] be drawn to have it sum over 1?

Here is my brute force approach using Scheme and recursion only:

(import (chicken random))

(set-pseudo-random-seed! 101)

(let loop ((n 1)
 (sum 0))
 (cond ((> sum 1) sum)
 (else (loop (add1 n) (+ sum (pseudo-random-real))))))

;; => 1.77838275494088

A more involved solution using (typed) Racket and for/fold is discussed on Stack Overflow.

How about the math? There are several solutions available on math.SO, more or less complicated. See also this application. In essence, the probability that the sum of $n$ variables is $> 1$ while the other $n - 1$ are $< 1$ is:

$$ \begin{equation} \begin{aligned} P(S > 1) & = \int_1^n P_{X_1 + \dots + X_n}(u)du - \int_1^n P_{X_1 + \dots + X_{n-1}}(u)du\cr & = \left( 1 - \frac{1}{n!} \right) - \left( 1 - \frac{1}{(n-1)!} \right)\cr & = \frac{1}{n(n-2)!} \end{aligned} \end{equation} $$

Then it follows that $\sum_{n=1}^{\infty}nP(S > 1)$ is equal to $\sum_{n=1}^{\infty}\frac{1}{(n-2)!}$, or after simplification $e$.

♪ Bauhaus • She’s in Parties

On wrapping and reflowing text

Thu, 01 Sep 2022 13:47:34 +0200

I stopped hard wrapping my plain text documents a long time ago, when I was primarily using Emacs for all-things related to text. I believe soft wrapping is better, but when you really need hard wraps it is no longer a solution. We hardly know when things get soft- or hard-wrapped unless we have some indication about the formatting option in our Emacs or Vim statusline. For the latter, this is easy fixed as suggested on ezyang’s blog.¹ That’s when you write for yourself, though.

Two things always bothered me. On the one hand, while it is easy to unwrap text in Emacs (or Vim), it is a pity to do that on more basic text editors (and, sometimes, even advanced ones). You can use awk, of course, but my colleagues on Windows won’t understand why they would need to install yet another program for that very specific purpose. On the other hand, there’s a netiquette and RFC 2022, which states:

There are two limits that this standard places on the number of characters in a line. Each line of characters MUST be no more than 998 characters, and SHOULD be no more than 78 characters, excluding the CRLF.
The 998 character limit is due to limitations in many implementations which send, receive, or store Internet Message Format messages that simply cannot handle more than 998 characters on a line. Receiving implementations would do well to handle an arbitrarily large number of characters in a line for robustness sake. However, there are so many implementations which (in compliance with the transport requirements of [RFC2821]) do not accept messages containing more than 1000 character including the CR and LF per line, it is important for implementations not to create such messages.
The more conservative 78 character recommendation is to accommodate the many implementations of user interfaces that display these messages which may truncate, or disastrously wrap, the display of more than 78 characters per line, in spite of the fact that such implementations are non-conformant to the intent of this specification (and that of [RFC2821] if they actually cause information to be lost). Again, even though this limitation is put on messages, it is encumbant upon implementations which display messages.

In other words, when writing emails, do hard-wrap your message (and don’t put your signature on top when group-replying in a mailing-list!). The same applies to Git commit messages, although we may wrap differently the header (or summary) and body of a commit message. Regarding the header line limiation, this is not an absolute rule and it is often not respected, as far as I can tell, especially when introducing Git leaders (feat, fix, docs or doc, etc.). I keep seeing people setting the number of max characters allowed on a line (textwidth in Vim parlance) below this limit, when composing email, probably to accomodate the quoted messages included in a group-reply. The OpenBSD even recommends Plain text, 72 characters per line. There’s another Git-related justification for the 72 vs. 78 char limit.² But see Use plaintext email.

The other option is to let the text “flow”, which leads to using the text-flowed format. There’s even an option for that in (Neo)mutt. However, there are some caveats with those conventions for ordinary emails in the 21th century: many mail readers, including Gmail³ or Outlook, don’t honor this setting apparently. Here’s how Gmail renders an email when we enforce a textwidth of 68 chars, despite asking for text-flowed content:

--8<--------
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
--8<--------
Hello,

I'm writing to myself to test my new Neovim settings for writing
mail according to the netiquette. This should end up being wrapped
correctly (but flowed for most mail reader, hopefully), with
signature at bottom to not bother people when group-replying.

--8<--------

That’s lame. The header does indicate that the text should be reflowed.

This also seems to cause trouble with quoted messages and on GitHub. There’s one solution to annoy those programs, but there may be a better alternative: don’t hard wrap for casual emails, but enforce 72 chars max (or whatever) for list-reply, which is what I ended up doing recently. Since I don’t know how to manage alternative settings in ftplugin/mail.vim depending on Neomutt reply action (message-reply or group-reply) and I don’t want to add folder hooks for each list I subscribed to, I defined a shortcut to change the format options when I am writing to a list rather than a single person:

set spell spelllang=fr,en

setlocal tw=0
set formatoptions+=wq

function! Reflow() abort
 setlocal tw=72 et cc=73
 setlocal list
endfunction

map <buffer> g= <Esc>:call Reflow()<CR>vipgq

iabbrev <buffer> =s --chl

Now, whenever I list-reply, I can just press g= (this is a mapping I use for formatting in LSP as well) to get the message correctly hard-wrapped to the desired hard limit (72 chars), and later I have gq to do the hard work for me. That may just be crap, but it looks like it works.

Better looking email or not? To be honest, I don’t really know since I quite like plain old good hard-wrapped text. But given that my editor and my terminal support so called wrap-margin, unlike Edward Z. Yang I do not mind writing long-lines as long as I know that I can soft- or hard-wrap them at any time withing a single key press.

♪ Jasmine Myra • New Beginnings

I use a different setting and I only display tw (with text-flowed indicator appended if any), cc and et: "[%{&tw}%{matchstr(&fo, 'w')}%{&cc?','.&cc:''}%{&et?',+':''}] ". ↩︎
On an 80 column terminal, if we subtract 4 columns for the indent on the left and 4 more for symmetry on the right, we’re left with 72 columns. ↩︎
I stopped writing email in Gmail UI so long time ago that I have vague memory that it was just plain crap, but see Dear Google, please fix plain text emails in Gmail. ↩︎

Micro posting in August

Wed, 31 Aug 2022 10:15:16 +0200

2022-08-09: The Frames, True.
2022-08-10:

Where things really got broken was when multithreaded applications became commonplace. That’s when ctrl-c basically stopping working at all. — Ctrl-C: Why Programmers Can’t “Reset” Programs With Ctrl-C, but Used to Be Able To, and Why They Should Be Able to Again

2022-08-10: Hot days (again). Salad parties almost every day.

2022-08-10: Technical reasons to choose FreeBSD over GNU/Linux. #unix
2022-08-10: Top 20 Latex Fonts. #tex
2022-08-10: more-itertools is yet another gem when working with Python’s iterables. #python
2022-08-11:

Expect to pay, now or later. — Give nothing, expect nothing: GitLab’s the latest punching bag for entitled users

2022-08-14: I’m done with Season 1 of Mindhunter. Beautiful script, brilliant actors, nicely orchestrated plots. Informal ratings of others TV shows available here.
2022-08-14: Stop using DICT dictionary apps (such as GNOME/MATE Dictionary). #unix
2022-08-16:

SQLite works with hundreds of millions of records just fine. Regular INSERTs show about 240K records per second on my laptop. And if you connect the CSV file as a virtual table (there is an extension for that) - inserts become 2 times faster. — SQLite is not a toy database

2022-08-16: Tom Mitchell’s Machine Learning book is a bit old but it’s now free. Note that the PDF version offered on the website is huge (60 Mo) since it’s a scanned copy.
2022-08-16: Deconstructing Bézier curves.
2022-08-16: Glmnet for python. I somewhat missed or forgot about that package. #python
2022-08-17: Jewel, Foolish Games.
2022-08-17: Howto: auto-start ssh-agent with systemd on Debian Bullseye — Debiania. #unix
2022-08-18:

The way I look at it, the apps themselves are the workspaces. On a smaller screen I usually full screen everything and on larger display scenarios, I have nearly everything laid out in front of me so I really just want to be able to switch between apps as fast as possible. — How I GNOME or Why I Don’t Use a Tiling Window Manager by Josh ShermanThat’s the conclusion I reached as well on OS X and Ubuntu when I started (re)using native window manager facilities. This remains so even after me trying i3 then Regolith, since Tmux can do a lot of tiling too, and the only thing I really need is to have apps maximized on screen, or split vertically (who need to fully split a 16:10 or 4:3 laptop screen horizontally these days?).

2022-08-18: Great! You can now use default keybindings from https://getpocket.com in Nyxt 3 (pre release).
2022-08-18: Watching with tmux | Arabesque. #unix
2022-08-22: FreeBSD - a lesson in poor defaults. #unix
2022-08-22: Readable s-expressions and sweet-expressions: Getting the infix fix and fewer parentheses in Lisp-like languages. #lisp
2022-08-22: Uncommon Uses of Python in Commonly Used Libraries. #python
2022-08-22: Visual Sum of Cubes.
2022-08-23: Magazine, Cut Out Shapes.
2022-08-23: Mannequin Pussy, Romantic.
2022-08-23:

I’ll take economy over excess any day - give me a nice, uncluttered interface, a simple man page and power to spare for those occasions when I need it, and I’m a happy guy. My dream email client is a lot like my dream car, the Ferrari F355 Spider: plain-vanilla seats, a simple dashboard and the ability to accelerate from 0 to 60 miles per hour in under six seconds. — A Man And His Mutt.

2022-08-23: Look what came up in the mailbox a few days ago:

2022-08-23: Playing with exceptions inside lambda expressions in Python . #python
2022-08-23: Searching this site using Pagefind. Looks like it works quite well with Hugo website.
2022-08-24: A simple way to make HTML websites: https://john-doe.neocities.org.
2022-08-24: Deconstructing the “Unix philosophy”. #unix
2022-08-24: Integers, decibels, and graphs.
2022-08-24: Should I use Signed or Unsigned Ints In C? (Part 1). #clang
2022-08-24: Use plaintext email.
2022-08-25: Marianne Faithfull, Guilt.
2022-08-25:

Beautiful code is likely to be simple – clear and easy to understand. Beautitful code is likely to be compact – just enough code to do the job and no more – but not cryptic, to the point where it cannot be understood. Beautiful code may well be general, solving a broad class of problems in a uniform way. One might even describe it as elegant, showing good taste and refinement. — A Regular Expression Matcher

2022-08-25: Macro elegance: the magical simplicity of Lisp macros. #lisp
2022-08-25: Why your website should be under 14kb in size.
2022-08-26: Michael R Oldham, Two Pieces Ripe Fruit.
2022-08-26:

With Vim as a sort of secondary editor, I want to be able to fire it up unconfigured and continue to be nearly as productive. A pile of remappings would prohibit this. In my mind this is like a form of emergency preparedness. Other people stock up food and supplies. I’m preparing myself to sit at a strange machine without any of my configuration so that I can start the rewrite of the software lost in the disaster, so long as that machine has vi, cc, and make. If I can’t code in C, then what’s the point in surviving anyway? — My Journey with Touch Typing and Vim

2022-08-26: The 10 stages of Vim. #vim
2022-08-27: MIKA, I See You.
2022-08-27: Hugo documentation theme as simple as plain book.
2022-08-27: A Lisper’s first impression of Julia .
2022-08-27: Combinator Formatting. #scheme
2022-08-27: Labeling mail with mutt. Interesting idea, especially for those who rely on notmuch for mail processing. But, well, it’s been a long time I left Gmail and I don’t regret those pesky virtual folders.
2022-08-27: Unifying fold left and fold right.
2022-08-28: Morcheeba, What New York Couples Fight About.
2022-08-28: And a great Firefox plugin to review soon: Vixen Vim.
2022-08-28: I somewhat forgot about this sticky TUI for processing spreadsheet data: VisiData is an interactive multitool for tabular data.
2022-08-28: Ubuntu Dock completely hidden now, which makes for a cleaner desktop!shellgsettings set org.gnome.shell.extensions.dash-to-dock autohide falsegsettings set org.gnome.shell.extensions.dash-to-dock dock-fixed falsegsettings set org.gnome.shell.extensions.dash-to-dock intellihide falseI also installed the Suckless terminal, as a popup terminal always opened on second workspace, with semi-transparency enabled thanks to Luke Smith’s patch and no border (Stein Gunnar Bakkeby ’s path). I can launch Cmus without messing my current Tmux workspace, and forget about it.

2022-08-28: Using FZF instead of DMENU.
2022-08-28: ssh, tmux and vim: A Simple Yet Effective Pair Programming Setup. #vim
2022-08-29:

All of a sudden, those brackets look kind of cool! — An Intuition for Lisp Syntax

2022-08-29: Here’s a rather long list of RSS feeds accumulated over the years. It has been purged on several occasions, depending on my mood and succesive jobs (e.g., rstats and stats stuff has been severely impacted in the past 6 years). It targets Newsboat, so all you have to do is put this file in your $HOME/.config/newsboat directory. Tags are not always very accurate, as the original posters may have changed their posts over time.
2022-08-29: Suckless life.
Ubuntu 22.04 LTS. Suckless terminal. Cmus music player.
2022-08-29: TIL Ubuntu and its default WM allows to move and resize windows without even touching the mouse with Alt-F7 and Alt-F8.
2022-08-29: Website’s look’n feel should now respect your system color theme. Dark theme heavily inspired from the Nord color palette of course. Last thing to fix is transparent background images.

2022-08-29: Mutt, the Vim Way.
2022-08-30:

If you spend a lot of time typing plain text, writing programs or HTML, you can save much of that time by using a good editor and using it effectively. — Seven habits of effective text editing

2022-08-30: TIL that I no longer need to worry with launching ssh-agent automagically in case it isn’t already running in the background using the following hack:

if [ -f ~/.ssh/agent.env ] ; then
 . ~/.ssh/agent.env > /dev/null
 if ! kill -0 $SSH_AGENT_PID > /dev/null 2>&1; then
 echo "Stale agent file found. Spawning a new agent. "
 eval `ssh-agent | tee ~/.ssh/agent.env`
 ssh-add
 fi
else
 echo "Starting ssh-agent"
 eval `ssh-agent | tee ~/.ssh/agent.env`
 ssh-add
fi

There’s probably a better way: Howto: auto-start ssh-agent with systemd on Debian Bullseye.
2022-08-30: Only fast languages are interesting. Don’t know if lush is still a thing nowadays, but some benchmarks were quite impressive back in 2011. #clojure #lisp
2022-08-30: SSH Tip: Connecting to a Private Network Without Trusting the Bastion Host. #unix
2022-08-31: New Order, Blue Monday.
2022-08-31: Just replaced ranger with fff. Looks great and lightweight, and ultra-fast. I had problem with displaying images with ranger now that I am under Wayland. With fff, I can simply open them in feh or eog and browse an entire directory using keyboard shortcuts only. (h/t DJ Adams, Exploring fff part 2 - get_ls_colors #unix
2022-08-31: Some interesting Bash notes. #unix
2022-08-31: OCaml at First Glance.
2022-08-31: Rewriting my blog in Rust for fun and profit.

How to get by without using a tiling WM

Tue, 30 Aug 2022 09:36:19 +0200

How do I do without a tiling Window Manager, after several months of using i3 and Regolith? A few weeks ago I started wondering whether I should give Sway a try now that Ubuntu defaults to Wayland and Gnome 42. However, I held back with the idea of better familiarizing myself with Ubuntu’s default WM. After all, I don’t have big requirements in terms of desktop management: I use a single workspace most of the time, I prefer to have windows maximized on screen or split side by side full-screen, there’s a launcher right under Super+S or Super+A which means I can just start typing the name of the application I want and press Enter, and the Super+Tab hot key allows me to switch between applications very easily. This is very much in spirit with points raised by Josh Sherman some time ago.

Yet I do not like the Snap system (nor the idea of sandboxed applications tied to an OS, as Apple did) and I removed all components on my previous install. They were not reinstalled when I upgraded to LTS 22.04. And I still feel more comfortable with terminal-oriented applications, or TUIs.

Anyway, here are some of the settings on my machine. First, as I said, I stand by maximized windows most of the time. When the application is up, if it is not maximized I can simply Super+Up to make it so. When I have two windows on screen, I can arrange them maximized and side by side using Super+Left and Super+Right. I yet have to find a way to automate this a little since at present I must hover on each window separately.

I use “focus follow mouse” (gsettings set org.gnome.desktop.wm.preferences focus-mode 'mouse') and Emacs keybindings (gsettings set org.gnome.desktop.interface gtk-key-theme "Emacs") for graphical applications like Firefox.
I modified the Caps lock key so that it acts as an Escape key (easier for me when using (Neo)vim) using gsettings set org.gnome.desktop.input-sources xkb-options "['caps:escape']". Previously, I was relying on setxkbmap -option caps:escape, which no longer works under Wayland. See also Remapping keys under Wayland.
I use two extensions for Gnome shell: auto-move windows, which allows to assign specific workspace to any applications, and simply workspace, which adds a workspace block in the menu bar, like i3bar or polybar.
I hided the dock entirely can probably be done by either removing Gnome dock from the system or through the extension preference panel. I choose to use a terminal incantation, instead.¹
I defined some shortcuts to make navigating workspaces (Ctrl+Super+Left and Right) and moving windows from one workspace to the other easier (Shit+Super+Left and Right). Since I learned recently that we can also move and resize window using keyboard shortcuts, I’m done with window and workspace management for now.
I generally try to limit myself to two windows per workspace (easier to tab switch, and in case you have more than two applications opened on your desktop they keep being sorted by most recent use which means they should end up one after the other in the app switcher), and likewise to two pane per Tmux or Neovim window (again, easier to M-o – a specific shortcut of mine – or C-w C-w).
Originally, I updated the default applications for specific filetypes (PDF, images, etc.), but finally I reverted to default settings and I launch Zathura (PDF), ranger (Files) or Feh (Images) from a running terminal.

I also use the gorgeous Nord theme for Gnome, which you can activate as follows:

% gsettings set org.gnome.desktop.interface gtk-theme "Nordic"
% gsettings set org.gnome.desktop.wm.preferences theme "Nordic"

It goes without saying that this changes the visual experience a lot.

Overall, I find that Gnome shell has been greatly improved over time, and it is now good enough for intensive daily use. I could hide Gnome terminal² window border as well (e.g., gsettings set org.gnome.Terminal.Legacy.Settings headerbar false), but I found that just going fullscreen (F11) is usually enough. I like the system tray icons, which is compact and remains interactive, and, contrary to polybar, I don’t have to bother configuring brightness, sound control, battery status, network settings and VPN myself.

♪ Foals • My Number

This can be done as follows: (forgot where I found the trick, sorry!)

gsettings set org.gnome.shell.extensions.dash-to-dock autohide false
gsettings set org.gnome.shell.extensions.dash-to-dock dock-fixed false\
gsettings set org.gnome.shell.extensions.dash-to-dock intellihide false

↩︎

By the way, you can see or backup all your Terminal settings using dconf dump /org/gnome/terminal/ > gnome_terminal_settings_backup.txt. ↩︎

Named let in Scheme

Sun, 28 Aug 2022 13:17:18 +0200

In one of my last posts, I discussed Common Lisp loop macro. I just finished reading The Adventures of a Pythonista in Schemeland, by Michele Simionato. Although it is now more than 10 years old, it is a very good read that I recommend to people interested in Scheme, especially if they want to learn more about macros. In this series of articles, the authors discusses some idiomatic construct in Scheme. Beside recursion, he shows an example of a for-loop form which I got interested in. Chicken Scheme, which is the current implementation of Scheme dialect I use,¹ has a for-each construct which allows to write stuff like this:

(define xs '(0 1 2 3 4))
(for-each print xs)

This is more limited than what’s available in Racket’s iterations and comprehensions forms but it has the merit of being there. Note that there are more involved constructs for iterating over sequences.

In Scheme, a named let is the standard way to build such a for-like expression. Let’s see an example:

(import srfi-48) ;; see also Chicken extras module

(let loop ((i 0))
 (unless (= i 5)
 (format #t "~a " i)
 (loop (+ i 1))))

This is well explained on Stack Overflow, especially how it relates to tail-recursion. See also Python to Scheme to Assembly, Part 1: Recursion and Named Let, for another use case of recursion and accumulators. Now, checking the SICP book, Abelson & coll. discuss an iterative Fibonacci procedure using a named let:

(define (fib n)
 (let fib-iter ((a 1)
 (b 0)
 (count n))
 (if (= coutn 0)
 b
 (fib-iter (+ a b) a (- count 1)))))

A named let can also be an alternative to the following procedure (Exercise 4.8):

(define (let->combination expr)
 (cons (make-lambda (let-vars expr) (let-body expr))
 (let-inits expr)))

Here is one solution from Scheme wiki:

(define (let->combination expr)
 (if (named-let? expr)
 (sequence->exp
 (list (named-let->func expr)
 (cons (named-let-func-name expr) (named-let-func-inits expr))))
 (cons (make-lambda (let-vars expr)
 (list (let-body expr)))
 (let-inits expr))))

See also Eli Bendersky’s solutions in Common Lisp.

Instead of a for loop, let’s consider a while loop. It is not much different, since in this case we have a predicate that helps stopping the iteration and a default case. See also this discussion on call/cc on SO. There’s a bunch of useful utilities in the “standard prelude” of Programming Praxis. It targets R5RS, and some of its forms are already available in Chicken Scheme or Racket, but it is worth taking a look at the code. For instance, it features a macro for a while loop construct:

(define-syntax while
 (syntax-rules ()
 ((while pred? body ...)
 (do () ((not pred?)) body ...))))

Example of use:

(let ((loop 4))
 (while (< 0 loop)
 (display loop)
 (set! loop (- loop 1))))
=> 4321

Incidentally, I just found that Jacques Chazarain also discussed the case of Fibonacci numbers and named let constructs, and he even proposed a different solution to the while loop macro, using pretty old syntax and non-hygienic macro:²

(define-macro (while test . body)
 `(letrec ((loop (lambda ()
 (if ,test
 (begin ,@body (loop))))))
 (loop)))

There is nevertheless a slight issue with the formulation above: the local function loop may happen to mask a variable bound in the let statement. To remedy this problem, one needs to use gensym to generate the name of the local loop construct, as illustrated below:

(define-macro (while test . body)
 (let ((loop (gensym)))
 `(letrec ((,loop (lambda ()
 (if ,test
 (begin ,@body (,loop))))))
 (,loop))))

A related topic is also discussed on Chicken Scheme website.

[2022-09-12]
Now that I’m catching up on my RSS feeds, I can suggest a further reading, especially for Lisp hackers: Named Lambda and Named Let, by Joe Marshall.

♪ MIKA • Blue Eyes

It is the first Scheme implementation I heard about, around 2007, via Jan de Leeuw’s mailing list on statistical computing. I wrote toy example, stopped using it and forgot about it until I got back to Scheme four or five years ago. ↩︎
Chazarain, J. Programmer avec Scheme. International Thomson Publishing, 1996. ↩︎

Usenet newsgroups with Neomutt

Sat, 27 Aug 2022 11:01:27 +0200

A few days ago I was looking at some old messages on Usenet’s comp.lang.scheme, now attached to Google groups. I quit all groups I was subscribed to a while ago already, but I renabled one of them just to see how it feels. Well, you still feel the same, Google, but I’m happy comp.lang.lisp hasn’t been shutdown completely, and it is still possible to dig into the archives:¹

Last time I installed slrn (official website) and configured it to use eternal-september NNTP server. It worked great, and it still works after a few months without opening slrn. I spend some time digging on DDG for users’ config for NNTP, and except for the official documentation, I found nothing. Noboody’s using (Neo)mutt for reading news via NNTP these days?

Apparently, there’s nothing much to do to get Neomutt access Usenet. First, Neomutt is able to read newsgroup in addition to standard Maildir inboxes, just press i and you’re right into Usenet provided the environment variable $NNTPSERVER is set correctly. With a few additional settings, as shown below, and a $HOME/.newsrc file, you’ll get a list of newsgroups you can subscribe to. Usually, it is enough to create the .newsrc file and indicate on each line the name of the group (e.g., comp.lang.lisp) followed by :. If you already have such a file, you can edit it by hand, which is what I ended up doing with the $HOME/.jnewsrc file created by slrn.

set newsrc = '~/.newsrc'
set news_cache_dir = '~/.mutt/news'
set news_server = 'news://{USER}:{PASS}@news.eternal-september.org'
set nntp_authenticators = 'user'
set nntp_listgroup = yes
set nntp_load_description = yes
set nntp_pass = '{PASS}'
set nntp_user = '{USER}'

Replace {USER} and {PASS} with your credentials form eternal-september.

I found the experience of reading news more pleasant under Neomutt, especially given that the color scheme is already set up and that I have my usual keyboard shortcut. Note, however, that no shortcuts are defined for browser and news-specific index views. You will have to define a few handful bindings or macros for better navigation.

Who ever said that Usenet is dead? It most certainly is not. Even though Usenet has changed it’s face over the last, say ten years, the simple idea of exchanging ideas and having discussions among a large number of people by posting articles on a global message-board has pertained. Yes, for those of you who still don’t know what this is all about, we are talking about Newsgroups, an important part of the Internet, albeit heavily neglected by the media, who usually equate “Internet” with “World Wide Web”. — Usenet is still a strange place

That’s it! Happy reading with good old NNTP.

♪ Miguel Zenzön • Quitate de la Vía

If like me you don’t want to be tied to Goggle for reading newsgroups, you can subscribe by email using the regular way: Just send an email to newsgroup+subscribe@googlegroups.com, where newsgroup is the name of the newsgroup your are interested to follow. ↩︎

More Neomutt little hacks

Fri, 26 Aug 2022 12:58:30 +0200

Some further tips and tricks on Neomutt after my original post. See also my related post on Tmux hacking.

I will probably never regret switching from mu4e to Neomutt two years ago. Chris Wellons did the same apparently. It’s the same keyboard-driven experience, but with all the functionalities you would expect from a mail processor. Moreover, as they use to say:

All mail clients suck. This one just sucks less. — Michael R. Elkins

I spent a lot of time configuring Neomutt to my liking, and to this end I happened to read a lot of blog posts,¹ the online documentation and the man pages, dotfiles available on Github, etc. Neomutt is great, but its configuration is harder than it seems because there are tons of options that you need to define carefully. Of course, default options are quite good, but you know when you start tinkering your setup… My previous post explained how I ended up setting my default config. Here are a few additional tricks that I learned along the way.

[2023-09-21]
Not mentioned here is how to get notified when you receive new emails. See this micro post for a solution that circumvents some of notify-send limitations.

using folder hooks: Folder hooks are actions that carried automatically when you enter a (local or remote) folder. I use a few of them, mostly to setup specific mailboxes. For instance, I have a different display mode depending on whether I’m in my Inbox or in the Archives folder (where I usually want a detailed timestamp, especially the year since messages can span more than 15 years), but see below:

folder-hook archives/* 'set read_inc=1000'
folder-hook archives/* 'set index_format="[%[%Y-%m-%d %H:%M]] [%Z %?X?A&-? %?l?%4l&%4c?] %-25.25F %?M?<%M> ?%s "'

folder-hook aliquote/* source ~/.config/neomutt/accounts/aliquote
folder-hook aliquote/* 'set index_format="[%[!%m-%d %H:%M]] %zt %-25.25F %?M?<%M>? %s "'
folder-hook aliquote/Sent 'set index_format="[%[!%m-%d %H:%M]] [%Z %?X?A&-?] %-25.25F %s "'
folder-hook aliquote/Sent 'set sort=reverse-date-sent'
folder-hook aliquote/Sent set strict_threads=yes

folder-hook aliquote/INBOX 'push "Tscipy-dev-request<enter><tag-prefix-cond><save-message>=digests<enter><end-cond>"'
--8<--------

If you write a lot of macros, sometimes it is worth considering folder-hook or variation thereof. You often just have to replace your macro statement with a push command to let Mutt press all the keys for you.

collapsing threads: I wanted all threads to be collapsed by default when entering my Archives folder. I originally used a hook for that, folder-hook archives/* 'push <collapse-all>\n', which has the undesired side-effect of making sidebar actions (navigating next and previous folders) to open the first message when entering the folder itself. Fortunately, there’s an even simpler solution using the following two options: set collapse_all = yes and set uncollapse_new = yes. This will affect all folder, but the last one will help in case there are new replies in the Inbox.
navigating the sidebar: After reading Joshua Stein configuration on Github, I ended up using two macros to toggle the sidebar on or off. However, there’s a simple command that do the trick: bind index,pager B sidebar-toggle-visible. I then use two macros to help navigating in the sidebar tree: macro index K "<enter-command>push <sidebar-prev><sidebar-open><enter>" and macro index J "<enter-command>push <sidebar-next><sidebar-open><enter>".
displaying all URLs in a message: I originally used urlscan to show all URLs found in a message, but I found that urlview is builtin and works better. By default, it is mapped on C-b.
colors: There are several sections to manage Neomutt color theme, the main parts being index, header and body. In the body section, you can even highlight Git diff or emojis if you like.
PGP signing/encryption: I like to have the possibility to sign and/or encrypt my message, but I don’t want them to be signed on by default. For this, you can unset the crypt_autoencrypt and crypt_autosign variables, and when you save your message, the magic key is p before confirming to send your message (y): you will be prompted to sign and/or encrypt your outgoing email. To configure PGP stufff in neomutt, source the relevant sample file, /usr/share/doc/neomutt/samples/gpg.rc.gz, which contains default values that should work if you have gpgme installed on your system, and the relevant settings in your muttrc file:
```
set crypt_autoencrypt = no
set crypt_autopgp = yes
set crypt_autosign = no
set crypt_replysign = yes
set crypt_replysignencrypted = yes
set crypt_use_gpgme = yes
set crypt_verify_sig = yes

set pgp_self_encrypt = yes
set pgp_sign_as = {PGP_KEY}
set pgp_use_gpg_agent = yes
```

Finally, some words about my workflow. The main structure of my mailboxes is shown below:

INBOX
Drafts
Sent
Trash
archives
digests
@chicken-users
--8<--------

I now use a single account setup because I found that juggling different accounts was kind of not that funny at times. My typical workflow is that every new mail end up in the INBOX, with some automatic filters to move messages from mailing-lists I subscribed to in their respective folders. Mailing digest may be archived manually into the “digests” folder, and I used to use a weird macro to automate this stuff.² The “archives” folder is an all-in-one archive of all the messages I received since around 2006 (Gmail time, remember?). It held more than 45K messages at some point, but I did a drastic pruning a few years ago so that now there are just about fifteen thousand messages, which is more manageable, even for Neomutt. I have some handful macros that help keeps the digests and mailing folders clean: one macro allows to delete all message older than a certain amount of days, while another macro is used to mark all messages as read.

macro index <Esc>x "<tag-pattern>~d >6m<enter><tag-prefix-cond><delete-message>" "delete old messages"
macro index \Cx "T~U<enter><tag-prefix><clear-flag>N<untag-pattern>.<enter>" "mark all messages as read"

[2022-08-27]
TIL some Vim keybindings are defined in /usr/share/doc/neomutt/vim-keys/vim-keys.rc. Unfortunately, there’s not much to see except some bindings for moving around, scrolling and managing threads (sort of).

♪ Maxence Cyrin • As the Darkness Falls

I may even have missed Steve Losh’s excellent blog post at some point. ↩︎

Example of a weird macro, that I converted to more manageable folder hooks a while ago:

macro index <Esc>s \
 "Tr-devel-request<enter><tag-prefix-cond><save-message>=digests<enter><end-cond> \
 T~Crabble@arXiv.org<enter><tag-prefix-cond><save-message>=digests<enter><end-cond> \
 Tnumpy-discussion-request<enter><tag-prefix-cond><save-message>=digests<enter><end-cond> \
 Tscipy-dev-request<enter><tag-prefix-cond><save-message>=digests<enter><end-cond> \
 --8<--------
 Tcryptography-request<enter><tag-prefix-cond><save-message>=digests<enter><end-cond>"

↩︎

Common Lisp loop

Thu, 25 Aug 2022 11:40:59 +0200

The Common Lisp loop macro is the most incredible thing I’ve seen so far when working with functional PL. It comes with so subtle variations that it is hard to summarize all of its functionalities in a few words. Let’s consider the following two statements:

(loop for i in '(0 1 2 3 4) do (format t "~a " i))
;; => 0 1 2 3 4
(loop for i on '(0 1 2 3 4) do (print i))
;; => (0 1 2 3 4)
;; => (1 2 3 4)
;; => (2 3 4)
;; => (3 4)
;; => (4)

The first instruction asks to iterate over each element of a list, while in the second we iterate over each cdr of a list. If you want to return a list for later consumption, you can use (append (list i i)) or simply “collect” the elements, but see below. Any other builtin or user function that accept the iterator variable (i) as an argument will work too. Little subtelties…

I don’t know if the loop macro in Common Lisp is genius or madness. It’s the idiomatic way to do iteration, but the syntax doesn’t resemble Lisp in any way, the number of variations is as long as your arm, and you can mix and match every which way. — A Closed and Common Lisp

A standard for loop construct is written:

(loop for i from 0.0 to 4.0 by 0.5 do (format t "~a " i))
;; => 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Note that loop works with real number too, unlike some other functions or macros, and that you are not limited to generating sequences of increasing values – replace start and end value and to with downto to get sequence in descending order.¹ If you are an Alexandria’s user, you probably already know the iota function, which helps in generating sequence of equally spaced integers (think of Python’s range function). Here it is in action, and its equivalent version using loop using keyword parameters:

(iota 5)
;; => (0 1 2 3 4)
(loop :for n :below 5 :collect n)
;; => (0 1 2 3 4)

The loop macro also supports conditional statement in between, like in (loop for i from 1 to 10 when (oddp i) collect i).

The only contender to CL’s loop macro is Racket’s iterations and comprehensions forms, which I use a lot for little scripts. Chicken Scheme and Clojure also have for loop constructs which are pretty handy. Many times, though, for loops are not really needed and can be safely replaced with map or fold, or variations thereof, even in Python. There was an interesting discussion on for loop in Scheme on the IRC #scheme channel recently, but I forgot to bookmark the conversation. It’s probably in the log, though.

If you are interested in macros, I can suggest the following blog posts:

Idiomatic Macros for Common Lisp, by Stuart Sierra
The Nature of Lisp, by Slava Akhmechet
Macro elegance: the magical simplicity of Lisp macros, by Linus Lee
Racket Macro Exercises, by Alexis King
Iterators and for loops with Guile Scheme, by Will Osborne
Macros in Racket, part one and The proper use of macros in Lisp, by Tim Bradshaw
Haskell doesn’t have macros, by Chris Done

♪ Brad Mehldau Trio • The Very Thought of You

They are known as the arithmetic-up and arithmetic-downto arguments. See the HyperSpec documentation. ↩︎

Manipulating images using Numpy

Tue, 23 Aug 2022 12:06:55 +0200

I always found that Python was great for manipulating strings, or even lists, which is why it remains my language of choice, as opposed to R, for bioinformatics stuff. Needless to say, I hate significant indentation, but other than that I find the language pleasant, and as they said it comes battery included. There’s lot to learn from the base module.

Today, I was trying to make some graphic with Matplotlib and I happened to find some interesting side-effects that we can produce using imshow() alone.

import matplotlib.pyplot as plt
import numpy as np

# Credit: https://www.publicdomainpictures.net/pictures/210000/velka/rose-grayscale.jpg
img = plt.imread("rose-grayscale.jpg")

idx = [[1,1,1], [1,0,0], [0,1,0], [0,0,1]]

for i in range(4):
 plt.subplot(2, 2, i+1)
 plt.imshow(img*idx[i])
 plt.axis('off')

Other manipulating RGB channels using simple matrix operations, we can also use more advanced techniques like adding blur or low-pass filter effects with OpenCV2. However, using Numpy alone we can also add some salt and pepper-like effect: (Credit: Shubham Pachori)

def noisy(noise_typ, image):
 if noise_typ == "gauss":
 row, col, ch = image.shape
 mean = 0
 var = 0.1
 sigma = var**0.5
 gauss = np.random.normal(mean, sigma, (row, col, ch))
 gauss = gauss.reshape(row, col, ch)
 noisy = image + gauss
 return noisy
 elif noise_typ == "s&p":
 row, col, ch = image.shape
 s_vs_p = 0.5
 amount = 0.004
 out = np.copy(image)
 num_salt = np.ceil(amount * image.size * s_vs_p)
 coords = [np.random.randint(0, i - 1, int(num_salt)) for i in image.shape]
 out[coords] = 1
 num_pepper = np.ceil(amount * image.size * (1.0 - s_vs_p))
 coords = [np.random.randint(0, i - 1, int(num_pepper)) for i in image.shape]
 out[coords] = 0
 return out
 elif noise_typ == "poisson":
 vals = len(np.unique(image))
 vals = 2 ** np.ceil(np.log2(vals))
 noisy = np.random.poisson(image * vals) / float(vals)
 return noisy
 elif noise_typ == "speckle":
 row, col, ch = image.shape
 gauss = np.random.randn(row, col, ch)
 gauss = gauss.reshape(row, col, ch)
 noisy = image + image * gauss
 return noisy

type = ["gauss", "s&p", "poisson", "speckle"]
for i in range(4):
 plt.subplot(2, 2, i+1)
 plt.imshow(noisy(type[i], img/255))
 plt.axis('off')

♪ Magazine • Back to Nature

Migrating to Systemd

Mon, 22 Aug 2022 14:37:44 +0200

I’ve been faithfully using cron jobs on Linux for twenty years now. It’s quite simple to schedule basic tasks using crontab -e: simply write your shell script or call the application you want to run at a desired time, and then forget about it. Currently, I only have two crontab entries running: one for checking my mail (mbsync), and the other for Kopia hourly backup. Most Linux distros now favor the use of systemd schedulers, also called “timers”, instead of cron, and so I decided to move all my running jobs to systemd.

One of the advantages of using systemd is that you can move your systemd files between your machine or over ssh. The other advantage, of course, is that you can manage your running jobs using systemctl, which means you can stop or pause tasks at any time, or reschedule a timer easily. Finally, you can check when your task were run for the last time, and when they will be run next.¹

As an example, consider the following cron task, which runs Kopia backup every hour on my machine:

0 * * * * kopia snapshot create --all

To use systemd, you need two files in $HOME/.config/systemd/user. Let’s call them kopia-backup.service and kopia-backup.timer. Here’s the content of the service file:

[Unit]
Description = Back up files using Kopia

[Service]
Type = oneshot
ExecStart = /usr/bin/env bash -c 'kopia snapshot create --all'

[Install]
WantedBy = default.target

This basically tells to run the service once (oneshot) using the command specified as in the above crontab entry. The default.target has to do with unit dependencies, but this may be omitted in our case. For the timer file, the most important setting is the schedule, which differs slightly from cron syntax:

[Unit]
Description = Run kopia-backup every hour
RefuseManualStart = no

[Timer]
Persistent = true
OnCalendar = *-*-* *:00:00
Unit = kopia-backup.service

[Install]
WantedBy = timers.target

The Persistent option informs systemd to launch the task if it was missed at the last start time because of some failure to run the task (e.g., system on sleep or done), much like anacron. There are also so-called monotonic timers, which just indicate how often a task should be run, not at what time precisely. This usually is performed with reference to system startup, using e.g., OnStartupSec=. Note that both approaches can be interleaved, and the task will run whenever any of the two timers occurs. By default, timers in systemd have an accuracy of 1 minute, but you can ask for a more tight schedule, for instance AccuracySec = 1us.

Here is a brief summary of most common settings for the scheduler:

Timer	Description
`::00`*	every minute
`:/30:00`	every 30 minute
`--* :00:00`	every hour
`--* /3:00:00`	every three hour
`--* 00:00:00`*	every day
`--* 01:00:00`*	every day at 1am
`Sun --* 00:00:00`*	every Sunday
`Mon...Fri --* 00:00:00`*	Monday to Friday
`Sun --* 00:00:00`*	every week
`* -01,07-01 00:00:00`	every 6 months
`* -01-01 00:00:00`	every year

Once you have these two files, you just need to reload the systemd configuration and enable the timer (which, in this case, is the unit of interest):²

% systemctl --user daemon-reload
% systemctl --user start kopia-backup.timer
% systemctl --user enable kopia-backup.timer

To check that the timer is indeed enabled or active, just issue:

% systemctl --user list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Mon 2022-08-22 19:15:00 CEST 14min left Mon 2022-08-22 19:00:08 CEST 21s ago mail-sync.timer mail-sync.service
Mon 2022-08-22 20:00:00 CEST 59min left Mon 2022-08-22 19:00:08 CEST 21s ago kopia-backup.timer kopia-backup.service

2 timers listed.

I implemented my two cron jobs (fetch mail every 15’ and backup my $HOME folder every hour) using systemd timer, and it works like a charm.

[2022-09-01]
See also this excellent review, in which I leaerned that we can check if our OnCalendar settings work as expected: Use systemd timers instead of cronjobs.

♪ Jan Lundgreen Trio • Rosemary’s Baby

One of the advantage of cron job is that the same job can be run again, in parallel, even if the previous instance hasn’t finished yet. This happens if you have long-running jobs that are launched at evry short interval. This at least is what I seem to remember from my early Linux days, and I didn’t check what the state of the rt is for this kind of situation. ↩︎
Thanks to Robert Lützner for having noticed an error in the wording of one of the commands. ↩︎

Simply Scheme

Sun, 21 Aug 2022 09:39:09 +0200

Simply Scheme, by Brian Harvey and Matthew Wright, is available on line in PDF and HTML versions. It is presented as an introductory textbook on computer science, with six chapters covering the basis of (interactive) functional programming: functions, compositions of functions, functions as data, recursion, abstraction, and sequential programming. It is not the first book I read on Scheme. I started with The Little series some years ago, followed by two textbooks in French, and this book is different in that it follows an example-driven approach to teaching Scheme. If you are looking for a such an introduction in Scheme to functional programming, go take a look at the first four chapters to get an idea. For people coming from Python and interested in learning a bit of Scheme, I would also recommend The Adventures of a Pythonista in Schemeland. If you’re a Racket user, there’s even a dedicated library: simply-scheme.

Note that the authors use older syntax for Scheme expressions than what is now known as R6 or R7RS (but see below), and they also provide some helper functions in a separate file, simply.scm (see Appendix A). It is important to download a copy of this file in order to be able to run the examples, since almost all of them use functions defined in that module. For instance, the accumulate function is defined as follows:

(define accumulate
 (let ((not not) (empty? empty?) (bf bf) (first first) (procedure? procedure?)
 (whoops whoops) (member member) (list list))
 (lambda (combiner stuff)
 (define (real-accumulate stuff)
 (if (empty? (bf stuff))
 (first stuff)
 (combiner (first stuff) (real-accumulate (bf stuff)))))
 (cond ((not (procedure? combiner))
 (whoops "Invalid first argument to ACCUMULATE (not a procedure):"
 combiner))
 ((not (empty? stuff)) (real-accumulate stuff))
 ((member combiner (list + * word se)) (combiner))
 (else
 (whoops "Can't accumulate empty input with that combiner"))))))

whoops in an error-handler function.

Example of use:

> (load "simply.scn")
> (define (acronym phrase)
 (accumulate word (every first phrase)))
> (acronym '(american civil liberties union))
aclu

As such, the reader doesn’t necessarily get into Scheme’s subtleties until chapter 3 or 4 maybe. The very first chapters are here to illustrate some general concepts from computing science: what a statement is, data structures such as lists, sets, the logic behind basic algorithms (e.g., permutations, factorial), branching patterns and function composition. The concepts of predicate and special form only appears in chapter 6, but that’s not a big deal since the authors favour an approach based on examples and introduce important concept one after the other (usually, one or two main concepts per chapter). Chapter 9 introduces lambda expression, and things start to get more interesting, especially when discussing named and anonymous functions.

An even more powerful use of lambda is to provide the value returned by some procedure that you write.

Chapter 10 deals with the Tic-Tac-Toe game and the author discuss how to solve the problem using reasoning and Scheme techniques available so far. This makes heavy use of helper functions available in simply.scm.

Next comes some chapters dealing with recursion, with complete functions defined along the way. As an example, consider the following snippet:

(define (evens sent)
 (if (<= (count sent) 1)
 '()
 (se (first (bf sent))
 (evens (bf (bf sent))))))

This function takes a sentence as its argument and returns a sentence of the even-numbered words of the original sentence. The authors write three versions of the same procedure before offering the best option. They later discuss the case of simplifying base case in recursive functions. The last chapter on recursion presents several functions written by the authors (e.g., every, keep, accumulate) and that were used previously. I found that this helps thinking back to previous exercises and how they could possibly be solved using the highlighted procedures.

The authors sometimes use diagrams or illustrations to explain the behavior of some functions. Most of the examples discussed in the book deal with the manipulation of strings. All chapters end up with a series of exercises, some of which are just applications of what have been discussed in the chapter. There are also mini-projects at the end of some chapters. Together with the heuristic or example-based approach, this reminds me of Practical Common Lisp, by Peter Seibel.

I wish I learned about this book earlier because it is both a good read (I like the authors’ writing style) and a valuable tool for those who want to practice in Scheme. Don’t stop at your first impression after chapter 4, things start to get really interesting with Parts III and IV. Reader of The Little Schemer may enjoy this book as well, especially the exercises and mini-projects. The only caveat is that following the textbook requires the authors’ toolbox, and not only a base Scheme installation – however, given that some of the concepts discussed in the book are quite advanced, it is somewhat expected to provide the reader with some additional tools as the authors acknowledge, which would otherwise deserve several chapters to expose.

♪ Led Zeppelin • You Shook Me

Nyxt V3

Sat, 20 Aug 2022 12:38:06 +0200

Two years ago I presented the Nyxt browser, and how I thought at that time it fit well in my text-oriented workflow. This is still the case, although I had neglected this browser a bit because of big web developments these last twenty months. Now I’m back to it as my daily driver for browsing, using the v3 (pre release), after trying out v2.2.4 for a week or so. I must admit it changes the surfing experience quite a bit, but it’s all good.

Last time, I only scratched the surface of this wonderful app. Let’s go into the details of what I like:

it’s quite fast, and largely less bloated than most other web browsers
keyboard shortcuts are useful and well thought (both for Vi and Emacs users)
you can easily jump to headings or opened buffers using dedicated panels, with fuzzy search
if you’re a Lisp user, you also get an REPL for free (I didn’t try it, though)
fuzzy search is available for almost everything (switching buffer, history, commands)

Vim users will get the comfort of modal mappings for free, and the normal mode can be defined for all but text input (e.g., input box in web document, status prompt in Nyxt). The Nyxt team even defined some g mappings, like in Vim. As I said in the other post, I can simply yt and yu to get the title and URL of a web page in my primary clipboard. It doesn’t look like much but it saves so much copy/paste back and forth between my text editor and the browser. The shortcut yy is used to copy (yank in Vim parlance) text, but you can safely add a C-c shortcut in addition.

Here is a list of shortcuts I use in my daily browsing activities using Nyxt:

Mapping	Mode	Command	Role
`C-w`*	`buffer-listing-mode`	`buffers-panel`	list all buffers in a separate buffer
`C-h`*	`history-mode`	`list-history`	list history in a separate buffer
`H`	`history-mode`	`history-backwards`	go backward in history
`L`	`history-mode`	`history-forwards`	go forward in history
`f`	`hint-mode`	`follow-hint`	prompt for hint and follow in current buffer
`F`	`hint-mode`	`follow-hint-new-buffer-focus`	prompt for hint and follow in new buffer
`/`	`search-buffer-mode`	`search-buffer`	search in current buffer
`C-j`	`download-mode`	`list-downloads`	show all dowload in a separate buffer
`+`	`document-mode`	`zoom-page`	zoom in
`-`	`document-mode`	`unzoom-page`	zoom out
`C-d`*	`document-mode`	`scroll-page-down`	scroll down (one page height)
`C-u`*	`document-mode`	`scroll-page-up`	scroll up (one page height)
`gh`	`document-mode`	`jump-to-heading`	jump to a given heading (H1, H2, etc.)
`}`	`document-mode`	`next-heading`	jump to next heading
`{`	`document-mode`	`previous-heading`	jump to previous heading
`C-c`*	`document-mode`	`copy`	copy highlighted text to clipboard
`o`		`set-url`	open URL in current buffer
`O`		`set-url-new-buffer`	open URL in new buffer
`D`		`delete-current-buffer`	close buffer
`R`		`reload-current-buffer`	reload page
`yt`		`copy-title`	save page title to clipboard
`yu`		`copy-url`	save URL to clipboard
`gb`		`switch-buffer`	switch buffer using fuzzy completion
`]`		`switch-buffer-next`	switch to next buffer
`[`		`switch-buffer-previous`	switch to previous buffer

In addition to the above mappings where custom mappings are highlighted using an asterisk, I also use a custom shortcut (C-o) to open the current buffer in Firefox:

(define-command-global open-external-browser ()
 "Open the current url in Firefox"
 (uiop:run-program (list "firefox" (render-url (url (current-buffer))))))

As I said, Nyxt is great for basic navigation but when it comes to inspecting HTML or CSS elements I’m more comfortable using Firefox developer tools. Time will tell if I can manage to do even basic web dev tasks in Nyxt. Also, Nyxt 3 introduced some changes in how we defined our configuration files (config.lisp versus init.lisp) and how we address some components. Basic stuff I showed in my earlier post no longer works, unfortunately. You can start with Artyom Bologov’s config to get an idea of how it feels to customize Nyxt a bit. I used some of his macros to define my own shortcuts, for example.

♪ Kendra Morris • This Life

Low level psychology of goldfish

Fri, 19 Aug 2022 11:51:16 +0200

I just finished reading an interesting book (in French), by Bruno Patino, on web and social network-induced attention disorder, and more generally the economy of attention: La civilisation du poisson rouge (Grasset, 2019).

Le poisson rouge tourne dans son bocal. Il semble redécouvrir le monde à chaque tour. Les ingénieurs de Google ont réussi à calculer la durée maximale de son attention : 8 secondes. Ces mêmes ingénieurs ont évalué la durée d’attention de la génération des millenials, celle qui a grandi avec les écrans connectés : 9 secondes. Nous sommes devenus des poissons rouges, enfermés dans le bocal de nos écrans, soumis au manège de nos alertes et de nos messages instantanés.
Une étude du Journal of Social and Clinical Psychology évalue à 30 minutes le temps maximum d’exposition aux réseaux sociaux et aux écrans d’Internet au-delà duquel apparaît une menace pour la santé mentale. D’après cette étude, mon cas est désespéré, tant ma pratique quotidienne est celle d’une dépendance aux signaux qui encombrent l’écran de mon téléphone. Nous sommes tous sur le chemin de l’addiction : enfants, jeunes, adultes.
Pour ceux qui ont cru à l’utopie numérique, dont je fais partie, le temps des regrets est arrivé. Ainsi de Tim Berners Lee, « l’inventeur » du web, qui essaie de désormais de créer un contre-Internet pour annihiler sa création première. L’utopie, pourtant, était belle, qui rassemblait, en une communion identique, adeptes de Teilhard de Chardin ou libertaires californiens sous acide.
La servitude numérique est le modèle qu’ont construit les nouveaux empires, sans l’avoir prévu, mais avec une détermination implacable. Au cœur du réacteur, nul déterminisme technologique, mais un projet qui traduit la mutation d’un nouveau capitaliste : l’économie de l’attention. Il s’agit d’augmenter la productivité du temps pour en extraire encore plus de valeur. Après avoir réduit l’espace, il s’agit d’étendre le temps tout en le comprimant, et de créer un instantané infini. L’accélération générale a remplacé l’habitude par l’attention, et la satisfaction par l’addiction. Et les algorithmes sont aujourd’hui les machines-outils de cette économie…

It is a quick read, go for it if you are interested in the psychology of conditioning and addictive or compulsive behaviors. Overall, the facts reported and discussed by the author agree with what I started feeling a few years ago. Figures speak for themselves, the style is clear, concise and punchy, which is not surprising from a journalist (specialist in new technologies). It would almost make me regret having thrown away most of my psychology books.

TL;DR We spend way too much time on the internet or social networks, for nothing. I mean, there’s no intellectual reward to compulsive activity on those medias, but this has been known for a long time by now. Still, many people and not only teenagers continue this desperate quest toward a virtual world where they may have a more interesting place – where “interesting” is only defined from one point of view, meaning this expectation is the only intra-personal aspect of those compulsive behaviors; the rest is sociology of crowds, or sociology, as you like to call it. That’s my personal take, of course. You may want to take a look at the more detailed perspective of the author who provides compelling evidence of the addiction induced by the raise of social web medias over time.

After four years of “web cleansing”, I am now free of Google, Twitter, Stack Exchange, Signal, Discord, and many other social networks. I can enjoy my hours spent in front of my computer with nothing else to do but produce material, certainly not always useful, but at least staying away from the potential distractions induced by a permanent connection to social networks and the internet. Don’t get me wrong, I spent some time there too in the past. My first phone was a Blackberry, you remember that small device that blinked as soon as you had a new email because Blackberry, the company, had push technology for its email system at the time. And I loved it, I really did. Now I just want to be left alone, and not be disturbed while I’m trying to concentrate.

The main point is that I always spend a lot of times on the internet, but now it is an asynchronous way of using the intertube. I check IRC channels from time to time, when I’m not engaged in public or private conversations, and Matrix is great for that very specific purpose since you get message logs for free – no need for a bouncer anymore. I also read blogs, via random searches on DDG or my loyal newsreader. But that’s about it.

Likewise, I watch TV shows on Apple TV or Netflix, but it’s limited to one or two episodes per night, and not every night. The downside is that I pay the rent for the year. If you don’t want to be brainwashed by tons of commercials on TV these days that’s the price you have to pay to have the choice to watch a series or a movie without being disturbed. I’m willing to pay in this case.

And now, let’s go back to reading good old printed books in hard copy.

♪ Joy Division • Transmission

Split-apply-combine in Scheme

Thu, 18 Aug 2022 15:57:09 +0200

I’ve been a devotee of the “split-apply-combine” way, popularized in the plyr (and legacy) package, of performing data operations in R. In its simplest form, and using base functions only, it amounts to:

data(crabs, package = "MASS")
spl = with(crabs, split(BD, sp))
apl = lapply(spl, mean)
cbn = rbind(x = apl)
cbn

Using plyr, this would simply be ddply(data = crabs, .(sp), summarize, x = mean(BD)). Related tools include subset and transform, and higher-order variations thereof (Filter, Reduce, and Map). I’ve seen few people using the later, but it really was not so common back in the days. Maybe David Springate talked about those functional recipes in one of his talks or tutorials, I don’t remember quite well, although I’m sure he blogged about functional programming in R. Thomas Mailund did this too. If I were to resume the above aspects of data processing using keywords that bear some resemblance with functional PLs, I would say this all amounts to filter, apply (or map), reduce and collect (the later is specific to Common Lisp, though).

On a related point, Eli Bendersky cites Ben Vandgrift and Alex Miller (Clojure Applied) in one of his Clojure-related blog post. Look at the following piece of code:

(defn revenue-by-department [carts]
 (->> (filter :settled? carts)
 (mapcat :line-items)
 (map line-summary)
 (group-by :dept)
 (reduce-kv dept-total)))

Doesn’t it look like what we discussed above?¹ The point of Eli Bendersky is that soon or later, when processing large amount of data, we need the power and expressiveness of SQL-inspired queries:

Applications that process and extract useful bits of information from large datasets all look alike in many programming languages, at least to some extent. What we really want in many cases is SQL-like primitives built into our languages, but this is often challenging (.NET’s LINQ is one example of a successful approach).

Of course, filtering rows or columns, or aggregating numerical quantities with or without grouping first are all builtin in DSLs for statistical computing (R, Stata, LispStat, to name a few). And there have been some attempt at producing reliable Lisp or Scheme packages for data munging. This will probably de dealt with in a future post, especially for Common Lisp and Racket.

In the mean time, since I’m reading a lot of Scheme books at the moment, I found a close analogy to this approach in chapter 8 (Higher-order functions) of Simply Scheme, where the author explains and illustrates the behavior of three procedures:

Function	Purpose	First argument
`every`	transform	one-arg transforming function
`keep`	select	one-arg predicate function
`accumulate`	combine	two-arg combining function

In this case, every will usually play the role of the R transform function, or Map; keep acts as a filter and does the job of (part of) subset or Filter; finally, accumulate resembles to many of R’s builtin functions (e.g., cumsum) and more generally acts as a reducer (Reduce). It is the combination of those higher-order functions that makes Scheme so powerful, as in the following examples:

(every square (keep even? '(87 4 7 12 0 5)))
(accumulate word (keep vowel? (every first '(and i love her))))

♪ Keith Jarrett & Charlie Haden • Goodbye

Clojure has been on my radar for 8 years now. Long time ago, I bought dozens of books, from which I barely scratched the surface. I never got into it really, except for writing toy examples based on my knowledge of Common Lisp and Scheme, but in a few months I think it will become my Lisp of choice for more serious development. ↩︎

Zathura viewer

Wed, 17 Aug 2022 10:51:22 +0200

Last year I briefly mentioned Zathura as my default PDF viewer. Since my last upgrade to Ubuntu 22.04 LTS, I reverted back to Gnome default settings for most things. I still have Zathura defined as my PDF viewer for BibTeX entries¹ and TeX documents, and I customized it a lot since then. I just reenabled it as my default PDF/PS viewer for a couple of reasons:

a single plain text configuration file
support for EPUB format
a nice index accessible via tab
easy bookmarks
synctex support for $\LaTeX$ editing in Neovim with vimtex
copy/pasting via mouse selection
keyboard-driven navigation, including zooming, adjusting, and page navigation
a jump list like in Vim (C-o/C-i)
and many other Vim-like features accessible form, e.g., the input bar.

Here are my personal settings in $HOME/.config/zathura/zathurarc:

set default-bg "#2E3440"
set default-fg "#D8DEE9"
set statusbar-fg "#D8DEE9"
set statusbar-bg "#2E3440"
set index-bg "#2E3440"
set index-fg "#D8DEE9"
set index-active-bg "#81A1C1"
set index-active-fg "#D8DEE9"
set inputbar-bg "#2E3440"
set inputbar-fg "#D8DEE9"
set notification-warning-bg "#EBCB8B"
set notification-warning-fg "#2E3440"
set highlight-color "#81A1C1"
set highlight-active-color "#81A1C1"
set completion-highlight-fg "#BF616A"
set completion-highlight-bg "#2E3440"
set completion-bg "#2E3440"
set completion-fg "#D8DEE9"
set notification-bg "#D8DEE9"
set notification-fg "#2E3440"

set recolor-darkcolor "#D8DEE9"
set recolor-lightcolor "#2E3440"
set recolor "true"

set incremental-search "true"
set adjust-open "best-fit"

set window-title-home-tilde "true"
set selection-clipboard "clipboard"

set statusbar-basename "true"
set statusbar-home-tilde "true"
set statusbar-h-padding "5"
set statusbar-v-padding "5"

set window-height 3000
set window-width 3000

set save_position "true"
set synctex-editor-command "nvim"

set font "Ubuntu Mono 13"

map <C-b> feedkeys ":bmark "
map D set "first-page-column 1:1"
map <C-d> set "first-page-column 1:2"

The color scheme follows the Nord dark theme, and it is on by default. If I want a light background as in Evince, I can C-r to disable recoloring. The window height and width are defined such that Zathura looks maximized on my screen. I think the remaining options speak for themselves.

♪ Jewel • Hands

I wrote a helper function in Vimscript that can xdg-open the PDF or Postscript or DJVU file associated to a BIbTeX entry, which I can trigger using gf shortcut.

function! text#open_bibtex_key() abort
 let s:uri = split(split(expand('<cWORD>'), '{')[1], ',')[0]
 if s:uri !=# ''
 silent exec "!xdg-open ~/Documents/papers/'".s:uri."'.* &"
 else
 echo 'No match found.'
 endif
endfunction

↩︎

Starship

Tue, 16 Aug 2022 14:20:29 +0200

I’ve been a heavy Zsh users for years. I tried Fish some years ago, and although it was a great experience I came back to Zsh at some point, especially given that I could get autosuggestions as well in a POSIX-compliant shell. On deskto.ps I noticed a very nice setup and I wondered WTF is this.

I spent some time configuring my Zsh prompt the way I like it to be: responsive and informative, no fuzzy stuff. The one we can see in the above screenshot looks too verbose to me, but that’s just me. My prompt-related Zsh mess can be viewed online, and it looks like this:

Specifically, I highlight Python virtual environment, command duration (when > 5s) and exit code, as well as Git branch with overall repo status. I don’t need to indicate whether files were added, modified or deleted, or any departure from a remote branch; I just want to know whethert he repo is clean or dirty. I use the same idea in my Neovim modeline.

Enters starship, which is supposed to be a

minimal, blazing-fast, and infinitely customizable prompt for any shell!

Being one of these blazing fast CLI utilities developed in Rust (think exa, fdfind, etc.), I thought I would give it a try. My idea was that since it only requires a single config file (in addition to the executable itself, which can be obtained using curl -sS https://starship.rs/install.sh | sh) I could reuse my setup on other machine quite easily.

I was able to reproduce my handmade Zsh prompt using the following settings shown below. The only difference is that I kept information regarding the virtual environment and Python version in the left prompt, and put evry transient information in the right prompt:

format = '$username$directory$python$haskell$git_branch$git_status$character'
right_format = '$cmd_duration$jobs$sudo'
add_newline = true

[character]
success_symbol = "[%](bold blue)"
error_symbol = "[%](bold red)"

[directory]
style = "bold blue"

[jobs]
symbol = '+'
format = '[\[$symbol$number\]]($style) '

[cmd_duration]
min_time = 5_000
format = "[$duration]($style) "
style = "bright-black"

[git_branch]
format = "[$branch(:$remote_branch)]($style)"
style = "bold purple"

[git_status]
format = "([$all_status$ahead_behind]($style)) "
conflicted = "!"
modified = "*"
ahead = ""
behind = ""
diverged = ""
untracked = ""
staged = ""
deleted = ""
renamed = ""
style = "bold purple"

[sudo]
symbol = "#"
format = '[$symbol]($style) '
style = "bold red"
disabled = false

[haskell]
symbol = ""
format = '[\[hs v$symbol($version)\]]($style) '
style = "bold yellow"
detect_extensions = ["cabal", "hs-boot"]

[python]
format = '[\[py ${pyenv_prefix}(${version})(\($virtualenv\))\]]($style) '
version_format = "v${major}(.${minor})"
detect_extensions = []
style = "bold yellow"

Cleaner, simpler to manage, no need to deal with Zsh precmd or use circomvoluted idioms to test whether we are in an SSH session or not. More to the point, once starship prompt is configured, it can be reused by any supported shell, including Fish.

The only thing that I wasn’t able to reproduce from my sh settings is that I used to use a subtle trick to show when there were multiple jobs running in the background: rather than displaying a symbol or the number of jobs in my left prompt, I just changed the color of the prompt character itself (from blue to orange when there’s at least one hidden job; likewise, I used a red color to indicate that the previous command exited with no zero return code). Bonus point: the right prompt doesn’t move when you resize your terminal, that is it remains right-aligned and does not stay in the middle of nowhere after you reduced then maximized your terminal window.

♪ The Jesus and Mary Chain • Darklands

Of signals and trap

Mon, 15 Aug 2022 22:54:54 +0200

I was reminded of signals in shell scripts, after reading some arcane Bash code a few months ago. What are signals, and how can they be used in shell scripts? The kernel sends signals to running processes about ongoing events. If you type trap -l in your terminal you’ll a list of all available signals, with both their number and their name. Either one can be used to reference a given signal. See the Bash manual for more information. Note that trap is also aware of ERR signals that occur when programs like tar exits with non zero status code.

I don’t remember what script caught my attention, but the gist of it is that you can easily monitor SIGHUP or SIGINT signals in a shell script, as you would do with other environmental variables or user input.

Without further ado, here’s a sample script which catch up eventual user interrupt (Ctrl-C) when an external program is running :

#!/usr/bin/env bash

trap "echo User interrupt; exit" SIGINT

ghc -e 'iterate (\x -> ((1103515245 * x) + 12345 `mod` 2^31)) 1'

Whatever follows trap can also be wrapped in a Bash function. And of course, you can capture more than one signals.

Of note, the congruential generator above is not great of course.¹ To generate an infinite sequence of random number generator, consider the more elaborated solution below:²

import System.Random

main = do
 g <- newStdGen
 (randoms g :: [Double])

A similar approach is discussed in this blog post. Other options are discussed on Stack Overflow. Another nice usage of the trap command is to clean up intermediate or temporary files in case the program crashes. See an exemple in this article.

I don’t have a clear idea of how to use trap for anything really involved in my Bash script, but I will surely consider using trap to clean up messy files from now on. In particular, I think the foloowing might be useful near the top of some of my Bash files (e.g., before carrying out large bioinformatics stuff), to offer a proper garbage collector to my scripts:

trap "rm -f /tmp/out" EXIT SIGINT

♪ Keith Jarrett • You’ve Changed

It was the original BSD rand() function, though. But see Random numbers fall mainly in the planes (PDF), by Marsaglia for a discussion of linear congruential generators. ↩︎
You will likely need to install the random package since it does not come with default install of Stack or GHCup. ↩︎

GHCup and Haskell tooling in 2022

Sat, 13 Aug 2022 21:23:04 +0200

For my Haskell toy scripts, I used to rely on Stack since Chris Done and Alexis King, that I believe to be highly respectable in the field, recommend it. I only write small scripts, like I write small Scheme scripts. Maybe I should consider using those languages as my default scripting tools, but see Why Haskell became my favorite scripting language.

Up to now, Stack was a great option : You can install global packages easily, so that System.Random is available after you launch a REPL with stack ghci, or you can manage small projects using stack directly. Getting HLS to work with external or local projects was of great service after years of hazards regarding Haskell’s proper LSP plugin. Once properly configured, Neovim with its builtin LSP facilities makes writing Haskell code really enjoyable, especially thanks to the hints and linting/formatting facilities.

Two or three months ago I heard about GHCup, which might be seens as what rustup is to Rust. Now, with only a single tool you’ll get everything you need, including ghc, cabal, stack and hls, which you no longer need to compile or download as a release tarball from Github every month or so.

After you installed GHCup for the first time, you should have a working Haskell stack on your machine. You can upgrade or update at any time, much like you would do with rustup:

% ghcup list
% ghcup upgrade
% ghcup install ghc 9.4.1
% ghcup install cabal 3.8.1.0

If you’re a Stack user, you can continue working as usual, you just got HLS for free.

♪ Keith Jarrett • Never Let Me Go

Recently on the commonplace book

Thu, 11 Aug 2022 13:30:56 +0200

I spent a little time reading Matthew Lee Hinman’s current blog, and I came across two interesting articles. The first one, Blogging Shift, is about blogging per se, which admittedly is in decline although there are up and down from one year to another. The author summarizes my own take on that particular matter, namely that the media has changed, along with the way we consume information. The internet, and more generally non-verbal communication, is not an all text place nowadays, and it has been largely impacted by the advent of social networks together with the (over)use of overpowered phones. Moreover, there’s this irrepressible need for some to give their opinion on everything and to follow sort of a normative approach. I left Twitter a few weeks ago, and I don’t regret it. It’s fallen into decay for years, IMHO, and I don’t see any value in checking in even once a week.

This is like a post-experience survey asking you to rank your experience on a scale of 1-to-5. It says “here is a pre-cut reactionary channel for some experience, it would be easiest to follow the template” and by doing so, limits the scope of considered responses. Simultaneously the pre-cut reactions provided to users by most content brokers (of which social networks are one of the largest) skew towards the passive—I’m registering my like or dislike in something with minimal effort—rather than the creationary. The bar for entry of opinion into the public sphere has been lowered at the cost of the diversity in the type of reactions.

Another blog post that attracted my attention has to do with what I though was known as the Zettelkasten note-taking way of archiving personal knowledge, but see How to Take Smart Notes. It happens to have yet another (older) name: a commonplace book. I did not go into the details to distinguish the etymology or the mode of data collection. However, I really like the idea of organizing ideas, thoughts or readings into a bulk of plain text notes, like much Org users I believe. I’m not really an Org user, but I do use Org for writing my notes and handouts. I once started collecting short notes into a master file, but these days I rarely find time to update it. Rather, I guess this blog entries serve as my commonplace book as well.

They differ from journals in that they aren’t necessarily a recording of personal action (though they can be), and they aren’t necessarily prone to introspective writing (though they may), but are more of a scrapbook of sorts, collecting ideas, quotations, diagrams, thoughts, observations, or anything else that may be useful for reference in the future. There is no prescribed format, and the author is also the intended reader, though in some cases, they have been published publicly.

Contrary to the author, I don’t get much from writing those notes, though. I mean, when I write some code to perform a task, I learn from writing the code itself, debugging and so on; but not from summarizing my experience into a blog post. However, when I need to find an old note or blog post, I usually have an idea of where to start, or what keywords to throw out in my terminal to retrieve that information.

♪ Hubert-Félix Thiéfaine • L’étranger dans la glace

Life in a few words

Tue, 09 Aug 2022 11:07:46 +0200

Just out of curiosity, I made a quick wordcloud using Mathematica with very few normalization (capital letters, punctuation characters, stop words, short words, etc.) to summarize some personal stuff I wrote during the last 15 years or so, and I thought I will just put it right there.

My very basic script is shown below: (More solutions are available on Stack Exchange.)

data = Import["~/Documents/home/writings/combined.txt"];
junk = {"comme", "quand", "enfin", "lorsque", "après", "jusqu",
 "sinon", "aujourd", "autour", "entre", "aussi", "toute", "petit",
 "avoir", "encore", "aurais", "étais", "avais", "juste", "leurs",
 "parce"};
cleanedData =
 StringReplace[#, RegularExpression[
 "\\b[0-9]+[/-][0-9]+([/-][0-9]+)?\\b"] -> ""] & /@
 DeleteStopwords[ToLowerCase[TextWords[data]]];
cleanedData =
 Select[Flatten[
 StringSplit[
 StringReplace[cleanedData, PunctuationCharacter -> " "]]],
 StringLength[#] > 4 && # != "comme" &];
cleanedData = DeleteCases[cleanedData, x_ /; MemberQ[junk, x]];
WordCloud[cleanedData, Disk[],
 ColorFunction -> ColorData["AtlanticColors"]]

The sieve for junk words was quickly devised by trial and error. Nothing’s perfect to be expected.

♪ Bod Dylan • Changing of the Guards

July in review

Tue, 02 Aug 2022 11:02:22 +0200

In which I summarize what I’ve read recently on the intertube.

I haven’t touched my Neovim config during 30 days, except for minor adjustments of my custom mappings. Well, it may well be because I was on holidays as well. Anyway, things start to come up right, I think. The more stuff I delete, the more it let me think about internal stuff, much like in this nicely set up blog post by Joe Nelson that I already mentioned somewhere on this site: History and effective use of Vim. Likewise, I reverted back to Ubuntu defualt settings (default display manager, defualt font, etc.), and it has been working great for me.
TIL about whonix, which is a full operating system, including common chat and office applications, that runs inside your current one via Tor, with added security layers. It might be a good way to get a secured OS on a USB stick.
If you’re looking for lightweight and minimalist, stop by the Inconsolation blog. You will likely find useful suggestion for old-fashioned but resilient software for pretty everything. But see this blog post if you look for a different take on so-called digital minimalism.
The Ghostscript suite got a new PDF interpreter. It’s been a while since I no longer rely on DVI+PS conversion for $\LaTeX$ typesetting – maybe I should, at least for the quality of the fonts in print form – but I still use Ghostscript utilities from time to time. A recurrent command I use if gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -dAutoRotatePages=/None -sOutputFile=merged.pdf *.pdf to assemble a series of individual PDF files.
Differentiable Programming from Scratch covers the basics of numerical and symbolic differentiation, as well as automatic differentiation, with lot of pretty illustrations.
An old but good post about page weight, by Chris Zacharias: Page Weight Matters. For a related discussion, check out Tom MacWright’s opinion on web design, especially his site redesign.
Clojure needs a Rails, but not for the reason you think. If I were to use a functional language to write my genomic data apps, I would certainly chose either Clojure or a combination of Elm+Elixir. I’m surpised there’s nothing already for starting right away with Clojure. I remember the Noir framework, but well it looks like 10 years old and unmaintained.
I always reading “naturalistic” data science or web scraping on real data. Here’s the most recent one: Famous HNers and Their Sites.
Reference Count, Don’t Garbage Collect: An interesting overview with pros and cons of reference counting vs. garbage collection.
The best average time complexity of Timsort is $\mathcal{O}(n)$. Hard to beat. See more in Timsort – the fastest sorting algorithm you’ve never heard of.
How a window manager with tiling layouts makes a 1024x600 resolution livable. And, yes, he lives in a cabin. Since I’m now on Wayland, I may try Sway at some point, but see Trying out Sway and Wayland.
SQLite Internals: Pages & B-trees: A technical review of SQLite internals. Pretty good job!
Flipping until you are lost: A summary of a recent paper on convex polygon triangulation flip walk (arXiv:2207.09972).

Still, the purists proclaim, it’s not enough. Python is not a replacement for Haskell. But does it matter? 90% of the impressive magic from early functional languages has been rolled into mainstream languages. That last 10%, well, it’s not clear that anyone is really wanting it or that the benefits are actually there. Purity has some advantages, but it’s so convenient and useful to directly modify a dictionary in Python. Fold and map are beautiful, but they work just as well in the guise of a foreach loop. — Functional Programming Went Mainstream Years Ago

♪ Deep Sea Diver • Shattering the Hourglass

More Tmux little hacks

Fri, 29 Jul 2022 21:54:57 +0200

A few months ago I learned that we can add popup menus to Tmux, using display-menu. I rarely use it, but I find it convenient. The more I learn about those little hacks in Tmux the more I like it. It’s worth all modern terminals by far.

Here’s mine:

bind-key -n M-q display-menu -x W -y S \
 "New Session" S "command-prompt -p \"New Session:\" \"new-session -A -s '%%'\"" \
 "Kill Session" x "kill-session" \
 "Kill Other Session(s)" X "kill-session -a" \
 "" \
 "New Window" ␍ new-window \
 "Kill Window" k "killw" \
 "Choose Window" w choose-window \
 "Previous Window" 🡠 previous-window \
 "Next Window" 🡢 next-window \
 "Swap Window Right" ↑ "swap-window -t -1" \
 "Swap Window Left" ↓ "swap-window -t +1" \
 "Horizontal Split" v "split-window -h" \
 "Vertical Split" s "split-window -v" \
 "" \
 "Layout Horizontal" h "select-layout even-horizontal" \
 "Layout Vertical" k "select-layout even-horizontal" \
 "" \
 "Swap Pane Up" < "swap-pane -U" \
 "Swap Pane Down" > "swap-pane -D" \
 "Break Pane" t break-pane \
 "Join Pane" j "choose-window 'join-pane -h -s \"%%\"'" \
 "#{?window_zoomed_flag,Unzoom,Zoom}" z "resize-pane -Z"

Copying/pasting in Tmux has always been a pain for me. In the end, I most of the time rely on what I call the “shift” selection and copy approach, meaning that you select text using your mouse while pressing the Shift key, and you copy the highlighted text using Shift+Ctrl+C. It works. Other than that, you can leverage Tmux’s facilities for copy-pasting text or whole buffers using the following shortcuts:

bind Y copy-mode
setw -g mode-keys vi
bind-key -T copy-mode-vi v send -X begin-selection
bind-key -T copy-mode-vi 'C-v' send -X rectangle-toggle
set -g set-clipboard off
bind-key -T copy-mode-vi y send -X copy-pipe-and-cancel "xclip -i -selection clipboard"
bind-key -T copy-mode-vi MouseDragEnd1Pane send-keys -X copy-pipe-and-cancel "xclip -i -selection clipboard"

In this case, you have two options: using the mouse to select text – that’s the MouseDragEnd1Pane snippet above, then you just Shift+Ctrl+V to paste the highlighted text wherever you want; this even works to paste text from a shell into a Vim buffer. Other than that, you can use the keyboard after pressing the Y key, then use Vim motion key to select your text – these are the very first keybindings in the above snippet.

If you don’t like window number in the status bar when you only have one window, you can add the following to your Tmux config:

set -g status off
set-hook -g after-new-window 'if "[ #{session_windows} -gt 1 ]" "set status on"'
set-hook -g pane-exited 'if "[ #{session_windows} -lt 2 ]" "set status off"'

It will first disable the status bar, and then rely on a hook to activate it when there’s more than one window available in the current session. I’ve been using this for years and it’s really great.

After much use, I found that binding the Alt key as a pseudo-prefix key is a must (it’s close to the Meta key used by Ubuntu). Here are some mappings I use almost every day:

bind-key -n M-! split-window -f -l 10 -c "#{pane_current_path}"
bind-key -n M-: command-prompt
bind-key -n M-< swap-pane -U
bind-key -n M-= choose-buffer
bind-key -n M-> swap-pane -D
bind-key -n M-D choose-client
bind-key -n M-Down swap-window -t +1
bind-key -n M-Enter new-window
bind-key -n M-Left previous-window
bind-key -n M-O switch-client -l
bind-key -n M-S command-prompt -p "New Session:" "new-session -A -s '%%'"
bind-key -n M-Right next-window
bind-key -n M-Up swap-window -t -1
bind-key -n M-k select-layout even-vertical
bind-key -n M-j choose-window 'join-pane -h -s "%%"'
bind-key -n M-h select-layout even-horizontal
bind-key -n M-l last-window
bind-key -n M-m command-prompt -p "Search man pages for:" "new-window 'exec man %%'"
bind-key -n M-o selectp -t :.+
bind-key -n M-p run "tmux split-window -p 10 'tmux send-keys -t #{pane_id} \"$(locate ~/ | fzf -m | paste -sd\\ -)\"'"
bind-key -n M-s split-window -v
bind-key -n M-t break-pane
bind-key -n M-v split-window -h
bind-key -n M-w choose-window
# bind-key -n M-x confirm-before -p "kill-pane #P? (y/n)" kill-pane
bind-key -n M-y copy-mode
bind-key -n M-z resize-pane -Z

[2022-08-27]
See my Tmux cheatsheet for more up to date mappings.

[2025-01-06]
See also this little gem on Will Richardson’s weblog.

Happy hacking on Tmux!

♪ Jean-Yves Thibaudet • Liszt Consolations, S. 172 - No. 3 in D-Flat Major. Lento placido

Micro posting in July

Fri, 29 Jul 2022 18:43:03 +0200

This was a pretty calm month, but I also wrote a lot of longer blog posts.

2022-07-04: 25,011 scrobbles on Last.fm in 9 months.
2022-07-06: Alain Kan, Speed My Speed.
2022-07-07: Mathematical Python. #python
2022-07-08: Nick Cave & The Bad Seeds, The Mercy Seat.
2022-07-08:

Git is built on a graph. Almost every Git command manipulates this graph. To understand Git deeply, focus on the properties of this graph, not workflows or commands. — Git from the inside out

2022-07-08: Mathematical Tours.
2022-07-08: What Comes After Git.
2022-07-10: Clipboard integration between tmux, nvim, zsh, x11, across SSH sessions. #unix

♪ Clan of Xymox • A Day

Scheming with Vim

Wed, 27 Jul 2022 20:50:59 +0200

As a matter of fact, I use Neovim to wrote all my Chicken Scheme and Racket code. It works quite great, even if I do not have as much facilities as I had under Emacs with Geiser.

I followed the recommendations available on Chicken website, as well as Evan Hanson’s useful blog post, Editing Scheme in Vim, especially for completion. I found that setting keywordprg to chicken-doc scheme instead of chicken-doc is most of the times what I really need.

My after/ftplugin/scheme.vim is short enough that I can post it right in this post:

let b:is_chicken=1

setl makeprg=csc\ %

setl complete+=,k~/.local/share/scheme/words

setl include=\^\(\\(use\\\|require-extension\\)\\s\\+
setl includeexpr=substitute(v:fname,'$','.scm','')
setl path+=/home/chl/.cache/chicken-install
setl suffixesadd=.scm

setl lispwords+=let-values,condition-case,with-input-from-string
setl lispwords+=with-output-to-string,handle-exceptions,call/cc,rec,receive
setl lispwords+=call-with-output-file

fun! Scheme_indent_top_sexp()
 let pos = getpos('.')
 silent! exec 'normal! 99[(=%'
 call setpos('.', pos)
endfun

nmap <buffer> <silent> == :call Scheme_indent_top_sexp()<cr>
nmap <buffer> gs :10 split term://csi -s %<cr>:startinsert<cr>

setlocal keywordprg=chicken-doc\ scheme

The file ~/.local/share/scheme/words was generated as advertised by Evan Hanson. I just use a different directory to store Scheme keywords. I update the word list periodically, especially after I install new packages.

For the time being, I’m not interested in other Scheme flavor, so I decided to target Chicken scheme only. Together with Racket, it is a solid Scheme implementation that further comes with a package manager and a nice FFI.

[2023-10-27]
I highly recommend installing vim-paredit (still maintained despite being listed on vim-scripts). For some time I use to rely on nvim-parinfer, and now there’s nvim-paredit (and parpar if you prefer a combo). Tehre’s also vim-sexp, with awful mappings, and vim-sexp-mappings-for-regular-people, with far better mappings but a longer nick). I’ve tried them all, noticed some issues with nvim-parinfer when deleting enclosed forms, which led me to rely on Vim dib and dab instead. Finally, there’s nvim-treesitter-sexp, but it seems to be limited to “modern” Lisp languages (Clojure, Fennel and the like); I haven’t tried it. With vim-paredit you get part of parinfer, i.e., when you delete a line or an s-expression inside a top-level form in Lisp you get the correct result, and many additional goodies like slurping and barfing the easy way (< and >), join/slice/wrap lists and sublists, in normal or visual mode. As a final remark, I should note that there’s a treesitter grammar for Scheme (as well as Common Lisp, Racket, and Clojure), which means indentation and formating should be all right.

A final word: The gs mapping is something that I implemented for most languages that offer some kind of a REPL or a quick compilation process. So far, I have it implemented for the folloing filetypes:

~/.../after/ftplugin % rg "<buffer> gs"
tex.vim
29:nmap <buffer> gs :VimtexCompile<CR>

r.vim
5:nmap <buffer> gs :10 split term://Rscript %<cr>:startinsert<cr>

org.vim
10:nmap <buffer> gs :10 split term://org-babel -pdf % && zathura %:r.pdf<cr>:startinsert<cr>

c.vim
12:nmap <buffer> gs :10 split term://zig cc % && ./a.out<cr>:startinsert<cr>

rmd.vim
1:nmap <buffer> gs :w<CR>:cd %:p:h <CR>:!Rscript -e 'library(knitr);knit("%:p")'<CR>:!latexmk -pdf -bibtex-cond -f %:r.tex <CR>:!xdg-open %:r.pdf <CR><C [... 0 more matches]

rnoweb.vim
10:nmap <buffer> gs :w<CR>:cd %:p:h <CR>:!Rscript -e 'library(knitr);knit("%:p")'<CR>:!latexmk -pdf -bibtex-cond -f %:r.tex <CR>:!xdg-open %:r.pdf <CR><C [... 0 more matches]

scheme.vim
23:nmap <buffer> gs :10 split term://csi -s %<cr>:startinsert<cr>

lisp.vim
1:nmap <buffer> gs :10 split term://ros -l %<cr>:startinsert<cr>

stata.vim
3:nmap <buffer> gs :10 split term://stata -q %<cr>:startinsert<cr>

markdown.vim
9:nmap <buffer> gs :10 split term://pandoc -s --pdf-engine=lualatex --filter pandoc-fignos --filter pandoc-citeproc --listings % -o %:r.pdf && zathura % [... 0 more matches]

racket.vim
80:nmap <buffer> gs :10 split term://racket %<cr>

python.vim
8:nmap <buffer> gs :10 split term://python3 %<cr>:startinsert<cr>

haskell.vim
6:nmap <buffer> gs :10 split term://chmod +x % && ./%<cr>:startinsert<cr>

rust.vim
3:nmap <buffer> gs :10 split term://cargo run<cr>:startinsert<cr>

Note that I also use iron to launch some REPL right into a running Neovim instance.¹ It also works great with the interactive interpreter csi.

[2023-10-27]
No more plugin providing REPL-like features. If I were to install one, I would choose either slimv (almost surely) or conjure. If this is just to send a line or a selection to a terminal running my favorite interpreter, I stand by my poor man mappings: vim.keymap.set("n", "ss", [[ Vy<C-w>wpa<CR><C-\><C-n><C-w>pj ]]) (current line), vim.keymap.set("x", "s", [[ y<C-w>wpa<CR><C-\><C-n><C-w>p ]]) (visual selection).

Happy scheming in (Neo)vim!

♪ Nick Cave & The Bad Seeds • Rock of Gibraltar

I locked Neovim packer at commit bc9c596d6a97955f0306d2abcc10d9c35bbe2f5b, but that should not stop you trying it out. ↩︎

Welch t-test in Scheme

Wed, 27 Jul 2022 09:41:22 +0200

Background

For two independant samples, the Student test statistic is defined as follows:

$$ t_{\text{obs}}=\frac{\bar x_1 - \bar x_2}{s_c\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}, $$

with $s_c=\left(\frac{(n_1-1)s^2_1+(n_2-1)s^2_2}{n_1+n_2-2}\right)^{1/2}$, and where $\bar x_i$ and $n_i$ represent the means and sample sizes of the two samples, and $s_c$ is the pooled variance for the difference of means. Under $H_0$, this test statistic follows a Student distribution with $n_1+n_2-2$ degrees of freedom.

A $100(1-\alpha)$% confidence interval for the difference $\bar x_1 - \bar x_2$ can be defined as

$$ \bar x_1 - \bar x_2\pm t_{\alpha, n_1+n_2-2}s_c\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}, $$

with $P(t<t_{\alpha,n_1+n_2-2})=1-\alpha/2$.

The above assumes equal variances at the population level.¹ This hypothesis can be relaxed by approximating the degrees of freedom using a linear combination of the sample variances, $\tfrac{s_1^2}{n_1}+\tfrac{s_2^2}{n_2}$.² The test statistic still follows a Student distribution, but with $\nu$ degrees of freedom.

The Satterthwaite approximation considers:

$$ \nu = \frac{ \left( s_1^2/n_1 + s_2^2/n_2 \right)^2 } { \frac{\left( s_1^2/n_1 \right)^2}{n_1-1} + \frac{\left( s_2^2/n_2 \right)^2}{n_2-1} }. $$

The formula proposed by Welch is close to the above formulation:³

$$ \nu = -2 + \frac{ \left( s_1^2/n_1 + s_2^2/n_2 \right)^2 } { \frac{\left( s_1^2/n_1 \right)^2}{n_1 + 1} + \frac{\left( s_2^2/n_2 \right)^2}{n_2 + 1} }. $$

Application in Scheme

Here is an simple application in Scheme. It follows the approach proposed by Larry Hunter in his Statistical functions in Common Lisp (Version 1.01 July 27, 2001), except that we don’t use a lookup table for the p-value as can be found in certain versions available on the web. Instead, we use the incomplete Beta distribution discussed in previous posts which is available from the GNU Scientific Library.

I will use Chicken Scheme since it is quite easy to use its FFI facilities (see, e.g., this post on the Murmurhash algorithm). For CL users, Matt Curtis provide a standalone incomplete beta function in his port of Larry Hunter’s code. Fortunately, there’s also some Chicken code for the original Lisp code, provided by Peter Lane. Here’s the gist of it, using the interpreter (csi):

> (import statistics)
> (t-test-two-sample 12 2 24 11 1.5 24 #f)
0.0561191363596448

Compare to the two-tailed p-value returned by Stata:

. ttesti 24 12 2 24 11 1.5, unequal

Two-sample t test with unequal variances
------------------------------------------------------------------------------
 | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]
---------+--------------------------------------------------------------------
 x | 24 12 .4082483 2 11.15547 12.84453
 y | 24 11 .3061862 1.5 10.36661 11.63339
---------+--------------------------------------------------------------------
combined | 48 11.5 .2627511 1.820393 10.97141 12.02859
---------+--------------------------------------------------------------------
 diff | 1 .5103104 -.0293791 2.029379
------------------------------------------------------------------------------
 diff = mean(x) - mean(y) t = 1.9596
Ho: diff = 0 Satterthwaite's degrees of freedom = 42.6558

 Ha: diff < 0 Ha: diff != 0 Ha: diff > 0
 Pr(T < t) = 0.9717 Pr(|T| > |t|) = 0.0566 Pr(T > t) = 0.0283

Now we would like to compute a 95% asymptotic confidence interval. According to Stata, using Satterthwaite’s approximation (42.7 df) it amounts to $[-0.0294;2.0294]$. Under the equal variance assumption (46 df), it is $[-.0272;2.0272]$. Not much of a difference between the two CIs. Here are the values computed using Scheme:

> (define (pooled-sd s1 n1 s2 n2)
 (sqrt (/ (+ (* (- n1 1) (* s1 s1))
 (* (- n2 1) (* s2 s2)))
 (+ n1 n2 -2))))
> (define sp (pooled-sd 2 24 1.5 24))

We need to be careful when using the one-sample confidence interval routine (normal-mean-ci), since it needs to get the right degrees of freedom. IT works great in the case of univariate statitics. The function reads:

(define (normal-mean-ci mean sd n alpha)
 (let ((t-value (t-distribution (- n 1) (- 1 (/ alpha 2)))))
 (values (- mean (* t-value (/ sd (sqrt n))))
 (+ mean (* t-value (/ sd (sqrt n)))))))

In this case, the degree of freedom is the sample size minus 1, and the standard error is computed as (/ sd (sqrt n)), which is defintely not what we want. The estimated standard error of the difference of means is the pooled SD, sp, times $\sqrt{1/24 + 1/24}=\sqrt{1/12}$, in the case of equal variances. Hence the correct 95% CI is computed as follows:

> (define se (* sp (sqrt (/ 1 12))))
> (define qt (t-distribution 46 0.975))
> (values (- 1 (* qt se)) (+ 1 (* qt se)))
-0.0272016886019351
2.02720168860194

Using separate variances, we would use the following standard error to compute te 95% CI:⁴

> (define se (sqrt (+ (/ (* 2 2) 24) (/ (* 1.5 1.5) 24))))
> (define nu (/ (square (+ (/ (* 2 2) 24) (/ (* 1.5 1.5) 24)))
 (+ (/ (square (/ (* 2 2) 24)) (- 24 1))
 (/ (square (/ (* 1.5 1.5) 24)) (- 24 1)))))
> (define qt (t-distribution nu 0.975))
> (values (- 1 (* qt se)) (+ 1 (* qt se)))
-0.0293791829211965
2.0293791829212

This agrees with Stata’s summary table.

Of course, it would be better to embed the computation of the confidence intervals into the testing procedure, which actually returns the p-value only.

♪ Caroles Daughter • My Mother Wants Me Dead

See the Behrens-Fisher problem. ↩︎
F. E. Satterthwaite (1946). An Approximate Distribution of Estimates of Variance Components. Biometrics Bulletin, 2, 110-114. ↩︎
B. L. Welch (1947). The Generalization of Student’s Problem When Several Different Population Variances Are Involved. Biometrika, 34, 28-35. ↩︎
See William D. Dupont (2009). Statistical Modeling for Biomedical Researchers (2nd ed.), Cambridge (p. 36). ↩︎

NewLISP and memoization

Tue, 26 Jul 2022 11:37:04 +0200

Surely, the Fibonacci recursive algorithm that is presented everywhere serves no purpose other than introducing the recursive way of thinking,¹ but anyway I’m here at writing another snippet of Lisp code to compute Fibonacci numbers. The real problem with those algorithms is that we are going to compute the same value over and over again. One solution is to rely on memoization, and in a previous post I discussed how to do so in Racket.

Using typed Racket, I wrote:

(: fib (-> Integer Integer))
(define fib (memoize (lambda ([n : Integer])
 (if (< n 2) 1
 (+ (fib (- n 1))
 (fib (- n 2)))))))

Now, it appears that newLISP, which I’m playing with for the Summer, also comes with a memoization macro:²

(define-macro (memoize mem-func func)
 (set (sym mem-func mem-func)
 (letex (f func c mem-func)
 (lambda ()
 (or (context c (string (args)))
 (context c (string (args)) (apply f (args))))))))

(define (fibo n)
 (if (< n 2) 1
 (+ (fibo (- n 1))
 (fibo (- n 2)))))

(memoize fibo-m fibo)

The later expands to:

(lambda-macro (mem-func func) (set (sym mem-func mem-func)
 (letex (f func c mem-func) (lambda () (or (context c (string (args))) (context
 c
 (string (args))
 (apply f (args))))))))
(lambda (n)
 (if (< n 2)
 1
 (+ (fibo (- n 1)) (fibo (- n 2)))))
(lambda () (or (context fibo-m (string (args))) (context fibo-m (string (args)) (
 apply fibo
 (args)))))

Benchmarking both functions is shown below:

> (time (fibo 30))
339,198
> (time (fibo-m 30))
320,943
> (time (fibo-m 30))
0,012

The execution time is almost instantaneous on the second run. Note, however, that it does not affect the first run, ince it first need to use a dedicated namespace to store intermediate results from recursive function calls.

♪ Cigarettes After Sex • Dreaming Of You

The same applies to the factorial, see, e.g., what’s up with the factorial ? ↩︎
The online documentation of NEWlisp reports that “when memoizing recursive functions, include the raw lambda specification of the function so recursive calls are memoized too”, which is what we did in our Racket example. ↩︎

ArXiving on July 2022

Sun, 24 Jul 2022 10:09:25 +0100

Here are a few papers that I read over the past months, in the CS and Stat category as usual.

Marginal Effects for Non-Linear Prediction Functions (https://arxiv.org/abs/2201.08837)

For everything related to marginal effects, I usually trust Stata margins command. By the way, there is a dedicated book for this very useful command written by Michael N. Mitchell. The present article deals with estimating marginal effects in the case of non-linear models, which usually involves derivatives as discussed in an old post. Here the authors argue in favor of forward differences, hence the approach coined “forward marginal effects.”

Better to be in agreement than in bad company: a critical analysis of many kappa-like tests assessing one-million 2x2 contingency tables (https://arxiv.org/abs/2203.09628)

I once gave a brief overview of recommend techniques for the cross tabulation of two categorical variables. This article goes a lot deeper by discussing agreement measures in 2x2 tables, inlcuding the cases of extreme tables where there’s a high level of agreement or a large imbalance between cells. The authors suggest that Holley and Guilford’s G and Gwet’s AC1 are the best candidates, among many other agreement measures (Cohen’s kappa, Pearson’s r, Yule’s Q and Y, Scott’s $\pi$, Shankar and Bangdiwala’s B, Dice’s F1 and McNemar’s $\chi^2$).

Selective inference for k-means clustering (https://arxiv.org/abs/2203.15267)

Unsupervised learning does not lead naturally to statistical inference, except maybe in the case of assessing clustering stability or choosing the optimal number of clusters. I discussed resampling-based appraoches in another post. However, with added constraints like in Mclust, we can work out the likelihood and optimize the number of clusters, for instance. Another inferential issue is that of testing whether cluster members differ on average on some attributes. Usually, we don’t test attributes that were used to build the partitioning: it is often meaningless since we are usually trying to maximize the separation between clusters based on those very specific attributes, and in any way this would lead to inflated Type I error rate. Most of the times, we establish and eventually test clustering profiles based on held out variables. In this paper, the authors propose a finite-sample p-value that controls the selective Type I error for a test of the difference in means between a pair of clusters obtained using k-means clustering.

A tutorial for using propensity score weighting for moderation analysis: an application to smoking disparities among LGB adults (https://arxiv.org/abs/2204.03345)

This aricle is in fact mostly a tutorial on moderation analysis using PS weighting, and illustrations in Stata (inline code) and R (external package) are provided.

Latent Trait Item Response Models for Continuous Responses (https://arxiv.org/abs/2204.03841)

Usually, latent trait models apply to categorical response variable (dichotomous or polytomous items, or rating scales). The objective is to build a continuous score reflecting the position of an individual on the ltent trait being measured. For a more detailed overview, see my answer on Cross Validated. This article deals with restricted continuous responses (e.g. positive only or interval-based responses), and extends response time models.

Uniformly Valid Inference Based on the Lasso in Linear Mixed Models (https://arxiv.org/abs/2204.03887)

The authors use restricted maximum likelihood (REML) estimators to separate the estimation of the regression coefficients and covariance parameters in order to estimate fixed-effects with L1 penalization in Gaussian linear mixed models.

The replication of non-inferiority and equivalence studies (https://arxiv.org/abs/2204.06960)

The authors use the TOST technique to analyze replication equivalence studies. A “success interval” for the relative effect size is used to compare the replicate to the original study effect.

Flexible Marginal Models for Dependent Data (https://arxiv.org/abs/2204.07188)

An interesting paper to complement one companion textbook I worked on long time ago.

♪ Benny Golson • Little Karin

Back to Matrix

Sat, 23 Jul 2022 21:26:51 +0200

I’m back to Matrix, finally. Last time I interacted on the network was in 2021, from the web app, Element. Previously, I tried Riot.im on my MacBook. Now that Element is available as a Desktop app for Linux, I see no reason to use the web app and I happily spent a few time configuring Element on my Linux laptop. I had to quit the channels I joined through my IRC bouncer to avoid duplicating my identities on the IRC channels I’m usually following (#scheme, #chicken, #lisp, #commonlisp, #haskell, #clojure), and joined them again from Element after having being authentified on the IRC network (Libera.chat). The process is well described on various blogs or the Matrix documentation itself. From what I understood, a bot is spawned from the Matrix network onto the IRC network, and from there on acts as if it was you.

It’s been two years now that I’m logged on IRC every day. Although I have few interactions with others, I like it a lot better than all other chat applications I used before, except maybe chat rooms on Cross Validated from those days where I was still active there. I also deleted my Twitter account a few weeks ago. I haven’t had any interaction in years, and I think it simply wasn’t for me anymore. The only things left in the cloud are my Apple account and my Gmail which I only use as a catchup email service for some external providers now. Everything else has been deleted. There’re some online account that I probably forgot but they will be too, soon or later.

So, why Matrix? As I said I like to stay using IRC but to keep up to date you either have to set up your own bouncer or rely on third-party ones. I’ve been very happy with BNC4FREE: you keep being logged in and get a backlog for the past 24 hours. You only need to connect once in a month, but of course you can launch your IRC application in the background on your local machine or a host server. I used both solutions, but finally I ended up spawning Irssi on my lab computer so it stays connected all time. The advantage of using the Matrix protocol is that you also stay connected all day and get a history of all messages (starting from the time you joined a channel). This also means you don’t need to have the application running in the background. And you get notification for free, via libnotify on Linux.

About Element itself. The UI is nicely set up. Nothing too fancy, especially after it is configured using the IRC layout (this can be found in the general settings). I haven’t explored all settings since I just installed the app yesterday, but it looks promising, especially the additional features (see below). Here’s a screenshot of Element on my desktop, using the Nord color theme:

Compared to the previous version of Element, there’s now a “lab features” of some sort, so that you can enable further options. For instance, you can define your custom UI theme or use one of from GitHub. Note that you will need to add a configuration file, $HOME/.config/Element/config.json, and not use the existing one ($HOME/.config/Element/electron-config.json), including the option "showLabsSettings": true (camel case here, and not snake case as shown on GH). Here’s mine:

{
 "settingDefaults": {
 "custom_themes": [
 {
 "name": "Nord dark theme",
 "is_dark": true,
 "colors": {
 "accent-color": "#a3be8c",
 "primary-color": "#88c0d0",
 "warning-color": "#bf616a",
 "sidebar-color": "#2e3440",
 "roomlist-background-color": "#3b4252",
 "roomlist-text-color": "#ebcb8b",
 "roomlist-text-secondary-color": "#e5e9f0",
 "roomlist-highlights-color": "#2e3440",
 "roomlist-separator-color": "#434c5e",
 "timeline-background-color": "#434c5e",
 "timeline-text-color": "#eceff4",
 "secondary-content": "#eceff4",
 "tertiary-content": "#eceff4",
 "timeline-text-secondary-color": "#81a1c1",
 "timeline-highlights-color": "#3b4252",
 "reaction-row-button-selected-bg-color": "#bf616a"
 }
 }
 ]
 },
 "showLabsSettings": true
}

♪ BADBADNOTGOOD • Signal from the Noise

Logistic fit in Mathematica

Tue, 19 Jul 2022 16:04:23 +0200

In an older post I showed how we could fit a Logistic regression using Racket builtin linear algebra routines.

It is not difficult to import a CSV file and get a working Table for a Logistic regression model, or any other model from the Exponential family really. Here’s how I did for the classical low birth weight dataset from Hosmer & Lemeshow¹, where I consider a single predictor (age of the mother):²

data = Import["~/Documents/work/tutors/CESAM/cours/Stata/birthwt2.csv"];
dt = data[[2 ;;, {2, 1}]];
m = GeneralizedLinearModelFit[dt, x, x, ExponentialFamily -> "Binomial"];

Note that we could also use SemanticImport to get a “Dataset” structured array, but since it is not callable from GeneralizedLinearModelFit, let’s stay with classical list of lists. From here now, almost all we need is now available in our “FittedModel” m, and we can even display the regression equation using the following incantation:

m // Normal

Now, what I really want is to display the logistic fitted curve together with observed proportions of low birth weights (< 2.5 kg) at each decile of age (see my other post to get an idea of this observed vs. fitted proportion display). In Mathematica, there are some grouping and binning functions (AggregatedEntityClass, SplitBy, GatherBy, BinLists, BinCounts) or one could also rely on histogram binning.

First we need to add a column indicating to which decile each value of age belongs to. Here is my solution, which relies on the empirical distribution function instead of the Quantile function. This is partly inspired by a solution proposed by Sjoerd C. de Vries to a similar problem:

ecdf = EmpiricalDistribution[dt[[All, 1]]];
f[x_] := 1 + CDF[ecdf, x] 10 // Floor
qvalues = Map[f, dt[[All, 1]]];
qvalues = qvalues /. {11 -> 10}; (* FIXME: account for max value *)

Here, an instruction like MapThread[Append, {dt, qvalues}] will simply append a new column holding the decile index for each row. We would like, however, to replace each decile index with the corresponding class center from the real values, which can be computed as follows:

tens = Quantile[dt[[All, 1]], Range[0, 1, .1]];
centers = Total[{Most[tens], Differences[tens]/2}];

Then, we need a replacement rule to tell Mathematica how to perform the desired mapping:

SetAttributes[g, Listable]
g[a_, b_] := a -> b

Finally, we will add a new column to our data set to reflect class membership for each observation, and plot both the prediction and the mean observed frequencies for each decile:

dt = MapThread[Append, {dt, qvalues /. g[Table[x, {x, 10}], centers]}]
obs = GroupBy[Sort[dt[[All, 2 ;;]]], Last -> First, Mean]
Show[ListPlot[obs], Plot[m[x], {x, 14, 45}], AxesLabel -> {"Age", "Pr(low)"}]

♪ Daft Punk • Da Funk

David W. Hosmer and Stanley Lemeshow, Applied Logistic Regression, Wiley, 2000. ↩︎
Note that the response variable comes last, unlike Stata’s model formulation. ↩︎

The unquantified self #24

Sun, 17 Jul 2022 16:12:24 +0200

I just finished watching Sense8 (Seasons 1 and 2). It is a great TV show that I highly recommend. I’m now looking for the next serie that will be as entertaining as this one. For the moment, I’m watching 3%.

I’m fiddling with Lisp and Scheme when I have some free time. Of course, I’m using Neovim for editing Lisp code and I yet have to write a blog post about that, but I don’t miss Emacs. Trust me, Neovim is quite a decent editor for Lisp code by any standards. Meanwhile, I’m reading or re-reading some good books on Lisp, and even flame wars on Hacker News from time to time. My idea remains the same: doing everything I used to perform in R or Stata with Lisp or Scheme. This may eventually take a decade or two, most probably.

I’m on vacation for the whole month of July but I need to rest. I don’t waste my time though since I spend a good part of my days traveling or cooking. Some ideas of appetizers below:

I also managed to get back to reading books more regularly than at the beginning of the year. I still don’t read enough in my opinion but I’m trying to correct that so that I can have decent statistics by the end of the year. One of my recent discoveries amazed me for a few days (I always need more time to absorb poetry than novels or essays):

Je suis cassée comme un automate
à la jambe fendue avance,
en grinçant, sur une seule patte.
Quoi qu’il en soit,
il continue. On le croirait ivre.
De son dos, il a su arracher la clé ;
il n’a plus besoin, c’est fini, d’être remonté.

— Cécile Coulon, Les ronces

I also have some books to review in some future posts, including Successful Lisp, Paradigms of Artificial Intelligence Programming, or Quickstart Molecular Biology. And a long list of TODOs to cleanup at some point…

♪ Arlo Parks • Too Good

Sequences in Lisp

Sat, 16 Jul 2022 19:35:36 +0200

In Common Lisp, sequences are made of lists, vectors, strings and [maybe other things]. In short, these are ordered collections of elements (or items for short). You can define a generic sequence using maque-sequence, or rely on specialized functions (make-list, make-array,¹ list, vector, etc.) as shown below:

* (defparameter *x* (make-sequence '(vector double-float) 5 :initial-element 1d0))
* *x*
#(1.0d0 1.0d0 1.0d0 1.0d0 1.0d0)
* (vector 1d0 1d0 1d0 1d0 1d0)
#(1.0d0 1.0d0 1.0d0 1.0d0 1.0d0)

Note that although they look the same, make-sequence in this case returned an array, and not a simple vector like vector. Also note that I’m using SBCL REPL so that ** means the last computed result.

* (equalp *x* **)
nil
* (type-of *x*)
(simple-array double-float (5))

To generate a regular sequence à la Python’s range function, we could define the following function:

* (defun iseq (end &optional (start 0) (step 1))
 (loop :for n :from start :below end :by step :collect n))
* (iseq 5)
(0 1 2 3 4)
* (iseq 6 1)
(1 2 3 4 5)

This will only work for numbers, of course. While playing with newLISP, I learned that there’s a builtin sequence function there, which works like LispStat’s iseq:²

> (setq x (sequence 1 5))
(1 2 3 4 5)
> (length x)
5
> (x 3)
4

There’s even a series function to create a geometric sequence of numbers. As shown in the above example, subsetting is a breeze (this is called implicit indexing per the documentation), but there’s more: slicing a list or an array is even easier than in most Lisp-like languages I know, except perhaps Clojure using subvec provided you’re using vectors:

> (2 x)
(3 4 5)

Of course, we could do the same with arrays as shown below:

> (setq z (array 2 2 '(1 2 3 4)))
((1 2) (3 4))
> (0 1 z)
((1 2))
> (1 1 z)
((3 4))

I think I’ll need to investigate more into this dialect to see if toy examples from LispStat could be ported to newLISP. Despite all the criticisms we can found on HN, I believe newLISP provides a handy way of writing quick scripts in Lisp, but see I am not afraid to admit that I’ve used Lisp for real work. See also newlisp - an intriguing dialect of Lisp, by Eli Bendersky, which also summarizes the point raised above. Other candidates on my list are PicoLisp and Janet. Stay tuned.

♪ As Animals • By My Side

See also this warning about portable scientific code in CL, especially when dealing with arrays. ↩︎
Or various implementation of the iota function in CL, e.g. Generating sequences in Common Lisp or the Alexandria library. ↩︎

Saint-Malo

Fri, 15 Jul 2022 21:12:31 +0200

De retour d’un petit séjour bien agréable avec mon fils à Saint-Malo. Nous avons eu beau temps, et surtout des températures beaucoup plus clémentes qu’en région parisienne. Nous avons quitté Saint-Malo le mercredi en fin d’après-midi par 24°C et nous sommes arrivés à Paris à 20h où il faisait encore 35°C !

Cette petite escapade nous a fait un bien fou. La mer était magnifique, et les galettes excellentes comme il se doit en Bretagne. Nous étions déjà allés à Saint-Malo chacun de notre côté, lui il y a quelques années avec sa mère et moi lorsque j’étais enfant avec mes parents. S’y retrouver tous les deux, après tant d’années sans vraies vacances à la mer, a été un vrai bonheur.

♪ Louis Ville & Mell • Hôtel Pourri

Palindromic numbers in Mathematica

Thu, 07 Jul 2022 21:13:59 +0200

I described some of my solutions to Project Euler in several posts I wrote in 2019. I’m now revisiting some of the code while playing with Mathematica. Indeed, Mathematica is an interesting language that is starting to interest me more than just for making small calculations or graphics on the fly. And it looks like many languages remind me of Stata – delayed evaluation for Mathematica, command abbreviations for VimL.

Problem 4 reads:

A palindromic number reads the same both ways. The largest palindrome made from the product of two 2-digit numbers is 9009 = 91 × 99. Find the largest palindrome made from the product of two 3-digit numbers.

With the Racket code provided in this previous post, I used the following for*/list construct to compute the solution using brute force:

(apply max
 (for*/list ([a (in-range 999 100 -1)]
 [b (in-range a 100 -1)]
 #:when (palindrome? (* a b)))
 (* a b)))

Computers are fast enough nowadays, although there are subtle optimizations available for such problems. Anyway, this means two functions and a main loop. Here is the same code in Mathematica, in short form without any consideration for optimization:

IsPalindrome[x_] := IntegerDigits[x] == Reverse[IntegerDigits[x]]
Max[Select[Flatten[Table[i * j, {i, 100, 999}, {j, 100, 999}]], IsPalindrome]]

I like the functional route we can take with Mathematica, although it’s slighty slower than Racket code in this case.

♪ Adam Wakeman • Pristina

On consulting

Thu, 30 Jun 2022 21:01:49 +0200

I have been working as a full-time statistical consultant (mostly in biomedical research) for something like 13 years. Sometimes I miss the job, but in the end I’m better served where I am now. The academic world is worth what it is worth, but when you are looking for a quiet place it is one of the best options. Consulting is both challenging and entertaining: you learn a lot, often in a short period of time, which means you get continuing education (almost) for free, and your network will rapidly grow provided your work is appreciated, which means you get a lot more opportunities, hence you may become more demanding when selecting new projects. I already mentioned this in a previous post.

After 13 years I had a serious accident that forced me to review my perspectives both professionally and from a more personal point of view. Concerning the first point, it turned out that I was getting tired of repeating the same thing over and over again to the doctors I was interacting with, not to mention the fact that from then on I was going to find myself in the position of a patient for a good part of my days.¹ But the real problem, it seems to me, is that at a certain point it becomes difficult to take the place of the principal investigator and to suggest alternative interpretations of the results. My job was to organize and analyze the data with sound statistical methods. It’s not that I didn’t care that much about what they might do with the results afterwards, but honestly that was not part of my job. I’m willing to learn a lot about domains I was not trained in, but it would be hard to become a specialist in each new domain in a few months, right? And finally, I was most often involved at later stages of clinical studies. You know the famous adage:

To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. — Sir R. A. Fisher

They rarely consulted me before carrying out their studies, so I was left with the eventually flawed design, and already collected data to analyze. No prior power analysis, no interim analysis, and so on. I always did the best I could do given the context, and I believe no one was ever unhappy with my job. However, I felt something was broken in this workflow. You get paid for analyzing data, without being involved in the design of the study, results come afterwards and people ask to carry out subgroup analysis or devise new hypotheses to explain or at least exploit unattended results. First it sounds at an angle with scientific reasoning, second I’m surely not the right person to ask. I ended up believing that some people would benefit from taking a course in experimental design and causal inference.

A few days ago, I came across an interview with John D. Cook where he described his move to mathematical consulting after years in Academia and work in private sector. He has better words than me regarding part of the specific issue I mentioned above:

Applied statistics may be closer to epistemology than mathematics. You have to question what you know, why you think you know it, and how confident you are (or should be) in that knowledge. You must make simplifying assumptions and determine whether they are justified. Statistical modeling is hard to do well. It’s easy to write down a model if you’re not overly concerned with the accuracy of your results. As the old saying goes, fools rush in where angels fear to tread.
Sometimes I get to work on statistical problems with crisp mathematical statements, such as finding an efficient way to compute something. These projects are fun because the epistemological concerns are someone else’s responsibility.
My work in applied mathematics has been more objective than my work in statistics. When you’re dealing with voltages or velocities, what you’re measuring (and what it means) is pretty clear. There’s much less anxiety around modeling, and your level of success at the end is usually obvious. I prefer mathematical work, but there are more statistical opportunities in consulting. These days I often hand over statistical projects to someone else; I work on statistical problems about data privacy but am moving away from medical statistics.

♪ Gerald Clayton • Like Water

I spent a total of 10 weeks in intensive care type hospitalization over a period of 5 months, 7 of which were consecutive. ↩︎

Micro posting in June

Tue, 28 Jun 2022 21:12:40 +0200

2022-06-01:

Saying that I’ve been maintaining my dotfiles for a decade is like saying that I’ve been decorating my home for a decade. I probably know what works for me, but that doesn’t mean I know what you’ll like! — A decade of dotfiles

2022-06-01: A vision of a multi-threaded Emacs. #emacs
2022-06-07: Swamp Children, Taste What’s Rhythm.
2022-06-14: Miley Cyrus, Nothing Else Matters.
2022-06-14: Now I know that even installing the rms package implies crap packages from so-called “modern R”…
2022-06-15: Lydia Loveless, Let’s Make Out.
2022-06-16:

(…) being overly tense works against giving your best performance. — The 85% Rule - When Giving It Your 100% Gets You Less than 85%

2022-06-16:

Even with a lot of help from a good text editor, writing HTML can be a drag. Nice documents end up as tag-swamps with little bits of content perched atop hills of tabs. — Write HTML Right

2022-06-16: Illustrating Python via Bioinformatics Examples. #python
2022-06-16: Learn Haskell by building a blog generator. #haskell
2022-06-17: Enrico Rava Quartet, Choctaw.
2022-06-17:

While it’s tempting to switch to one of the endless new apps out there, there are good reasons to trust old tools. — Cool desktops don’t change

2022-06-17: Run any code on any client. With WebAssembly and Wasmer.
2022-06-17: Two understandable deficiencies in Common Lisp. #lisp
2022-06-17: Unix History Repository. #unix
2022-06-19: Meredith Brooks, Bitch.
2022-06-25: And another round of cleansing for my Neovim setup: down to 15 plugins now (the screenshot below includes popup.nvim which is no longer needed).

2022-06-25: Just started watching Sense8 a few days ago. Looks great so far. It’s the fourth TV shows I watch after a long break (December 2021).
2022-06-25: Time to quit, I guess.

2022-06-25: Crafting Interpreters: A Review.
2022-06-25: Oh Shit, Git!?!. #git
2022-06-27: Iggy Pop, Some Weird Sin.

♪ XXX • XXX

June in review

Sat, 25 Jun 2022 21:08:13 +0200

Things from the internet that I heard about recently, a year after.

If I weren’t so satisfied of my current laptop, I would surely look for another one from System76: this one, for example, looks gorgeous in terms of specs.
Start all of your commands with a comma: I like the idea of distinguishing user commands from system commands, but I find it weird to prefix all of them with a comma. Perl or Emacs Lisp users are used to use the my prefix, I guess, and Zsh has underscore for builtin functions.
https://pyscript.net/ looks great. Wasm is the way to go IMHO, and it is surely as promising as the Electron framework some years ago. BTW, atom is dead.
Effective Shell. As they said: “This book is for anyone who is interested in computing, and wants to learn more about the exciting, but sometimes daunting world of The Shell.”
One-liner for running queries against CSV files with SQLite. Interesting tibbits. Together with dsq, there’s plenty of opportunity to run SQL queries on flat files from the command-line.
If you feel nostalgic at times, you can always download the mini vMac 36.04.

Using ftplugin to tidy my Vim configuration. I used it a lot too:

~/.config/nvim % tree after
after
└── ftplugin
 ├── bib.vim
 ├── cmake.vim
 ├── css.vim
 ├── c.vim
 ├── gitcommit.vim
 ├── haskell.vim
 ├── help.vim
 ├── lisp.vim
 ├── lua.vim
 ├── markdown.vim
 ├── org.vim
 ├── python.vim
 ├── qf.vim
 ├── racket.vim
 ├── rmd.vim
 ├── rnoweb.vim
 ├── rust.vim
 ├── r.vim
 ├── scheme.vim
 ├── sql.vim
 ├── stata.vim
 ├── text.vim
 ├── tex.vim
 ├── vim.vim
 └── zsh.vim

1 directory, 25 files

Inconsistent thoughts on database consistency. I used to do a lot more SQL back in the days, so whenever I came across a post about SQL or NoSQL database I read it. This one is about consistency in ditributed DBs, which may be understodd differently depending on the point of view of the reader.

Your language isn’t broken, it’s doing floating point math. Computers can only natively store integers, so they need some way of representing decimal numbers. This representation is not perfectly accurate. This is why, more often than not, 0.1 + 0.2 != 0.3. — Floating Point Math

♪ The Clash • Groovy Times

Org and Beamer

Thu, 23 Jun 2022 22:36:50 +0200

A few days ago, I decided to rewrite a MArkdown template that I occasionally use for talks or courses. Part of this has already been discussed in an another post. Since I always found Org documents better than Markdown ones, I wanted an Org template much like I have a template for short handouts. I followed the same idea and I created a small shell script that I call from a terminal or from a Vim shortcut:

#!/usr/bin/env bash

FILE=$1
ELISP="/home/chl/Documents/notes/assets/org-beamer.el"

emacs --batch -l "$ELISP" --eval "(progn (find-file \"$FILE\") (org-beamer-export-to-pdf))"

The org-beamer.el file holds all the default settings I wish to use. I changed things a bit, especially the main fonts. I could use xelatex or lualatex as before, but after all Bitstream Vera is a nice sans serif font, and it comes with builtin math support thanks via the arev package. My Emacs Lisp setup file now basically reads: (I skipped basic settings and org-babel related stuff.)

(eval-after-load "ox-latex"
 '(add-to-list 'org-latex-classes
 '("beamer"
 "\\documentclass[presentation,9pt]{beamer}
 \\usepackage{arev}
 \\usepackage[backend=bibtex,style=numeric-comp,citestyle=numeric-comp,autocite=plain]{biblatex}
 \\addbibresource{/home/chl/Documents/notes/references.bib}
 \\usecolortheme[named=black]{structure}
 \\setbeamertemplate{navigation symbols}{}
 \\makeatletter
 \\setbeamertemplate{title page}[default][left,colsep=-4bp]
 \\makeatother
 \\usepackage{tcolorbox}
 \\usepackage{textcomp}
 \\usepackage{ragged2e}
 \\usepackage{listings}
 \\definecolor{SlateGrey}{HTML}{708090}
 \\definecolor{CornflowerBlue}{HTML}{6495ed}
 \\definecolor{SandyBrown}{HTML}{f4a460}
 \\definecolor{White}{HTML}{ffffff}
 \\usepackage[font={color=SlateGrey,small}]{caption}
 \\setbeamercolor{frametitle}{bg=SlateGrey,fg=White}
 \\setbeamercolor{alerted text}{fg=SlateGrey}
 \\setbeamertemplate{itemize items}{\\scalebox{1.4}{\\raisebox{-.25ex}{\\text{\\textbullet}}}}
 \\addtobeamertemplate{block begin}{}{\\justifying\\small\\sffamily}
 \\captionsetup[figure]{labelformat=empty}
 \\captionsetup[table]{labelformat=empty}
 \\captionsetup[lstlisting]{labelformat=empty}
 \\renewcommand{\\textbf}{\\alert}
 \\definecolor{low}{named}{SandyBrown}
 \\definecolor{high}{named}{CornflowerBlue}
 \\newtcbox{\\texthigh}{nobeforeafter,colframe=high!15!white,colback=high!5!white,boxrule=0.5pt,arc=4pt,
 boxsep=0pt,left=2pt,right=2pt,top=2pt,bottom=2pt,tcbox raise base}
 \\newtcbox{\\textlow}{nobeforeafter,colframe=low!15!white,colback=low!5!white,boxrule=0.5pt,arc=4pt,
 boxsep=0pt,left=2pt,right=2pt,top=2pt,bottom=2pt,tcbox raise base}
 \\newcommand{\\mathhigh}[1]{\\textcolor{high}{#1}}
 \\newcommand{\\mathlow}[1]{\\textcolor{low}{#1}}
 \\newcommand{\\goto}[2]{\\quad\\hyperlink{#1}{\\beamerbutton{#2}}}
 \\renewcommand{\\footnoterule}{%
 \\kern -3pt
 {\\color{SlateGrey}\\hrule width \\textwidth height .25pt}
 \\kern 2.5pt
 }
 \\renewcommand{\\footnotesize}{\\scriptsize}
 \\author{\\url{https://aliquote.org}\\\\ \\url{mailto:chl@aliquote.org}}
 \\lstset{
 basicstyle=\\small\\ttfamily,
 commentstyle=\\color{SlateGrey}\\textit,
 numbers=left,
 stepnumber=1,
 numbersep=8pt,
 numberfirstline=true,
 firstnumber=1,
 numberstyle=\\color{SlateGrey}\\footnotesize{},
 captionpos=t,
 abovecaptionskip=3ex,
 aboveskip=20pt,
 identifierstyle=\\ttfamily,
 keywordstyle=\\ttfamily,
 escapeinside={(*@}{@*)}
 }
 [NO-DEFAULT-PACKAGES]
 [PACKAGES]
 [EXTRA]"
 ("\\section{%s}" . "\\section*{%s}")
 ("\\subsection{%s}" . "\\subsection*{%s}")
 ("\\subsubsection{%s}" . "\\subsubsection*{%s}")
 ("\\paragraph{%s}" . "\\paragraph*{%s}")
 ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))))

Short preview below.

After having reviewed the new features of Context I do not exclude the idea of returning to pure tex slides soon, though. I still have some handouts to read.

♪ Meredith Brooks • Bitch

20 years with R: A retrospective

Tue, 14 Jun 2022 09:03:52 +0200

I gave my first statistics lecture using R almost 20 years ago. At that time, I mostly working on behavioral data and R appeared as nice replacement for Statistica or SPSS. At that time, Octave was not as good a pretender to MATLAB as it is nowadays, and Python’s data stack has not even started. Old times good times. That was really exciting to dive into a new programming languages dedicated to applied statistics, with a dedicated graphical device, a vibrant community of senior statisticians, and a central repository of packages modeled after CTAN and CPAN.

However, I lately became tired of all the new ways of doing R these days, and how the intrusion of modern packages impact at various degree the way I liked to do things in R. Although R has gained some popularity thanks to those packages, the whole ecosystem has been impacted, IMHO. I noticed there are now some hot debate between proponents of base R and those who prefer the tidyverse, much like the old R vs. Python flame that emerged 5 to 10 years ago. It’s probably time to stop worrying about all those things since I rarely need R these days, after all. But something is going on, for sure.

Here are a series of plots I made in R for statistical reports, articles, and handouts. That’s it (for the retrospective).

On a related point, I recently started drafting a short tutorial on how not to use “modern R” tools, and then I came across Frank Harrell’s excellent handout on his R workflow. Since I was going to make the apology of his packages, I think it’s best to start working on another handout. I will leave the foreword untouched below.

Although the tidyverse provides lot of packages that may be useful to perform common data preparation and univariate or multivariate statistical summaries, other solutions do exist. I am pretty confident that base-only R packages have been developed since I last used R for intensive data analysis (I can think of strengejacke’s packages for data visualization and data manipulation, for instance), but the Hmisc packages existed long before plyr, then dplyr, were published on CRAN, and it already worked pretty well. It’s still the best piece of software ever written for R after MASS, in my view. Together with the rms and ggplot2 packages, you will get the data munging triumvirate almost for free. Almost because you will have to learn a lot and to choose wisely among the numerous options. In addition, remember that “lightweight is the right weight”. If you can perform 80% of your tasks with three packages and builtin stuff, then you’re on the right side of the Pareto law. Here’s your starter kit for Hmisc.

Disclaimer: I have nothing against the tidyverse way of doing things (I just barely understand why we really need another “rlang”, and I regret the problems of reverse dependencies that it may have caused in the past). If that suits your needs, that’s all fine. If on the contrary you get stuck on basic data manipulation stuff, or built-in one-liner R functions don’t play well with “tibble”, then you’re probably going in the wrong direction. Start with Phil Spector’s textbook \cite{spector-2008-data-manip-r}, then eventually learn a bit of plyr \cite{wickham-2011-split-apply}. The 80% of time spent in data manipulation, as we use to say, is not a joke: You will likely spend most of your time switching from long to wide format, aggregating data all over again and again, and not just before even you start building your super nice statistical model.

[2023-01-04]
See also Norm Matloff’s take on the tidyverse: Teach Base-R, Not Just the Tidyverse.

♪ IAMDDB • Silver Lines

Du paradoxe des anniversaires

Sun, 12 Jun 2022 21:20:39 +0200

Petite note de synthèse rédigée à l’attention de mon fils pour son grand oral du BAC.

Le paradoxe des anniversaires, qui n’en est pas vraiment un, s’intéresse à la probabilité que deux personnes soient nées le même jour, plus précisément au nombre de personnes que l’on doit interroger avant que la probabilité de rencontrer deux personnes nées le même jour soit supérieure ou égale à 50 %.

Les hypothèses que l’on fera sont les suivantes : l’année comporte 365 jours et on néglige les années bisextiles.

On notera $p(n)$ la probabilité que au moins deux personnes prises parmi $n$ soient nées le même jour, et $n(p)$ le nombre de personnes nécessaires pour obtenir une probabilité $p$ donnée.

Le paradoxe vient du fait que les gens s’imaginent que la probabilité recherchée porte sur l’événement “deux personnes sont nées un même jour donné”, comme dans la question “quelle est la probabilté que le jour du grand oral de maths tombe le jour de mon anniversaire, en supposant que celui-ci puisse être organisé n’importe quel jour de l’année”. On va considérer le complémentaire de cette probabilité, ce qui nous évite d’énumérer tous les cas, et le soustraire de 1, et on dénotera $n$ le nombre de personnes attendues. On cherche donc $n$ tel que :

$$ 1-\left(\frac{364}{365}\right)^n = \frac{1}{2}. $$

La résolution de cette équation montre que si $n=253$, alors la probabilité recherchée est $> 1/2$. Qu’en est-il s’il l’on ne considère pas de date d’anniversaire fixe dans l’année ? C’est le problème énoncé plus haut.

Le plus simple est encore une fois de passer par le complémentaire en se demandant quelle est la probabilité que toutes les personnes interrogées ait un jour d’anniversaire différent.¹ Par dénombrement, on constate que le nombre de cas où $n$ personnes ont des anniversaires différents équivaut à un arrangement de $n$ objets pris parmi 365, soit $\frac{365!}{(365-n)!}$. Pour chaque personne il y a 365 possibilités de jour d’anniversaire, et donc comme il y a $n$ personnes il y a au total $365^n$ choix possibles. Le nombre de cas où $n$ personnes ont des anniversaires différents divisé par le nombre de possibilités donne donc la probabilité que tout le monde soit né un jour différent. Le complémentaire de cette probabilité équivaut à la probabilité de l’événement “au moins deux personnes sont nées le même jour”, soit :

$$ p(n) = 1 - \frac{365!}{(365-n)!} \cdot \frac{1}{365^n}. $$

 In[2]= N[1 - 365! / ((365-23)!365^23)]
out[2]= 0.507297

 In[3] = p[k_] := 1 - 365! / ((365-k)!365^k);
 Plot[p[k], {k, 1, 100}, GridLines -> {{23}, {p[23]}}];
 Table[N[p[k]], {k, 22, 25}]
Out[3] = {0.475695,0.507297,0.538344,0.5687}
 ^^^^^^^^

L’expression $p(n)$ ci-dessus peut être approximée par $1-e^{-\frac{n^2}{2|E|}}$. Cette approximation s’obtient par le développement limité de $e^x = 1 + x + \mathcal{o}(x)$ pour $x$ au voisinage de 0. Comme discuté sur Wikipédia, ceci donne une approximation du nombre de personnes nécessaires pour avoir une probabilité $p$ d’observer au moins deux personnes avec le même jour d’anniversaire :

$$ n(p) \approx \sqrt{2\cdot 365\;\text{ln}\left(\frac{1}{1-p}\right)}. $$

 In[4]= n[k_] := Sqrt[2 365 Log[1 / (1-k)]];
 Table[N[n[k]], {k, 0.4, 0.6, 0.1}]
Out[4]= {19.3107,22.4944,25.863}
 ^^^^^^^

Il existe d’autres approximations, et également d’autres façon de poser le problème (p.ex., nombre moyen de personnes nécessaires pour observer une paire d’anniversaires identiques, en supposant un échantillonnage avec ou sans remise).

Le paradoxe des anniversaires trouve des applications dans divers domaines, puisqu’il suffit de remplacer le concept de jour d’anniversaire par n’importe quel élément pris dans un entier dénombrable, par exemple des mots de passe générés aléatoirement.

[2022-06-13]
Exemples d’application en bioinformatique :

♪ The Vaselines • Jesus Wants Me for a Sunbeam

Le paradoxe des anniversaires se généralise au problème suivant : si l’on considère un ensemble $E$ fini, quelle est la probabilité $p(n)$ que parmi $n$ éléments tirés uniformément et de manière indépendante dans $E$, deux éléments au moins soient identiques. On montre que celle-ci vaut : $$ p(n) = 1 - \frac{|E|!}{(|E|-n)!} \cdot \frac{1}{|E|^n}. $$ ↩︎

Rhythmbox

Sat, 11 Jun 2022 17:59:02 +0200

I’ve been a faithful user of cmus for the past two years. It is fast and responsive, it handles tons of music with no worries, and it is quite easy to configure. Using cmusfm, we can even scrobble the tracks we listen to to Last.fm, and it all works great.

Since I upgraded to Ubuntu 22.04 LTS I decided to try another music player. I’ve been trying Audacious on i3, but that’s was just to try it out. Last week, I decided to give Rhythmbox a try. I think it should normally be instaled on a full-featured Ubuntu distro, like Libreoffice, Thunderbird and the like. Since I decided to install the bare essentials, I don’t have default application for managing media files on Ubuntu.

[2022-09-29]
TIL about Lollypop (via Reddit), which also seems to handle metadata quite well, and provides audio scrobbling facilities. There apparently is the Sayonara player in the race as well.

I imported my main libraries into Rhythmbox with no problem at all. All tracks were correctly classified by author/album/genre apparently. I could check with beets, except that it no longer works since I switched to Python 3.10. I then created new playlists using the following command, for each playlist folder:

~/Media/Music/garbage % find . -name "*.mp3" -type f | shuf > garbage.m3u

I activated the Last.fm plugin and installed (later on) the LastfmPlaycount 3rd party plugin. This allows me to get a synchronized view of my lsitening activities from Last.fm, which also means I would get desynchronized track counts when switching back and forth between cmus and Rhythmbox, unless I found something equivalent for cmus. Anyway, I’ve been using Rhythmox as my daily music player for two weeks now, and I’m pretty happy with it. The current song that is being played is nicely displayed in the notification widget in the middle of the top bar (I forgot about its name), with basic control (start/pause, previous/next track, and the album cover). Same happens when using Rhythmbox: you get a small album cover art that you can hover on, classification by genre, artist, and so on. Everything you would expect from a basic music player in fact.

The advantage of usign Last.fm to get my track counts is that it doesn’t matter whether some tracks are included on more than one playlists, unlike cmus whose playcount is really a physical track count. In other words, I’m seeing an increase in my all-time playcount as I write this post. Meanwhile, I’m slowly updating track counts by listening to my music library again and again. Not a bad deal after all.

♪ Timo Lassy • Mountain Man Exit

Computing the inverse CDF of a Gaussian

Wed, 08 Jun 2022 16:08:28 +0200

There’s no closed-form solution to the inverse CDF for the normal distribution, also known as the “quantile” function, that was discussed in a previous post. By glancing at the statistics module from CPython, I noticed the authors actually rely on the rational approximation described in Wichura’s Algorithm AS 241: The Percentage Points of the Normal Distribution.¹

The same algorithm is used by the authors of the Racket math module, with an additional remark:

Added asymptotic expansions to increase range when given log probabilities, and a Newton iteration to fix up answers in the tricky spot between -20000 and -744.

Note that test cases are provided in the original article such that you can easily verify that your own implementation provides the expected results:

$$ \begin{equation} \begin{aligned} z_{0.25} &= -0.6744897501960817,\cr z_{0.001} &= -3.090232306167814,\cr z_{10^{-20}} &= -9.262340089798408\cr \end{aligned} \end{equation} $$

Here is what I got from Racket:

> (require math/distributions)
> (for ([q (in-list '(0.25 0.001 10.0e-20))])
 (displayln (flnormal-inv-cdf 0.0 1.0 q #f #f)))
-0.6744897501960817
-3.090232306167813
-9.013271153126675

The last value does not agree with test values provided in the reference paper, and I got the same results in R using qnorm(10e-20)).² When working with very low p-values, or when you suspect that something could get wrong with floating-point arithmetic, it’s always wise to double check the result you get. Mathematica, for instance, returns the expected result:

In[10]:= N[InverseCDF[NormalDistribution[], 10^-20], 16]
Out[10]= -9.262340089798408

I learned a long time ago that for everything related to statistical distributions, it’s always worth double checking with Mathematica.

On a related point, in the case of R, some strange things may happen, although the authors usually took care of the details for you. For instance, there’s a lower.tail argument to the p* and q* family functions, which allows to correct for catastrophic cancellation; it even works when distribution are not symmetric (otherwise we could simply use qnorm(1-p)).

♪ Shawn Mullins • Lullaby

Wichura, Michael J. Algorithm AS 241: The Percentage Points of the Normal Distribution. Journal of the Royal Statistical Society. Series C (Applied Statistics), 37(3): 477-484 (1988). PDF ↩︎
Same for Stata (invnormal(10e-20)), or Python using scipy (norm.ppf(10e-20)). ↩︎

Ubuntu 22.04

Mon, 06 Jun 2022 20:59:59 +0200

In a recent post, I said that I will probably upgrade my Ubuntu soon or later. The day after I published it I finally did it. It took me a few hours to upgrade my laptop full of useless applications and libraries installed all along the last year, but finally I got the welcome screen before lunch. There are many reviews available on the web, so I will just mention what I did and what I find the most interesting updates for my use cases.

There are important updates to the kernel, glibc and various other system libraries, but I noticed that now clang and Python are versions 14.0 and 3.10.4. Python 3.10 is supposed to be faster than previous versions, but I didn’t run any benchmarks so far. Nothing new regarding Neovim: still at version 0.6 from the official apt repositories, so I’m using the Neovim PPA for the unstable version.
The window manager has been improved, IMHO. The calendar widget is able to display all events stored in Gnome calendar, there’s a processor scheduler that can alternate between energy saving and high performance policy, and virtual desktops are now displayed horizontally rather than vertically. Gnome screen capture got a few enhancements as well.

I deleted a bunch of unused or defunct applications, including Docker and bitlbee. I also manually cleanup up the old LLVM files that came with the previous installation of Ubuntu 20.04 and that were not cleaned up during the upgrade process. Finally, I re enabled Tracker, which I disabled a while ago, and some default settings, which I also disabled for i3 and Regolith desktop. I did not reinstall the Snap store. In fact, I deleted everything related to snap when I configured my laptop the first time, and then never looked back. At this point, I don’t even know how to reinstall snap on my machine.
There were some quircks here and there: IPython was not happy with the missing Qt backend for matplotlib, so I now use matplotlib.use("GTK4Cairo") instead, which is not a bad idea after all. Of course, switching to Python 3.10 instead of 3.8 also means that I had to reinstall every little applications that sit in my $HOME/.local/bin folder, mostly from pip3 install --user. To remap the CAPS lock key to , I was previously invoking setxkbmap -option caps:escape in my shell init scripts. This no longer works under Wayland, but I discovered that you can just ask Gnome shell to do that for you: gsettings set org.gnome.desktop.input-sources xkb-options "['caps:escape']".
I spent another couple hours fixing my Hugo theme, which stopped working with version 0.92. I am now aware of the fact: after each update of Hugo something goes wrong. I had not encountered such problem the last two years since I was on the 20.04 LTS, which means no update at all for most applications. However, I tor my hair out for a long time with rolling release on Apple Homebrew. Anyway, I fixed it, again, and now I hope we are quiet for some long months.
I also managed to install the Nordic theme, which is beautiful (especially compared to the default dark theme in Ubuntu) and reminds me of the one I had when I was using Regolith desktop. I didn’t reinstalled it yet, since I want write to try the default settings yet another time again.

♪ Big Spider’s Back • Black Chow

Kopia backup

Sun, 05 Jun 2022 20:36:43 +0200

For everything related to backup I used to use Arq back in my Apple days. I tried Borg on both Apple and Ubuntu, and I also tried Timeshift, but finally I decided to use Duplicity and deja-dup for my weekly backup on Ubuntu 20.04. It worked well, although the cache files took a lot of place. At that time I was backing up my entire $HOME system on an external HD connected on an USB-C port. I didn’t add a cron task for that and I managed to launch the backup process more or less carefully every Sunday.

A week ago, I deleted my ~/.local/share folder by mistake. Fortunately, I was able to recover everything thanks to those weekly backups. However, the only way to recover this specific folder from Duplicity snapshots was to extract the full snapshot in a temporary folder and stopped the extraction process when I noticed that the folder in question was extracted. I was in a hurry, there may be a better option or better tool to use than deja-dup. Unlike Arq, there does not seem to be an easy way to view the inner structure of each snapshot.

Yesterday, I decided to give Kopia a try. It uses deduplication, compression, encryption — all the state of the art in terms of backup strategies. Moreover it allows to list all snapshots, mount them on your local filesystem, diff one snapshot against the other, and restore any snaphot with a few commands or via a nice GUI (Kopia UI). Of course, you can set exclusion rules, schedule how often snapshots are taken, and so on. Finally, you can synchronize your local backup with other sources, like an external HD or a local home server. I choose to backup my HOME directory on my HOME directory, and to synchronize it to an external HD, while previously I only use external HD to store my backup. This way, I have a local copy from which I can restore anything if something goes wrong – no need to take my external HD with me, and I use deduplication with the sftp transfer protocol. The documentation is also excellent, and there’s a Discourse group.

One thing I noticed is that it is blazing fast, especially compared to deja-dup.¹ I excluded static folders which are unlikely to change more than once a week (pictures, music, and PDFs) and I ended up with a 30 Go backup repository.

~ % du -sh ./* | sort -h
364K ./bin
1,7G ./tmp
2,1G ./Sites
6,0G ./cwd
6,9G ./Documents
18G ./Images
33G ./backup
74G ./Media

I’m aware of the 3-2-1 backup strategy, except that I no longer use any offsite storage (Dropbox or Google, previously). I’m expecting to setup a central backup on a NAS, and a local copy on an external HD. I also have a cron job to schedule backup every hour, with default retention policy (48 hourly snapshots are kept in the central repository).

~ % kopia snapshot list
chl@aliquote:/home/chl
 2022-06-05 19:05:21 CEST k295d3d6f42eeb3fa08c5d47e58af91bf 58.4 GB drwxr-xr-x files:504927 dirs:73697 (hourly-17,daily-3,weekly-2)
 2022-06-06 20:22:20 CEST k96b2fb21131fbbfc0815dd5fb373a763 44 GB drwxr-xr-x files:506393 dirs:73202 (hourly-16)
 2022-06-06 21:00:02 CEST kf916e1b0c60fa4f409a42684fb93019a 44 GB drwxr-xr-x files:506629 dirs:73202 (hourly-15)
 2022-06-06 22:00:01 CEST k728bb034a39a4ff650182b3059d49991 44 GB drwxr-xr-x files:506712 dirs:73203 (hourly-14)
 2022-06-06 23:00:01 CEST k5f4017166b44946ca6259a4ae17f1e38 44 GB drwxr-xr-x files:506714 dirs:73203 (hourly-13,daily-2)
 2022-06-07 00:00:01 CEST kbf3b9941f81be51b1165cfb6806b4251 44.1 GB drwxr-xr-x files:506716 dirs:73203 (hourly-12)
 2022-06-07 01:00:01 CEST k97b62af55128160101b828d76e26256c 44.1 GB drwxr-xr-x files:506718 dirs:73203 (hourly-11)
 2022-06-07 02:00:01 CEST k39386d816ded10cbc656a2e4c161e1bc 44.1 GB drwxr-xr-x files:506720 dirs:73203 (latest-10,hourly-10)
 2022-06-07 03:00:01 CEST kc012f6b2bf5e8678e89b5d891603a168 44.1 GB drwxr-xr-x files:506722 dirs:73203 (latest-9,hourly-9)
 2022-06-07 04:00:01 CEST k8947a49d41f253dd2938d29ffbd7f1a3 44.1 GB drwxr-xr-x files:506724 dirs:73203 (latest-8,hourly-8)
 2022-06-07 05:00:02 CEST ka91ca4f30e5411edfa2ff196a4e0905f 44.1 GB drwxr-xr-x files:506726 dirs:73203 (latest-7,hourly-7)
 2022-06-07 06:00:01 CEST k19705c3b296d13670f0cc9ac7e91ffb5 44.2 GB drwxr-xr-x files:506728 dirs:73203 (latest-6,hourly-6)
 2022-06-07 07:00:02 CEST kadc58cb442efbd727441de745cbebd13 44.2 GB drwxr-xr-x files:506730 dirs:73203 (latest-5,hourly-5)
 2022-06-07 08:00:01 CEST kdb87258b93ae8e3d3d83ac5a4d281426 44.2 GB drwxr-xr-x files:506973 dirs:73244 (latest-4,hourly-4)
 2022-06-07 10:00:01 CEST k7d9b1c6f0d746e9b9ffdb63011299a2c 44.2 GB drwxr-xr-x files:507123 dirs:73244 (latest-3,hourly-3)
 2022-06-07 11:00:01 CEST k22b5115235d9dceb0ec9f204be6e3ae7 44.3 GB drwxr-xr-x files:507782 dirs:73254 (latest-2,hourly-2)
 2022-06-07 12:00:01 CEST k7dc38129e4011ef6c0e870e220094459 44.3 GB drwxr-xr-x files:507743 dirs:73229 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)

It all looks very promising. I’m waiting to see what happens next, especially in case of accidental deletions or corruptions.

[2022-08-25]
Small update: I effectiveley had to use Kopia to restore some files that were accidently deleted, and to compared two directories at different points in time. The latter is made really easy thanks to the diff utility inside Kopia. Also, after two months of daily backup, I came to the conclusion that compression (zstd or pgzip) is not that good as it consumes a lot of RAM and keep RAM cache increasing when your backup are schduled on an hourly basis. Finally, to keep your backup repository at a reasonable size don’t forget to set up appropriate exclusion rules using kopia set policy.

♪ Washed Out • Amor Fati

Backups to a local SSD are expected to be faster than backups whose destination is an external HD, though. ↩︎

The unquantified self #23

Sat, 28 May 2022 21:19:49 +0200

I spent a few hours reconfiguring my Macbook pro from scratch, with Ubuntu 22.04 LTS. Indeed, last year I happily installed Ubuntu 20.04 LTS on my a Macbook pro dating from 2014, but I never really used it so far. The idea was to use it as a home server, which I never configured for that purpose anyway. It’s been dormant since then. Now it is up to date, and mostly synchronized with my work machine so that I can use one laptop or the other depending on my mood. The synchronization is a mix of rsync between static folders (Documents and Media that are not supposed to evolve too much over time), GitHub for version controlled projects, and kopia for backup. I used to use borg, and then Déjà Dup, which leverages Duplicity and its rsync backend. However, Kopia looks cool since we can have multiple repositories over different backends that can be synced between each other on schedule.

For my work machine, I still haven’t upgraded to the 22.04 LTS release. On the one hand, I’m still confused that Canonical is orienting itself toward snap-based distribution of applications. It sounds like a new AppStore in some ways. For instance, Firefox is just blowing slow on fist launch on my Macbook. On the other hand, Regolith Desktop is not available for the latets LTS release. I could set up a Nordic GTK theme for Gnome shell and switched back to i3-gaps, but I also customized my OS a lot (remove snap, disable tracker, and so on), which means I may just reinstall everythng from scratch using an USB stick at some point. Let’s wait and see.

I watched Severance two weeks ago, and now I’m already on Season 2 of Altered Carbon. I didn’t watch any TV shows alone since last December at least. It’s been a while. With my son, we are watching Vikings: Valhalla when we have soem free time together. In the meantime, I’m blowing up the counters on last.fm with around 800 tracks every week. So far so good. I still have some vinyls to listen to late in the night when I’m at home.

Random pictures from the past days:

I spent a few days off, due to public holidays here. I’m eagerly waiting for the longer Summer break. In the meantime, I still have lot of work to do, and lot of side projects to tackle on. Anyway, I’m now listening to Berlin, The Metro. Let’s all have a good evening then.

♪ ‘Til Tuesday • Voices Carry

Micro posting in May

Fri, 27 May 2022 22:20:22 +0200

2022-05-05:
4,000 static pages!
2022-05-05:

Core values are self-reinforcing. They attract like-minded people, who will then defend them. — The values of Emacs, the Neovim revolution, and the VSCode gorilla

2022-05-05: Pollen’s sequel? Quad: document processor. #racket
2022-05-05: Parquet and Postgres in the Data Lake. #database
2022-05-05: Substrait: Cross-Language Serialization for Relational Algebra. #database
2022-05-05: What we can deduce from a leaked PDF.
2022-05-11:
Lush plants on the loggia…
2022-05-11: Marianne Faithfull, Love more or less.
2022-05-11:

The /bin vs /usr/bin split (and all the others) is an artifact of this, a 1970’s implementation detail that got carried forward for decades by bureaucrats who never question why they’re doing things. It stopped making any sense before Linux was ever invented, for multiple reasons (…) — Understanding the bin, sbin, usr/bin , usr/sbin split

2022-05-11: How to Use Autocompletion in Vim. #vim
2022-05-11: My story on “worse is better”.
2022-05-11: RSS Feed Best Practises.
2022-05-12:

Running a personal website is an emotional roller coaster. This fact seems strange to me, because I feel I do a fairly good job of keeping my ego disengaged from the process. I realize that whether people like what I write or hate it has absolutely nothing to do with my worth as a human being. Yet, receiving comments and emails from readers who say they enjoy my website and understand its value feels good. — The Joys and Sorrows of Maintaining a Personal Website

2022-05-12:

The point is to support programming interactively. You don’t want to have to kill your program and rebuild it from scratch just because you changed a definition. That’s silly; adding and changing definitions is most of what you do! If your development environment is going to support interactive development, then it had better know how to keep your program running when you change some definitions. — On repl-driven programming

2022-05-12: Making Things Flow: Making the behaviour of Vim flow . #vim
2022-05-12: Roll A Lisp In C - Evaluation. #clang #lisp
2022-05-12: Use Fast Data Algorithms.
2022-05-12: Vim’s useful lists. #vim
2022-05-13: Vampire Weekend, This Life.
2022-05-13: I just rewrote (again) the build policy for my Stata lecture notes. Org markup language is fantastic after all!
2022-05-14:
Tuning and overoptimizing Neovim startup time (19 plugins, including 7 opt ones).
2022-05-14:
Giant salad for today’s lunch!
2022-05-14: The Cure, The Hanging Garden.
2022-05-14:

Lisp has a specific world view which makes it less compatible with the current outside world. — Why Lisp is different

2022-05-14: Why Did Heroku Fail?
2022-05-14: htmx gives you access to AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext.
2022-05-16:

2022-05-16: Aaron Neville, Tell It Like It Is.
2022-05-16: > My conclusion after using Julia for many years is that there are too many correctness and composability bugs throughout the ecosystem to justify using it in just about any context where correctness matters. — Why I no longer recommend Julia
2022-05-16: R Workflow by Frank Harrel. #rstats
2022-05-16: The Modern JavaScript Tutorial.
2022-05-17: Real-time LaTeX Using (Neo)Vim. #vim #tex
2022-05-17: Starting with Common Lisp in 2020. #lisp
2022-05-17: menyoki: Screen{shot,cast} and perform ImageOps on the command line.
2022-05-18: Iggy Pop, Some Weird Sin.
2022-05-18:

So, having spent hundreds of frustrating hours developing a working OAUTH2 solution for GMail, I am defeated at the final hurdle. Google’s demands mean that I am simply not going to be able to support GMail past May 31st, however much it hurts me to feel that I am letting my users down.> My deepest apologies to you all. — http://www.pmail.com/newsflash.htm

2022-05-18: Professional Programming: The First 10 Years.
2022-05-19: Kajagoogoo, Too Shy.
2022-05-19: > I suppose this is another one of those things we all end up doing: Write a printf() implementation, create an IRC bot, and author a book about Open Source. — Uncurled
2022-05-19: Org-babel as an alternative to Jupyter notebooks.
2022-05-19: The Big Six Matrix Factorizations.
2022-05-19: Unfinished Business with Postgres. #database
2022-05-20: Laurie Anderson, Slip Away.
2022-05-20: A maximally-dense encoding for n-choose-k.
2022-05-20: Logging C Functions. #clang
2022-05-23:

Shy away from numbers or concrete analysis during design. Good engineers estimate. Great engineers speculate. — The mindless tyranny of ‘what if it changes?’ as a software design principle

2022-05-23: Modern Python: Big ideas, Little Code. #python
2022-05-23: Python Standard Library changes in recent years. #python
2022-05-23: Useless Use Of dd. #unix
2022-05-23: What’s in which Python. #python
2022-05-24: Elliott Smith, Waltz #2.
2022-05-24:

The big endian advantages tend to be cosmetic or convention based, or of minor usefulness. The little endian advantages offer real world performance boosts in many cases (given the right algorithm or encoding to take advantage of it). — On Endianness

2022-05-24: 10 Tips for Writing Fast Mathematica Code.
2022-05-24: The unreasonable effectiveness of f‍-‍strings and re.VERBOSE. #python
2022-05-24: Useful Python decorators for Data Scientists. #python
2022-05-25: Bit Twiddling Hacks.
2022-05-26: Donald Knuth on work habits, problem solving, and happiness.

Random sampling by the inverse CDF transformation

Thu, 26 May 2022 11:45:44 +0200

There are many ways to generate random samples from a given distribution, including Monte Carlo simulation. In this case, the random number generation process consists in generating i.i.d. uniform $\mathcal{U}(0,1)$ RVs, convert them to independent $\mathcal{N}(0,1)$ RVs, and eventually convert them to correlated $\mathcal{N}(0,1)$ RVs. Once we get uniform random variables on $(0,1)$, we can use one of the following methods to convert them to gaussian RVs: the Box-Muller or the Marsaglia polar method, or the inverse CDF transformation.

Regarding the Box-Muller approach the idea is as follows: You generate two independent $\mathcal{U}(0,1)$ RVs on $(0,1)$ and then you define two new independent $\mathcal{N}(0,1)$ RVs, $x_1$ and $x_2$, using this transformation:

$$ \begin{equation} \begin{aligned} x_1 &= \sqrt{-2\log(y_1)}\; \text{cos}(2\pi y_2)\cr x_2 &= \sqrt{-2\log(y_1)}\; \text{sin}(2\pi y_2). \end{aligned} \end{equation} $$

To get ride of the trigonometric functions, the Marsaglia’s polar method instead considers two independent $\mathcal{U}(0,1)$ RVs on $(-1,1)$ and accept them if $r^2 = y_1^2 + y_2^2 < 1$.¹ Once accepted, generate $x_1$ and $x_2$ as follows:

$$ \begin{equation} \begin{aligned} x_1 &= \sqrt{-2\log(r^2)/r^2}\; y_1\cr x_2 &= \sqrt{-2\log(r^2)/r^2}\; y_2. \end{aligned} \end{equation} $$

Finally, the inverse CDF transformation is a little more involved. Recall that a standard gaussian variate $Z$, with mean 0 and variance 1, has the following probability density function (PDF):

$$ \phi(z) = \frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}z^2\right). $$

Its cumulative distribution function (CDF) is then $\Phi(z) = \Pr(Z<z) = \int_{-\infty}^z\phi(s)ds$. The inverse CDF method is used to compute $x = \Phi^{-1}(y)$, with $y$ distributed uniformly on $(0,1)$.

Note that the normal CDF $\Phi(x)$ is related to the error function $\text{erf}(x)$, as discussed on Wikipedia. Indeed, $\Phi(x) = \frac{1}{2} + \frac{1}{2}\text{erf}(x/\sqrt{2})$, which implies:

$$ \Phi^{-1}(y) = \sqrt{2}\;\text{erf}^{-1}(2y-1). $$

If your programming language provides you with either an inverse erf or normal CDF, then you can generate a sequence of random normal deviates easily. Here is an example using Racket:

(require math/distributions)
(define (replicate f times)
 (for ([x (in-range times)])
 (displayln (f))))
(define (rnorm [mean 0.0] [sd 1.0])
 (let ([x (random)])
 (flnormal-inv-cdf mean sd x #f #f)))
(replicate rnorm 10)

Note that Racket’s random produces random variates in the open interval $(0,1)$. If you need to include 0 and 1, for whatever reason, checkout soegaard’s answer on SO.

And here are two random samples of size 1000, depicted using a boxplot: (I yet have to write a quantile-quantile renderer for Racket’s plot.)

(require plot)
(plot-new-window? #t)
(define xs (build-list 1000 (lambda (x) (rnorm))))
(define ys (build-list 1000 (lambda (x) (rnorm))))
(plot (for/list ([data (list xs ys)]
 [label (list "a" "b")]
 [index (in-naturals)])
 (box-and-whisker data
 #:x index
 #:width 3/4)))

Relevant links for Scheme users:

Random Number Generation, from the old Science Collection or the SRFI 27
MIT Scheme documentation
Philip Bewig’s own implementation, and some of his answers on SO
the Mersenne Twister algorithm for Chicken Scheme

♪ A Certain Ratio • Do the Du

A rejection rate of 20% can be expected. ↩︎

Wolfram Language Server

Tue, 24 May 2022 12:07:17 +0200

Today I learned that Mathematica 13.0 ships with a builtin language server. There are actually two VS Code plugins that provide client-side code for Wolfram LSP: lsp-wl and the official wolfram extension.

I first tried to configure lsp-wl with Neovim builtin lspconfig, using a custom server config. Neither of wolframscript or netcat proved to work in this case. However, I got something close to what we have in VS Code with the official extension using the following setup:

local nvim_lsp = require("lspconfig")
local configs = require("lspconfig.configs")

configs.lsp_wl = {
 default_config = {
 cmd = {
 "wolfram",
 "kernel",
 "-noinit",
 "-noprompt",
 "-nopaclet",
 "-noicon",
 "-nostartuppaclets",
 "-run",
 'Needs["LSPServer`"];LSPServer`StartServer[]',
 },
 filetypes = { "mma", "wl" },
 root_dir = nvim_lsp.util.path.dirname,
 },
}

nvim_lsp.lsp_wl.setup({ on_attach = lsp_attach({}) })

The lsp_attach function does take care of formatting on save, completion and common key mappings across LSPs.

I mostly use the mma filetype because I rely on mash for batch processing Mathematica code in Org document, as described in a minimal Org setup to write scientific notebooks.¹

I still have to manage how to get omnifunc completion working, but for the moment I have hover information, diagnostics and formatting.

♪ Heather Nova • Heal

This also seems to work with iron.nvim, at least as well as with wolframscript when I last tried. ↩︎

Org and Jupyter

Wed, 18 May 2022 12:48:23 +0200

I do like Org in Vim, but I need Emacs to process my code chunks via Org babel. I already described my setup in earlier posts of mine, but I reached the point where I think that ESS is no longer the definite answer for Org + R or Org + Stata documents.

I also realized that ESS support for Stata is going to an end (like was the case of xlispstat a few years ago). It’s probably time to rebase all subsequent batch processing of Org files to the corresponding Jupyter kernel.

stata support is now obsolete since we were unable to elicit FSF paperwork from some of the original authors: see the lisp/obsolete sub-directory on the ESS github repo — Changes and New Features in 19.04 (unreleased)

Needless to say, using Jupyter for interactive work in the terminal is also a better option since the terminal version of Stata does not have any support for readline. Since I pushed a PR for Stata base support to iron.nvim, I should better rewrite a proper alternative using the stata kernel.

As mentioned in the introduction, I still relies on Emacs for most of the work. Ubuntu 20.04 LTS comes with Emacs 26.3, which is pretty old by any standard, and does not support modules, which are a prerequisite for installing emacs-jupyter. Since my Emacs config was outdated and quite a mess actually (I installed some packages via apt, others manually, and yet a few other packages via Emacs package-install), I decided it was time to install a fresh new version of Emacs. There are prebuilt binaries (Emacs 27 and 28) available for Ubuntu via a dedicated PPA:

$ sudo add-apt-repository ppa:kelleyk/emacs
$ sudo apt install emacs28

Then, I installed a bunch of packages with package-install. I know I should be using straight but since I will only be using Emacs as a postprocessor for my Org files, I don’t want to bother with an alembicated init.el file. Part of my config file is shown below:

(when (and (version< emacs-version "26.3") (>= libgnutls-version 30603))
 (setq gnutls-algorithm-priority "NORMAL:-VERS-TLS1.3"))

(require 'package)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/"))
(with-eval-after-load 'package (add-to-list 'package-archives '("nongnu" . "https://elpa.nongnu.org/nongnu/")))

(package-initialize)
(unless package-archive-contents
 (package-refresh-contents))

(setq package-list '(almost-mono-themes evil geiser slime paredit rainbow-delimiters org-contrib jupyter ess))
(dolist (package package-list)
 (unless (package-installed-p package)
 (package-install package)))

So far, so good. I should now be able to use either ESS or Jupyter to process my Org code blocks. Let’s get back to Stata.

Ironically, I reverted back to using ESS and ob-stata when I reworked my stata-sk handouts. Originally, I was interested in Jupyter because of bad support for Stata, even after some hacking (which no longer applies, by the way), and because at that same time I was using Stata 15 which has native support for SVG graphics, which are the default backend in stata_kernel. After blaming my GitHub repo once again, I reverted the changes I introduced in some of the previous commits.

First diff:

- #+begin_src stata :session :results output :exports both
+ #+BEGIN_SRC jupyter-stata :exports both
 sysuse auto, clear
 summarize price mpg
 regress price mpg
- #+end_src
+ #+END_SRC

Doom Emacs used to use lowercase Org blocks.

Second diff:

- #+BEGIN_SRC jupyter-stata :exports both
+ #+BEGIN_SRC stata
 sysuse auto, clear
 summarize price mpg
 regress price mpg

I moved common header arguments to a global #+PROPERTY.

Third diff will be essentially identical to the first one.

And so I’m back to using jupyter-stata in my SRC block, instead of stata. Besides replacing a few header options here and there, one of the consequence of using Stata 13 is that I also have to get ride of SVG or even PNG output. This means that I need to quietly generate the charts and then save them to EPS. This was already what I was doing when using ob-stata, but now I need to silence out the graph otherwise Jupyter complains. All EPS files are then converted to PNG using a shell script:

for i in fig-*.eps; do convert -density 300 -quality 85 "$i" "dist/${i%%.*}.png"; done

So far it seems to work great. I just need to update my default Org config, in addition to my Stata handouts once again.

♪ Hoodoo Gurus • What’s My Scene

Common Lisp Hyperspec in Vim

Tue, 17 May 2022 21:33:20 +0200

I really like the keywordprg mapping in Vim. It’s the capital K that you press to get help for the word under the cursor (known as cword in Vim parlance). If you use an LSP client (builtin Neovim LSP, Coc or ALE, for example), you likely use it everyday since it is generally mapped onto the Hover method of the LSP server.

When not in an LSP-aware environment, I write my own keywordprg. You can find plenty of nice ideas on the web. Since I use some programming languages that don’t come with an LSP yet, I have the following in my after/ftplugin/stata.vim settings, for instance:

setlocal keywordprg=stata_help

As mentioned in a footnote of one my older posts, I query Stata’s online documentation when working from TUIs. In this case, stata_help is just a wrapper shell script for links https://www.stata.com/help.cgi\?"${1}". This way, when I’m over a Stata command, I just press K and I get its online help in a tab, with a proper text-mode browser. Note that I prefer Links over Lynx for text-mode browsing the internet.

This morning I wrote a little Lisp script using Roswell, since I wanted to try it for a moment. I already configured Roswell to fetch the latest release of SBCL on my laptop because Ubuntu’s default version is pretty outdated. It works great but I never use the scripting facilities that Roswell offers. I wrote two versions of a small utility that allows to fetch the relevant documentation from Lispworks Hyperspec online documentation. The first version was a simple lookup through the documentation to retrieve the URL corresponding to the symbol at point and to return a system call to open it in the default browser. The lookup step is performed thanks to the hyperspec package. I used ASDF builtin facilities to launch external program (uiop:run-program) but one could also makes use of SBCL’s builtin extensions (sb-ext:run-program). Finally, I decided to just return the URL to the standard output, in case I need to reuse the script for another purpose. Both versions are shown below:

#!/bin/sh
#|-*- mode:lisp -*-|#
#|
exec ros -Q -- "$0" "$@"
|#

(with-output-to-string (*standard-output*)
 (asdf:load-system :hyperspec))

;; (defun main (cword &rest argv)
;; (declare (ignore argv))
;; (uiop:run-program
;; (list "/usr/bin/xdg-open"
;; (funcall #'hyperspec:lookup (read-from-string cword)))
;; :output nil))

(defun main (cword &rest argv)
 (declare (ignore argv))
 (format t "~D~%" (hyperspec:lookup (read-from-string cword))))

Since I just fetch the URL, I need a way to tell Vim to display the online doc right inside a tab. This is done by adding setlocal keywordprg=hyperspec_lookup in my after/ftplugin/lisp.vim file, where hyperspec_lookup is just a wrapper shell script:

#!/usr/bin/env bash

rc=$(hyperspec_lookup.ros "$1")
links "$rc"

And now I get the full CLHS documentation for free when working in neovim!

[2022-08-22]
An even better soluton is to use dasht and the dasht.vim plugin. The later detects filetype automagically and provides help for the relevant docset, although it is still possible to search the whole collection of docsets. It works great for Common Lisp and Racket, but I’m still looking for R5, R6 and R7RS docsets for Scheme. Note that for CSS? HTML and Javascript, the documentation is taken from Mozilla MDN and it’s a bit crapy since the whole page header is returned together with the result. Also, if you prefer to use Fzf from the command line, the following shortcut may come handy:

dasht-query-line $@ | perl -0777 -pe ’s/name = (.?)\ntype = (.?)\nfrom = (.?)\nurl = (.?)\n/ $4 | $1 | $3 | $3\n\n/g’ | column -ts’|’ | fzf –no-sort –with-nth=2.. –bind ’enter:execute(w3m {1})+accept’

. (Don’t remember where I found this snippet, sorry!)

♪ Echo & The Bunnymen • All That Jazz

Neovim colorscheme switching and Tmux

Fri, 13 May 2022 11:30:11 +0200

At work I am sometimes bothered by the ambient light. This was not much of a problem until recently when I switched to a dark theme. The Nord theme is really great, but admittedly a light theme is more suitable in a bright room. Since I wanted to try out the vimbones theme from the zenbones plugin, I thought it would be a good idea to set up a dual theme for Neovim and Gnome terminal. Most of my other TUI settings depend on $TERM colors, so I only need the correct values for the terminal (16 colors). The zenbones plugin comes with extra configuration for various terminal emulators, including Kitty. I just grabbed the relevant settings for Gnome terminal, and copy the colorscheme Vim file. At this point, I do not want to rely on extra packages, like Lush, so I deleted part of the vim script file.¹ And here we are:

The light theme is clean and elegant, and it matches almost all of my own settings for the Nord theme, which I modified heavily to discard fancy stuff that I am not interested in.

Regarding automatic colorscheme switching in (Neo)vim, I saw various topics on the internet, including settings global environment variable to define the colorscheme from within (Neo)vim config file, see Set Vim background based on your terminal’s background color. I don’t really like depending on environment variable, so I continued to explore a bit on the internet and I discovered that we can simply test for the &background value in Vim. Indeed, Vim and Neovim can detect the background color of the terminal where they are running, and set background accordingly. Here’s the gist, as a vimL function:

function! misc#set_theme() abort
 if &background ==# 'dark'
 colorscheme nord
 else
 colorscheme bones
 endif
endfunction

This has to be used in an autocommand like this:

augroup colortheme
 autocmd!
 autocmd OptionSet background ++nested call misc#set_theme()
augroup END

At this point, I’m not sure we really need the ++nested option. This works great when Neovim runs in Gnome terminal. However, it doesn’t work in Tmux because it doesn’t handle OSC 11!

There’s a hacky solution, though. The gist comes from user erw7 on GitHub:

vim.loop.fs_write(2, "\27Ptmux;\27\27]11;?\7\27\\", -1, nil)

I wrapped the above in a conditional statement that checks whether we are running Neovim inside Tmux or not. Since I set Tmux as “tmux-256color”, this amounts to something like if $TERM ==# "tmux-256color". And, well, that’s it!

♪ Arctic Monkeys • Fluorescent Adolescent

After two years using Neovim, I still use less than 20 plugins in my config set. No friction, no reverse dependencies to solve every two weeks. ↩︎

Micro posting in April

Wed, 27 Apr 2022 21:42:05 +0200

2022-04-01:
Hot off the kitchen
2022-04-01:

Like many here, I turned off commenting years ago. My motivation had less to do with spam. (I’m on WordPress and it does a pretty good job with that.) It had more to do with the conversations moving from my site to the social web. Folks wanted to talk about stuff where they already were, rather than centralizing that conversation on individual blogs. — Ask HN: Are blog comments a thing of the past?

2022-04-01:

Note that “semantic understanding” is mostly unrelated to the traditional interpretation of “IDE” as Integrated Development Environment. I personally don’t feel that the “Integrated” bit is all that important. I commit&push from the command line using Julia scripts, rebase in magit, and do code reviews in a browser. If anything, there’s an ample room for improvement for the integration bits. For me, I in “IDE” stands for “intelligent”, smart. — Why an IDE?

2022-04-01: Building a Full Stack Application with Flask and HTMx.
2022-04-01: Implementing cosine in C from scratch. #clang
2022-04-02:

Atom Editor was, and still is, an amazing piece of technology. It was the first practical example of web technologies running locally, applied to a really hard problem: text editors. It it was, and still is, darn good at handling code. — The impossibility to maintain Atom

2022-04-02:

Consume input from stdin, produce output to stdout. Put another way, your program should be a filter. Filters are easily integrated into shell pipelines, arguably the most important utility for Unix tools composition. — Hints for writing Unix tools

2022-04-02: Apparently, the formers developers of Atom (whose status remains unclear at the time of this writing) are now developing a new text editor, Zed. Note that there’s also Lapce in the radar.
2022-04-04: Principal Component Analysis for Data Science .
2022-04-04: Roll your own Ngrok with Nginx, Letsencrypt, and SSH reverse tunnelling.
2022-04-04: Using Vim/Emacs is like a refactoring process. #vim
2022-04-04: Why we don’t use a staging environment.
2022-04-06:

Sometimes, though, I think there’s a value in doing things the old fashioned way, if only to get a little “closer to the ground” on what our tools are doing behind the scenes. — The Simple Joy of Coding by Hand

2022-04-07:

Textbook examples are there to illustrate a method. The recommended approach is to first give two examples where the method works, then one where it fails. But we, the textbook writers, are busy, so we’ll often just give the one that works and stop. As I wrote once, “the book has a chapter on regression discontinuity designs. That’s fine. But all we see in the chapter are successes.” — The examples you see in the textbooks are not representative of the sorts of problems you see in the real world

2022-04-08: P!nk, So What.
2022-04-08: After Nova, yet another code editor for Mac: Zas Editor.
2022-04-08: An Intuitive Guide to Linear Algebra.
2022-04-08: The 0,5 MB of nothing in all Apple Music files.
2022-04-11:

2022-04-11: What Every Programmer Should Know About Memory?.
2022-04-11: What scientists must know about hardware to write fast code.
2022-04-12: An Ode to Apple’s Hide My Email.
2022-04-12: Starship: The minimal, blazing-fast, and infinitely customizable prompt for any shell!
2022-04-12: What Is Rust’s unsafe? #rust
2022-04-14:

Unlike systems software, the values demanded from text editors or IDEs vary greatly depending on who you ask. These are much more personal tools and make room for a diverse set of desires. — The values of Emacs, the Neovim revolution, and the VSCode gorilla

2022-04-14: How I Vim: Andy Stewart, a.k.a airblade. #vim
2022-04-14: Prettify your screenshot using imagemagick.
2022-04-14: TypeScript / How the compiler compiles.
2022-04-14: Vim Key Combinations. #vim
2022-04-14: What are the dark corners of Vim your mom never told you about?. #vim
2022-04-17: Akira Kosemura, Mercy.
2022-04-17: A Quick Guide to Mutt.
2022-04-17: Git Jump to Vim Quickfix. #vim
2022-04-19: Alain Kan, Speed My Speed.
2022-04-19:

Being `pure functional’ does not eradicate state; it merely changes the way state is handled – which could be either more or less clearer and convenient. — Grasping ‘all-the-apples-at-once’

2022-04-19: Nixery provides ad-hoc container images that contain packages from the Nix package manager.
2022-04-20: Alice in CHains, Down in a Hole.
2022-04-21: Choosing the Right Integers. #clang #rust
2022-04-21: Exponentially Better Rotations.
2022-04-21: Helix: A post-modern modal text editor. Looks cool.
2022-04-21: LaTeX A Modern Approach. #tex
2022-04-21: Memray is a memory profiler for Python. #python
2022-04-21: Scheme for scientific computing.
2022-04-25: Jiři Horák, Ravens Home.
2022-04-25:

I don’t think it was a big technical innovation (it’s obvious that you want to separate a language-agnostic editor and a language-specific server). I think it’s a rather bad (aka, “good enough”) technical implementation (stay tuned for “Why LSP sucks?” post I guess?). But it moved us from a world where not having a language IDE was normal and no one was even thinking about language servers, to a world where a language without working completion and goto definition looks unprofessional. — Why LSP?

2022-04-25:

You dumbass. You thought you needed media queries to be responsive, but no. Responsive means that it responds to whatever motherfucking screensize it’s viewed on. This site doesn’t care if you’re on an iMac or a motherfucking Tamagotchi. — This is a motherfucking website

2022-04-25: Creating a tmux Colour Theme.
2022-04-26: Dirk Maassen, Still.
2022-04-26: Working at producing elegant (or at least less common) notebooks using Org babel.
2022-04-27: The Bela Session by Bauhaus
2022-04-27:

I know you’re going to be tempted to reply with “you’re using it wrong” - but I’m not. This is how I like to use my computer. And it is clear that the MacBook isn’t my computer - it is Apple’s. — Things I can’t do on MacOS which I can do on Ubuntu

2022-04-27: Now I’m calculating with constructive reals!

Helix editor

Sat, 23 Apr 2022 20:28:05 +0200

And here’s a fresh new text editor: Helix. In fact, I heard about about Helix one year ago, when it was still in early alpha, but I didn’t pay much attention to it. It is written in Rust, as many of the little apps (fdfind, exa, tig, etc.) I use all the day, much like one of the best-selling UI for Neovim(not tested).

Apparently, Helix got inspired by both Vim and Kakoune. It ships with LSP-ready configurations, as well as a default picker. How long will it stay on the “market”, I don’t know, but I like the idea. The best comment I found on Hacker News thus far was the one by rodgerd:

If you’re trying to get people to try your new editor, no matter how awesome the ideas that you’re trying, you’re going to lose a chunk of your audience if it doesn’t support C syntax highlighting, a chunk if it doesn’t support Python intellisense, and so on and so forth. You can have the best idea in the world, but people will play with it, and decide that they love it but can’t use it for “real work” unless you slog through adding native support for a bunch of languages and runtimes/compiler suites.
That’s a huge burden.
Now, if you implement LSP, you get “for free” support for C#, C, Python, Rust, etc etc. People can evaluate your editor on its merits as a text editor, not on whether you had the time and energy to add support for their favourite language.

Soon after this comment, the whole comment thread starts to be a replay of the Vim vs. Emacs flame, and point-and-click vs. keyboard-driven approaches to text editing. I hope HN will still be alive after Twitter burned out of Hell.

I spent a few hours playing with Helix today. I compiled it from scratch using cargo and followed the instructions to set up its runtime (865 Mo!) and so on. It works really great, I mean for a text editor written from scratch in Rust. I really like the experience, even if some keybindings are not that intuitive when coming from (Neo)vim. This is easily configurable, though. Modal editing is great in any case, and I suspect it would look even more natural for people coming from Kakoune which I never tried. For those coming from Neovim, I would say it feels the same, with sane defaults. The built-in LSP and Treesitter integrations work as advertised, and you also get some DAP debugging for Rust and C/C++. The configuration is easy to manage (this is a simple toml file), and there are plenty of gorgeous color themes to choose from. I do not really like the popup window that presents available combination of keypresses (much like which-key in Emacs or Vim), but I guess some like it. I am more comfortable with my own config for Neovim, but I would surely recommend Helix for anyone interested in a zero-config powerful text editor for coding. Fron what I’ve read over GitHub, the authors do not plan to add support for third-party plugins. Maybe it’s a good idea after all. Eventually, there may be a GUI at some point (one screenshot comes from this webiste, btw).

[2023-06-22]
Here is an excellent review of Helix, from an experimented (Neo)vim user, and how it compares to Neovim or Kakoune when it comes to performing simple or more complex editing tasks.

♪ Ida Sand • Burning

Random links from April

Wed, 20 Apr 2022 21:45:28 +0200

Here are some random links collected the past few days, mostly about Scheme and related languages/tools.

Various scientific packages for Chicken Scheme:

iraikov/chicken-dataframe, so we get some useful data structure like in Racket.
raikov/chicken-mpi, some MPI bindings for Chicken Scheme.
iraikov/chicken-lapack, Chicken Scheme bindings for ATLAS and LAPACK; looks old but maybe it is worth a try.
dieggsy/csl, the mandatory toolbox for rapid prototyping; must check if everything is included.

Others:

Essentials of Programming Languages Language in Racket (see chenyukang/eopl for Chicken solutions).
The Scheme and Functional Programming Workshop for 2020.
Lispy in Scheme
Getting started with Scheme by solving an Advent of Code 2018 challenge
Chez Scheme vs. SBCL: a comparison
Fun with speech synthesis and Chicken Scheme
Scheme-ing

Lisp is a great language and you too can trust it if you write numerical programs of 5000 lines or less. You can probably trust it for other tasks as well but I don’t have such an experience with Lisp. Please tell the world that Lisp is a great language for numerical/statistical analysis. Those who dismiss it are wrong. There’s absolutely no reason to not give it a trial. I am not afraid to admit that I’ve used Lisp for real work

♪ Lisa Hilton • More Than Another Day

On choosing your colorscheme

Tue, 19 Apr 2022 19:29:19 +0200

I devised my own variant of the Nord color theme for Neovim. I started with the most up to date package I found on GitHub (shaunsingh/nord.nvim) and updated the default settings, which means that I removed almost all colors. Surely Nord is a dark theme, but a pretty decent one. The only competitor to date is zenbones, IMHO. Both Neovim plugins provide Treesitter-specific highlighting, and a bunch of highlight groups for various plugins.

Previously, I devised a similar dark theme for Doom Emacs, and later on a light duotone colorscheme for Vim. Both themes rely on similar principles: highlight constant (strings and numbers, with a dimmed color for the latter) using soft (pastel) color, and “top-level” reserved keywords, e.g. functions or classes, import statements and things like that, mostly using bold typeface or a lighter color. The rest is left as is, as otherwise it becomes distracting in my view. And losing attentional sight is not a good idea when you spend most of your day in front of your editor. In this regard, Nicolas Rougier’s article, On the design of text editors, is a great read, especially section 4 on colorization:

Wayne goes a step further and denounces the use of syntax highlighting since it is a waste of an important information channel and suggest several alternative uses of color, among which, rainbow parenthesis, context highlighting, import highlighting, argument highlighting, type highlighting, etc. Instead of syntax or semantic colorization based on content, a simple alternative would be to adapt colorization to the reader, taking attentional constraints into account.

[2022-09-08]
For a more extreme take, see A case against syntax highlighting. See also Your syntax highlighter is wrong.

Of course, your mileage may vary. What is important is that your colorscheme fits with your own perception and way of working, or in other words that it brings a little more to the table than no color at all.

♪ Arno • Help Me Mary

The unquantified self #22

Sun, 17 Apr 2022 21:52:46 +0200

Things are growing, outside. Compare those two pictures, taken two months apart:

It’s already April and I feel like I haven’t written that much. Neither on this blog, nor on my digital notebook by the way. Anyway, time passes even if it is not to catch up with it. Regarding side projects, I still have in mind the idea of writing a statistical library for Scheme. It will probably target Racket and/or Chicken Scheme. I started playing again with the latter, which is now Chicken 5. I first learned about Chicken Scheme more than 10 years on Jan de Leeuw’s tech listserv. I played with it a little bit, but that’s all. Now that I reinstalled it, reread the documentation, and discovered that the IRC room is pretty active on Libera.chat, I thought it would be a good idea to investigate more time into the language itself. I don’t really want to investigate too much into Chez Scheme for it lacks tooling and packaging (although there are some external package managers, e.g., akku, but see Travis Hankelman).

I spent a fair amount of time tinkering my Neovim setup lately. Unlike my last adventures with Doom Emacs, though, I’m never restarting from scratching, just refining things here and there. I tried a lot of packages, then uninstalled them right away. Likewise, I try to stay close to the default key mappings so as to learn Vim the right way, or I think so. I don’t know if I made that much progress since I started using Vim, but surely I’m more productive with this text editor than I was with Emacs. I’m on the HEAD version of Neovim, and it never crashed a single time in one year. The only minor annoyances that I encountered occasionally were from minor updates regarding core helper functions or updated color scheme elements. I have other opinions on Neovim versus Emacs, but I’ll keep it for a separate post. All this also goes hand in hand with my switch to the Regolith tiling window manager, which makes everything super easier compared to Gnome desktop

My son and I discovered we missed some episodes of The Walking Dead, Season 10. Things are now settled since we just finished watching them the night before last. We now just have to find our way thru Season 11. I was surprised to hear AC/DC in the middle of one of the episodes, by the way.

♪ Jean-Yves Thibaudet • Adagio, Op. 11

Tinkering my Neovim setup

Thu, 14 Apr 2022 22:16:31 +0200

Yesterday, I spent a fair amount of time cleaning up my Neovim config files. This mostly consisted in removing non-essential plugins, and replacing old Vimscript with Lua code, thanks to the mini.nvim project. If you’re using vim-surround, hop or other jumper, or even clever-f, please have a look at what mini.nvim has to offer. It even features a minimal completion engine that I adopted right away since the successor of nvim-compe is a no-brainer for me (why would we need to install 4 to 7 plugins instead of one?!).

At some point I was about to discard packer and just rely on the builtin packadd approach, together with a set of Git submodules, that Evgeni Chasnovski uses for his own Neovim setup, but packer is great for it manages everything so gently, including filetype-specific plugins. Lazy loading is really great sometimes.

Finally, the number of packages I rely on is limited and I believe my startup time remains quite decent (< 100 ms):¹

~ % tree -L 2 .local/share/nvim/site/pack/packer
.local/share/nvim/site/pack/packer
├── opt
│   ├── iron.nvim
│   ├── magma-nvim
│   ├── neogen
│   ├── parinfer-rust
│   ├── vim-table-mode
│   ├── vim-test
│   └── vimtex
└── start
 ├── Comment.nvim
 ├── gitsigns.nvim
 ├── null-ls.nvim
 ├── nvim-dap
 ├── nvim-lightbulb
 ├── nvim-lspconfig
 ├── nvim-treesitter
 ├── packer.nvim
 ├── plenary.nvim
 ├── popup.nvim
 ├── telescope-bibtex.nvim
 ├── telescope-fzf-native.nvim
 ├── telescope-github.nvim
 ├── telescope.nvim
 ├── telescope-symbols.nvim
 └── telescope-ui-select.nvim

25 directories, 0 files
~ % vim-profiler.py nvim
Running nvim to generate startup logs... done.
Loading and processing logs... done.
Plugin directory: /home/chl/.local/share/nvim/site/pack/packer/start
=====================================
Top 10 plugins slowing nvim's startup
=====================================
1 0.774 nvim-treesitter
2 0.183 telescope.nvim
3 0.093 nvim-lspconfig
4 0.044 plenary.nvim
=====================================
times in msec
 clock self+sourced self: sourced script
 clock elapsed: other lines

Results from vim --startuptime:

000.020 000.020: --- NVIM STARTING ---
000.965 000.945: locale set
001.487 000.523: inits 1
001.499 000.012: window checked
--8<--------
-------->8--
076.311 001.266: loading rtp plugins
076.683 000.085 000.085: sourcing /home/chl/.local/share/nvim/site/pack/packer/start/nvim-lspconfig/plugin/lspconfig.vim
077.273 000.501 000.501: sourcing /home/chl/.local/share/nvim/site/pack/packer/start/nvim-treesitter/plugin/nvim-treesitter.vim
077.435 000.045 000.045: sourcing /home/chl/.local/share/nvim/site/pack/packer/start/plenary.nvim/plugin/plenary.vim
077.813 000.183 000.183: sourcing /home/chl/.local/share/nvim/site/pack/packer/start/telescope.nvim/plugin/telescope.vim
077.952 000.827: loading packages
077.986 000.034: loading after plugins
077.998 000.013: inits 3
081.488 003.489: reading ShaDa
081.647 000.159: opening buffers
081.673 000.027: BufEnter autocommands
081.676 000.003: editing files in windows
082.048 000.373: VimEnter autocommands
082.052 000.003: UIEnter autocommands
082.432 000.280 000.280: sourcing /usr/share/nvim/runtime/autoload/provider/clipboard.vim
082.440 000.108: before starting main loop
083.690 001.250: first screen update
083.692 000.003: --- NVIM STARTED ---

I rewrote 50% of my old pure Vim config. The remaining is just the after/ftplugin/*, ftplugin/*, and autoload/* stuff, which I keep as Vimscript for compatibility issues.

♪ Adam Wakeman • Lean on Me

vim-profiler is available from GitHub. ↩︎

Micro posting in March

Mon, 28 Mar 2022 10:10:12 +0200

2022-03-02:

Every few years a new company says you should use their special format. You have to pay them a monthly fee to use it — or keep all of your documents in their care. They offer some convenience or features, but at the cost of flexibility, portability, and independence. — Write plain text files

2022-03-02: I’ve long been ignorant of i3 scratchpad. Now that I use it, my music dispatcher looks much more comfortable.
See also I3 Scratchpad to access often used applications.
2022-03-02: BLAS-level CPU Performance in 100 Lines of C. #clang
2022-03-02: Debugging with GDB. See also GDB dashboard. #clang
2022-03-02: Symbols vs. Strings (via Irreal). #lisp
2022-03-03: Roswell: Hidden feature of “-s” option. #lisp
2022-03-04: Steve Cardenas, Language of Love.
2022-03-06: Yumi Zouma, Give It Hell.
2022-03-06:

When the software changes, people want to know why and how. — Keep a Changelog

2022-03-06: SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm.
2022-03-06: You’re using email wrong. My philosophy is much simpler: Inbox zero, block toxic people as well as spams and hoaxes.
2022-03-09: Practical Python Programming. #python
2022-03-09: [Unix] Maildir, mutt and vim.
2022-03-11:

By carefully examining every album you own, you get to become more familiar with your library, its extent, its variation, and its quirks. People used to spend hours lovingly sorting and resorting their shelves of LPs. In the iTunes age, many of us toss our music into a heap and forget about it. This is great for some people. But there’s value in intimate, complete familiarity with your collection. So instead of a chore, try thinking of correcting tags as quality time with your music collection. That’s what I do. — Using the Auto-Tagger

2022-03-11:

I love Emacs, but doing everything in it is the wrong move. Doing it with this site was proof of that for me. — Wherein org-mode was a pain

2022-03-11: Binary search with confidence.
2022-03-11: Python built-in functions to know. #python
2022-03-11: Software Design for Flexibility: a review. #scheme
2022-03-11: The Art of Assembly language programming.
2022-03-14: Happy Mondays, Step On.
2022-03-14:

The two most important ways to make your text easy to read are a short line-length and the copious use of paragraphs. Viewing a single large block of run-away text with no line breaks immediately puts stress on the reader, as absorbing the information provided therein requires a high degree of concentration and eye movement. — The art of plain text

2022-03-14: Amateur archiving activities, January 2022.
2022-03-14: Python Design Patterns. #python
2022-03-14: Why Dark and Light is Complicated in Photographs.
2022-03-15: John Cale, Chinese Envoy.
2022-03-15:

Many services have self-service account deletion tools, but often hide them so well that I failed to locate them without deferring to their customer support. On average, it took 24 interactions (menu clicks, password entries, etc.) after an initial login to request and confirm a deletion request. — Some discouraging anecdotes on how services handle account deletions

2022-03-15:

My initial reasons for giving up on OS X / macOS were due to my frustrations around developer tooling and package management, problems now largely solved by Homebrew. I got tired of being able to easily do certain kind of development on Linux and then struggle to get all the required pieces set up in the same way on OS X. Then, as soon as Homebrew began to resolve all that, Apple decided to ruin its laptops through the introduction of the awful and annoyingly-loud butterfly keyboard and touchbar. — Linux Developer Laptops: Dell’s Precision 5500 series reigns supreme

2022-03-15: An Introduction to Just Enough Cython to be Useful. #python
2022-03-17: The Cluttered Web: A Scrapbook of Sreenshots.
2022-03-21: Nicole Glover, Hive Queen.
2022-03-21: Foliate: A simple and modern eBook viewer for Linux desktops. I should note that Zathura handles Epub documents perfectly well.
2022-03-21: Rust’s Unsafe Pointer Types Need An Overhaul. #rust
2022-03-21: T>T: When Double Precision is Not Enough. #python
2022-03-21: XTerm: It’s Better Than You Thought. #unix
2022-03-22:
First peony on the balcony.
2022-03-22: Structured Concurrency.
2022-03-23: Alice in Chains, Rooster.
2022-03-23:

Authors always say the best way to write better is to read more, which I can vouch for in blogging. Building up some RSS feeds in an aggregator like The Old Reader can be a great source of inspiration.
> But it’s also important to stress that this is only one approach. 8,000 posts sounds superficially impressive, but doesn’t speak to quality or detail. Writing a few posts a year is just as valid and worthwhile. If you’re having fun doing it, who cares. — 8,000 post feedback, and regular writing

2022-03-23:

Random effects/mixed effects models shine for multi-level data such as measurements within cities within counties within states. They can also deal with measurements clustered within subjects. There are at least two contexts for the latter: rapidly repeated measurements where elapsed time is not an issue, and serial measurements spaced out over time for which time trends are more likely to be important. An example of the first is a series of tests on a subject over minutes when the subject does not fatigue. An example of the second is a typical longitudinal clinical trial where patient responses are assessed weekly or monthly. For the first setup, random effects are likely to capture the important elements of within-subject correlation. Not so much for the second setup, where serial correlation dominates and time ordering is essential. — Longitudinal Data: Think Serial Correlation First, Random Effects Second

2022-03-23:

[I]n biology, when one research team publishes something useful, then other labs want to use it too. Important work in biology gets replicatedall the time—not because people want to prove it’s right, not because people want to shoot it down, not as part of a “replication study,” but just because they want to use the method. So if there’s somethingthat everybody’s talking about, and it doesn’t replicate, word will get out. — Biology as a cumulative science, and the relevance of this idea to replication

2022-03-23: Nice add-on to hide Firefox tab when there’s only one.
2022-03-23: Rust ownership, the hard way. #rust
2022-03-23: Two Logging Options Better than Print Statements. #python
2022-03-24: LaTeX2e unofficial reference manual.
2022-03-25: Mo-Dettes, White Mice.
2022-03-25:

Some developers won’t use an editor without a debugger, or linting, or Git built-in. For me, these aren’t hard requirements but are bonuses. Indeed, most good editors have these, or plugins that enable them. How useful they’re all depends on the language and platform you’re developing for. — Why Neovim is the best code editor / IDE for developers

2022-03-25: Hot off the kitchen.

Some handy shortcuts for Vim

Wed, 23 Mar 2022 14:07:02 +0100

Over the past two years where I’ve been using Neovim exclusively, I settled upon a few keybindings that I found on the internet and that I came to appreciate. I should note that I am now using Telescope instead of vim-fzf, and so many mappings are provided by this Neovim plugin:

<C-s> in insert mode, this is used to correct the last typo (using the first suggestion) while in normal mode it allows to replace the word under the cursor.¹
<C-p> to navigate files (Telescope find_files).
<C-q> calls a little function to close or open the quickfix (if there’re some items to show).
<leader>g and <leader>G to quickly check git status (for the whole directory or the current buffer).
<C-f> as a shortcut for Telescope live_grep; using the Ivy theme and a 20% window at the bottom, it really shines. See the screenshot at the end of the post.
<C-x> is actually a shortcut for "_dP, and it is pretty cool since it allows to visually select a region and paste it over and over again on visual blocks selected afterwards.
<C-h>, <C-j>, <C-k> and <C-l> to increase/decrease split width or height (e.g., <C-h> stands for :vertical resize -4<CR>).
<C-Left> and <C-Right> to navigate between tabs, since I can use w/b and W/B for horizontal word-wise motion. I also use <localleader><tab> to create new tabs.
<leader><leader> to switch between opened buffers (Telescope buffers). Note that <C-b> would be better, but it’s already the primary key for Tmux and I became used to the space-space combo after 3 years of Doom Emacs.
<leader>e which is basically :e <C-R>=expand("%:p:h") . "/" <CR>; I used it most of the day, since it allows to quickly open a file in the current working directory, with tab completion.
<leader>r as a shortcut for Telescope oldfiles.
<leader>$ and <leader>! to open a terminal in a new tab or in a split, respectively.
<localleader>s or <localleader>S to sort a region in lexicographic order (increasing or decreasing).²
<localleader>" for Neogen

See also this related post: Custom Vim shortcuts (2/n).

In addition to the above, I also have handy shortcuts for Magma, vim-surround-like stuff,³ etc.

Finally, I don’t really like or need snippet engines, so I just defined some handy abbreviations, like this:

iabbrev +w []()
iabbrev +@ chl@aliquote.org
iabbrev +% --8<--------<CR>-------->8--<Up>
iabbrev +' ``
iabbrev +" ```<CR><CR>```<Esc>2k
iabbrev +:: ¯\_(ツ)_/¯
iabbrev +:( ಠ_ಠ
iabbrev +:) ಠ‿ಠ

♪ Charlie Haden • Conception

<c-g>u<Esc>[s1z=`]a<c-g>u (Credit: Gilles Castel, see also Francis Tseng or SO), and :let @s='\<'.expand('<cword>').'\>'<CR>:%s/<C-r>s//<Left> and sy:%s/<C-r>s//<Left> (Credit: bluz71) ↩︎
vip:sort u<CR> in visual mode and :sort u<CR> in normal mode (Credit: strager) ↩︎
E.g., vnoremap _( :call text#surround('(', ')')<CR> ↩︎

Auto-tagging my music library

Sat, 12 Mar 2022 21:12:02 +0100

I use Cmus as my main media player, and occasionally mpv although I mostly use the later for video playback (including streamlink). I’m pretty happy with my current setup. Everything runs from a terminal, there’s no fancy UI nor unwanted distraction, just the raw output. My music library actually weighs about 62 Go and it’s full of MP3, some OGG as well as some videos, but not everything is in order. I mean, I miss a lot of tags for some albums that I ripped from my personal CD collection. Here comes the idea of auto-tagging everything. I tried three solutions. Here are my very first impressions.

I tested all three software on a specific folder in my database, which I know is not the easiest one, since it features a mix of A Certain Ratio and Joy Division’s live shows, among others. And I warmly recommend everything from The Factory Records.

Beets

At first, I was impressed by beets, especially since it is a command-line only utility, which comes with many add-ons (aka plugins). I configured its settings based on the documentation and what I could find on the intertube.

Tagging:
 Various Artists - Live at Leigh Rock Festival 1979
URL:
 https://musicbrainz.org/release/e01790b2-77ba-4b7d-8383-118a6cdc700d
(Similarity: 47.9%) (missing tracks) (2xCD, 2006, GB, Ozit)
CD 1
Missing tracks (29/35 - 82.9%):
 ! Leigh Festival Stage Announcements/Disorder (# 1) (3:59)
 ! Leaders of Men (# 2) (2:35)
 ! Colony (# 3) (3:56)
 ! Insight (# 4) (3:39)
 ! Digital (# 5) (2:48)
 ! Dead Souls (# 6) (4:18)
 ! Shadowplay (# 7) (3:30)
 ! She's Lost Control (# 8) (3:24)
 ! Transmission (# 9) (4:04)
 ! Interzone (#10) (2:08)
 ! Intro of Sound of Music (#11) (0:25)
 ! Tony Wilson Fucker (#12) (3:01)
 ! Messages (#19) (4:36)
 ! Bunker Soldiers (#20) (2:52)
 ! Julia's Song (#21) (4:16)
 ! VCBXL (#22) (4:07)
 ! Almost (#23) (3:56)
 ! Red Frame White Light (#24) (3:45)
 ! Mystery Reality (#25) (3:28)
 ! Electricity (#26) (3:54)
 ! Second Head (#27) (4:00)
 ! Brave Boys Keep Their Promises (#28) (2:31)
 ! Sleeping Gas (#29) (3:45)
 ! Take a Chance (#30) (2:30)
 ! Ha Ha I'm a Drowning (#31) (4:16)
 ! Read It in Books (#32) (2:32)
 ! Straight Reigns (#33) (2:26)
 ! Camera Camera (#34) (3:07)
 ! Went Crazy (#35) (3:20)
Apply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort, plaY? M
Finding tags for album "Various Artists - Live at Leigh Rock Festival 1979".
Candidates:
1. Various Artists - Live at Leigh Rock Festival 1979 (47.9%) (missing tracks) (2xCD, 2006, GB, Ozit)
2. Joy Division - Leigh Rock Festival (21.6%) (tracks, id, missing tracks, ...) (12" Vinyl, 2008, GB, Ozit-Morpheus Records, OZITLP8796)
3. Whitesnake - Live at Reading Rock Festival 1979 (19.4%) (tracks, id, artist, ...) (CD, 2011, XE, EMI, 50999 679550 2 2)
4. Bernard O Neill - Zumzeaux Live at Edinburgh Folk Festival 1993 (15.6%) (tracks, id, missing tracks, ...) (bandcamp, Vinyl, 2021, FR, Bernard O Neill)
5. Betsy Bell & Mary Gray, Come A Maying, Gemma Khawaja - Live at Leigh Folk Festival 2014 (14.9%) (tracks, id, artist, ...) (bandcamp, Digital Media, 2014, GB, Gemma Khawaja)
6. H.E.A.T - Live at Sweden Rock Festival (14.9%) (tracks, missing tracks, id, ...) (Digital Media, 2019, XW, earMUSIC)
7. Various Artists - Rock On: 1979 (14.1%) (tracks, missing tracks, id, ...) (1996, US)
8. Various Artists - Suomi-rock 1979 (13.5%) (missing tracks, tracks, id, ...) (CD, 1994, FI, Poko Rekords, CDROCK 79)
9. Crazyhead - Live at Reading Rock Festival 1989 remastered audio. (13.3%) (id, unmatched tracks, artist, ...) (bandcamp, Digital Media, 2016, GB, Crazyhead)
10. Steel Pulse - Live at Montreux Jazz Festival 1979 (11.9%) (tracks, id, artist, ...) (1979, CH)
11. ETRON FOU LELOUBLAN - Live At The Rock In Opposition Festival, 1978 (11.8%) (tracks, id, artist, ...) (bandcamp, Vinyl, 2015, FR, Replica Records)
12. 1978, Etron Fou Le Loublan,  “ Live At The Rock In Opposition Festival - New London Theatre »
par Dino (6.4%) (id, unmatched tracks, artist, ...) (bandcamp, Digital Media, 2018, FR, Revue & Corrigée)
# selection (default 1), Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort, plaY?

It is nice that it asks us to decide what to do next. However, it may soon become cumbersome for mass retagging. Moreover, the returned results are not always very accurate, even if the above example doesn’t exemplify this claim. On the first run, the correct 2 CD set did not figure on the first position, although I don’t why it is the case when I replicated the import process later.

Onetagger

I came across this software by chance. I was surprised there was a Linux ready binary to download. The UI is quite nicely done, and it is easy to start the auto-tagging process. You must select one or more external sources, including iTunes (with limited queries) or Spotify (needs a free account). The results were quite deceptive. I got a lot of failed downloads from one source, and even if all queries to Apple Music were okay (but not that successful), I ended up with all ID3 infos removed from my tracks. Don’t know why.

--8<--------
Mar 12 20:57:18.106 INFO [ITunes] State: Ok, Accuracy: Some(1.0), Path: /home/chl/Media/Music/Albums/A Certain Ratio/Live At Leigh Rock Festiv
al '79 [Disc 1]/1-17 Crippled Child.mp3, module: onetagger::tagger
Mar 12 20:57:18.106 INFO iTunes rate limit delay: 2956, module: onetagger::tagger::itunes
Mar 12 20:57:21.489 INFO [ITunes] State: Ok, Accuracy: Some(1.0), Path: /home/chl/Media/Music/Albums/A Certain Ratio/Live At Leigh Rock Festiv
al '79 [Disc 1]/1-15 All Night Party.mp3, module: onetagger::tagger
Mar 12 20:57:21.490 INFO iTunes rate limit delay: 2952, module: onetagger::tagger::itunes
Mar 12 20:57:24.748 INFO [ITunes] State: Ok, Accuracy: Some(1.0), Path: /home/chl/Media/Music/Albums/A Certain Ratio/Live At Leigh Rock Festiv
al '79 [Disc 1]/1-13 Faceless.mp3, module: onetagger::tagger
Mar 12 20:57:24.748 INFO iTunes rate limit delay: 2963, module: onetagger::tagger::itunes
Mar 12 20:57:28.075 INFO [ITunes] State: Ok, Accuracy: Some(1.0), Path: /home/chl/Media/Music/Albums/A Certain Ratio/Live At Leigh Rock Festiv
al '79 [Disc 1]/1-18 Flight.mp3, module: onetagger::tagger
Mar 12 20:57:28.076 INFO iTunes rate limit delay: 2954, module: onetagger::tagger::itunes
Mar 12 20:57:31.489 INFO [ITunes] State: Ok, Accuracy: Some(1.0), Path: /home/chl/Media/Music/Albums/A Certain Ratio/Live At Leigh Rock Festiv
al '79 [Disc 1]/1-14 I Feel.mp3, module: onetagger::tagger
Mar 12 20:57:31.491 INFO iTunes rate limit delay: 2955, module: onetagger::tagger::itunes
Mar 12 20:57:35.100 INFO [ITunes] State: Ok, Accuracy: Some(1.0), Path: /home/chl/Media/Music/Albums/A Certain Ratio/Live At Leigh Rock Festiv
al '79 [Disc 1]/1-16 The Choir.mp3, module: onetagger::tagger
Mar 12 20:57:35.101 INFO iTunes rate limit delay: 2957, module: onetagger::tagger::itunes
Mar
-------->8--

Picard

MusicBrainz is a recognized platform for tagging. Picard uses its database to fetch track’s tags, album cover, etc. It is easy to install on Ubuntu (you just have to add the recommended PPA and ask for sudo apt update && sudo apt install picard). Last time I gave it a try, I found it not so intuitive. But now I realize that (1) it is fast and (2) it provides reliable results. It even added the lyrics. Deal.

~/.../A Certain Ratio/Live At Leigh Rock Festival '79 [Disc 1] % id3v2 -l 1-13\ Faceless.mp3
id3v1 tag info for 1-13 Faceless.mp3:
Title : Faceless Artist: A Certain Ratio
Album : Live at Leigh Rock Festival 19 Year: 2006, Genre: Rock (17)
Comment: Track: 13
id3v2 tag info for 1-13 Faceless.mp3:
TIT2 (Title/songname/content description): Faceless
TPE1 (Lead performer(s)/Soloist(s)): A Certain Ratio
TRCK (Track number/Position in set): 13/18
TALB (Album/Movie/Show title): Live at Leigh Rock Festival 1979
TPOS (Part of a set): 1/2
TCON (Content type): Rock (17)
TBPM (BPM (beats per minute)): 0
TCMP (): frame
TMED (Media type): CD
TDAT (Date): 2407
TORY (Original release year): 2006
TPUB (Publisher): Ozit
TYER (Year): 2006
IPLS (Involved people list): ,
TXXX (User defined text information): (SCRIPT): Latn
TPE2 (Band/orchestra/accompaniment): Various Artists
TSO2 (): frame
TSOP (): frame
TXXX (User defined text information): (ASIN): B000F9RME8
TXXX (User defined text information): (originalyear): 2006
TXXX (User defined text information): (ARTISTS): A Certain Ratio
UFID (Unique file identifier): http://musicbrainz.org, 36 bytes
TXXX (User defined text information): (MusicBrainz Album Status): official
TXXX (User defined text information): (MusicBrainz Album Release Country): GB
TXXX (User defined text information): (MusicBrainz Album Type): album/compilation/live
TXXX (User defined text information): (Acoustid Id): fcbc4ae0-1982-41d7-9961-1da4663c3021
TXXX (User defined text information): (MusicBrainz Album Id): e01790b2-77ba-4b7d-8383-118a6cdc700d
TXXX (User defined text information): (MusicBrainz Artist Id): 73577e86-4da8-4d76-949d-34c757e053be
TXXX (User defined text information): (MusicBrainz Album Artist Id): 89ad4ac3-39f7-470e-963a-56509c546377
TXXX (User defined text information): (MusicBrainz Release Group Id): 0ac9fd10-73e7-3d48-b1e0-c45b788130bb
TXXX (User defined text information): (MusicBrainz Release Track Id): 34e13319-80ff-32ca-9d07-73675a0d37b0
USLT (Unsynchronized lyric/text transcription): ()[XXX]: Tekst piosenki:
Go on, close the door
Leave me alone again
Hide the sun, bring me darkness
Let the walls come swallow me

I breathe in deep and taste my senses
For one last time, I know I'll be free
I breathe in deep, vigilant senses
I'm not enslaved, I'll fly again in my dreams

Feeling sad and blue and lost and cold
Counting hours, building towers, watch them all collapse

I breathe in deep and taste my senses
For one last time, I know I'll be free
I breathe in deep, vigilant senses
I'm not enslaved, I'll fly again in my dreams

All the cowards let's step up and speak our minds, will you listen?
Be the judge and face the fact. How you and me are both to blame

My spirit fades, my need remains untouched
My night has come, and it will be my last

I breathe in deep and taste my senses
For one last time, I know I'll be free
I breathe in deep, vigilant senses
I'm not enslaved, I'll fly again in my dreams

I breathe in deep and taste my senses
For one last time, I know I'll be free
I breathe in deep, vigilant senses
I'm not enslaved, I'll fly again in my dreams
Dodaj interpretację do tego tekstu »

Historia edycji tekstu
APIC (Attached picture): (e)[, 3]: image/jpeg, 166930 bytes

I yet have to tag my all library database. Wishing me luck, then!

♪ Joy Bogat • Sometimes

Vim and clangd

Tue, 08 Mar 2022 21:35:53 +0100

Bioinformatics is what I officially do for a living nowadays, along web dev – yes, those pretty little command-line tools are often better served using a Bootstrap-based Flask or Django framework, you know. Bioinformatics is cool, but it takes as much time as running a multi-level Bayesian IRT model, especially on those low cost servers we got at the University. Today I decided to enjoy my free time while processing several tasks in parallel in the background (read mapping and genome assembly) and to dive into my Neovim config.

I removed a few unused packages, as usual. I tried a few other ones, because why not. The most surprising finding was that the successor of ‘hrsh7th/nvim-compe’ (now defunct), which is what I use as a basic completion engine for all things related to LSP, buffer and command-line management, is nvim-cmp. It is maintained by the same author but this plugin is just asking us to install a ton of additional plugins (e.g., one for LSP stuff, one for buffer, one for : stuff, and so on). I can understand why (one target, one package), but I find it weird that we can’t get a unified UI for things like completion. After all, we only have to manage one package to get all LSP settings, isn’t it? Anyway, I tried this approach and just gave up: too much noise (why do we need to enable a snippet engine system, even if we don’t need one, BTW?), not that many improvements over my bare use of completion needs. I reverted back to nvim-compe.

Later on, I wanted to test my c/c++ config more extensively, with two prerequisites: It should rely on clang-based workflow, and it should support both standard Makefile (for C) and CMake (for C and C++, but since I don’t do C++ the later is mostly to ensure that I can compile others’ codebase).

I came across a lot of posts dealing with how to setup Vim or Neovim and LSP/Clangd. I already had the bare config for ’neovim/nvim-lspconfig’, which is (mostly):

local nvim_lsp = require('lspconfig')

local on_attach = function(client, bufnr)
 local function buf_set_keymap(...) vim.api.nvim_buf_set_keymap(bufnr, ...) end
 local function buf_set_option(...) vim.api.nvim_buf_set_option(bufnr, ...) end

 buf_set_option('omnifunc', 'v:lua.vim.lsp.omnifunc')

 local opts = {noremap=true, silent=true}

 buf_set_keymap("n", "<localleader>w=", "<cmd>lua vim.lsp.buf.formatting_sync(nil, 100)<CR>", opts)
 buf_set_keymap('n', '<localleader>wa', '<cmd>lua vim.lsp.buf.code_action()<CR>', opts)
 buf_set_keymap('n', '<localleader>wd', '<cmd>lua vim.diagnostic.open_float(buffer, {scope="line"})<CR>', opts)
 buf_set_keymap('n', '<localleader>wl', '<cmd>lua vim.lsp.codelens.refresh()<CR>', opts)
 buf_set_keymap('n', '<localleader>wL', '<cmd>lua vim.lsp.codelens.run()<CR>', opts)
 buf_set_keymap('n', '<localleader>ws', '<cmd>Telescope lsp_document_symbols<CR>', opts)
 buf_set_keymap('n', '<localleader>wS', '<cmd>Telescope lsp_workspace_symbols<CR>', opts)
 buf_set_keymap('n', '<localleader>wr', '<cmd>lua vim.lsp.buf.references()<CR>', opts)
 buf_set_keymap('n', '<localleader>wq', '<cmd>lua vim.diagnostic.setqflist()<CR>', opts)
 buf_set_keymap('n', 'K', '<cmd>lua vim.lsp.buf.hover()<CR>', opts)
 buf_set_keymap('i', '<C-h>', '<cmd>lua vim.lsp.buf.signature_help()<CR>', opts)
 buf_set_keymap('n', '[d', '<cmd>lua vim.diagnostic.goto_prev()<CR>', opts)
 buf_set_keymap('n', ']d', '<cmd>lua vim.diagnostic.goto_next()<CR>', opts)
 buf_set_keymap('n', 'gD', '<cmd>lua vim.lsp.buf.declaration()<CR>', opts)
 buf_set_keymap('n', 'gd', '<cmd>lua vim.lsp.buf.definition()<CR>', opts)
 buf_set_keymap('n', 'gr', '<cmd>Telescope lsp_references<CR>', opts)
 buf_set_keymap('n', 'gR', '<cmd>lua vim.lsp.buf.rename()<CR>', opts)
 buf_set_keymap('n', 'gi', '<cmd>lua vim.lsp.buf.implementation()<CR>', opts)
 buf_set_keymap('n', 'gy', '<cmd>lua vim.lsp.buf.type_definition()<CR>', opts)
end

local servers = {
 "pylsp", "clangd", "hls", "r_language_server",
 "racket_langserver", "clojure_lsp", "tsserver",
 "julials", "rust_analyzer", "purescriptls", "quick_lint_js"
}

for _, lsp in ipairs(servers) do
 nvim_lsp[lsp].setup {
 on_attach = on_attach,
 flags = {debounce_text_changes = 150}
 }
end

And, that’s it: As I said in a previous post, the built-in LSP tools together with a few well crafted package usually make my day. I mean, I get all the things we usually need form an IDE: diagnostics, goto definition/references, and so on. And with the progress of code actions, at least in Rust, C and Haskell, we will soon get powerful refactoring tools. Now, what additional packages are useful in the context of LSP/Clangd. Well, clangd-extensions comes to the rescue, especially by adding inlay hints (you’ll need Clang-13 at least), type hierarchy and AST, and more.

Regarding Makefile and CMake, I rely on bare Vim config in after/ftplugin/c.vim, namely:

compiler gcc
set makeprg=cd\ build\ &&\ cmake\ -DCMAKE_BUILD_TYPE=debug\ -DCMAKE_EXPORT_COMPILE_COMMANDS=1\ ..\ &&\ cmake\ --build\ .

autocmd QuickFixCmdPost [^l]* nested cwindow
autocmd QuickFixCmdPost l* nested lwindow

nmap gZ :10 split term://zig cc % && ./a.out<cr>

I use make in case I’m working with CMake (this assumes there’s a already a build directory that was defined beforehand; and you’ll need to link the compile_command.json file at the root directory). For standard Makefile-based project, I use :!make from Vim command prompt. Lastly, I tend to use the same mapping, gZ, across multiples langauges to just fire a quick terminal and run the program.

♪ Midnight Sister • Satellite

Micro posting in February

Sat, 26 Feb 2022 20:42:17 +0100

2022-02-01: Elbow, The Seldom Seen Kid.
2022-02-01:

Fundamentally I think C trusts developers while C++ trusts compilers. This is a massive difference that sharing the same native types or syntax for while loop cannot hide. — The problem with C

2022-02-01: Update on Regolith Desktop: Now installed on my own laptop for more precise control. Back to a dark theme (Nord) – long time no see. One of the main advantage of Regolith over, say, i3 or i3-gaps, is that all Ubuntu apps and settings remain available, including controllers that would have to be defined manually otherwise (e.g., brightness or volume control, screen sharing or screen capture). The integration of native apps (e.g., nautilus or gnome-terminal) is also really nicely handled by the WM without excessive RAM usage (< 800 Mo).
2022-02-01: Installing Every Arch Package.
2022-02-01: Magma: Interact with Jupyter from NeoVim. #vim
2022-02-01: Radix Sort Revisited. #clang
2022-02-01: Regolith Desktop looks awesome. I’m currently testing it on a work machine: it worked right out of the box after reboot, and it provides sensible default settings to i3 WM. #unix
2022-02-02:

Mike Caulfield invented the term “a chorus of explanations” several years ago to explain why sites like Stack Overflow are useful. Where a lesson typically explains something once, SO and similar sites present several explanations with different levels of detail, different assumptions about the reader’s background, and quite possibly different solutions to the original problem. Most readers may be satisfied by the top answer, but others can scroll down to find one that’s a better fit for who they are, what they are ready to understand, and what they’re trying to do. — A Cacaphony of Explanations

2022-02-02: The Common Lisp Document Repository is a repository of documents that are of interest to the Common Lisp community. The most important property of a CDR document is that it will never change: if you refer to it, you can be sure that your reference will always refer to exactly the same document. #lisp
2022-02-03: Whit Dickey, Nothingness.
2022-02-03:

In December, in my article When dropping support feels like sabotage, I was complaining that Apple was limiting the support of security updates to just the two previous versions of Mac OS, and not extending such support to even older versions like High Sierra or Mojave given the amount of people that are still using them. But I was forgetting that on iOS the situation is even worse. — → No more security updates for iOS 14

Lovely, #apple. I’m still on Mojave and iOS 12.5 ;-)
2022-02-05: Bush Tetras, Too Many Creeps.
2022-02-05: I still have 57 RSS feeds out of 727 unread in my inbox. Work in progress.
2022-02-05: A boring statusline for Neovim. #neovim
2022-02-05: Reducing the size of large PDFs.
2022-02-08:

Time finds problems, people fix them. — Software Deployment, Speed, and Safety

2022-02-10: Brave New Hacktoberfest — Learn4Haskell. #haskell
2022-02-10: CR+LF has a long history…
2022-02-10: How Postgres Stores Rows. #database
2022-02-10: Understanding LSM Trees: What Powers Write-Heavy Databases. #database
2022-02-16: Graham Collier, Red Sky in the Morning.
2022-02-16:

Monad is the name of the class for “and_then”, defined in a sensible way, with some laws for how it should behave predictably, and then a bunch of library code works on anything that implements “and_then”. — A plain English description of monads without Haskell code

2022-02-16:

What you don’t realize is that you’ve been self-censoring yourself from searching most of the things you would have wanted to search. You already know subconsciously that Google isn’t going to return a good result. — Google Search Is Dying

2022-02-16: Bash Pitfalls.
2022-02-16: Music theory for nerds.
2022-02-17: The Vaselines, Son of a Gun.
2022-02-17:

Fancy algorithms are slow when n is small, and n is usually small. — Rob Pike’s 5 Rules of Programming

2022-02-17:

What makes old manuals so fascinating to me is that they document the history of applied computer science. This gives the entire field, including tech comms, the resemblance of a past. Let’s not dismiss that: in a sector where startups seem to build new things on top of blurry or nonexistent foundations, reading about old software reminds us that there’s indeed a giant upon whose shoulders we’ve climbed. — Why I collect and read old computer manuals

2022-02-18: Siouxsie & The Banshees, Jigsaw Feeling.
2022-02-18: Fred’s ImageMagick Scripts.
2022-02-18: My journey from macOS to FreeBSD. #apple #unix
2022-02-18: vim-wordy: Uncover usage problems in your writing. #vim
2022-02-19: Howard Devoto, Rainy Season.
2022-02-19: Tired of Tmux prefix (C-b or C-a, depending on your taste)? Let’s rebind everything we used on the Alt key using bind-key -n M- whatever. It may not work on all terminals, but let’s that not stop us adding more aliases to our Tmux config file.
2022-02-21: Lee Ritenour, Dreamcatcher.
2022-02-21:

I generally consider theory to be useful background knowledge but fastidious study a poor substitute for firsthand experience. I “learn with my hands,” is the way I’ve always put it. it has its upsides and downsides — how I think when I think about programming

2022-02-21:

Personally, in every activity I’ve participated in where it’s possible to get a rough percentile ranking, people who are 95%-ile constantly make mistakes that seem like they should be easy to observe and correct. “Real world” activities typically can’t be reduced to a percentile rating, but achieving what appears to be a similar level of proficiency seems similarly easy. — 95%-ile isn’t that good

2022-02-21: My Notebook System.
2022-02-22: Live Streaming a Macintosh Plus (or Any Compact Mac). Amazing stuff as always.
2022-02-22: magma-nvim: Interact with Jupyter from NeoVim. Great plugin! #vim
2022-02-23:

Read three books on a topic and you know more about it than 99% of the world. Watch news all day for years and you have a distant, water-cooler-level awareness of thousands of stories, at least for the few weeks each is popular. — Five Things You Notice When You Quit the News

2022-02-23: Vim’s useful lists. #vim
2022-02-23: vim cheatsheet. #vim
2022-02-24: Lingua Ignota, Wicked Game.
2022-02-24: Finally got a decent Twitter client… No more unsolicited tweets from nowhere/nobody, maybe?
Spoiler: twterm
2022-02-24: I thought I would write a short post to explain how to fetch genome data from NCBI the old way, but here’s the gist, assuming you want to fetch all bacterial genomes:

% wget ftp://ftp.ncbi.nih.gov/genomes/refseq/bacteria/assembly_summary.txt
% mkdir bacteria_refseq
% awk -F '\t' '{if($12=="Complete Genome") print $20}' assembly_summary.txt > assembly_summary_complete_genomes.txt
% for record in $(cat assembly_summary_complete_genomes.txt); \do wget -P bacteria_refseq -e robots=off -r --no-parent -A "*genomic.fna.gz" "$record"/; done

Then wait, like me.
2022-02-24: Parsing series of integers with SIMD.
2022-02-24: The web is overrun by pop-ups and blockers haven’t worked in years.
2022-02-24: irssi (IRC client) - Questions and Answers.
2022-02-25: How to speed up the Rust compiler in 2022. #rust
2022-02-25: jless — a command-line JSON viewer.

ArXiving on January 2022

Fri, 25 Feb 2022 10:09:25 +0100

Here are a few papers that I read over the past two months, in the CS and Stat category as usual.

At the time of this writing you can replace the x in arxiv URL to get a pretty HTML rendering of the article. See below for the second article discussed above.

Type Stability in Julia: Avoiding Performance Pathologies in JIT Compilation (https://arxiv.org/abs/2109.01950)

I still haven’t really gotten into the Julia language. I follow the news here and there and I’m glad to know that there is now a native graphical backend, not just web output. The last time I tested Julia (v1.6), I found the runtime of a first script to be quite substantial, although I understand why that is the case. The language server is slow too, because of the initial indexing of the symbols contained in all the installed packages. Nevertheless, I think Julia has a future and I hope there will soon be a consolidated set of tools for statistical modeling, much like Frank Harrell’s rms package. As I said in a previous post, for the moment I want to investigate Racket and Lisp-like languages, or maybe Haskell, to see if they could provide a good fit for my own workflow (even if I rarely perform statistical analyses nowadays). Julia remains, however, an interesting approach to statistical computing nowadays.

An Introduction to Quantum Computing for Statisticians (https://arxiv.org/abs/2112.06587)

Just out of curiosity, and because I was reading Introduction to Classical and Quantum Computing (PDF) by Thomas G. Wong, I fetched this paper on quantum computing for statisticians. Well, I haven’t read it actually, but this look interesting as far as I can tell.

Aitchison’s Compositional Data Analysis 40 Years On: A Reappraisal (https://arxiv.org/abs/2201.05197)

It’s been a long time since I last read a paper written by Michael Greenacre. Last time was probably when I was working on multi-block methods back in 2016. This paper discusses the original approach to compositional data analysis, which is usually associated with lot of zeros in the dataset, and suggests that, contrary to the classical use of isometric logratios, soft constraints may yield correct and almost identical interpretation of the data in most cases. I have only skimmed the article but I will probably come back to it if I need to go deeper into the subject and the relationship between this type of approach and the techniques based on correspondence analysis.

How ISO C became unusable for operating systems development (https://arxiv.org/abs/2201.07845)

The take-away message is that the primary reason why ISO C is poorly designed for OS development relates to a “design approach in the ISO standard that has given priority to certain kinds of optimization over both correctness and the ‘high-level assembler’ intentions of C, even while the latter remain enshrined in the rationale.” Among other things, pointers (casting from and aliasing to) remain problematic in several edge cases.

I still use C because this is one of the first languages I learned when I was in grad school (after Turbo Pascal, you named it). I like it because of its syntax and its simplicity, but I usually only write toy programs so I don’t have to bother with all its “pitfalls” regarding low-level and hard-core programming. I don’t use C++, but I’m rather interested in learning Rust because I like the tooling as well as rust-analyzer. Yet, I know it’s a hard path.

♪ Laura-Mary Carter • Town Called Nothing

Jupyter kernels in Neovim

Tue, 22 Feb 2022 21:28:53 +0100

I spend a certain amount of time writing Python, Racket or Lisp scripts for my daily jobs or musings around PLs. I’m using Neovim since last year and I tested a number of plugins that bring REPL-like experience to (Neo)Vim. Lately, this was mostly neoterm and iron. Both plugins are great, at least they do the job which amounts to taking a visual selection or whatever line is under the cursor and sending it to a live shell subprocess running IPython, Racket or SBCL. I even contributed a PR to neoterm for adding Stata support since I wanted to use this workflow with Stata code. Now, what if we could use an unified API to provide REPL-like experience in (Neo)vim, without relyingon the built-in terminal?

Enter Jupyter! It’s the next iteration of IPython, with support for multiple languages. You can think of it as the LSP of all REPLs. In fact there’s a kernel available for every language I use to use. The advantage are clear: you have the same UI for any PL, and you can dispatch your IO operations in a console or in a web browser. Most people I know generally using the “notebook” version, which means working in your default web browser. I don’t like it, and I prefer to work from a console, maybe an enriched Qt console. I’ve tried it with pleasure in the past in the case of Stata, for example.¹ You can also call Jupyter kernel directly from your plain text editor, which is what I did for a long time when I was working with Emacs, especially for Mathematica and Stata in addition to R and Python.

I’ve been looking for a similar solution in Neovim. This would alleviate the need for and additional terminal window, especially if it could benefit from virtual text features, or things like that. And I discovered Magma, which is a Neovim plugin that provides exactly what I need: a simple way to start a given kernel (MagmaInit, then select the language), no interference with a running LSP server, the possibility to evaluate the current line, a region or a motion.

Like for iron I settled upon the following mappings, which I find convenient for most use cases:

vim.cmd[[nnoremap <silent><expr> gz :MagmaEvaluateOperator<CR>]]
vim.cmd[[xnoremap <silent> gz :<C-u>MagmaEvaluateVisual<CR>]]
vim.cmd[[nnoremap <silent> gz :MagmaEvaluateLine<CR>]]

I know there are other actions available, e.g. for reevaluating a certain code chunk or clearing current output, but I don’t need them often. Combined with LSP, however, this plugin provides decent IDE-like features, with the comfort of Vim editing.

Following the suggestions available on the Jupyter project wiki, I installed the following kernels: common-lisp, haskell, iclojure, ipurescript, ir, javascript, python3, racket, rust, stata, wolframlanguage13. The hardest part was to get CL kernel working since I kept getting an error with ironclad when using quicklisp alone. Finally, everything went fine with Roswell.

Note that you can still send the whole buffer to a terminal and run the appropriate program, thanks to ftplugin directives. For instance, I have the following settings for Python, defined in after/ftplugin/python.vim:

setlocal makeprg=python3\ %
setlocal errorformat=

nmap gZ :13 split term://python3 %<cr>
nmap <buffer><localleader>k :!pydoc3 <cword><cr>

ia pp print("")<Left>

I use more or less similar mappings for Haskell, C, Rust and Racket.

♪ Tkay Maidza • Onto Me

Unfortunately, the Stata kernel requires Stata 14 or higher for image display, and I only have Stata 13 MP. ↩︎

The unquantified self #21

Sun, 20 Feb 2022 20:23:59 +0100

It’s already February. It also looks like Spring is coming, at least on my balcony:

We started watching a new TV show with my son, See. I originally thought it would make a good sequel to Vikings, but this was only based on the poster of the TV show on Apple TV. Now that I realized that the main characters are all blind, I find the series even more interesting. When my son is not with me, I’m just back to the basics: I read or watch Twitch, mostly strager and togglebit — give them a follow, you won’t regret it. I still listen to music all day long, and my latest streak according to Last.fm was around 1000 tracks a week.

As discussed in one of my previous posts, I’m pretty happy with my current setup. Indeed I switched to Regolith desktop and I can now enjoy a useable window manager and Ubuntu built-in stuff (with a revamped GTK theme). This means I’m also back to a dark theme for my TUI stack, but I kept Firefox untouched because all dark theme I came across look so bad for most websites with lot of pictures or complex layout.

Other than that, I remained diligent in the kitchen:

That’s all for now!

♪ Anja Lechner & François Couturier • Vague / E la nave va

Multi-blast and parallel processing

Thu, 10 Feb 2022 20:57:39 +0100

Last year I was involved in running long series of blast of individual DNA sequences against one or more reference genome or ITS therein. For instance, I would be given a multi-fasta file with hundreds of ITS from fungi species which I have to blast individually against a builtin database of fungi ITS. I call this “multi-blast” but I’m not a bioinformatician and there might be a better term. The idea is to find the closest match for each sequence to the ones already available in our in-house database. Even if those individual sequences are not really big (400 to 800 bp), this is an embarassly parallel problem, which can be solved using Python multiprocessing capabilities or GNU parallel.

The following code uses Python, but since GNU parallel is invoked via a shell subprocess, it doesn’t really matter. Here is a simple function to perform blast queries in parallel:

import os


def parallel_blast_its(file, db):
 """Parallelized version of one-against-all Blast"""
 cmd = (
 "parallel --recstart '>' --recend '\n' -N 1 --pipe "
 + "blastn -query - -out '"
 + file
 + "_{#}' "
 + " -outfmt 0 -max_target_seqs 5 -db "
 + db
 + " < "
 + file
 )
 os.system(cmd)

The key here is how to pass individual record from a multi-fasta file to GNU parallel. This is done by specifying recstart and recend options, which allow to pass a raw string to blastn as for the sequence and then output each result in a temporary folder (out) with the name of the original multi-fasta file suffixed with the record number.

And here’s a complete program to blast individual sequence from a multi-fasta file against a reference db, which was used for a metagenomics study:

import multiprocessing
import os
import random

from Bio import SeqIO

SAMPLES = "representatives.fasta"
CPUS = multiprocessing.cpu_count()
PARA = False


def chunks(lst, n):
 for i in range(0, len(lst), n):
 yield lst[i : i + n]


def blast(record):
 for r in record:
 key = str(r.id).split(";")[0]
 SeqIO.write(r, "out/tmp.fasta", "fasta")
 cmd = (
 "blastn -num_threads 4 -out out/blast/"
 + key
 + " -outfmt 0"
 + " -db db/db-its-2020-12 -query out/tmp.fasta"
 + " -evalue 1e-60 -qcov_hsp_perc 90 -max_target_seqs 1 2>/dev/null"
 )
 os.system(cmd)


if __name__ == "__main__":
 records = list(SeqIO.parse(SAMPLES, "fasta"))
 if PARA:
 nc = int(len(records) / CPUS - 2)
 data = chunks(records, nc)
 p = multiprocessing.Pool(processes=CPUS - 2)
 result = [p.apply_async(blast, args=(list(x),)) for x in data]
 result = [item.get() for item in result]
 else:
 blast(records)

The idea is to write each DNA sequence as a Fasta file in a temporary directory, and to process it from there using blastn. Note that we could use the subprocess module instead of os. Likewise, the Biopython is only used to read Fasta files. Given the standardized format of those files, it would be easy to write a custom parser.

♪ Metallica • Nothing Else Matters

No more VS Code

Thu, 03 Feb 2022 10:23:13 +0100

I think I reinstalled VS Code on my laptop one or two years ago. Since then, I never used it, and I found myself only updating the application and the limited set of extensions I installed alongside for one year long without even touching it. In fact, I was mostly interested in Language Server add-ons, but now that Neovim builtin LSP is quite stabilized, I believe I no longer need to check whether it works as expected (i.e., as it is expected to behave in VS Code or IntelliJ). In fact, I have everything at hand under Neovim, and it sparks joy.

I have a long series of posts about how I configured Neovim during last summer. Most of those settings are still relevant today, but I added a few packages, including debugging capabilities, $\TeX$ (see Texing in Vim) and Org (see Org in Vim) stuff, etc. Regarding LSP, I added specifically the following packages:

nvim-lightbulb, because I wanted to have some idea of where code action were available. Since I don’t like intrusive help or notification, I choose to show them only for the current line thanks to an autocommand.
lsp_extensions.nvim, only to display inlay for Rust code.
lsp_codelens_extensions.nvim, to get pretty code lens actions right to a Rust test fn or for inline Haskell snippets.
nvim-dap, nvim-dap-ui, nvim-dap-virtual-text, for debugging. I only trigger dap UI on demand, but I have virtual text activated automagically when I started a debugging session. I use it mostly for Python scripts. As an update to a previous micro-post, I even get this working for Haskell.
neogen, because everyone likes docstrings.

Below are some demos (Python, Rust, Haskell) of what’s available in a modern Vim world nowadays.

Left: highlight dap debugging with virtual text and LSP error/warning highlights in the status line (syntax and formatting errors deliberately added);
Middle: Rust with inlay hints and code lens for test cases;
Right: Haskell with Wingman code action.

[2021-03-08]
See also Neovim LSP to replace VSCode, or Neovim and Rust, which was featured on the Rust blog this month.

Happy coding in Neovim!

♪ Memory Cassett • Asleep at a Party

Regolith Desktop

Wed, 02 Feb 2022 20:48:05 +0100

Out of curiosity, I tried Regolith Desktop yesterday. I was looking at Pop OS for a long time before deciding on i3 last year, mostly because Gnome shell and Gnome WM eat up too much memory to my taste, and because I was looking for a keyboard centric experience to manage and navigate my desktop environment. I must say that Regolith Desktop rocks!

Regolith Desktop is originally based on i3-gaps, but it adds further keyboard centric and TUIs specific to i3. Above all, it comes with sensible default settings and its installation is a breeze. Just issue sudo add-apt-repository pa:megalith-Linux/release && budo apt install megalith-desktop-standard in your terminal and you’re done after a reboot.

If you have an existing i3 configuration file, you can reuse (part of) it for Regolith, whose config files are located under $HOME/.config/regolith. The main config file is an Xresources file. Note that your own $HOME/.Xresources file is also read by Regolith at startup. Once you made some changes or update default settings, you can update the WM with regolith-look refresh. For instance, I changed the default font for the terminal and various other settings:

gnome.terminal.font : JetBrains Mono 11
gnome.wallpaper : $HOME/Media/Pictures/waves.jpg
i3-wm.font : pango:JetBrains Mono Medium 10
i3-wm.gaps.inner.size : 8
i3-wm.gaps.outer.size : 5
i3xrocks.value.font : JetBrains Mono Medium 10
rofi.font : JetBrains Mono Medium 10

Not all i3 options are manageable from this file, though. You can copy the default config file from /etc/regolith/i3/config and update its settings under your own regolith directory.

Except that I didn’t find any pleasant light theme for Regolith – I currently use my beloved Nord theme – the experience is close to what I was used to with my own i3 settings. I only tweak the default GTK theme so that Firefox is not using a dark theme as the default. And if I happen to get tired (again) of dark theme for the terminal and my editor, I can still switch back to Kitty since it uses its own color settings. I also disabled a lot of applets for the i3bar.

What I really like, though, is that all Gnome applications work as if you were using the default Gnome shell desktop, with a pretty GTK theme. Here’s the default look’n feel of Gnome applications (here, the control panel):

Gnome terminal nicely fits in the tiling framework (no menu bar, no title bar, etc.), and of course Tmux makes session and terminal management even easier:

♪ Marianne Faithfull • Crazy Love

Texing in Vim

Wed, 26 Jan 2022 15:05:55 +0100

I rarely have to write $\LaTeX$ these days. I usually let Pandoc do the harder stuff, and simply write my plain text documents using Org or Markdown markup language. I used to rely on $\LaTeX$ in the case of R+Sweave/knitr reports, but I rarely have to write any statistical report these days. However, I may happen to write plain $\TeX$ or edit $\LaTeX$ documents, in which case I appreciate a solid workflow to edit, compile and keep my text in sync with the PDF renderer.

Under Emacs I had everything I needed thanks to Auctex and Pandoc, but under Vim I always used the bare minimum so far, that is native syntax highlighting and Vim motion. I don’t use snippets but I have a set of iabbrev that alleviates the need to write cumbersome and recurrent $\LaTeX$ expressions:

iabbrev <<@ <<LABEL>>=<CR>@<Esc>?LABEL<CR>cw
iabbrev chapter@ \begin{chapter}
iabbrev section@ \begin{section}
iabbrev subsection@ \begin{subsection}
iabbrev subsubsection@ \begin{subsubsection}
iabbrev paragraph@ \paragraph{TITLE}<Esc>?TITLE<CR>cw
iabbrev equation@ \begin{equation}<CR>
iabbrev align@ \begin{align}
iabbrev tabular@ \begin{tabular}{ALG}<CR><Esc>?ALG<CR>cw
iabbrev table@ \begin{table}[POS]<CR>\centering<CR>\caption{}<CR>\label{}<CR><Esc>?POS<CR>cw
" FIXME: Add figure environment
iabbrev texttt@ \texttt{DATA}<Esc>?DATA<CR>cw
iabbrev frac@ \frac{DATA}{}<Esc>?DATA<CR>cw

Enters Vimtex which I discovered when I was reading How I’m able to take notes in mathematics lectures using LaTeX and Vim a while ago. Note that vimtex+zathura is the winning combo here. Contrary to the author of the blog post, I don’t use concealing, nor snippets (see above).

[2023-10-25]
See this excellent tutorial to get started with using vimtex: Getting started with the VimTeX plugin. I also tried texlab and ltex-ls, but finally vimtex alone already features everything we need (chktex or textidote configured via makeprg, auto-formatting with latexindent, omni-completion). The documentation is also gorgeous, I’d be surprised if you didn’t find an answer to your questions inside.

Things I like: ]] will automagically close the current environment, Bibtex keys and labels can be autocompleted using builtin omnicomplete (C-x C-o), you can toggle on/off a table of contents in a dedicated sidebar, the % matching operator is redefined to highlight opening and closing $\LaTeX$ delimiters, you can compile and preview your file with pre-defined mappings, and there are specific motion operators (e.g., ic, id, ie, i$). And on top of that, you can use reverse (Ctrl+click from Zathura) and forward (<localleader>lr from Vim) search. And it just works! Zathura is configured as the defaut PDF viewer, and all you have to do is to ensure that synctex is active when compiling your $\LaTeX$ document. This is already defined in Vimtex default options. See also this review: A Complete Guide on Writing LaTeX with Vimtex in Neovim. (Note that you no longer need to spawn a server since Neovim 0.5+.)

To compile Knitr standalone documents, I use the following mapping:

map <localLeader>k :w<CR>:cd %:p:h <CR>:!Rscript -e 'library(knitr);knit("%:p")'<CR>:!latexmk -pdf -bibtex-cond -f %:r.tex <CR>:!xdg-open %:r.pdf <CR><CR>

This is basically what I have in my Makefile, except that I use texi2pdf instead of latexmk. My latexmk settings are as follows ($HOME/.latexmkrc):

@default_files = ('main.tex');
$clean_ext = "bbl nav out snm";

$latex = 'latex -interaction=nonstopmode -shell-escape';
$pdflatex = 'pdflatex -shell-escape -interaction=nonstopmode -synctex=1 -file-line-error';
$pdf_previewer = 'zathura';
set_tex_cmds('-synctex=1 -interaction=nonstopmode -shell-escape %O %S');

Finally, I use the Bibtex extension for Telescope, which provides a nice alternative to Helm-bibtex to display a list of available references (globally or in the current directory). Inserting the reference by pressing the enter key will take care of formatting the bibliographic key depending on the filetype (Markdown, Latex or plain text).

♪ Sonic Youth • The World Looks Red

15 years and 600 posts

Tue, 25 Jan 2022 21:31:06 +0100

After 15 years, I realize I have now written 600 posts on this blog. On 4 different blogging platforms or site generators: Wordpress, Jekyll, Textpattern, and Hugo. I redesign the site several times, but there has been two main layouts over the time. I miss the good old days when I edited all pages, except the blog, by hand using HTML and CSS. I mostly write in English although this is not my mother tongue language; surely I make a lot of typos but also write many inconsistent expressions, but I hope that French and non-French speaking readers will forgive me. It always amazed me that I am able to find minor typos in others’ writings, even English native speakers, and continue to make so many mistakes myself because I’m too lazy to reread what I’ve just written or to look for a better wording when I’ve already moved on to the next idea.

Originally called Memos, this section has become a blog per se even if the fashion of blogs comes and goes but is certainly not what it was in the 2000s. As I said in earlier posts, people start writing very nice blog posts and then stop. I always find it sad that it ends up so early, but that’s life. Personally, i don’t have a problem with writing, on the contrary: i have done practically nothing else for 30 years, despite a rather prolonged hiatus about 15 years ago. But basically, keeping the link with “public writing” makes it possible to keep a trace of one’s own peregrinations in life (professional most of the time, at least as far as i’m concerned) and of one’s constantly renewed learnings.

After digging into the archives, I discovered that one of the oldest file in the public directory on this server dates back from 2005 and it happens to be a… Flash applet. There’re also some gems from more than ten years ago, before Gist were available on GitHub. Note that many files were uploaded at a later time so that we cannot trust all timestamps for sure. This is the case, for instance, of my PhD dissertation in Postscript which was moved from one folder to another one without archive mode.

testParamPass2.swf 2005-08-26 15:11 3.1K
test_minted.tex 2010-12-26 19:18 1.2K

I’ve published various quick stats, especially in this anniversary post. Rather than publishing yet another histogram of post size or count, let’s just plot an aggregated content in the form of a wordcloud, which is inspired from one my recent reading. I generated a wordcloud of all those 600 posts, and I plan to make a nice video when I have time. Here are individual wordcloud for the last three posts:

That’s it.

♪ Mischa Blanos • Steppe

Micro posting in January

Tue, 25 Jan 2022 20:53:12 +0100

2022-01-02:
From Last.fm, this week
2022-01-02:

It’s all well and good that SQLite is scalable and reliable enough for most needs, but what’s the point of using it instead of a more traditional client/server database? — Consider SQLite

2022-01-02:

It’s been my experience that our intuition of what is and isn’t a pathological or malicious input is about as accurate as time estimates. Requirements change and any and all attack vectors will be found and exploited. The mental burden of remembering the assumptions made to correctly check for pathological or malicious inputs in all cases and keep them updated during refactoring is far too enormous to successfully maintain. — Almost Always Unsigned

2022-01-02: Haskell tutorials:- Course on functional programming - Haskell (Clément Hurlin)- Haskell training: 101 and 102 (Google)
2022-01-02: Seeing Theory: A visual introduction to probability and statistics. I already mentioned this amazing website but it is still worth a retweet.
2022-01-02: Using databases with Shiny. #rstats
2022-01-03: Joy Division, Glass.
2022-01-03:

All the functions in Abramowitz & Stegun are now fully computable in the Wolfram Language. — Launching Version 13.0 of Wolfram Language + Mathematica

2022-01-03: Fast CSV processing with SIMD.
2022-01-03: Idle puzzles. #lisp
2022-01-04: A Closed and Common Lisp. #lisp
2022-01-04: Julia Data Science. #julia
2022-01-05: Firewater, One Of Those.
2022-01-05:

All I want to say with all these abstractions is that proper logging architecture is hard. It is not just about writing logger.log() here and there. It is a complex process of creating proper abstractions, composing them, and maintaining strict layers of pure and impure code. — Do not log

2022-01-05: Old times good times. I came across this (archived) post where comments on a blog really add values to the post itself. #scheme
2022-01-05: A graphic overview of the events in the history of data visualization, by Michael Friendly & Daniel J. Denis. #dataviz
2022-01-05: Handbook of Graph Drawing and Visualization.
2022-01-05: The complexity of dealing with missing data, variable selection, and cross validation for prediction models.
2022-01-06: Müncher Rundfunkorchester, Arvo pärt: Works.
2022-01-06: I will always miss you.Circa. 2011
2022-01-07: Orange Juice, I Can’t Help Myself.
2022-01-07: Orchestra of the Swan, David Le Page, Eleanor Turner & Daniele Rosina, Comptine d’un autre été, l’après-midi.
2022-01-07:

The inferential analogy here is that our knowledge grows by a process that is partly random but also involves a struggle for survival amongst ideas. You May Believe You Are a Bayesian But You Are Probably Wrong (PDF)

2022-01-07: A succinct cheat sheet for newbie linux coders and system administrators. #unix
2022-01-07: Hugo documentation theme as simple as plain book.
2022-01-07: Strange little diff.The “blast+” term get recognised as part of the diff even if I deleted the whole line beforehand.
2022-01-07: Bash Style Guide for Mac and Linux. ̀#unix
2022-01-07: Beej’s Guide to Unix IPC. #unix
2022-01-07: Optimizing Docker image size and why it matters.
2022-01-08: Finally:

2022-01-08: Summary of the morning cooking session.

2022-01-08: Fast and Elegant Clojure: Idiomatic Clojure without sacrificing performance. #clojure
2022-01-08: First Steps With Clojure. #clojure
2022-01-09:

During discussions with my friends and colleagues, whenever the topic of chat protocols comes up, I often remark how simple the Internet Relay Chat (IRC) protocol is and how this simplicity has fostered creativity in the lives of many young computer hobbyists growing up in the late 1990s and early 2000s. — Simplicity of IRC

2022-01-09: Bayesian Modeling and Computation in Python. #python #statistics
2022-01-09: IRC technology news from the second half of 2021.
2022-01-10: Eusebius, Water Lily.
2022-01-10: Maxence Cyrin, As the Darkness Falls.
2022-01-10:

… there are thousands of reusable patterns I’ve picked up … Unfortunately, I’ve forgotten about 95% of them. … The point is to reflect on what actually stuck, so that others may save time by spendingtheir time learning what is more likely to stick. — Practical Shell Patterns I Actually Use (via John D. Cook)

2022-01-10: Idle puzzles 2: Revenge of the Shift. #lisp
2022-01-10: xsv: A fast CSV command line toolkit written in Rust. Handy (and faster) alternative to csvkit.
2022-01-11:

The real enemy addressed by inlining is unexpected dependency and mutation of state, which functional programming solves more directly and completely. However, if you are going to make a lot of state changes, having them all happen inline does have advantages; you should be made constantly aware of the full horror of what you are doing. When it gets to be too much to take, figure out how to factor blocks out into pure functions (and don.t let them slide back into impurity!). — John Carmack on Inlined Code

2022-01-11: While browsing GitHub, I noticed that the nvim-lua organization offers a “starter kit” init.lua file for Neovim users. It is a single Lua file, which is quite handy when you want to quickly bootstrap a sane config on external servers. I tested it in my own environment and it’s pretty good actually. If you are new to Neovim, and especially Lua config files, I recommend taking a look at how this file is organized. Except for a few packages (theme and snippets), and the default key mappings, you will likely find a working setup for your daily editing tasks. I like the idea of having a single file for the whole config, although I can understand why we may want to split our whole config in separate files, which is actually what I do.
2022-01-11: csview: A high performance csv viewer with cjk/emoji support.
2022-01-12:

Cognitive load is important. I don’t expand much on this here, but if a programming language or a library makes me feel stupid, then I’m less likely to use it or like it. C++ does that a lot :) — “Modern” C++ Lamentations

2022-01-12: Scientific computing with Rust:- Rust Vectors- From Julia to Rust
2022-01-12: Database-like ops benchmark. Interesting to note that Polars wins over well-established competitors from Apache or Python. #rust
2022-01-12: The Art of Lisp & Writing. #lisp
2022-01-12: The Rust Performance Book. #rust
2022-01-12: Topological Data Analysis.
2022-01-12: Why I Use a Debugger. #rust
2022-01-13: The Feelies, The Boy With the Perpetual Nervousness.
2022-01-14: Glenn Gould, Goldberg Variations, BWV 988.
2022-01-14: Halfway through my RSS reading list of the week. It’s amazing how much I learn from all those posts, even if I don’t necessarily have enough time to recollect all the interesting stuff I read. I should take more note, but I’ve became so lazy lately.> YYMMDDHHMM … will no longer fit into a signed 32 bit number.
> The last value for last year is December 31, 2021 at 23:59, and in this format it becomes 2112312359. That just fits into a signed 32 bit number (which tops out at 2147483647).
> The first value for this year is January 1, 2022 at 00:00, which turns into 2201010000, and that does not fit! — YYMMDDHHMM just overflowed a signed 32 bit int
2022-01-14: My ideal Rust workflow. #rust
2022-01-14: Pointers/references in Python. #python
2022-01-15:
Tasty
2022-01-15: Anja Lechner & François Coutu, Vague / E la nave va.
2022-01-16: Les Ogres de Barback, Nos vies en couleurs. Clin d’oeil à Peyo.
2022-01-17:

The worst possible way to learn Rust is by vaguely looking at it, or trying some small stuff, in short scattered sessions at night after your demanding work. Sure you did learn a few languages like this before, so you may be confident in your abilities. But at some point in Rust, and it may come soon, you’ll encounter a higher step and if you don’t fight it with concentration and dedication, you risk not overcome it. — How not to learn Rust

2022-01-17: Automating the annotation of PDFs.
2022-01-17: Climbing Mount SICP.
2022-01-17: Counting with martingales.
2022-01-17: Digit Sum Patterns and Modular Arithmetic.
2022-01-17: Lesser Known PostgreSQL Features. #database
2022-01-17: Ways I Use Testing as a Data Scientist .
2022-01-18: Best resources to learn about data and distributed systems. #database
2022-01-19: Current 93, All the Pretty Little Horses.
2022-01-19: What goes into making an OS to be Unix compliant certified?. #apple
2022-01-20:

Luschny’s webpage cites, for example, recent treatments of the subject by leading mathematicians such as Terence Tao. And his most compelling argument, from my personal perspective, is the way he unveils the early publications: I learned from him that my own presentation of the story, in The Art of Computer Programming and much more extensively in Concrete Mathematics, was a violation of history! I had put words and thoughts into Bernoulli and Euler’s minds that were not theirs at all. This hurt, because I’ve always tried to present the evolution of ideas faithfully; in this case I’d fooled myself, by trying to conform what they wrote to what I’d learned. — Concrete Mathematics and Bernoulli

2022-01-20: How vectorization speeds up your Python code. #python
2022-01-21: Stefano di Battista, Cosa avete fatto a Solande.
2022-01-21: Postgres Indexes for Newbies. #database
2022-01-24: Jacques Loussier Trio, Gymnopédie No. 1, Variation 1.
2022-01-24: A Handful Of Memories by Adam Wakeman
2022-01-24: Hot end of the week in the kitchen ;-)

2022-01-24: Do you feel like Google search results keep getting worse?
2022-01-25:

What Gemini is doing, is saying “we don’t need no videos, images, stylesheets, nor JavaScripts, because we want to have a lightweight web experience, so we throw all that crap out!”. Fine, sounds great. But why does it require a new protocol for that? Why couldn’t one simply build on top of existing HTTP infrastructure, throw away all the baggage and instead implement a new Content-Type, which existing browsers then could parse? — Gemini is Solutionism at its Worst

Tips and tricks for Neomutt

Wed, 19 Jan 2022 20:15:24 +0100

A few months ago, I talked about my new email client, neomutt, and the default settings I settled upon. Nothing has changed much, except that I now use a single catchup email address, which makes managing emails more comfortable over time.

Here are a few other tricks that I learned (the hard way) in the meantime:

If you use Vim or Neovim as your main editor for writing emails, you can enable spell checking as follows: set editor = "nvim +8 -c 'set textwidth=0' -c 'set spell spelllang=fr,en'". Note that you can use more than one language.
I don’t like notification, but you can get live notifications on your desktop using inotify.
If you have multiple accounts (past or present), it is useful to define alternates, e.g. alternates (john@smith.com)|(john.smith@(mac\.com|gmail\.com)).
Folding may be enabled automagically for threads provided you ask for it, e.g., write folder-hook archives/* 'push <collapse-all>\n' in your muttrc config file, after replacing archives with the mailbox of interest.
If you prefer reading plain text message over, say, HTML version, don’t forget to add alternative_order text/plain text/enriched text/html.
If you use mu as your main backend to process your email archives, you can benefit from its search functionalities in addition to Neomutt builtin ones. I use the following bindings to query and retrieve results:
```
macro index F "<shell-escape>mu find --clearlinks --format=links --linksdir=~/.mu/results " "mu find"
macro index ,f "<change-folder-readonly>~/.mu/results<enter>" "mu find results"
```
Rather than reading then deleting a message, you can mark it as read and delete it altogether using the following macro: macro index,pager X ":set confirmappend=no delete=yes resolve=no\n<clear-flag>N<tag-prefix><delete-message>:set confirmappend=yes delete=ask-yes resolve=yes\n<next-undeleted>". This way, to delete a message just press the X key; it will be marked as read and moved to your Trash folder. No more unread message in your Trash folder!
To mark all messages as read, you can use the following macro: macro index \Cr "T~U<enter><tag-prefix><clear-flag>N<untag-pattern>.<enter>" "mark all messages as read".

Finally, to view specific attachments, you can use the following settings:

application/pdf; pdftotext -layout %s -; copiousoutput;
# application/pdf; /usr/bin/xdg-open %s ; copiousoutput
application/postscript ; /usr/bin/xdg-open %s ; copiousoutput
application/msword; pandoc --from docx --to plain %s; copiousoutput
application/rtf; pandoc --from rtf --to plain %s; copiousoutput

♪ Chris Bailey • Bring It On Home to Me

Un problème de rencontres

Fri, 14 Jan 2022 20:17:44 +0100

Asking for the number of permutations of $n$ distinct elements such that element $k$ is not in the $k$th position, $k=1, \dots, n$, is just the “problème des rencontres”.¹ As stated on Wikipedia, it amounts to determine the probability that a random permutation is a derangement.²

The total number of objects to be considered is $N=n!$, by definition. All those $a_n$ permutations are distinguished by the fact that element $a_k$ is in the $k$th position. The problem is symmetric because the selection of a set of size $j$ determines a number independent of the particular set that is being selected. For instance, if $b_1, b_2, \dots, b_j$ is such a subset, then $N(b_1b_2\dots b_j) = (n-j)!$. From there, the relative sum $S_j$ over such a subset turns out to be:

$$ S_j = \frac{{n \choose j}(n-j)!}{n!} = \frac{1}{j!}. $$

Riordan relies on symbolic development (pp. 52-53) to show that:

$$ \begin{align} p_n(0) &= 1 - S_1 + S_2 - S_3 + \dots + (-1)^nS_n\cr &= 1 - 1 + \frac{1}{2!} - \frac{1}{3!} + \dots + \frac{(-1)^n}{n!} \end{align} $$

and also that:

$$ \begin{align} p_n(k) &= S_k - (k+1)S_{k+1} + {k+2 \choose 2}S_{k+2} - \dots + (-1)^{n-k}{n \choose k}S_n\cr &= \frac{1 - 1 + \frac{1}{2!} - \dots + \frac{(-1)^{n-k}}{(n-k)!}}{k!} \cr &= \frac{p_{n-k}(0)}{k!} \end{align} $$

The above indicates that all probabilities are determined by $p_n(0)$ which is the truncated exponential series $e^{-1}$,³ whose mean is $S_1=1$. For large $n$, the distribution is approximated by a Poisson distribution with mean 1 since $p_n(k) \approx e^{-1}/k!$.

Riordan, John. Introduction to Combinatorial Analysis. Dover Publications Inc., 2002. ↩︎
An $\mathcal{O}(N)$ algorithm to find a particular derangement of a sequence of $n$ items consists in dividing the original sequence in pairs of items and then swap each item from each pair; if $n$ is odd the last pair must be swapped twice, which means there are at most $n/2+1$ swaps. It does not help generating a random derangement, but it finds the derangement with the least perturbation in lexicographic order. ↩︎
Recall that well-known Taylor and Maclaurin series includes $e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \dots + \frac{x^n}{n!} = \sum_{n=0}^\infty \frac{x^n}{n!}$ for all $x$. A truncated series is just an approximation of an infinite series using some maximum for the series. For instance, in the case of a geometric series, $1 + t + t^2 + t^3 + \dots = \sum_{n=0}^\infty t^n$, we know that when $|t|<1$, the solution converges to $\frac{1}{1-t}$. Instead of this, we could consider $\sum_{n=0}^N t^n$, which can be evaluated numerically. ↩︎

Pandoc templates are great

Mon, 10 Jan 2022 21:17:41 +0100

In my previous post, I talked about writing Org file in Vim. While I much prefer the Org format, I often fall back on Markdown as the default markup for plain text files. I devised several Pandoc templates over the years, especially for RMarkdown documents. As discussed in an older post of mine, I use a Makefile and a YAML header to process my text MArkdown or RMarkdown documents. Having a separate YAML header as well as plain $\LaTeX$ file to add custom macros make it easy to define a common template for a series of handouts. The main font is Fira Sans, and its derivation (Fira Sans Condensed and Fira Code). It took me a longer time to understand how to use ligatures in listings-based code block. I’ve used various workarounds, like using Unicode characters (e.g., for the R assignment operator, <-) or additional $\LaTeX$ packages for verbatim environments. Finally, the solution is quite easy once you learn it: Just add the lstfiracode package, and customize your listings settings as follows: \lstset{basicstyle=\small\ttfamily, style=FiraCodeStyle, ...}.

Lately, I’ve also used the Eisvogel template for some handouts. It works great, its design favors simple (sans) defaults, but I don’t really like the default settings for code chunks, for example. Needless to say, I wrote my own custom template to export Org file to Latex Tufte handout via Pandoc, as discussed elsewhere on this blog. I also have a similar workflow for Md -> Tex export, without using the Tufte handout Latex class. It reimplements part of the Tufte layout, but it relies on different base fonts (Fira and STIX Two Math) and verbatim environments. Here is an excerpt:

\usepackage[a4paper, left=1cm, right=8cm, top=1cm, bottom=1cm,
marginparwidth=6.5cm]{geometry}
\usepackage{sidenotes}
\usepackage{sectsty}
\subsectionfont{\color{DarkSlateGrey}}
\usepackage{titlesec}
\titlespacing{\section}{0pt}{\parskip}{-\parskip}
\titlespacing{\subsection}{0pt}{\parskip}{-\parskip}
\renewcommand{\baselinestretch}{1}
\usepackage[justification=justified, singlelinecheck=false]{caption}

\DeclareCaptionFont{white}{\color{white}}
\DeclareCaptionFormat{listing}{\hspace*{-3.5pt}%
 \parbox{\textwidth}{\colorbox{LightSlateGrey}{\parbox{\dimexpr\textwidth+1pt}{#1#2#3}}\vskip2.8pt}}
\captionsetup[lstlisting]{labelformat=empty,format=listing,labelfont=white,textfont=white}

\renewenvironment{figure}[1][width=\marginparwidth]
 {\begin{marginfigure}}
 {\end{marginfigure}}

\renewcommand{\footnote}[1]{\let\thesidenote\relax\sidenotetext{#1}}

Pandoc templates are great because it reduces the $\LaTeX$ you have to write. I know most text editors will handle snippets or shortcuts that will create new environment or section or whatever on the fly. However, If you can stick to Markdown or Org syntax for 80% of the document, that’s not so bad either.

Finally, a long time ago (if memory serves it was in 2009), I got inspired by Kieran Healy’s own templates and I devised a simple Xe$\LaTeX$ article-based Pandoc template which used another set of great fonts:¹

\usepackage[xetex,
 colorlinks=true,
 urlcolor=BlueViolet,
 citecolor=greeny,
 filecolor=,
 linkcolor=greeny,
 plainpages=false,
 pdfpagelabels,
 bookmarksnumbered,
 pdftitle={\mytitle},
 %pagebackref,
 pdfauthor={\myauthor},
 pdfkeywords={\mykeywords}]{hyperref}

\setromanfont[Mapping={tex-text},Numbers={OldStyle},%
 Ligatures={Common}]{Minion Pro}
\setsansfont[Mapping={tex-text},Scale=0.9]{Myriad Pro}
\setmonofont[Mapping=tex-text,Colour=AA0000,Scale=0.9]{Inconsolata}

Today, I decided to rehabilitate this template using a much simpler Pandoc template, and to replace the Fira font triumvirate with the above settings. Rather than writing a separate YAML header, I put the header directly into the Markdown file.

---
mainfont: Minion Pro
sansfont: Myriad Pro
monofont: Fira Code
mathfont: STIX Two Math
monofontoptions: >
 [Scale = 0.8, Numbers = Lining,
 BoldFont = Fira Code Medium,
 Contextuals = Alternate]
mainfontoptions: [Numbers = OldStyle, Ligatures = Rare]
mathfontoptions: [Scale = MatchLowercase]
fontsize: 9pt
linestretch: 1
classoption: [svgnames]
lang: fr-FR
citecolor: DarkSlateGrey
urlcolor: SlateGrey
linkcolor: SlateGrey
link-citations: true
bibliography: /home/chl/Documents/notes/references.bib
csl: apa-fr-provost.csl
header-includes:
 - \defaultfontfeatures{Extension = .otf}
 - \usepackage{fontawesome}
 - \usepackage{lstfiracode}
 - \usepackage{setspace}
 - >
 \AtBeginEnvironment{quote}{\sffamily\raggedright
 \frenchspacing\setstretch{1.0}}
 - >
 \lstset{basicstyle=\small\ttfamily,
 style=FiraCodeStyle,
 commentstyle=\color{DarkSlateGrey},
 frame = single,
 rulecolor = \color{LightSlateGrey}}
---

Here is a side by side preview of both variants (Adobe fonts on the left, Fira on the right):

Of course, the same template can be applied to an Org file, with minor adaptation, using Latex header options.

See also What best combination of fonts for Serif, Sans, and Mono do you recommend? ↩︎

Generating partitions

Sun, 09 Jan 2022 10:19:18 +0100

In a previous post we showed how to efficiently generate the power set of a given set of elements. Other than its relation to Binomial coefficient as documented on Wikipedia, I don’t know if they are of any use on statistics. We also mentioned the case of partitions, which are more interesting from a statistical point of view since they are at the heart of partitioning methods in cluster analysis. How do we generate all partitions? Don Knuth discusses two algorithms to generate integer (algorithm P) or set (algorithm H) partitions,¹ but see also The Algorithm Design Manual (or this older course by Steven Skiena).

The main idea of algorithm H to generate set partition is as follows:

Set $a_1\dots a_n \leftarrow 0\dots 0, b_1\dots b_{n-1} \leftarrow 1\dots 1, \text{and } m \leftarrow 1$.
Visit the restricted growth string $a_1\dots a_n$,² which represents a partition into $m + [a_n=m]$ blocks. Then go to (4) if $a_n=m$.
Set $a_n \leftarrow a_n + 1$ and return to (2).
Set $j \leftarrow n - 1$; then, while $a_j = b_j$, set $j \leftarrow j - 1$.
Terminate if $j = 1$. Otherwise set $a_j \leftarrow a_j + 1$.
Set $m \leftarrow b_j + [a_j=b_j]$ and $j \leftarrow j + 1$. Then, while $j < n$, set $a_j \ leftarrow 0$, $b_j \leftarrow m$, and $j \leftarrow j + 1$. Finally set $a_n \leftarrow 0$ and go back to (2).

An Haskell solution to multiset partitions is available in Generating Multiset Partitions, by Brent Yorgey.

What about the total number of partitions? The number of partition, $p(n)$, can be formulated using recursion. Let $p_1(k) = 1$ ($1\le k \le n$), then

$$ p_m(k) = \begin{cases} p_{m-1}(k) & \mbox{if } k > n,\cr p_{m-1}(k) + p_m(k-m) & \mbox{if } k \ge m > 1, \end{cases} $$

and we evaluate $p_m(k)$ for $1 \le m \le n$ and $m \ge k \ge n$. Note that $p(n) = p_n(n)$. Berstel et al. offer an implementation in Pascal to compute $p(n)$ as $\alpha 10^4 + \beta$, whose literal translation in C looks like this:³ (untested code)

#include <stdlib.h>

void npartitions(int n, int *alpha, int *beta) {
 int nmax = 100;
 int base = 10000;
 int a[nmax];
 int b[nmax];
 int k, m, s;

 for (k = 0; k < n; ++k) {
 a[k] = 0;
 b[k] = 1;
 }

 for (m = 2; m < n; ++m) {
 for (k = m; k < n; ++k) {
 s = b[k] + b[k - m];
 b[k] = s % base;
 div_t q = div(s, base);
 a[k] = a[k] + a[k - m] + q.quot;
 }
 }

 *alpha = a[n];
 *beta = b[n];
}

♪ Leisure • Take You Higher

D. E. Knuth. The Art of Computer Programming, Vol. 4A: Combinatorial Algorithms. Addison-Wesley, 2011. ↩︎
A restricted growth string is a string $a_1a_2\dots a_n$ of nonnegative integers in which we have $a_1=0$ and $a_{j+1} \le 1 + \text{max}(a_1,\dots, a_j)$ for $1 \le j < n$. ↩︎
J. Berstel, J.-E. Pin, and M. Pocchiola. Mathématiques et Informatique : Combinatoire et Arithmétique. Ediscience International, 1991. ↩︎

Org in Vim

Thu, 06 Jan 2022 19:55:45 +0100

It’s been a few years since I last discussed the benefits of using Org rather than, say, Markdown. In short, the syntax is cleaner, it add further structural editing elements (like example, verse, etc.), it adds Babel code block (which works quite well under Emacs), Org tables are easier to manage than Markdown or Pandoc ones, and it is well interfaced with Pandoc. And for those who like GTD and stuff like that it has its own agenda and appointment management tool chain.

I wrote my last post about it when I stopped using Emacs full time.¹ At this time, there were two competitors to Emacs org-mode in Vim: vim-orgmode and vimwiki. These plugins were surely not as powerful as org-mode, and neither provided agenda-like or code block facilities. I ended up writing Org mode as I do write Markdown, as plain text. Now, things have changed a little as far as Markdown is concerned since a Treesitter parser has been made available. And today I just came across orgmode, which is specifically designed for Neovim users and relies on Treesitter for the most part.

I installed this plugin alongside vim-table-mode, and everything works as advertised. I didn’t tested the agenda facilities since I no longer rely on it in my daily workflow.² In particular, I like the fact that I get automatic folding as one would expect when working with Emacs. I do not use the concealing feature for links and other emphasized pieces of text, but they look to work too, provided you set conceallevel=2 in your config or from the command line when editing your file. For editing purpose at least, it is enough for me. Below is a screenshot of my Vim 101 tutorial, which I wrote in Emacs two years ago.

If you take a look at the online help for orgmode, you will soon notice that it focus on agenda functionalities. I hope that automatic evaluation of code blocks will be integrated soon. I tried sniprun breifly in the afternoon, and although it is very approachable and it provides a concise way to evaluate code block and displaying results using modern features like virtual text, it does not support Lisp-like PLs (Common Lisp, Scheme or Clojure). We’ll see how it goes then. In the meantime, the vim-table-mode is a killer feature since it even works for any filetype in Vim.

This is my first post of the year, by the way. Happy new year to all of you!

♪ Camille Berthollet & Julie Berthollet • The Leftovers, Suite

See also Emacs Org and Tufte handout, Org custom CSS, or Org exports. ↩︎
I’ve been using Org agenda for two years, and I used to manage everything (meeting notes, projects tasks and TODO lists, appointments) in Emacs using org-mode. Then I realized once again that GTD and workflow like that are definitely not for me. A simple TODO list in plain text and Changelog are enough for me, together with Git history. ↩︎

Let's close the year

Fri, 31 Dec 2021 20:04:04 +0100

And so, this is already the end of the year.

No year in review today, I’m definitely too lazy tonight. All I could say is that the year went by very quickly in the end. I did the job I had planned to do, besides the move and the unexpected medical care, and I could have tinkered a bit more with Scheme and Lisp, but that’s for another time probably. Today, I’m sick (no COVID, though, just a heavy cold) and alone at home. I have, however, enough alcohol to finish my evening looking at the Christmas tree, reading a novel before falling asleep quietly until tomorrow.

I wish I still felt like drawing, but I don’t think I feel like it anymore.

Same for writing.

En tirant sur sa mèche
Je vis apparaître des flocons encore vierges
$\quad\quad\quad$ de toute trace d’encre
Je décidais de prendre sa main
Et tout mon bras se vida de son sang
$\quad\quad\quad$ pour n’être plus qu’une aile repliée
Lorsque j’ai caressé ses lèvres
Le sol se déroba sous mes pieds
$\quad\quad\quad$ me laissant en suspension au-dessus de son corps
Elle prit enfin mon autre main
Pour ne plus que je m’évapore
Mais au lieu de la serrer
Elle a souri et ses yeux sont devenus un reflet
où je voyais danser de minuscules filaments
$\quad\quad\quad$ de mon désir

Quand je m’éveillais et que je ne sentis plus sa main
Je sus qu’elle m’avait dévoré durant mon sommeil

I upgraded my Python, Racket and R packages yesterday, mad made a complete backup of my $HOME directory today. While everything went fine with R and Racket, I broke my pylsp installation, which occurs more often than I would. Every. Single. Time. (This time, the problem came from the tomli dependency). I also added ‘kosayoda/nvim-lightbulb’ to my list of Neovim plugins in order to make code actions more discoverable. Actually, they are available for Haskell, Rust, Python (although it looks like garbage most of the time), and Clojure. At least, those are the languages I use the most with LSP. I also added ’eraserhd/parinfer-rust’ recently. It works like a charm and it makes working with s-expressions much smoother, especially when deleting part of a form. Finally, I tried another polyglot REPL: ‘hkupty/iron.nvim’ (as a replacement to neoterm, which appears to be on hold since a few months).

If you haven’t listened to Led Zeppelin recently, I can recommend this beautiful cover of Stairway to heaven.

Happy new year to all of you!

♪ Nick Cave & The Bad Seeds • Slowly Goes The Night

Generating power sets in Lisp

Wed, 29 Dec 2021 20:16:50 +0100

A power set is the set of all subsets composed of one or more elements of the original set. Consider the set $\{1,2,3\}$, then its power set is: $\{\}$, $\{1\}$, $\{2\}$, $\{3\}$, $\{1,2\}$, $\{1,3\}$, $\{2,3\}$, $\{1,2,3\}$. It is different from a partition which considers mutually exclusive elements, excluding empty sets and considering only the union of all possible subsets. In this particular case, the corresponding partition would be: $\{\{1\}, \{2\}, \{3\}\}$, $\{\{1, 2\}, \{3\}\}$, $\{\{1, 3\}, \{2\}\}$, $\{\{1\}, \{2, 3\}\}$ and $\{\{1, 2, 3\}\}$.

How to generate such a power set in a recursive manner? First, the power set of an empty list is of course the empty list. Second, the power set of $A = \{a,\dots\}$, where $\dots$ represents the cdr of $A$ using Lisp notation (i.e., all elements after the first one, $a$) amounts to concatenating the power set of $A - \{a\}$ – which means all subset of $A$ which do not include $a$ – and, again, the power set of $A - \{a\}$, this time with a prepended to each subset. In the above example, the later point yields, on the one hand, $\{\}$, $\{2\}$, $\{3\}$, $\{2,3\}$, and on the other hand, $\{1\}$, $\{1,2\}$, $\{1,3\}$, $\{1,2,3\}$. I don’t remember the name of this algorithm.

In Lisp, we can write the following:¹

(defun power-set (lst)
 (if (eq nil lst) '(())
 (let ((power-set-cdr (power-set (cdr lst))))
 (append power-set-cdr
 (mapcar #'(lambda (subset) (cons (car lst) subset))
 power-set-cdr)))))

[2023-02-06]
Daniel P. Friedman suggested this implementation of power set in Scheme, see his slides on The Joys of Scheme (PS):

(define power-set
 (lambda (set)
 (cond
 ((null? set) '(()))
 (else (extend (car set) (power-set (cdr set)))))))
(define extend
 (lambda (item power-set
 (append
 (map (lambda (set) (cons item set)) power-set)
 power-set))

Using our base example, this gives:

CL-USER(11): (power-set '(1 2 3))

(nil (3) (2) (2 3) (1) (1 3) (1 2) (1 2 3))

[2023-03-14]
For a related approach, see Set of All Partitions of a Finite Set by Atabey Kaygun.

What’s more interesting is that it works with characters as well, so that we can compose an infinite number of words given a constrained alphabet. In the following example, I will consider the most frequently used letters in French:

(defparameter *set* '(#\e #\a #\i #\s #\n #\r #\t #\o))

And here is the result:

 CL-USER(22): (mapcar #'(lambda (x) (format nil "~{~a~}" x)) (power-set *set*))

("" "o" "t" "to" "r" "ro" "rt" "rto" "n" "no" "nt" "nto" "nr" "nro" "nrt"
 "nrto" "s" "so" "st" "sto" "sr" "sro" "srt" "srto" "sn" "sno" "snt" "snto"
 "snr" "snro" "snrt" "snrto" "i" "io" "it" "ito" "ir" "iro" "irt" "irto" "in"
 "ino" "int" "into" "inr" "inro" "inrt" "inrto" "is" "iso" "ist" "isto" "isr"
 "isro" "isrt" "isrto" "isn" "isno" "isnt" "isnto" "isnr" "isnro" "isnrt"
 "isnrto" "a" "ao" "at" "ato" "ar" "aro" "art" "arto" "an" "ano" "ant" "anto"
 "anr" "anro" "anrt" "anrto" "as" "aso" "ast" "asto" "asr" "asro" "asrt"
 "asrto" "asn" "asno" "asnt" "asnto" "asnr" "asnro" "asnrt" "asnrto" "ai" "aio"
 "ait" "aito" "air" "airo" "airt" "airto" "ain" "aino" "aint" "ainto" "ainr"
 "ainro" "ainrt" "ainrto" "ais" "aiso" "aist" "aisto" "aisr" "aisro" "aisrt"
 "aisrto" "aisn" "aisno" "aisnt" "aisnto" "aisnr" "aisnro" "aisnrt" "aisnrto"
 "e" "eo" "et" "eto" "er" "ero" "ert" "erto" "en" "eno" "ent" "ento" "enr"
 "enro" "enrt" "enrto" "es" "eso" "est" "esto" "esr" "esro" "esrt" "esrto"
 "esn" "esno" "esnt" "esnto" "esnr" "esnro" "esnrt" "esnrto" "ei" "eio" "eit"
 "eito" "eir" "eiro" "eirt" "eirto" "ein" "eino" "eint" "einto" "einr" "einro"
 "einrt" "einrto" "eis" "eiso" "eist" "eisto" "eisr" "eisro" "eisrt" "eisrto"
 "eisn" "eisno" "eisnt" "eisnto" "eisnr" "eisnro" "eisnrt" "eisnrto" "ea" "eao"
 "eat" "eato" "ear" "earo" "eart" "earto" "ean" "eano" "eant" "eanto" "eanr"
 "eanro" "eanrt" "eanrto" "eas" "easo" "east" "easto" "easr" "easro" "easrt"
 "easrto" "easn" "easno" "easnt" "easnto" "easnr" "easnro" "easnrt" "easnrto"
 "eai" "eaio" "eait" "eaito" "eair" "eairo" "eairt" "eairto" "eain" "eaino"
 "eaint" "eainto" "eainr" "eainro" "eainrt" "eainrto" "eais" "eaiso" "eaist"
 "eaisto" "eaisr" "eaisro" "eaisrt" "eaisrto" "eaisn" "eaisno" "eaisnt"
 "eaisnto" "eaisnr" "eaisnro" "eaisnrt" "eaisnrto")

Next step would be to ask how many of those “artificial” words are real words? In other words, how many of the above pseudo-words are found in your Unix or Mac dictionary (/usr/share/dict/french)?

[2022-02-03]
See also Wordle letters for more “classical”, Perl-based, approaches to filtering dict files on Linux or Mac.

[2024-02-01]
Yet another shell-based workflow to solve wordle puzzle: Wordle With Grep.

♪ Neon Indian • Should Have Taken Acid With You

Newlisp makes it even easier:

(define (power-set x)
 (if (empty? x)
 (list '())
 (let ((elt (first x))
 (p (power-set (rest x))))
 (append (map (fn (s) (cons elt s)) p) p))))

↩︎

Mon maître et mon vainqueur

Tue, 28 Dec 2021 22:14:40 +0100

Vous, je ne sais pas, mais moi je vois la vie comme deux lignes parallèles : la première représente ce à quoi l’on aspire, ce que l’on voudrait être ; la seconde, ce que l’on est réellement. Et bien sûr elles ne se superposent jamais tout à fait, mais tout l’enjeu est d’en réduire l’écart autant que possible. On en mesure pas la réussite d’une vie à l’écart entre ces deux lignes, mais à l’effort consenti pour le réduire. — François-Henri Désérable

Alors évidement on pense tout de suite à l’idée de la régression linéaire, n’est-ce pas ? Mais peu importe. J’écris des articles techniques la plupart du temps, alors parler de littérature, même si c’est une grande passion pour moi, serait un peu hors de propos sur ce blog.

Au fond, j’écris sur des sujets scientifiques dont je ne retire quasiment rien, après coup, sinon la satisfaction d’avoir approfondi des sujets que je souhaitais maîtriser un peu plus, et d’avoir quelques notes dans mes archives. C’est la raison pour laquelle cette section du site s’appelait initialement Memos et non Blog. On peut aussi voir ça comme une forme de formation continue, en quelque sorte, ou une manière de m’échapper des contraintes du “bureau ordinaire” (comme dans métro-boulot-dodo). Rester éveillé “au chevet des mots”, au bout de la nuit, en revanche, ça c’est quelque chose qui est plus enrichissant, et voici encore un ouvrage, parmi la soixantaine de livres lus cette année, que j’ai bien apprécié. À bon entendeur !

♪ A Certain Ratio • Berlin

lost+found 2021

Mon, 27 Dec 2021 21:39:37 +0100

It is very unlikely these posts will ever end in anything other than rough drafts, so closing them right now.

2020-10-04

I was about to benchmark some common statistical programs used to perform boosting, then forgot about it.

There has been a renewed interest in Boosting techniques over the past few years, possibly through Kaggle contests where it ranks among the top algorithms currently in use. LightGBM and XGBoost (and CatBoost) are probably the most used algorithms at the time of this writing.¹ Packages are available for R, Python, Clojure, and probably other scientific PLs.

Boosting tree models have a number of advantages.² They are (quite) easy to setup and to interpret (for simple datasets), they deal with continuous and categorical predictors (or monotonic transformations thereof) whether there are missing values or not, they are not that much sensitive to outliers compared to other supervised learning approaches, and they can capture simple or higher-order interactions between features. They, however, suffer from higher variance and lack of stability (like almost any tree-based models) and have poorer predictive performance compared to, say, well regularized linear or GLM models.

The lead article is of course the one written by Hastie and coll. in 2001, but see one of Hastie’s talk (given at MIT in 2003). The H20 team also provides a nice summary of the algorithm. Briefly, the idea of this ensemble method is to aggregate multiple classifiers built sequentially on a training dataset where individual weights are updated online. Classifiers are then weighted based on their own performance, before delivering an averaged decision.

In what follows, I’m going to use a dataset from the medical literature from the UCI archive to fit a GBM using Python, with the help of the H2O package. It should be possible to use Spark (with MLlib) instead. Originally, I thought about using a data set from Statistical Learning for Biomedical Data (Malley, Malley & Pajevic, Cambridge University Press, 2011), where the authors used Random Forest using subsampling to account for class imbalance. Unfortunately, I realized that the “Stroke-A dataset” is not in the public domain.³

import h2o
from h2o.estimators import H2OGradientBoostingEstimator
h2o.init()

stroke = h2o.import_file()

2021-06-07

Some rants against this “new” R (which will soon be called the modern R, but not as in Modern Applied Statistics with S).

Something that has always struck me for years is that: Learning R is hard. However, I would say that it’s like the Vim learning curve, not the Emacs’s one, unless you learned it the rlang’s way. You know, there’s some business behind the “tidyverse” way of doing things. Above that, there are too much shortcuts to learn how to speak R. I’m not thinking of the R inferno or things like that, just the language itself.

For the record, I learned R when it was at version one, and I’ve been teaching R for biostatistics during almost 15 years. A crude estimate suggests that I was successful 30 to 40% of the time. Still, I believe I was a good teacher. However I should note that I was not teaching R the easy way, but rather how to think in R for statistical computing. Maybe statistical computing is the problem here.

♪ Alboran Trio • Tema di pinicchio

More or less since I started drafting this post in April, 2020. ↩︎
Murphy, P., Kevin. 2012. Machine Learning: A Probabilistic Perspective. Cambridge, Massachusetts London, England: The MIT Press. ↩︎
Other interesting datasets are available on the Vanderbilt server/ ↩︎

Micro posting in December

Sun, 26 Dec 2021 21:47:15 +0100

2021-12-01:

It took little under a decade for the headline feature developer Jon Skinner added to Sublime Text’s second version to become one of the defining features of this decade’s software. — The History of Command Palettes: How Typing Commands Became The Norm Again

2021-12-01:

When I first started using Rust, I really missed monads. But here’s the thing. Having used lots of monads in Haskell, and read lots of blog posts about monads, I’ve learned that in systems contexts, it’s often best to just have a simple monad stack that just consists of Reader + IO (and Maybe’s and Option’s sprinkled about occasionally). Huge monad transformer stacks often raise more problems than they solve. But Reader + IO is essentially the “default monad stack” of Rust. — Haskell’s Children

2021-12-01: Design Issues for Foreign Function Interfaces: A survey of existing native interfaces for several languages and some suggestions.
2021-12-01: Emacs Docs: The modern documentation website Emacs deserves. #emacs
2021-12-01: Introducing Coalton: How to Have Our (Typed) Cake and (Safely) Eat It Too, in Common Lisp. #lisp
2021-12-01: Julia 1.7 Highlights. #julia
2021-12-02: An Illustrated Guide to Elliptic Curve Cryptography Validation.
2021-12-02: GDB dashboard. #clang
2021-12-02: The Fastest FizzBuzz Implementation.
2021-12-07:

2021-12-07: https://plotdigitizer.com/: really handy when you do not have raw data and want to revisit a 2D chart.
2021-12-07:

Using vi/vim properly, you don’t use it modally. You are always in normal mode, and only enter insert mode for short bursts of typing text, after which you press to go to normal mode. Thus, the remembering-the-mode problem just doesn’t exist: you don’t answer the phone in insert mode to get back to vi and not remember where you were. If you are typing text and the phone rings, you exit insert mode and then answer the phone. Or you press when you come back. But you never think about insert mode as a mode where you stay. — Why, oh WHY, do those #?@! nutheads use vi?

2021-12-07: Testing Firefox more efficiently with machine learning.
2021-12-08: Data Integration, Manipulation and Visualization of Phylogenetic Trees. #rstats #bioinfo
2021-12-08: What’s Functional Programming All About?
2021-12-14: Fiona Apple, Under The Table.
2021-12-14:

Dependencies matter. Every dependency you add to your project is an invitation to break your project. — Lightweight is the right weight

2021-12-14: Orgdown (in short “OD) is a lightweight markup language similar to Markdown but it’s consistent, easy to learn, simple. (via Irreal)
2021-12-14: Rethinking Email.
2021-12-14: jc: CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. (via Simon Willison)
2021-12-15: Section 25, Looking from a Hilltop.
2021-12-15: Biological Modeling: A free course in modeling biological systems at multiple scales.
2021-12-15: Blockchains don’t solve problems that are interesting to me.
2021-12-15: One Reason Typeclasses Are Useful. #lisp
2021-12-15:

Day & Night.

2021-12-17: Again, amazing work by Nicolas Rougier hacking on GNU Emacs: GNU Emacs - Notebook. #emacs
2021-12-17: Polars: Lightning-fast DataFrame library for Rust and Python. #rust
2021-12-20:

You are here because you are somehow interested in what the “Web3” or “NFT” thing is about. Maybe someone gave you this link, maybe you follow me somewhere. In this document I’ll try to explain what those terms mean, what ideas and politics they are based on and what I think about them. I will do my best to represent the Web3/NFT ideas as fairly as possible but for transparency’s sake I should note that I am not a fan. — The Third Web

2021-12-20: Common Lisp - Oldie but goldie . #lisp
2021-12-20: Unix Shell Programming: The Next 50 Years (The Future of the Shell, Part I). #unix
2021-12-20: Wingman for Haskell. #haskell
2021-12-20: r-ci: Easy and portable CI for R packages. #rstats
2021-12-22:

2021-12-22: Common Saints, Idol Eyes.
2021-12-22: Off for Christmas.
2021-12-26:
I’m back.
2021-12-26: Nick Cave & The Bad Seeds, The Ship Song.
2021-12-26: Vellum is a data frame library build for Common Lisp. #lisp

The unquantified self #20

Sat, 25 Dec 2021 20:43:20 +0100

Yet another post about how life is going while I’m about to complete this year 2021. I’ll be on holidays for the next two weeks. I haven’t had time to write as much blog posts than I originally thought last month, but I’m not worried. I will have plenty of time later on, probably, and if not this means those blog posts can wait. In the meantime, I managed to finish Homeland, and I listened to a lot of nice playlists of jazz but also from the post-punk era (close to 500 tracks each week for a month). You can check it out on Last.fm.

Sometimes I miss my old good Nord dark theme. As an alternative to my current setup (kitty + zsh), I considered using xterm + tmux. This might provide an easy-to-getup setup for remote machine since I only need to copy two config files. I grabbed xterm settings on GitHub and I started hacking my xterm config by adding a few options in $HOME/.Xresources:

xterm*faceName: JetBrains Mono
xterm*faceSize: 11
xterm*selectToClipboard: true
XTerm*metaSendsEscape: true

This set up xterm with a decent font, while allowing to copy from the clipboard (using Shift-Insert) and using the alt key (which I need for my tmux keybindings). My tmux setup is almost unchanged except that I wanted to replicate some kitty facilities, like the fact that we can hide the window status bar if there’s only one window opened. Here comes a little hack based on tmux hooks, which I found after reading this gist:

set -g status off
set-hook -g after-new-window 'if "[ #{session_windows} -gt 1 ]" "set status on"'
set-hook -g pane-exited 'if "[ #{session_windows} -lt 2 ]" "set status off"'

Now I have a nifty CLI setup based xterm and tmux, which remains available on almost all machines I deal with everyday.

Other than those Unix geekeries, here are some good dishes cooked lately:

♪ Mazzy Star • Fade into you

Micro posting in November

Fri, 26 Nov 2021 20:36:25 +0100

2021-11-02:

BSD is what you get when a bunch of Unix hackers sit down to try to port a Unix system to the PC. Linux is what you get when a bunch of PC hackers sit down and try to write a Unix system for the PC. — BSD vs Linux

2021-11-02: Clojonic: Pythonic Clojure. #clojure
2021-11-02: Let’s Decentralize: Alternatives to the standard internet stack.
2021-11-02: OpenBSD from a veteran Linux user perspective. #unix
2021-11-05: How to learn compilers: LLVM Edition.
2021-11-05: Rewind the web to 2002. Thanks, that was one of my preferred website back in 1999. This also remember me of those CD covers I was crafting together.

2021-11-15: Algorithms animated.
2021-11-15: Bayesian Optimization Book.
2021-11-15: Scientific visualization Course, by Nicolas Rougier.

2021-11-16: Project Euler, problem 10:

Mathematica: Total @ Prime @ Range @ PrimePi[2*^6]
Racket: ̀(apply + (filter prime? (range 1 2e6))) ((require math/number-theory))
2021-11-17: Old times, good times.

2021-11-17: Computing prime numbers with Clojure. #clojure
2021-11-17: The proper use of macros in Lisp. #lisp
2021-11-23:

It is oddly satisfying to know that no matter the kind of machine I find myself on or the kind of restrictions it has (no network to download Emacs and/or my custom configuration, corporate policy against fetching packages from MELPA, etc.), that there will always be an editor that I can happily use without needing to make it “my own”. — vi, my favorite config-less editor

2021-11-23: How Python List Works. #python
2021-11-23: How many principal components?.
2021-11-23: Learning Containers From The Bottom Up.
2021-11-23: Preserving k-mer frequencies in sequence shuffles.
2021-11-24: Alice in Chains, Facelift. Long time no see.
2021-11-24: If you are looking for some small or largish dataset for your statistical machinery, don’t forgetStatSci.org.
2021-11-24: Not to say, but this should be the 2958th post since 2018-02-15, when I first launch this micro-blog. Other than that there are 586 posts available on the regular blog, of which 319 were written the last 4 years. #this
2021-11-24: Lesser Known PostgreSQL Features.
2021-11-24: The Arcane Algorithm Archive.
2021-11-24: The Kelly Coin-Flipping Game: Exact Solutions. Also worth taking some time to explore the whole site which is beautifully designed.
2021-11-24:

//
// Dear maintainer:
//
// Once you are done trying to 'optimise' this routine,
// and have realized what a terrible mistake that was,
// please increment the following counter as a warning
// to the next guy://// total_hours_wasted_here = 16
//

https://stackoverflow.com/q/184618/420055

The unquantified self #19

Mon, 22 Nov 2021 19:30:26 +0100

As I’m (temporarily) tired of reviewing the intertubes and posting more or less interesting links on this blog, I decided to focus on Lisp/Racket and statistical programming until the end of the year at least. Ideally, I would like to translate some of the APSTAT algorithms in Scheme, maybe write some bindings for the Apache Arrow library for Racket, use Racket sketching or CL sketch for data visualization, and explore related packages for data processing in Racket. Actually, I really like Chicken Scheme, since it was the first Scheme I used for real applications 10 years ago, after Jan de Leeuw mentioned it in one of his regular emails on the UCLA statcompute list. Its FFI module is easy to use, which could ease the integration of Fortran or C code into reliable statistical libraries. Still, I like Racket better, for it has a lot of convenient procdeure built-in, and it comes with a powerful graphical backend. Stay tuned!

I’m done with Homeland Season 6. When I drafted this post, I was not quite sure to go on with the next seasons, but I didn’t find anything really exciting to fill my evenings. I guess I might as well read the pile of books waiting for me. Finally I did both: I started Season 7 and read a bunch of books.

I listen to music (thanks to Cmus) and Bandcamp almost all day for three weeks now, and it’s been great. When I look at my Last.fm stats, I see that I’m currently listening to 500 tracks each week. This does not include vinyls played at home, nor some of the iTunes tracks that don’t get scrobbled for whatever reason. Anyway, I’m quite happy with my current setup, and writing those blog posts has never been as comfortable now that I’m more used to (Neo)vim and that my blogging infrastructure doesn’t get updated every two weeks (which meant lot of breaking changes when I was on Homebrew). The only thing left from my Apple days is my photo library, which I haven’t relocated on my home server yet.

♪ Caroline Davis • Respite

Student's t distribution

Fri, 19 Nov 2021 21:23:27 +0100

In a previous post, we discussed how p-values are computed in the case of Student t-test in LispStat, R, Stata and Python. John Monahan has more to say about the Student’s t distribution in his book Numerical Methods in Statistics,¹ and how it can be approximated.

Specifically, let’s denote $Q(t\mid k) = \Pr(X > t)$, assuming $X$ follows a Student’s t distribution with $k$ df. As we noted earlier, the fact that $k$ ($\eta$ in our previous post) is an integer or not plays a role when one looks for an analytical solution. If $k$ is an integer, Abramowitz and Stegun gave the following formula:²

$$ Q(t\mid k) = \begin{cases} \frac{1}{2} - \frac{\theta}{\pi}, \quad k=1 \cr \frac{1}{2} - \frac{1}{\pi}\left[\theta + \sin\theta\left(\cos\theta + \frac{2}{3}\cos^3\theta + \dots + \frac{2\times 4\times\dots\times (k-3)}{1\times 3\times\dots\times (k-2)}\cos^{k-2}\theta\right)\right], \quad k\ \text{odd} \cr \frac{1}{2} - \sin\theta\left[1 + \frac{1}{2}\cos^2\theta + \frac{1\times 3}{2\times 4}\cos^4\theta + \dots + \frac{1\times 3\times\dots\times (k-3)}{2\times 4\times\dots\times (k-2)}\cos^{k-2}\theta\right], \quad k\ \text{even} \end{cases} $$

where $\theta = \text{arctan}(t/\sqrt{k})$.³ With large value of $k$, the incomplete beta function may be used as follows:

$$ Q(t\mid k) = \frac{1}{2}I_x(k/2,1/2) \quad \text{for}\; x = k/(k+t^2). $$

Note that the use of a Beta distribution was also at hand in our previous post. Furthermore, for large value of $t$, some of the trigonometric expressions above may be avoided by using the fact that $\sin(\theta) = 1/\sqrt{1+w}$ and $\cos(\theta) = \sqrt{\frac{w}{1+w}}$, with $w=k/t^2$. When $k$ is even, we have:

$$ Q(t\mid k) = \frac{\sqrt{1+w} - B}{2\sqrt{1+w}}, \quad \text{where}\ B = \sum_{j=1}^{n/2-1} c_j\left(\frac{w}{1+w}\right)^j $$

with $c_j = (2j-1)c_{j-1}/(2j)$, and $c_0 = 1$. To avoid cancellation, we can use the following power series: $\sqrt{1+w} = 1 + \frac{1}{2}w - \frac{1}{8}w^2 + \frac{1}{16}w^3 - \frac{5}{128}w^4 + \frac{7}{256}w^5 - \dots$, which allows to compute $\sqrt{1+w}-B$ by matching powers of $w$.

Similarly, when $k$ is odd,

$$ Q(t\mid k) = \frac{1}{\pi}\left[\frac{\pi}{2} - \text{arctan}\left(\frac{t}{\sqrt{k}}\right) - \frac{\sqrt{w}}{1+w}\sum_{j=1}^{(n-3)/2} d_jw^j\right], $$

where $d_j = 2jd_{j-1}/(2j+1)$ and $d_0 = 1$. Again, the following series can be used: $\sqrt{1+w} = \frac{\pi}{2} - \text{arctan}\left(\frac{t}{\sqrt{k}}\right) = \frac{1}{\sqrt{w}}\left[1-\frac{w}{3} + \frac{w^2}{5} - \frac{w^3}{7} + \frac{w^4}{9} - \dots \right]$.

The tail probabilities for Student’s $t$ distribution can also be represented using the Gamma distribution as:

$$ Q(t\mid k) = \frac{\Gamma\big((k+1)/2\big)}{\Gamma(k/2)\sqrt{\pi}}u^{-k/2}\sum_{j=0}^\infty\frac{c_j}{(k+2)u^j}, $$

where $c_0 = 1$, $c_j = \frac{2j-1}{2j}c_{j-1}$, and $u = 1 + \frac{t^2}{k}$.

Currently, there’s no way to compute tail probability from the Student’s $t$ distribution in Racket math library, although it has everything we need for the Gamma or Beta distribution. Let’s do this by hand using Racket’s builtin facilities to compute the regularized incomplete Beta function:

(require math/special-functions)

(define (pt t df [two-tailed? #f])
 (let ([beta (* 0.5 (beta-inc (* 0.5 df) 0.5 (/ df (+ df (sqr t))) #f #t))])
 (if two-tailed? (* 2 beta) beta)))

(pt 2.41 36)
; => 0.010594171035128275

R returns $Q(2.41 \mid 36) = 0.01059417$ for the one-tailed probability.

[2022-10-17]
For those more versed into the Python ecosystem, John Cook wrote a nice summary of what’s available in Scipy: Using Python as a statistical calculator.

John F. Monahan. Numerical Methods in Statistics. Cambridge University Press (2011). ↩︎
Abramowitz and Stegun. Handbook of Mathematical Functions ↩︎
As noted by Monahan, for large values of $t/\sqrt{k}$, which may happen for very large test statistic under low df distribution, this expression may yield to potential cancellation. ↩︎