Generalised to multiple runtime directories with priorities (#5411)

* Generalised to multiple runtime directories with priorities

This is an implementation for #3346.

Previously, one of the following runtime directories were used:

1. `$HELIX_RUNTIME`
2. sibling directory to `$CARGO_MANIFEST_DIR`
3. subdirectory of user config directory
4. subdirectory of path to helix executable

The first directory provided / found to exist in this order was used as a
root for all runtime file searches (grammars, themes, queries).

This change lowers the priority of `$HELIX_RUNTIME` so that the user
config runtime has higher priority. More significantly, all of these
directories are now searched for runtime files, enabling a user to override
default or system-level runtime files. If the same file name appears
in multiple runtime directories, the following priority is now used:

1. sibling directory to `$CARGO_MANIFEST_DIR`
2. subdirectory of user config directory
3. `$HELIX_RUNTIME`
4. subdirectory of path to helix executable

One exception to this rule is that a user can have a `themes`
directory directly in the user config directory that has higher piority
to `themes` directories in runtime directories. That behaviour has been
preserved.

As part of implementing this feature `theme::Loader` was simplified
and the cycle detection logic of the theme inheritance was improved to
cover more cases and to be more explicit.

* Removed AsRef usage to avoid binary growth

* Health displaying ;-separated runtime dirs

* Changed HELIX_RUNTIME build from src instructions

* Updated doc for more detail on runtime directories

* Improved health symlink printing and theme cycle errors

The health display of runtime symlinks now prints both ends of the
link.

Separate errors are given when theme file is not found and when the
only theme file found would form an inheritence cycle.

* Satisfied clippy on passing Path

* Clarified highest priority runtime directory purpose

* Further clarified multiple runtime details in book

Also gave markdown headings to subsections.

Fixed a error with table indentation not building
table that also appears present on master.

---------

Co-authored-by: Paul Scott <paul.scott@anu.edu.au>
Co-authored-by: Blaž Hrastnik <blaz@mxxn.io>

book/src/install.md

@@ -137,8 +137,8 @@ cargo install --path helix-term --locked
 ```
 
 This command will create the `hx` executable and construct the tree-sitter
-grammars either in the `runtime` folder, or in the folder specified in `HELIX_RUNTIME`
+grammars in the local `runtime` folder. To build the tree-sitter grammars requires
-(as described below). To build the tree-sitter grammars requires a c++ compiler to be installed, for example `gcc-c++`.
+a c++ compiler to be installed, for example `gcc-c++`.
 
 > 💡 If you are using the musl-libc instead of glibc the following environment variable must be set during the build
 > to ensure tree-sitter grammars can be loaded correctly:
@@ -149,11 +149,13 @@ grammars either in the `runtime` folder, or in the folder specified in `HELIX_RU
 
 > 💡 Tree-sitter grammars can be fetched and compiled if not pre-packaged. Fetch
 > grammars with `hx --grammar fetch` (requires `git`) and compile them with
-> `hx --grammar build` (requires a C++ compiler).
+> `hx --grammar build` (requires a C++ compiler). This will install them in
+> the `runtime` directory within the user's helix config directory (more
+> [details below](#multiple-runtime-directories)).
 
 ### Configuring Helix's runtime files
 
-- **Linux and macOS**
+#### Linux and macOS
 
 Either set the `HELIX_RUNTIME` environment variable to point to the runtime files and add it to your `~/.bashrc` or equivalent:
 
@@ -167,7 +169,7 @@ Or, create a symlink in `~/.config/helix` that links to the source code director
 ln -s $PWD/runtime ~/.config/helix/runtime
 ```
 
-- **Windows**
+#### Windows
 
 Either set the `HELIX_RUNTIME` environment variable to point to the runtime files using the Windows setting (search for
 `Edit environment variables for your account`) or use the `setx` command in
@@ -190,6 +192,20 @@ Or, create a symlink in `%appdata%\helix\` that links to the source code directo
 > 💡 On Windows, creating a symbolic link may require running PowerShell or
 > Cmd as an administrator.
 
+#### Multiple runtime directories
+
+When Helix finds multiple runtime directories it will search through them for files in the
+following order:
+
+1. `runtime/` sibling directory to `$CARGO_MANIFEST_DIR` directory (this is intended for
+  developing and testing helix only).
+2. `runtime/` subdirectory of OS-dependent helix user config directory.
+3. `$HELIX_RUNTIME`.
+4. `runtime/` subdirectory of path to Helix executable.
+
+This order also sets the priority for selecting which file will be used if multiple runtime
+directories have files with the same name.
+
 ### Validating the installation
 
 To make sure everything is set up as expected you should run the Helix health

helix-loader/src/grammar.rs

@@ -67,8 +67,9 @@ pub fn get_language(name: &str) -> Result<Language> {
 #[cfg(not(target_arch = "wasm32"))]
 pub fn get_language(name: &str) -> Result<Language> {
     use libloading::{Library, Symbol};
-    let mut library_path = crate::runtime_dir().join("grammars").join(name);
+    let mut rel_library_path = PathBuf::new().join("grammars").join(name);
-    library_path.set_extension(DYLIB_EXTENSION);
+    rel_library_path.set_extension(DYLIB_EXTENSION);
+    let library_path = crate::runtime_file(&rel_library_path);
 
     let library = unsafe { Library::new(&library_path) }
         .with_context(|| format!("Error opening dynamic library {:?}", library_path))?;
@@ -252,7 +253,9 @@ fn fetch_grammar(grammar: GrammarConfiguration) -> Result<FetchStatus> {
         remote, revision, ..
     } = grammar.source
     {
-        let grammar_dir = crate::runtime_dir()
+        let grammar_dir = crate::runtime_dirs()
+            .first()
+            .expect("No runtime directories provided") // guaranteed by post-condition
             .join("grammars")
             .join("sources")
             .join(&grammar.grammar_id);
@@ -350,7 +353,9 @@ fn build_grammar(grammar: GrammarConfiguration, target: Option<&str>) -> Result<
     let grammar_dir = if let GrammarSource::Local { path } = &grammar.source {
         PathBuf::from(&path)
     } else {
-        crate::runtime_dir()
+        crate::runtime_dirs()
+            .first()
+            .expect("No runtime directories provided") // guaranteed by post-condition
             .join("grammars")
             .join("sources")
             .join(&grammar.grammar_id)
@@ -401,7 +406,10 @@ fn build_tree_sitter_library(
             None
         }
     };
-    let parser_lib_path = crate::runtime_dir().join("grammars");
+    let parser_lib_path = crate::runtime_dirs()
+        .first()
+        .expect("No runtime directories provided") // guaranteed by post-condition
+        .join("grammars");
     let mut library_path = parser_lib_path.join(&grammar.grammar_id);
     library_path.set_extension(DYLIB_EXTENSION);
 
@@ -511,9 +519,6 @@ fn mtime(path: &Path) -> Result<SystemTime> {
 /// Gives the contents of a file from a language's `runtime/queries/<lang>`
 /// directory
 pub fn load_runtime_file(language: &str, filename: &str) -> Result<String, std::io::Error> {
-    let path = crate::RUNTIME_DIR
+    let path = crate::runtime_file(&PathBuf::new().join("queries").join(language).join(filename));
-        .join("queries")
-        .join(language)
-        .join(filename);
     std::fs::read_to_string(path)
 }

helix-loader/src/lib.rs

@@ -2,11 +2,12 @@ pub mod config;
 pub mod grammar;
 
 use etcetera::base_strategy::{choose_base_strategy, BaseStrategy};
-use std::path::PathBuf;
+use std::path::{Path, PathBuf};
 
 pub const VERSION_AND_GIT_HASH: &str = env!("VERSION_AND_GIT_HASH");
 
-pub static RUNTIME_DIR: once_cell::sync::Lazy<PathBuf> = once_cell::sync::Lazy::new(runtime_dir);
+static RUNTIME_DIRS: once_cell::sync::Lazy<Vec<PathBuf>> =
+    once_cell::sync::Lazy::new(prioritize_runtime_dirs);
 
 static CONFIG_FILE: once_cell::sync::OnceCell<PathBuf> = once_cell::sync::OnceCell::new();
 
@@ -25,31 +26,83 @@ pub fn initialize_config_file(specified_file: Option<PathBuf>) {
     CONFIG_FILE.set(config_file).ok();
 }
 
-pub fn runtime_dir() -> PathBuf {
+/// A list of runtime directories from highest to lowest priority
-    if let Ok(dir) = std::env::var("HELIX_RUNTIME") {
+///
-        return dir.into();
+/// The priority is:
-    }
+///
-
+/// 1. sibling directory to `CARGO_MANIFEST_DIR` (if environment variable is set)
+/// 2. subdirectory of user config directory (always included)
+/// 3. `HELIX_RUNTIME` (if environment variable is set)
+/// 4. subdirectory of path to helix executable (always included)
+///
+/// Postcondition: returns at least two paths (they might not exist).
+fn prioritize_runtime_dirs() -> Vec<PathBuf> {
+    const RT_DIR: &str = "runtime";
+    // Adding higher priority first
+    let mut rt_dirs = Vec::new();
     if let Ok(dir) = std::env::var("CARGO_MANIFEST_DIR") {
         // this is the directory of the crate being run by cargo, we need the workspace path so we take the parent
         let path = std::path::PathBuf::from(dir).parent().unwrap().join(RT_DIR);
         log::debug!("runtime dir: {}", path.to_string_lossy());
-        return path;
+        rt_dirs.push(path);
     }
 
-    const RT_DIR: &str = "runtime";
+    let conf_rt_dir = config_dir().join(RT_DIR);
-    let conf_dir = config_dir().join(RT_DIR);
+    rt_dirs.push(conf_rt_dir);
-    if conf_dir.exists() {
+
-        return conf_dir;
+    if let Ok(dir) = std::env::var("HELIX_RUNTIME") {
+        rt_dirs.push(dir.into());
     }
 
     // fallback to location of the executable being run
     // canonicalize the path in case the executable is symlinked
-    std::env::current_exe()
+    let exe_rt_dir = std::env::current_exe()
         .ok()
         .and_then(|path| std::fs::canonicalize(path).ok())
         .and_then(|path| path.parent().map(|path| path.to_path_buf().join(RT_DIR)))
-        .unwrap()
+        .unwrap();
+    rt_dirs.push(exe_rt_dir);
+    rt_dirs
+}
+
+/// Runtime directories ordered from highest to lowest priority
+///
+/// All directories should be checked when looking for files.
+///
+/// Postcondition: returns at least one path (it might not exist).
+pub fn runtime_dirs() -> &'static [PathBuf] {
+    &RUNTIME_DIRS
+}
+
+/// Find file with path relative to runtime directory
+///
+/// `rel_path` should be the relative path from within the `runtime/` directory.
+/// The valid runtime directories are searched in priority order and the first
+/// file found to exist is returned, otherwise None.
+fn find_runtime_file(rel_path: &Path) -> Option<PathBuf> {
+    RUNTIME_DIRS.iter().find_map(|rt_dir| {
+        let path = rt_dir.join(rel_path);
+        if path.exists() {
+            Some(path)
+        } else {
+            None
+        }
+    })
+}
+
+/// Find file with path relative to runtime directory
+///
+/// `rel_path` should be the relative path from within the `runtime/` directory.
+/// The valid runtime directories are searched in priority order and the first
+/// file found to exist is returned, otherwise the path to the final attempt
+/// that failed.
+pub fn runtime_file(rel_path: &Path) -> PathBuf {
+    find_runtime_file(rel_path).unwrap_or_else(|| {
+        RUNTIME_DIRS
+            .last()
+            .map(|dir| dir.join(rel_path))
+            .unwrap_or_default()
+    })
 }
 
 pub fn config_dir() -> PathBuf {

helix-term/src/application.rs

@@ -31,6 +31,7 @@ use crate::{
 use log::{debug, error, warn};
 use std::{
     io::{stdin, stdout},
+    path::Path,
     sync::Arc,
     time::{Duration, Instant},
 };
@@ -113,10 +114,9 @@ impl Application {
 
         use helix_view::editor::Action;
 
-        let theme_loader = std::sync::Arc::new(theme::Loader::new(
+        let mut theme_parent_dirs = vec![helix_loader::config_dir()];
-            &helix_loader::config_dir(),
+        theme_parent_dirs.extend(helix_loader::runtime_dirs().iter().cloned());
-            &helix_loader::runtime_dir(),
+        let theme_loader = std::sync::Arc::new(theme::Loader::new(&theme_parent_dirs));
-        ));
 
         let true_color = config.editor.true_color || crate::true_color();
         let theme = config
@@ -162,7 +162,7 @@ impl Application {
         compositor.push(editor_view);
 
         if args.load_tutor {
-            let path = helix_loader::runtime_dir().join("tutor");
+            let path = helix_loader::runtime_file(Path::new("tutor"));
             editor.open(&path, Action::VerticalSplit)?;
             // Unset path to prevent accidentally saving to the original tutor file.
             doc_mut!(editor).set_path(None)?;

helix-term/src/commands/typed.rs

@@ -1565,7 +1565,7 @@ fn tutor(
         return Ok(());
     }
 
-    let path = helix_loader::runtime_dir().join("tutor");
+    let path = helix_loader::runtime_file(Path::new("tutor"));
     cx.editor.open(&path, Action::Replace)?;
     // Unset path to prevent accidentally saving to the original tutor file.
     doc_mut!(cx.editor).set_path(None)?;

helix-term/src/health.rs

@@ -52,7 +52,7 @@ pub fn general() -> std::io::Result<()> {
     let config_file = helix_loader::config_file();
     let lang_file = helix_loader::lang_config_file();
     let log_file = helix_loader::log_file();
-    let rt_dir = helix_loader::runtime_dir();
+    let rt_dirs = helix_loader::runtime_dirs();
     let clipboard_provider = get_clipboard_provider();
 
     if config_file.exists() {
@@ -66,17 +66,31 @@ pub fn general() -> std::io::Result<()> {
         writeln!(stdout, "Language file: default")?;
     }
     writeln!(stdout, "Log file: {}", log_file.display())?;
-    writeln!(stdout, "Runtime directory: {}", rt_dir.display())?;
+    writeln!(
-
+        stdout,
-    if let Ok(path) = std::fs::read_link(&rt_dir) {
+        "Runtime directories: {}",
-        let msg = format!("Runtime directory is symlinked to {}", path.display());
+        rt_dirs
+            .iter()
+            .map(|d| d.to_string_lossy())
+            .collect::<Vec<_>>()
+            .join(";")
+    )?;
+    for rt_dir in rt_dirs.iter() {
+        if let Ok(path) = std::fs::read_link(rt_dir) {
+            let msg = format!(
+                "Runtime directory {} is symlinked to: {}",
+                rt_dir.display(),
+                path.display()
+            );
             writeln!(stdout, "{}", msg.yellow())?;
         }
         if !rt_dir.exists() {
-        writeln!(stdout, "{}", "Runtime directory does not exist.".red())?;
+            let msg = format!("Runtime directory does not exist: {}", rt_dir.display());
+            writeln!(stdout, "{}", msg.yellow())?;
+        } else if rt_dir.read_dir().ok().map(|it| it.count()) == Some(0) {
+            let msg = format!("Runtime directory is empty: {}", rt_dir.display());
+            writeln!(stdout, "{}", msg.yellow())?;
         }
-    if rt_dir.read_dir().ok().map(|it| it.count()) == Some(0) {
-        writeln!(stdout, "{}", "Runtime directory is empty.".red())?;
     }
     writeln!(stdout, "Clipboard provider: {}", clipboard_provider.name())?;
 

helix-term/src/ui/mod.rs

@@ -280,10 +280,10 @@ pub mod completers {
     }
 
     pub fn theme(_editor: &Editor, input: &str) -> Vec<Completion> {
-        let mut names = theme::Loader::read_names(&helix_loader::runtime_dir().join("themes"));
+        let mut names = theme::Loader::read_names(&helix_loader::config_dir().join("themes"));
-        names.extend(theme::Loader::read_names(
+        for rt_dir in helix_loader::runtime_dirs() {
-            &helix_loader::config_dir().join("themes"),
+            names.extend(theme::Loader::read_names(&rt_dir.join("themes")));
-        ));
+        }
         names.push("default".into());
         names.push("base16_default".into());
         names.sort();

helix-view/src/theme.rs

@@ -1,5 +1,5 @@
 use std::{
-    collections::HashMap,
+    collections::{HashMap, HashSet},
     path::{Path, PathBuf},
     str,
 };
@@ -37,19 +37,21 @@ pub static BASE16_DEFAULT_THEME: Lazy<Theme> = Lazy::new(|| Theme {
 
 #[derive(Clone, Debug)]
 pub struct Loader {
-    user_dir: PathBuf,
+    /// Theme directories to search from highest to lowest priority
-    default_dir: PathBuf,
+    theme_dirs: Vec<PathBuf>,
 }
 impl Loader {
-    /// Creates a new loader that can load themes from two directories.
+    /// Creates a new loader that can load themes from multiple directories.
-    pub fn new<P: AsRef<Path>>(user_dir: P, default_dir: P) -> Self {
+    ///
+    /// The provided directories should be ordered from highest to lowest priority.
+    /// The directories will have their "themes" subdirectory searched.
+    pub fn new(dirs: &[PathBuf]) -> Self {
         Self {
-            user_dir: user_dir.as_ref().join("themes"),
+            theme_dirs: dirs.iter().map(|p| p.join("themes")).collect(),
-            default_dir: default_dir.as_ref().join("themes"),
         }
     }
 
-    /// Loads a theme first looking in the `user_dir` then in `default_dir`
+    /// Loads a theme searching directories in priority order.
     pub fn load(&self, name: &str) -> Result<Theme> {
         if name == "default" {
             return Ok(self.default());
@@ -58,7 +60,8 @@ impl Loader {
             return Ok(self.base16_default());
         }
 
-        let theme = self.load_theme(name, name, false).map(Theme::from)?;
+        let mut visited_paths = HashSet::new();
+        let theme = self.load_theme(name, &mut visited_paths).map(Theme::from)?;
 
         Ok(Theme {
             name: name.into(),
@@ -66,16 +69,18 @@ impl Loader {
         })
     }
 
-    // load the theme and its parent recursively and merge them
+    /// Recursively load a theme, merging with any inherited parent themes.
-    // `base_theme_name` is the theme from the config.toml,
+    ///
-    // used to prevent some circular loading scenarios
+    /// The paths that have been visited in the inheritance hierarchy are tracked
-    fn load_theme(
+    /// to detect and avoid cycling.
-        &self,
+    ///
-        name: &str,
+    /// It is possible for one file to inherit from another file with the same name
-        base_theme_name: &str,
+    /// so long as the second file is in a themes directory with lower priority.
-        only_default_dir: bool,
+    /// However, it is not recommended that users do this as it will make tracing
-    ) -> Result<Value> {
+    /// errors more difficult.
-        let path = self.path(name, only_default_dir);
+    fn load_theme(&self, name: &str, visited_paths: &mut HashSet<PathBuf>) -> Result<Value> {
+        let path = self.path(name, visited_paths)?;
+
         let theme_toml = self.load_toml(path)?;
 
         let inherits = theme_toml.get("inherits");
@@ -92,11 +97,7 @@ impl Loader {
                 // load default themes's toml from const.
                 "default" => DEFAULT_THEME_DATA.clone(),
                 "base16_default" => BASE16_DEFAULT_THEME_DATA.clone(),
-                _ => self.load_theme(
+                _ => self.load_theme(parent_theme_name, visited_paths)?,
-                    parent_theme_name,
-                    base_theme_name,
-                    base_theme_name == parent_theme_name,
-                )?,
             };
 
             self.merge_themes(parent_theme_toml, theme_toml)
@@ -148,7 +149,7 @@ impl Loader {
         merge_toml_values(theme, palette.into(), 1)
     }
 
-    // Loads the theme data as `toml::Value` first from the user_dir then in default_dir
+    // Loads the theme data as `toml::Value`
     fn load_toml(&self, path: PathBuf) -> Result<Value> {
         let data = std::fs::read_to_string(path)?;
         let value = toml::from_str(&data)?;
@@ -156,25 +157,35 @@ impl Loader {
         Ok(value)
     }
 
-    // Returns the path to the theme with the name
+    /// Returns the path to the theme with the given name
-    // With `only_default_dir` as false the path will first search for the user path
+    ///
-    // disabled it ignores the user path and returns only the default path
+    /// Ignores paths already visited and follows directory priority order.
-    fn path(&self, name: &str, only_default_dir: bool) -> PathBuf {
+    fn path(&self, name: &str, visited_paths: &mut HashSet<PathBuf>) -> Result<PathBuf> {
         let filename = format!("{}.toml", name);
 
-        let user_path = self.user_dir.join(&filename);
+        let mut cycle_found = false; // track if there was a path, but it was in a cycle
-        if !only_default_dir && user_path.exists() {
+        self.theme_dirs
-            user_path
+            .iter()
+            .find_map(|dir| {
+                let path = dir.join(&filename);
+                if !path.exists() {
+                    None
+                } else if visited_paths.contains(&path) {
+                    // Avoiding cycle, continuing to look in lower priority directories
+                    cycle_found = true;
+                    None
                 } else {
-            self.default_dir.join(filename)
+                    visited_paths.insert(path.clone());
+                    Some(path)
                 }
+            })
+            .ok_or_else(|| {
+                if cycle_found {
+                    anyhow!("Theme: cycle found in inheriting: {}", name)
+                } else {
+                    anyhow!("Theme: file not found for: {}", name)
                 }
-
+            })
-    /// Lists all theme names available in default and user directory
-    pub fn names(&self) -> Vec<String> {
-        let mut names = Self::read_names(&self.user_dir);
-        names.extend(Self::read_names(&self.default_dir));
-        names
     }
 
     pub fn default_theme(&self, true_color: bool) -> Theme {