Indentation rework (#1562)

* WIP: Rework indentation system

* Add ComplexNode for context-aware indentation (including a proof of concept for assignment statements in rust)

* Add switch statements to Go indents.toml (fixes the second half of issue #1523)
Remove commented-out code

* Migrate all existing indentation queries.
Add more options to ComplexNode and use them to improve C/C++ indentation.

* Add comments & replace Option<Vec<_>> with Vec<_>

* Add more detailed documentation for tree-sitter indentation

* Improve code style in indent.rs

* Use tree-sitter queries for indentation instead of TOML config.
Migrate existing indent queries.

* Add documentation for the new indent queries.
Change xtask docgen to look for indents.scm instead of indents.toml

* Improve code style in indent.rs.
Fix an issue with the rust indent query.

* Move indentation test sources to separate files.
Add `#not-kind-eq?`, `#same-line?` and `#not-same-line` custom predicates.
Improve the rust and c indent queries.

* Fix indent test.
Improve rust indent queries.

* Move indentation tests to integration test folder.

* Improve code style in indent.rs.
Reuse tree-sitter cursors for indentation queries.

* Migrate HCL indent query

* Replace custom loading in indent tests with a designated languages.toml

* Update indent query file name for --health command.

* Fix single-space formatting in indent queries.

* Add explanation for unwrapping.

Co-authored-by: Triton171 <triton0171@gmail.com>
This commit is contained in:
Triton171 2022-03-30 17:08:07 +02:00 committed by GitHub
parent c18de0e8f0
commit 58758fee61
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
66 changed files with 1160 additions and 724 deletions

View file

@ -16,3 +16,4 @@
- [Guides](./guides/README.md)
- [Adding Languages](./guides/adding_languages.md)
- [Adding Textobject Queries](./guides/textobject.md)
- [Adding Indent Queries](./guides/indent.md)

79
book/src/guides/indent.md Normal file
View file

@ -0,0 +1,79 @@
# Adding Indent Queries
Helix uses tree-sitter to correctly indent new lines. This requires
a tree-sitter grammar and an `indent.scm` query file placed in
`runtime/queries/{language}/indents.scm`. The indentation for a line
is calculated by traversing the syntax tree from the lowest node at the
beginning of the new line. Each of these nodes contributes to the total
indent when it is captured by the query (in what way depends on the name
of the capture).
Note that it matters where these added indents begin. For example,
multiple indent level increases that start on the same line only increase
the total indent level by 1.
## Scopes
Added indents don't always apply to the whole node. For example, in most
cases when a node should be indented, we actually only want everything
except for its first line to be indented. For this, there are several
scopes (more scopes may be added in the future if required):
- `all`:
This scope applies to the whole captured node. This is only different from
`tail` when the captured node is the first node on its line.
- `tail`:
This scope applies to everything except for the first line of the
captured node.
Every capture type has a default scope which should do the right thing
in most situations. When a different scope is required, this can be
changed by using a `#set!` declaration anywhere in the pattern:
```scm
(assignment_expression
right: (_) @indent
(#set! "scope" "all"))
```
## Capture Types
- `@indent` (default scope `tail`):
Increase the indent level by 1. Multiple occurences in the same line
don't stack. If there is at least one `@indent` and one `@outdent`
capture on the same line, the indent level isn't changed at all.
- `@outdent` (default scope `all`):
Decrease the indent level by 1. The same rules as for `@indent` apply.
## Predicates
In some cases, an S-expression cannot express exactly what pattern should be matched.
For that, tree-sitter allows for predicates to appear anywhere within a pattern,
similar to how `#set!` declarations work:
```scm
(some_kind
(child_kind) @indent
(#predicate? arg1 arg2 ...)
)
```
The number of arguments depends on the predicate that's used.
Each argument is either a capture (`@name`) or a string (`"some string"`).
The following predicates are supported by tree-sitter:
- `#eq?`/`#not-eq?`:
The first argument (a capture) must/must not be equal to the second argument
(a capture or a string).
- `#match?`/`#not-match?`:
The first argument (a capture) must/must not match the regex given in the
second argument (a string).
Additionally, we support some custom predicates for indent queries:
- `#not-kind-eq?`:
The kind of the first argument (a capture) must not be equal to the second
argument (a string).
- `#same-line?`/`#not-same-line?`:
The captures given by the 2 arguments must/must not start on the same line.

View file

@ -1,6 +1,10 @@
use std::collections::HashMap;
use tree_sitter::{Query, QueryCursor, QueryPredicateArg};
use crate::{
chars::{char_is_line_ending, char_is_whitespace},
syntax::{IndentQuery, LanguageConfiguration, Syntax},
syntax::{LanguageConfiguration, RopeProvider, Syntax},
tree_sitter::Node,
Rope, RopeSlice,
};
@ -186,103 +190,405 @@ pub fn indent_level_for_line(line: RopeSlice, tab_width: usize) -> usize {
len / tab_width
}
/// Find the highest syntax node at position.
/// This is to identify the column where this node (e.g., an HTML closing tag) ends.
fn get_highest_syntax_node_at_bytepos(syntax: &Syntax, pos: usize) -> Option<Node> {
let tree = syntax.tree();
// named_descendant
let mut node = tree.root_node().descendant_for_byte_range(pos, pos)?;
while let Some(parent) = node.parent() {
if parent.start_byte() == node.start_byte() {
node = parent
} else {
break;
}
}
Some(node)
}
/// Calculate the indentation at a given treesitter node.
/// If newline is false, then any "indent" nodes on the line are ignored ("outdent" still applies).
/// This is because the indentation is only increased starting at the second line of the node.
fn calculate_indentation(
query: &IndentQuery,
node: Option<Node>,
line: usize,
newline: bool,
) -> usize {
let mut increment: isize = 0;
let mut node = match node {
Some(node) => node,
None => return 0,
};
let mut current_line = line;
let mut consider_indent = newline;
let mut increment_from_line: isize = 0;
/// Computes for node and all ancestors whether they are the first node on their line.
/// The first entry in the return value represents the root node, the last one the node itself
fn get_first_in_line(mut node: Node, byte_pos: usize, new_line: bool) -> Vec<bool> {
let mut first_in_line = Vec::new();
loop {
let node_kind = node.kind();
let start = node.start_position().row;
if current_line != start {
// Indent/dedent by at most one per line:
// .map(|a| { <-- ({ is two scopes
// let len = 1; <-- indents one level
// }) <-- }) is two scopes
if consider_indent || increment_from_line < 0 {
increment += increment_from_line.signum();
if let Some(prev) = node.prev_sibling() {
// If we insert a new line, the first node at/after the cursor is considered to be the first in its line
let first = prev.end_position().row != node.start_position().row
|| (new_line && node.start_byte() >= byte_pos && prev.start_byte() < byte_pos);
first_in_line.push(Some(first));
} else {
// Nodes that have no previous siblings are first in their line if and only if their parent is
// (which we don't know yet)
first_in_line.push(None);
}
increment_from_line = 0;
current_line = start;
consider_indent = true;
}
if query.outdent.contains(node_kind) {
increment_from_line -= 1;
}
if query.indent.contains(node_kind) {
increment_from_line += 1;
}
if let Some(parent) = node.parent() {
node = parent;
} else {
break;
}
}
if consider_indent || increment_from_line < 0 {
increment += increment_from_line.signum();
let mut result = Vec::with_capacity(first_in_line.len());
let mut parent_is_first = true; // The root node is by definition the first node in its line
for first in first_in_line.into_iter().rev() {
if let Some(first) = first {
result.push(first);
parent_is_first = first;
} else {
result.push(parent_is_first);
}
increment.max(0) as usize
}
result
}
// TODO: two usecases: if we are triggering this for a new, blank line:
// - it should return 0 when mass indenting stuff
// - it should look up the wrapper node and count it too when we press o/O
pub fn suggested_indent_for_pos(
/// The total indent for some line of code.
/// This is usually constructed in one of 2 ways:
/// - Successively add indent captures to get the (added) indent from a single line
/// - Successively add the indent results for each line
#[derive(Default)]
struct Indentation {
/// The total indent (the number of indent levels) is defined as max(0, indent-outdent).
/// The string that this results in depends on the indent style (spaces or tabs, etc.)
indent: usize,
outdent: usize,
}
impl Indentation {
/// Add some other [IndentResult] to this.
/// The added indent should be the total added indent from one line
fn add_line(&mut self, added: &Indentation) {
if added.indent > 0 && added.outdent == 0 {
self.indent += 1;
} else if added.outdent > 0 && added.indent == 0 {
self.outdent += 1;
}
}
/// Add an indent capture to this indent.
/// All the captures that are added in this way should be on the same line.
fn add_capture(&mut self, added: IndentCaptureType) {
match added {
IndentCaptureType::Indent => {
self.indent = 1;
}
IndentCaptureType::Outdent => {
self.outdent = 1;
}
}
}
fn as_string(&self, indent_style: &IndentStyle) -> String {
let indent_level = if self.indent >= self.outdent {
self.indent - self.outdent
} else {
log::warn!("Encountered more outdent than indent nodes while calculating indentation: {} outdent, {} indent", self.outdent, self.indent);
0
};
indent_style.as_str().repeat(indent_level)
}
}
/// An indent definition which corresponds to a capture from the indent query
struct IndentCapture {
capture_type: IndentCaptureType,
scope: IndentScope,
}
#[derive(Clone, Copy)]
enum IndentCaptureType {
Indent,
Outdent,
}
impl IndentCaptureType {
fn default_scope(&self) -> IndentScope {
match self {
IndentCaptureType::Indent => IndentScope::Tail,
IndentCaptureType::Outdent => IndentScope::All,
}
}
}
/// This defines which part of a node an [IndentCapture] applies to.
/// Each [IndentCaptureType] has a default scope, but the scope can be changed
/// with `#set!` property declarations.
#[derive(Clone, Copy)]
enum IndentScope {
/// The indent applies to the whole node
All,
/// The indent applies to everything except for the first line of the node
Tail,
}
/// Execute the indent query.
/// Returns for each node (identified by its id) a list of indent captures for that node.
fn query_indents(
query: &Query,
syntax: &Syntax,
cursor: &mut QueryCursor,
text: RopeSlice,
range: std::ops::Range<usize>,
// Position of the (optional) newly inserted line break.
// Given as (line, byte_pos)
new_line_break: Option<(usize, usize)>,
) -> HashMap<usize, Vec<IndentCapture>> {
let mut indent_captures: HashMap<usize, Vec<IndentCapture>> = HashMap::new();
cursor.set_byte_range(range);
// Iterate over all captures from the query
for m in cursor.matches(query, syntax.tree().root_node(), RopeProvider(text)) {
// Skip matches where not all custom predicates are fulfilled
if !query.general_predicates(m.pattern_index).iter().all(|pred| {
match pred.operator.as_ref() {
"not-kind-eq?" => match (pred.args.get(0), pred.args.get(1)) {
(
Some(QueryPredicateArg::Capture(capture_idx)),
Some(QueryPredicateArg::String(kind)),
) => {
let node = m.nodes_for_capture_index(*capture_idx).next();
match node {
Some(node) => node.kind()!=kind.as_ref(),
_ => true,
}
}
_ => {
panic!("Invalid indent query: Arguments to \"not-kind-eq?\" must be a capture and a string");
}
},
"same-line?" | "not-same-line?" => {
match (pred.args.get(0), pred.args.get(1)) {
(
Some(QueryPredicateArg::Capture(capt1)),
Some(QueryPredicateArg::Capture(capt2))
) => {
let get_line_num = |node: Node| {
let mut node_line = node.start_position().row;
// Adjust for the new line that will be inserted
if let Some((line, byte)) = new_line_break {
if node_line==line && node.start_byte()>=byte {
node_line += 1;
}
}
node_line
};
let n1 = m.nodes_for_capture_index(*capt1).next();
let n2 = m.nodes_for_capture_index(*capt2).next();
match (n1, n2) {
(Some(n1), Some(n2)) => {
let same_line = get_line_num(n1)==get_line_num(n2);
same_line==(pred.operator.as_ref()=="same-line?")
}
_ => true,
}
}
_ => {
panic!("Invalid indent query: Arguments to \"{}\" must be 2 captures", pred.operator);
}
}
}
_ => {
panic!(
"Invalid indent query: Unknown predicate (\"{}\")",
pred.operator
);
}
}
}) {
continue;
}
for capture in m.captures {
let capture_type = query.capture_names()[capture.index as usize].as_str();
let capture_type = match capture_type {
"indent" => IndentCaptureType::Indent,
"outdent" => IndentCaptureType::Outdent,
_ => {
// Ignore any unknown captures (these may be needed for predicates such as #match?)
continue;
}
};
let scope = capture_type.default_scope();
let mut indent_capture = IndentCapture {
capture_type,
scope,
};
// Apply additional settings for this capture
for property in query.property_settings(m.pattern_index) {
match property.key.as_ref() {
"scope" => {
indent_capture.scope = match property.value.as_deref() {
Some("all") => IndentScope::All,
Some("tail") => IndentScope::Tail,
Some(s) => {
panic!("Invalid indent query: Unknown value for \"scope\" property (\"{}\")", s);
}
None => {
panic!(
"Invalid indent query: Missing value for \"scope\" property"
);
}
}
}
_ => {
panic!(
"Invalid indent query: Unknown property \"{}\"",
property.key
);
}
}
}
indent_captures
.entry(capture.node.id())
// Most entries only need to contain a single IndentCapture
.or_insert_with(|| Vec::with_capacity(1))
.push(indent_capture);
}
}
indent_captures
}
/// Use the syntax tree to determine the indentation for a given position.
/// This can be used in 2 ways:
///
/// - To get the correct indentation for an existing line (new_line=false), not necessarily equal to the current indentation.
/// - In this case, pos should be inside the first tree-sitter node on that line.
/// In most cases, this can just be the first non-whitespace on that line.
/// - To get the indentation for a new line (new_line=true). This behaves like the first usecase if the part of the current line
/// after pos were moved to a new line.
///
/// The indentation is determined by traversing all the tree-sitter nodes containing the position.
/// Each of these nodes produces some [AddedIndent] for:
///
/// - The line of the (beginning of the) node. This is defined by the scope `all` if this is the first node on its line.
/// - The line after the node. This is defined by:
/// - The scope `tail`.
/// - The scope `all` if this node is not the first node on its line.
/// Intuitively, `all` applies to everything contained in this node while `tail` applies to everything except for the first line of the node.
/// The indents from different nodes for the same line are then combined.
/// The [IndentResult] is simply the sum of the [AddedIndent] for all lines.
///
/// Specifying which line exactly an [AddedIndent] applies to is important because indents on the same line combine differently than indents on different lines:
/// ```ignore
/// some_function(|| {
/// // Both the function parameters as well as the contained block should be indented.
/// // Because they are on the same line, this only yields one indent level
/// });
/// ```
///
/// ```ignore
/// some_function(
/// parm1,
/// || {
/// // Here we get 2 indent levels because the 'parameters' and the 'block' node begin on different lines
/// },
/// );
/// ```
pub fn treesitter_indent_for_pos(
query: &Query,
syntax: &Syntax,
indent_style: &IndentStyle,
text: RopeSlice,
line: usize,
pos: usize,
new_line: bool,
) -> Option<String> {
let byte_pos = text.char_to_byte(pos);
let mut node = syntax
.tree()
.root_node()
.descendant_for_byte_range(byte_pos, byte_pos)?;
let mut first_in_line = get_first_in_line(node, byte_pos, new_line);
let new_line_break = if new_line {
Some((line, byte_pos))
} else {
None
};
let query_result = crate::syntax::PARSER.with(|ts_parser| {
let mut ts_parser = ts_parser.borrow_mut();
let mut cursor = ts_parser.cursors.pop().unwrap_or_else(QueryCursor::new);
let query_result = query_indents(
query,
syntax,
&mut cursor,
text,
byte_pos..byte_pos + 1,
new_line_break,
);
ts_parser.cursors.push(cursor);
query_result
});
let mut result = Indentation::default();
// We always keep track of all the indent changes on one line, in order to only indent once
// even if there are multiple "indent" nodes on the same line
let mut indent_for_line = Indentation::default();
let mut indent_for_line_below = Indentation::default();
loop {
// This can safely be unwrapped because `first_in_line` contains
// one entry for each ancestor of the node (which is what we iterate over)
let is_first = *first_in_line.last().unwrap();
// Apply all indent definitions for this node
if let Some(definitions) = query_result.get(&node.id()) {
for definition in definitions {
match definition.scope {
IndentScope::All => {
if is_first {
indent_for_line.add_capture(definition.capture_type);
} else {
indent_for_line_below.add_capture(definition.capture_type);
}
}
IndentScope::Tail => {
indent_for_line_below.add_capture(definition.capture_type);
}
}
}
}
if let Some(parent) = node.parent() {
let mut node_line = node.start_position().row;
let mut parent_line = parent.start_position().row;
if node_line == line && new_line {
// Also consider the line that will be inserted
if node.start_byte() >= byte_pos {
node_line += 1;
}
if parent.start_byte() >= byte_pos {
parent_line += 1;
}
};
if node_line != parent_line {
if node_line < line + (new_line as usize) {
// Don't add indent for the line below the line of the query
result.add_line(&indent_for_line_below);
}
if node_line == parent_line + 1 {
indent_for_line_below = indent_for_line;
} else {
result.add_line(&indent_for_line);
indent_for_line_below = Indentation::default();
}
indent_for_line = Indentation::default();
}
node = parent;
first_in_line.pop();
} else {
result.add_line(&indent_for_line_below);
result.add_line(&indent_for_line);
break;
}
}
Some(result.as_string(indent_style))
}
/// Returns the indentation for a new line.
/// This is done either using treesitter, or if that's not available by copying the indentation from the current line
#[allow(clippy::too_many_arguments)]
pub fn indent_for_newline(
language_config: Option<&LanguageConfiguration>,
syntax: Option<&Syntax>,
indent_style: &IndentStyle,
tab_width: usize,
text: RopeSlice,
pos: usize,
line: usize,
new_line: bool,
) -> Option<usize> {
line_before: usize,
line_before_end_pos: usize,
current_line: usize,
) -> String {
if let (Some(query), Some(syntax)) = (
language_config.and_then(|config| config.indent_query()),
syntax,
) {
let byte_start = text.char_to_byte(pos);
let node = get_highest_syntax_node_at_bytepos(syntax, byte_start);
// TODO: special case for comments
// TODO: if preserve_leading_whitespace
Some(calculate_indentation(query, node, line, new_line))
} else {
None
if let Some(indent) = treesitter_indent_for_pos(
query,
syntax,
indent_style,
text,
line_before,
line_before_end_pos,
true,
) {
return indent;
};
}
let indent_level = indent_level_for_line(text.line(current_line), tab_width);
indent_style.as_str().repeat(indent_level)
}
pub fn get_scopes(syntax: Option<&Syntax>, text: RopeSlice, pos: usize) -> Vec<&'static str> {
@ -326,156 +632,4 @@ mod test {
let line = Rope::from("\t \tfn new"); // 1 tab, 4 spaces, tab
assert_eq!(indent_level_for_line(line.slice(..), tab_width), 3);
}
#[test]
fn test_suggested_indent_for_line() {
let doc = Rope::from(
"
use std::{
io::{self, stdout, Stdout, Write},
path::PathBuf,
sync::Arc,
time::Duration,
}
mod test {
fn hello_world() {
1 + 1;
let does_indentation_work = 1;
let test_function = function_with_param(this_param,
that_param
);
let test_function = function_with_param(
this_param,
that_param
);
let test_function = function_with_proper_indent(param1,
param2,
);
let selection = Selection::new(
changes
.clone()
.map(|(start, end, text): (usize, usize, Option<Tendril>)| {
let len = text.map(|text| text.len()).unwrap() - 1; // minus newline
let pos = start + len;
Range::new(pos, pos)
})
.collect(),
0,
);
return;
}
}
impl<A, D> MyTrait<A, D> for YourType
where
A: TraitB + TraitC,
D: TraitE + TraitF,
{
}
#[test]
//
match test {
Some(a) => 1,
None => {
unimplemented!()
}
}
std::panic::set_hook(Box::new(move |info| {
hook(info);
}));
{ { {
1
}}}
pub fn change<I>(document: &Document, changes: I) -> Self
where
I: IntoIterator<Item = Change> + ExactSizeIterator,
{
[
1,
2,
3,
];
(
1,
2
);
true
}
",
);
let doc = doc;
use crate::diagnostic::Severity;
use crate::syntax::{
Configuration, IndentationConfiguration, LanguageConfiguration, Loader,
};
use once_cell::sync::OnceCell;
let loader = Loader::new(Configuration {
language: vec![LanguageConfiguration {
scope: "source.rust".to_string(),
file_types: vec!["rs".to_string()],
shebangs: vec![],
language_id: "Rust".to_string(),
highlight_config: OnceCell::new(),
config: None,
//
injection_regex: None,
roots: vec![],
comment_token: None,
auto_format: false,
diagnostic_severity: Severity::Warning,
grammar: None,
language_server: None,
indent: Some(IndentationConfiguration {
tab_width: 4,
unit: String::from(" "),
}),
indent_query: OnceCell::new(),
textobject_query: OnceCell::new(),
debugger: None,
auto_pairs: None,
}],
});
// set runtime path so we can find the queries
let mut runtime = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
runtime.push("../runtime");
std::env::set_var("HELIX_RUNTIME", runtime.to_str().unwrap());
let language_config = loader.language_config_for_scope("source.rust").unwrap();
let highlight_config = language_config.highlight_config(&[]).unwrap();
let syntax = Syntax::new(&doc, highlight_config, std::sync::Arc::new(loader));
let text = doc.slice(..);
let tab_width = 4;
for i in 0..doc.len_lines() {
let line = text.line(i);
if let Some(pos) = crate::find_first_non_whitespace_char(line) {
let indent = indent_level_for_line(line, tab_width);
assert_eq!(
suggested_indent_for_pos(
Some(&language_config),
Some(&syntax),
text,
text.line_to_char(i) + pos,
i,
false
),
Some(indent),
"line {}: \"{}\"",
i,
line
);
}
}
}
}

View file

@ -92,7 +92,7 @@ pub struct LanguageConfiguration {
pub indent: Option<IndentationConfiguration>,
#[serde(skip)]
pub(crate) indent_query: OnceCell<Option<IndentQuery>>,
pub(crate) indent_query: OnceCell<Option<Query>>,
#[serde(skip)]
pub(crate) textobject_query: OnceCell<Option<TextObjectQuery>>,
#[serde(skip_serializing_if = "Option::is_none")]
@ -220,17 +220,6 @@ impl FromStr for AutoPairConfig {
}
}
#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub struct IndentQuery {
#[serde(default)]
#[serde(skip_serializing_if = "HashSet::is_empty")]
pub indent: HashSet<String>,
#[serde(default)]
#[serde(skip_serializing_if = "HashSet::is_empty")]
pub outdent: HashSet<String>,
}
#[derive(Debug)]
pub struct TextObjectQuery {
pub query: Query,
@ -404,13 +393,13 @@ impl LanguageConfiguration {
self.highlight_config.get().is_some()
}
pub fn indent_query(&self) -> Option<&IndentQuery> {
pub fn indent_query(&self) -> Option<&Query> {
self.indent_query
.get_or_init(|| {
let language = self.language_id.to_ascii_lowercase();
let toml = load_runtime_file(&language, "indents.toml").ok()?;
toml::from_slice(toml.as_bytes()).ok()
let lang_name = self.language_id.to_ascii_lowercase();
let query_text = read_query(&lang_name, "indents.scm");
let lang = self.highlight_config.get()?.as_ref()?.language;
Query::new(lang, &query_text).ok()
})
.as_ref()
}
@ -557,7 +546,7 @@ impl Loader {
pub struct TsParser {
parser: tree_sitter::Parser,
cursors: Vec<QueryCursor>,
pub cursors: Vec<QueryCursor>,
}
// could also just use a pool, or a single instance?
@ -1180,7 +1169,7 @@ struct HighlightIter<'a> {
}
// Adapter to convert rope chunks to bytes
struct ChunksBytes<'a> {
pub struct ChunksBytes<'a> {
chunks: ropey::iter::Chunks<'a>,
}
impl<'a> Iterator for ChunksBytes<'a> {
@ -1190,7 +1179,7 @@ impl<'a> Iterator for ChunksBytes<'a> {
}
}
struct RopeProvider<'a>(RopeSlice<'a>);
pub struct RopeProvider<'a>(pub RopeSlice<'a>);
impl<'a> TextProvider<'a> for RopeProvider<'a> {
type I = ChunksBytes<'a>;
@ -2126,7 +2115,7 @@ mod test {
#[test]
fn test_load_runtime_file() {
// Test to make sure we can load some data from the runtime directory.
let contents = load_runtime_file("rust", "indents.toml").unwrap();
let contents = load_runtime_file("rust", "indents.scm").unwrap();
assert!(!contents.is_empty());
let results = load_runtime_file("rust", "does-not-exist");

View file

@ -0,0 +1 @@
../../../src/indent.rs

View file

@ -0,0 +1,13 @@
# This languages.toml should contain definitions for all languages for which we have indent tests
[[language]]
name = "rust"
scope = "source.rust"
injection-regex = "rust"
file-types = ["rs"]
comment-token = "//"
roots = ["Cargo.toml", "Cargo.lock"]
indent = { tab-width = 4, unit = " " }
[[grammar]]
name = "rust"
source = { git = "https://github.com/tree-sitter/tree-sitter-rust", rev = "a360da0a29a19c281d08295a35ecd0544d2da211" }

View file

@ -0,0 +1,105 @@
use std::{
io::{self, stdout, Stdout, Write},
path::PathBuf,
sync::Arc,
time::Duration,
};
mod test {
fn hello_world() {
1 + 1;
let does_indentation_work = 1;
let mut really_long_variable_name_using_up_the_line =
really_long_fn_that_should_definitely_go_on_the_next_line();
really_long_variable_name_using_up_the_line =
really_long_fn_that_should_definitely_go_on_the_next_line();
really_long_variable_name_using_up_the_line |=
really_long_fn_that_should_definitely_go_on_the_next_line();
let (
a_long_variable_name_in_this_tuple,
b_long_variable_name_in_this_tuple,
c_long_variable_name_in_this_tuple,
d_long_variable_name_in_this_tuple,
e_long_variable_name_in_this_tuple,
): (usize, usize, usize, usize, usize) =
if really_long_fn_that_should_definitely_go_on_the_next_line() {
(
03294239434,
1213412342314,
21231234134,
834534234549898789,
9879234234543853457,
)
} else {
(0, 1, 2, 3, 4)
};
let test_function = function_with_param(this_param,
that_param
);
let test_function = function_with_param(
this_param,
that_param
);
let test_function = function_with_proper_indent(param1,
param2,
);
let selection = Selection::new(
changes
.clone()
.map(|(start, end, text): (usize, usize, Option<Tendril>)| {
let len = text.map(|text| text.len()).unwrap() - 1; // minus newline
let pos = start + len;
Range::new(pos, pos)
})
.collect(),
0,
);
return;
}
}
impl<A, D> MyTrait<A, D> for YourType
where
A: TraitB + TraitC,
D: TraitE + TraitF,
{
}
#[test]
//
match test {
Some(a) => 1,
None => {
unimplemented!()
}
}
std::panic::set_hook(Box::new(move |info| {
hook(info);
}));
{ { {
1
}}}
pub fn change<I>(document: &Document, changes: I) -> Self
where
I: IntoIterator<Item = Change> + ExactSizeIterator,
{
[
1,
2,
3,
];
(
1,
2
);
true
}

View file

@ -0,0 +1,68 @@
use helix_core::{
indent::{treesitter_indent_for_pos, IndentStyle},
syntax::Loader,
Syntax,
};
use std::path::PathBuf;
#[test]
fn test_treesitter_indent_rust() {
test_treesitter_indent("rust.rs", "source.rust");
}
#[test]
fn test_treesitter_indent_rust_2() {
test_treesitter_indent("indent.rs", "source.rust");
// TODO Use commands.rs as indentation test.
// Currently this fails because we can't align the parameters of a closure yet
// test_treesitter_indent("commands.rs", "source.rust");
}
fn test_treesitter_indent(file_name: &str, lang_scope: &str) {
let mut test_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
test_dir.push("tests/data/indent");
let mut test_file = test_dir.clone();
test_file.push(file_name);
let test_file = std::fs::File::open(test_file).unwrap();
let doc = ropey::Rope::from_reader(test_file).unwrap();
let mut config_file = test_dir;
config_file.push("languages.toml");
let config = std::fs::read(config_file).unwrap();
let config = toml::from_slice(&config).unwrap();
let loader = Loader::new(config);
// set runtime path so we can find the queries
let mut runtime = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
runtime.push("../runtime");
std::env::set_var("HELIX_RUNTIME", runtime.to_str().unwrap());
let language_config = loader.language_config_for_scope(lang_scope).unwrap();
let highlight_config = language_config.highlight_config(&[]).unwrap();
let syntax = Syntax::new(&doc, highlight_config, std::sync::Arc::new(loader));
let indent_query = language_config.indent_query().unwrap();
let text = doc.slice(..);
for i in 0..doc.len_lines() {
let line = text.line(i);
if let Some(pos) = helix_core::find_first_non_whitespace_char(line) {
let suggested_indent = treesitter_indent_for_pos(
indent_query,
&syntax,
&IndentStyle::Spaces(4),
text,
i,
text.line_to_char(i) + pos,
false,
)
.unwrap();
assert!(
line.get_slice(..pos).map_or(false, |s| s == suggested_indent),
"Wrong indentation on line {}:\n\"{}\" (original line)\n\"{}\" (suggested indentation)\n",
i+1,
line.slice(..line.len_chars()-1),
suggested_indent,
);
}
}
}

View file

@ -2240,17 +2240,16 @@ fn open(cx: &mut Context, open: Open) {
)
};
// TODO: share logic with insert_newline for indentation
let indent_level = indent::suggested_indent_for_pos(
let indent = indent::indent_for_newline(
doc.language_config(),
doc.syntax(),
&doc.indent_style,
doc.tab_width(),
text,
line_end_index,
new_line.saturating_sub(1),
true,
)
.unwrap_or_else(|| indent::indent_level_for_line(text.line(cursor_line), doc.tab_width()));
let indent = doc.indent_unit().repeat(indent_level);
line_end_index,
cursor_line,
);
let indent_len = indent.len();
let mut text = String::with_capacity(1 + indent_len);
text.push_str(doc.line_ending.as_str());
@ -2703,19 +2702,16 @@ pub mod insert {
let curr = contents.get_char(pos).unwrap_or(' ');
let current_line = text.char_to_line(pos);
let indent_level = indent::suggested_indent_for_pos(
let indent = indent::indent_for_newline(
doc.language_config(),
doc.syntax(),
&doc.indent_style,
doc.tab_width(),
text,
current_line,
pos,
current_line,
true,
)
.unwrap_or_else(|| {
indent::indent_level_for_line(text.line(current_line), doc.tab_width())
});
let indent = doc.indent_unit().repeat(indent_level);
);
let mut text = String::new();
// If we are between pairs (such as brackets), we want to
// insert an additional line which is indented one level
@ -2727,7 +2723,7 @@ pub mod insert {
.is_some();
let new_head_pos = if on_auto_pair {
let inner_indent = doc.indent_unit().repeat(indent_level + 1);
let inner_indent = indent.clone() + doc.indent_style.as_str();
text.reserve_exact(2 + indent.len() + inner_indent.len());
text.push_str(doc.line_ending.as_str());
text.push_str(&inner_indent);

View file

@ -19,7 +19,7 @@ impl TsFeature {
match *self {
Self::Highlight => "highlights.scm",
Self::TextObject => "textobjects.scm",
Self::AutoIndent => "indents.toml",
Self::AutoIndent => "indents.scm",
}
}

View file

@ -0,0 +1,33 @@
[
(compound_statement)
(field_declaration_list)
(enumerator_list)
(parameter_list)
(init_declarator)
(case_statement)
(expression_statement)
] @indent
[
"case"
"}"
"]"
] @outdent
(if_statement
consequence: (_) @indent
(#not-kind-eq? @indent "compound_statement")
(#set! "scope" "all"))
(while_statement
body: (_) @indent
(#not-kind-eq? @indent "compound_statement")
(#set! "scope" "all"))
(do_statement
body: (_) @indent
(#not-kind-eq? @indent "compound_statement")
(#set! "scope" "all"))
(for_statement
")"
(_) @indent
(#not-kind-eq? @indent "compound_statement")
(#set! "scope" "all"))

View file

@ -1,16 +0,0 @@
indent = [
"compound_statement",
"field_declaration_list",
"enumerator_list",
"parameter_list",
"init_declarator",
"case_statement",
"condition_clause",
"expression_statement",
]
outdent = [
"case",
"}",
"]",
]

View file

@ -0,0 +1,10 @@
[
(if_condition)
(foreach_loop)
(while_loop)
(function_def)
(macro_def)
(normal_command)
] @indent
")" @outdent

View file

@ -1,12 +0,0 @@
indent = [
"if_condition",
"foreach_loop",
"while_loop",
"function_def",
"macro_def",
"normal_command",
]
outdent = [
")"
]

View file

@ -0,0 +1,3 @@
; inherits: c
(access_specifier) @outdent

View file

@ -1,17 +0,0 @@
indent = [
"compound_statement",
"field_declaration_list",
"enumerator_list",
"parameter_list",
"init_declarator",
"case_statement",
"condition_clause",
"expression_statement",
]
outdent = [
"case",
"access_specifier",
"}",
"]",
]

View file

@ -0,0 +1,20 @@
[
(class_body)
(function_body)
(function_expression_body)
(declaration)
(initializers)
(switch_block)
(if_statement)
(formal_parameter_list)
(formal_parameter)
(list_literal)
(return_statement)
(arguments)
] @indent
[
"}"
"]"
")"
] @outdent

View file

@ -1,20 +0,0 @@
indent = [
"class_body",
"function_body",
"function_expression_body",
"declaration",
"initializers",
"switch_block",
"if_statement",
"formal_parameter_list",
"formal_parameter",
"list_literal",
"return_statement",
"arguments"
]
outdent = [
"}",
"]",
")"
]

View file

@ -0,0 +1,12 @@
[
(function_definition)
(while_statement)
(for_statement)
(if_statement)
(begin_statement)
(switch_statement)
] @indent
[
"end"
] @outdent

View file

@ -1,12 +0,0 @@
indent = [
"function_definition",
"while_statement",
"for_statement",
"if_statement",
"begin_statement",
"switch_statement",
]
outdent = [
"end"
]

View file

@ -0,0 +1,19 @@
[
(init_declarator)
(compound_statement)
(preproc_arg)
(field_declaration_list)
(case_statement)
(conditional_expression)
(enumerator_list)
(struct_specifier)
(compound_literal_expression)
] @indent
[
"#define"
"#ifdef"
"#endif"
"{"
"}"
] @outdent

View file

@ -1,19 +0,0 @@
indent = [
"init_declarator",
"compound_statement",
"preproc_arg",
"field_declaration_list",
"case_statement",
"conditional_expression",
"enumerator_list",
"struct_specifier",
"compound_literal_expression"
]
outdent = [
"#define",
"#ifdef",
"#endif",
"{",
"}"
]

View file

@ -0,0 +1,26 @@
[
(import_declaration)
(const_declaration)
(type_declaration)
(type_spec)
(func_literal)
(literal_value)
(element)
(keyed_element)
(expression_case)
(default_case)
(type_case)
(communication_case)
(argument_list)
(field_declaration_list)
(block)
(type_switch_statement)
(expression_switch_statement)
] @indent
[
"case"
"}"
"]"
")"
] @outdent

View file

@ -1,30 +0,0 @@
indent = [
"import_declaration",
"const_declaration",
#"var_declaration",
#"short_var_declaration",
"type_declaration",
"type_spec",
# simply block should be enough
# "function_declaration",
# "method_declaration",
# "composite_literal",
"func_literal",
"literal_value",
"element",
"keyed_element",
"expression_case",
"default_case",
"type_case",
"communication_case",
"argument_list",
"field_declaration_list",
"block",
]
outdent = [
"case",
"}",
"]",
")"
]

View file

@ -0,0 +1,13 @@
[
(object)
(block)
(tuple)
(for_tuple_expr)
(for_object_expr)
] @indent
[
(object_end)
(block_end)
(tuple_end)
] @outdent

View file

@ -1,13 +0,0 @@
indent = [
"object",
"block",
"tuple",
"for_tuple_expr",
"for_object_expr"
]
outdent = [
"object_end",
"block_end",
"tuple_end"
]

View file

@ -0,0 +1,22 @@
[
(array)
(object)
(arguments)
(formal_parameters)
(statement_block)
(object_pattern)
(class_body)
(named_imports)
(binary_expression)
(return_statement)
(template_substitution)
(export_clause)
] @indent
[
"}"
"]"
")"
] @outdent

View file

@ -1,28 +0,0 @@
indent = [
"array",
"object",
"arguments",
"formal_parameters",
"statement_block",
"object_pattern",
"class_body",
"named_imports",
"binary_expression",
"return_statement",
"template_substitution",
# (expression_statement (call_expression))
"export_clause",
# typescript
"enum_declaration",
"interface_declaration",
"object_type",
]
outdent = [
"}",
"]",
")"
]

View file

@ -0,0 +1,9 @@
[
(object)
(array)
] @indent
[
"]"
"}"
] @outdent

View file

@ -1,9 +0,0 @@
indent = [
"object",
"array"
]
outdent = [
"]",
"}"
]

View file

@ -0,0 +1,2 @@
(block_mapping_pair) @indent

View file

@ -1,3 +0,0 @@
indent = [
"block_mapping_pair",
]

View file

@ -0,0 +1,3 @@
(basic_block) @indent
(label) @outdent

View file

@ -1,7 +0,0 @@
indent = [
"basic_block",
]
outdent = [
"label",
]

View file

@ -0,0 +1,6 @@
[
(function_body)
(instruction)
] @indent
"}" @outdent

View file

@ -1,8 +0,0 @@
indent = [
"function_body",
"instruction",
]
outdent = [
"}",
]

View file

@ -0,0 +1,24 @@
[
(function_definition)
(variable_declaration)
(local_variable_declaration)
(field)
(local_function)
(function)
(if_statement)
(for_statement)
(for_in_statement)
(repeat_statement)
(return_statement)
(while_statement)
(table)
(arguments)
(do_statement)
] @indent
[
"end"
"until"
"}"
")"
] @outdent

View file

@ -1,24 +0,0 @@
indent = [
"function_definition",
"variable_declaration",
"local_variable_declaration",
"field",
"local_function",
"function",
"if_statement",
"for_statement",
"for_in_statement",
"repeat_statement",
"return_statement",
"while_statement",
"table",
"arguments",
"do_statement",
]
oudent = [
"end",
"until",
"}",
")",
]

View file

@ -0,0 +1,18 @@
[
; "function",
(bind)
(assert)
(with)
(let)
(if)
(attrset)
(list)
(indented_string)
(parenthesized)
] @indent
[
"}"
"]"
] @outdent

View file

@ -1,18 +0,0 @@
indent = [
# "function",
"bind",
"assert",
"with",
"let",
"if",
"attrset",
"list",
"indented_string",
"parenthesized",
]
outdent = [
"}",
"]",
]

View file

@ -0,0 +1,12 @@
[
(let_binding)
(type_binding)
(structure)
(signature)
(record_declaration)
(function_expression)
(match_case)
] @indent
"}" @outdent

View file

@ -1,13 +0,0 @@
indent = [
"let_binding",
"type_binding",
"structure",
"signature",
"record_declaration",
"function_expression",
"match_case",
]
outdent = [
"}",
]

View file

@ -0,0 +1,15 @@
[
(function)
(identifier)
(method_invocation)
(if_statement)
(unless_statement)
(if_simple_statement)
(unless_simple_statement)
(variable_declaration)
(block)
(list_item)
(word_list_qw)
] @indent
"}" @outdent

View file

@ -1,17 +0,0 @@
indent = [
"function",
"identifier",
"method_invocation",
"if_statement",
"unless_statement",
"if_simple_statement",
"unless_simple_statement",
"variable_declaration",
"block",
"list_item",
"word_list_qw"
]
outdent = [
"}"
]

View file

@ -0,0 +1,17 @@
[
(array_creation_expression)
(arguments)
(formal_parameters)
(compound_statement)
(declaration_list)
(binary_expression)
(return_statement)
(expression_statement)
(switch_block)
(anonymous_function_use_clause)
] @indent
[
"}"
")"
] @outdent

View file

@ -1,17 +0,0 @@
indent = [
"array_creation_expression",
"arguments",
"formal_parameters",
"compound_statement",
"declaration_list",
"binary_expression",
"return_statement",
"expression_statement",
"switch_block",
"anonymous_function_use_clause",
]
oudent = [
"}",
")",
]

View file

@ -0,0 +1,11 @@
[
(messageBody)
(enumBody)
(oneofBody)
(serviceBody)
(rpcBody)
(msgLit)
] @indent
"}" @outdent

View file

@ -1,12 +0,0 @@
indent = [
"messageBody",
"enumBody",
"oneofBody",
"serviceBody",
"rpcBody",
"msgLit",
]
outdent = [
"}",
]

View file

@ -0,0 +1,38 @@
[
(list)
(tuple)
(dictionary)
(set)
(if_statement)
(for_statement)
(while_statement)
(with_statement)
(try_statement)
(import_from_statement)
(parenthesized_expression)
(generator_expression)
(list_comprehension)
(set_comprehension)
(dictionary_comprehension)
(tuple_pattern)
(list_pattern)
(argument_list)
(parameters)
(binary_operator)
(function_definition)
(class_definition)
] @indent
[
")"
"]"
"}"
(return_statement)
(pass_statement)
(raise_statement)
] @outdent

View file

@ -1,39 +0,0 @@
indent = [
"list",
"tuple",
"dictionary",
"set",
"if_statement",
"for_statement",
"while_statement",
"with_statement",
"try_statement",
"import_from_statement",
"parenthesized_expression",
"generator_expression",
"list_comprehension",
"set_comprehension",
"dictionary_comprehension",
"tuple_pattern",
"list_pattern",
"argument_list",
"parameters",
"binary_operator",
"function_definition",
"class_definition",
]
outdent = [
")",
"]",
"}",
"return_statement",
"pass_statement",
"raise_statement",
]
ignore = ["string"]

View file

@ -0,0 +1,25 @@
[
(argument_list)
(array)
(begin)
(block)
(call)
(class)
(case)
(do_block)
(elsif)
(if)
(hash)
(method)
(module)
(singleton_class)
(singleton_method)
] @indent
[
")"
"}"
"]"
"end"
"when"
] @outdent

View file

@ -1,25 +0,0 @@
indent = [
"argument_list",
"array",
"begin",
"block",
"call",
"class",
"case",
"do_block",
"elsif",
"if",
"hash",
"method",
"module",
"singleton_class",
"singleton_method",
]
outdent = [
")",
"}",
"]",
"end",
"when",
]

View file

@ -0,0 +1,80 @@
[
(use_list)
(block)
(match_block)
(arguments)
(parameters)
(declaration_list)
(field_declaration_list)
(field_initializer_list)
(struct_pattern)
(tuple_pattern)
(unit_expression)
(enum_variant_list)
(call_expression)
(binary_expression)
(field_expression)
(tuple_expression)
(array_expression)
(where_clause)
(token_tree)
(macro_definition)
(token_tree_pattern)
(token_repetition)
] @indent
[
"}"
"]"
")"
] @outdent
; Indent the right side of assignments.
; The #not-same-line? predicate is required to prevent an extra indent for e.g.
; an else-clause where the previous if-clause starts on the same line as the assignment.
(assignment_expression
.
(_) @expr-start
right: (_) @indent
(#not-same-line? @indent @expr-start)
(#set! "scope" "all")
)
(compound_assignment_expr
.
(_) @expr-start
right: (_) @indent
(#not-same-line? @indent @expr-start)
(#set! "scope" "all")
)
(let_declaration
.
(_) @expr-start
value: (_) @indent
(#not-same-line? @indent @expr-start)
(#set! "scope" "all")
)
(if_let_expression
.
(_) @expr-start
value: (_) @indent
(#not-same-line? @indent @expr-start)
(#set! "scope" "all")
)
(static_item
.
(_) @expr-start
value: (_) @indent
(#not-same-line? @indent @expr-start)
(#set! "scope" "all")
)
; Some field expressions where the left part is a multiline expression are not
; indented by cargo fmt.
; Because this multiline expression might be nested in an arbitrary number of
; field expressions, this can only be matched using a Regex.
(field_expression
value: (_) @val
"." @outdent
(#match? @val "(\\A[^\\n\\r]+\\([\\t ]*(\\n|\\r).*)|(\\A[^\\n\\r]*\\{[\\t ]*(\\n|\\r))")
)

View file

@ -1,28 +0,0 @@
indent = [
"use_list",
"block",
"match_block",
"arguments",
"parameters",
"declaration_list",
"field_declaration_list",
"field_initializer_list",
"struct_pattern",
"tuple_pattern",
"unit_expression",
"enum_variant_list",
"call_expression",
"binary_expression",
"field_expression",
"tuple_expression",
"array_expression",
"where_clause",
"macro_invocation"
]
outdent = [
"where",
"}",
"]",
")"
]

View file

@ -0,0 +1,22 @@
[
(block)
(arguments)
(parameter)
(class_definition)
(trait_definition)
(object_definition)
(function_definition)
(val_definition)
(import_declaration)
(while_expression)
(do_while_expression)
(for_expression)
(try_expression)
(match_expression)
] @indent
[
"}"
"]"
")"
] @outdent

View file

@ -1,23 +0,0 @@
indent = [
"block",
"arguments",
"parameter",
"class_definition",
"trait_definition",
"object_definition",
"function_definition",
"val_definition",
"import_declaration",
"while_expression",
"do_while_expression",
"for_expression",
"try_expression",
"match_expression"
]
outdent = [
"}",
"]",
")"
]

View file

@ -0,0 +1,17 @@
[
(element)
(if_statement)
(each_statement)
(await_statement)
] @indent
[
(end_tag)
(else_statement)
(if_end_expr)
(each_end_expr)
(await_end_expr)
">"
"/>"
] @outdent

View file

@ -1,18 +0,0 @@
indent = [
"element"
"if_statement"
"each_statement"
"await_statement"
]
outdent = [
"end_tag"
"else_statement"
"if_end_expr"
"each_end_expr"
"await_end_expr"
">"
"/>"
]
ignore = "comment"

View file

@ -0,0 +1,3 @@
(statement) @indent
"}" @outdent

View file

@ -1,7 +0,0 @@
indent = [
"statement",
]
outdent = [
"}",
]

View file

@ -0,0 +1,7 @@
; inherits: javascript
[
(enum_declaration)
(interface_declaration)
(object_type)
] @indent

View file

@ -1 +0,0 @@
../javascript/indents.toml

View file

@ -0,0 +1,2 @@
(block_mapping_pair) @indent

View file

@ -1,3 +0,0 @@
indent = [
"block_mapping_pair",
]

View file

@ -0,0 +1,16 @@
[
(Block)
(BlockExpr)
(ContainerDecl)
(SwitchExpr)
(AssignExpr)
(ErrorUnionExpr)
(Statement)
(InitList)
] @indent
[
"}"
"]"
")"
] @outdent

View file

@ -1,16 +0,0 @@
indent = [
"Block",
"BlockExpr",
"ContainerDecl",
"SwitchExpr",
"AssignExpr",
"ErrorUnionExpr",
"Statement",
"InitList"
]
outdent = [
"}",
"]",
")"
]