bash heredoc ending sign bug

Heredocs are impossible to handle correctly with micro's highlighting system because we can't capture the start token to be used later as the end token, so we have been using something that works for the most common cases:

https://github.com/micro-editor/micro/blob/b75cfc79a175524839492053dab80fcee18b41bf/runtime/syntax/sh.yaml#L59-L63

Heredocs are impossible to handle correctly with micro's highlighting system because we can't capture the start token to be used later as the end token, so we have been using something that works for the most common cases:

https://github.com/micro-editor/micro/blob/b75cfc79a175524839492053dab80fcee18b41bf/runtime/syntax/sh.yaml#L59-L63

I filed the same bug report a couple weeks ago. Since then, I've learned that it is not limited to heredoc, there is just clearly a bug in the way it handles quotation in general.

If you're in a hurry, the last two lines of this comment contain what I think should be an insight into how micro's parser fails for bash quotation marks.

# Handles this fine:
SED_FILE() { printf '%s' "${@}" | grep -oP "\bsed\s[^;|&]+" | head -1 | awk '{print $NF}' | sed "s/['\"]//g"; }

# The exact same content breaks highlighting:
SED_FILE="$(printf '%s' "${@}" | grep -oP "\bsed\s[^;|&]+" | head -1 | awk '{print $NF}' | sed "s/['\"]//g")"

NOTE: There is a common misconception that ' "double quoted string inside single quotes" ' are invalid syntax in bash, but this is not true.

The only "rules" baked into the language are (paraphrasing):

1. You get interpolation inside double-quotes. If your string is wrapped with double-quotes, then you have to escape any inner double-quotes that you want to be preserved in the output, otherwise they are interpreted no differently than if you ran the command outside of whatever wrapping is around them.

ie: echo "hey" does not preserve the " characters, but echo "\"hey\"" and echo '"hey"' do.

2. Single-quotes are literally multi-character escapes. Thus single-quotes can not wrap single-quotes - not even if you escape them (see: last example below)

Here's a few examples of how bash handles nested quotation marks:

echo hey                             #  hey
echo $(echo $(echo "hey"))           #  hey
echo "$(echo "$(echo "hey")")"       #  hey
echo "$(echo "$(echo \"hey\")")"     #  "hey"
echo "$(echo \"$(echo "hey")\")"     #  "hey"
echo "$(echo "$(echo '"hey"')")"     #  "hey"
echo "$(echo "$(echo "'hey'")")"     #  'hey'
echo "$(echo \"$(echo "'hey'")\")"   #  "'hey'"
echo "$(echo \"$(echo 'hey')\")"     #  "hey"
echo "$(echo \"$(echo hey)\")"       #  "hey"

echo $(echo '"$(echo '"hey"')')      #  "$(echo hey)

# notice this does not break
echo '$(echo "$(echo '"hey"')")'     #  $(echo "$(echo hey)")

# this breaks:
echo $(echo '$(echo \'"hey"\')')     #  > 
# same thing:
echo '\'hey\''                       #  > 

Notice how github's syntax highlighting correctly handles the two breaking examples at the end there. Copy the contents of the entire example code block into micro and notice that syntax highlighting does not detect this.

SUMMARY: We have confirmed that syntax highlighting for bash in micro breaks when it shouldn't and does not break when it should.

I filed the same bug report a couple weeks ago. Since then, I've learned that it is not limited to heredoc, there is just clearly a bug in the way it handles quotation in general.

If you're in a hurry, the last two lines of this comment contain what I think should be an insight into how micro's parser fails for bash quotation marks.

# Handles this fine:
SED_FILE() { printf '%s' "${@}" | grep -oP "\bsed\s[^;|&]+" | head -1 | awk '{print $NF}' | sed "s/['\"]//g"; }

# The exact same content breaks highlighting:
SED_FILE="$(printf '%s' "${@}" | grep -oP "\bsed\s[^;|&]+" | head -1 | awk '{print $NF}' | sed "s/['\"]//g")"

NOTE: There is a common misconception that ' "double quoted string inside single quotes" ' are invalid syntax in bash, but this is not true.

The only "rules" baked into the language are (paraphrasing):

1. You get interpolation inside double-quotes. If your string is wrapped with double-quotes, then you have to escape any inner double-quotes that you want to be preserved in the output, otherwise they are interpreted no differently than if you ran the command outside of whatever wrapping is around them.

ie: echo "hey" does not preserve the " characters, but echo "\"hey\"" and echo '"hey"' do.

2. Single-quotes are literally multi-character escapes. Thus single-quotes can not wrap single-quotes - not even if you escape them (see: last example below)

Here's a few examples of how bash handles nested quotation marks:

echo hey                             #  hey
echo $(echo $(echo "hey"))           #  hey
echo "$(echo "$(echo "hey")")"       #  hey
echo "$(echo "$(echo \"hey\")")"     #  "hey"
echo "$(echo \"$(echo "hey")\")"     #  "hey"
echo "$(echo "$(echo '"hey"')")"     #  "hey"
echo "$(echo "$(echo "'hey'")")"     #  'hey'
echo "$(echo \"$(echo "'hey'")\")"   #  "'hey'"
echo "$(echo \"$(echo 'hey')\")"     #  "hey"
echo "$(echo \"$(echo hey)\")"       #  "hey"

echo $(echo '"$(echo '"hey"')')      #  "$(echo hey)

# notice this does not break
echo '$(echo "$(echo '"hey"')")'     #  $(echo "$(echo hey)")

# this breaks:
echo $(echo '$(echo \'"hey"\')')     #  > 
# same thing:
echo '\'hey\''                       #  > 

Notice how github's syntax highlighting correctly handles the two breaking examples at the end there. Copy the contents of the entire example code block into micro and notice that syntax highlighting does not detect this.

SUMMARY: We have confirmed that syntax highlighting for bash in micro breaks when it shouldn't and does not break when it should.