Syntax highlighting severely broken after upgrade from 5.4.0 to 6.x

Created by: okdana

(I don't know if this is a worthwhile report, but i can't find any others like it, so at the very least maybe somebody can find it in search)

Yesterday we did an incremental update from 5.4.0 to 6.8.0 (5.4.0 -> 6.0.1 -> 6.7.5 -> 6.8.0). Prior to this, syntax highlighting in the file viewer was fabulous. However, after the initial update to 6.0.1, i noticed that it was broken. It has remained broken for us through to 6.8.0.

Specifically:

(1) None of the highlighting themes match their preview images under Profile > Design. Monokai is the one that i use and have the most experience with, and it looks like this now (example is bash, obv):

Every Monokai implementation i've ever seen, including the one in GitLab's preview image, uses yellow for strings, and i've rarely if ever seen that unreadable burgundy colour, so this looks totally wrong to me.

(2) Highlighting of multi-line statements, like doc comments and multi-line strings, is broken. The appearance of such a statement can break the entire page below it. Probably same issue as #6546 (closed).

(3) It seems like maybe nested quotes, or nested parentheses, or something, is broken in bash highlighting. I notice often, but intermittently, that constructions like var="$( foo "$( bar "${baz}" )" )" are broken, with the highlighter apparently not closing the quotes correctly — but it doesn't happen every time, so it may be dependent on some other bug.

(4) Highlighting of bash variables or function names that contain the names of built-ins (or common utilities maybe) is broken. For example, a statement assigning a variable called source is highlighted as if it were calling the built-in source. If a function contains the word echo, that word alone is highlighted as a built-in, while the rest of the function remains un-highlighted.

(5) Detection of variables is inconsistent. You can see in the screenshot above that it doesn't pick up "${@}". Other times it doesn't pick up a lone ${foo}, sometimes it does. Sometimes it picks up ${foo} when surrounded by other text, sometimes it doesn't. This again may be dependent on another bug.

(6) Sometimes back-ticks are treated like strings, sometimes not.

(7) In PHP, many tokens don't seem to be highlighted at all. It picks up some common/reserved words like namespace, return, true, and null, and it gets at least some function names, but everything else is un-highlighted.

I have tried rake assets:clean assets:precompile cache:clear to no effect. Since i can see most of the same problems affecting the official GitLab site (here for example), though, i'm guessing it's not a configuration problem.

The syntax highlighting in 6.x is so dysfunctional that i think i am simply going to have to try to disable it.