> For the complete documentation index, see [llms.txt](https://linux.microcisco.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://linux.microcisco.com/linux-how-to-do/untitled/grep-and-regex/regex2.md). # REGEX2 This is mainly for my understanding of Regex as well as its functions and uses predominantly in Python3 as well as in its use in BASH shell scripts and UNIX grep, awk and sed ## Resources [Regular Expressions 101](https://regex101.com/)\ [Tutorial - regular-expressions.info/tutorial.html](https://www.regular-expressions.info/tutorial.html)
![](https://2463554094-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LuYJISZ_oyxYSs-KjPX%2F-MauYl_U8zL38jxvtkV7%2F-MawprrqQ6lllDnleSW_%2Fimage.png?alt=media\&token=203d12a3-1d79-4d0c-8b30-d6fc921bf73a) ## Terminology The name can be abbreviated any number of ways such as **`regex, regexp or regexes.`** We will just call it **"regex"**\ Regex (as per Wiki) is a sequence of characters that specifies a search pattern. Usually such patterns are used by string-searching algorithms for "find" or "find and replace" operations on strings, or for input validation. Using a regex expression such as ***`reg(ular expressions|ex(p|es)?)`*** we are wanting to search for **"regular expression(s)", "regex", "regexp", and "regexes"**. By using one expression we have covered 5 combinations of searches. Using plain text search we would have had to do a search 5 times. ### Flavours As in the opening statement I mentioned Python in particular, however different programm applications invoke regular engines slightly differently and are not compatible with each other.\ So the regular expression engine used by JavaScript is invoked differently by Python. These are called **"Flavours of Regex engines"** ### **How does a RegEx Engine Work?** We need to know how regex works so that we can understand why a particular regex does not do what you intend it to do.\ With all the different flavours of Regex Engines, there are fundamentally two types: * text-directed engines * regex-directed engines All modern flavours belong to the regex-directed engines because only regex-directed engines have very useful features such as "lazy quantifiers" and "backreferences" (Will come to those later) A regex-directed engine walks through the regex, attempting to match the next token in the regex to the next character. If a match is found, the engine advances through the regex and the subject string. If a token fails to match, the engine backtracks to a previous position in the regex and the subject string where it can try a different path through the regex. This tutorial will talk a lot more about backtracking later on. Modern regex flavours using regex-directed engines have lots of features such as [atomic grouping](https://www.regular-expressions.info/atomic.html) and [possessive quantifiers](https://www.regular-expressions.info/possessive.html) that allow you to control this backtracking. A text-directed engine walks through the subject string, attempting all permutations of the regex before advancing to the next character in the string. A text-directed engine never backtracks. #### **The Regex Engine Always Returns the Leftmost Match** This is a very important point to understand: a regex engine always returns the leftmost match, even if a “better” match could be found later. When applying a regex to a string, the engine starts at the first character of the string. It tries all possible permutations of the regular expression at the first character. Only if all possibilities have been tried and found to fail, does the engine continue with the second character in the text. Again, it tries all possible permutations of the regex, in exactly the same order. The result is that the regex engine returns the leftmost match. When applying *`cat`* to `He captured a catfish for his cat.` ,the engine tries to match the first token in the regex `c` to the first character in the match `H`. \ This fails. There are no other possible permutations of this regex, because it merely consists of a sequence of literal characters. So the regex engine tries to match the `c` with the `e.` \ This fails too, as does matching the `c` with the `space`. \ Arriving at the 4th character in the string, `c` matches `c.` The engine then tries to match the second token `a` to the 5th character, `a.` This succeeds too. \ But then, `t` fails to match `p`. At that point, the engine knows the regex cannot be matched starting at the 4th character in the string. \ So it continues with the 5th: `a`. Again, `c` fails to match here and the engine carries on. \ At the 15th character in the string, `c` again matches `c`. The engine then proceeds to attempt to match the remainder of the regex at character 15 and finds that `a` matches `a` and `t` matches `t.` The entire regular expression could be matched starting at character 15. The engine is “eager” to report a match. It therefore reports the first three letters of catfish as a valid match. The engine never proceeds beyond this point to see if there are any “better” matches. The first match is considered good enough. In this first example of the engine’s internals, our regex engine simply appears to work like a regular text search routine. However, it is important that you can follow the steps the engine takes in your mind. In following examples, the way the engine works has a profound impact on the matches it finds. Some of the results may be surprising. But they are always logical and predetermined, once you know how the engine works. ## Characters ### Literal Characters The most basic regex is a single character such as **a**. \ A string **"Jack is a boy"** it matches the 1st **'a'** after the 'J'. \ To match the other 'a' we would need to tell the regex engine to carry on searching - in JavaScript they use flags as in **\ \ g** where g=global, ie search the entire file.\ In Python we would need to do **re.findall** Similarly the regex **cat** matches 'cat' in "About **cat**s and dogs". The regular expression consists of a series of 3 literal characters. The engine is saying "find a 'c' followed immediately by an 'a' and immediately followed by a 't" (Regex is case sensitive unless stipulate otherwise) ### Special Characters There are 12 characters with special meanings and are called "METACHARACTERS". These metacharacters add more functionality and meaning when matching strings. They are: | **METACHARACTERS** | **DESCRIPTION** | EXAMPLE | | -------------------- | ---------------------------------------------------------------------------- | ------------------------------------ | | `\ Backslash` | `Signals a special sequence (can also be used to escape special characters)` | `\d` | | `^ Caret` | `Starts with` | `^hello` | | `$ Dollar Sign` | `Ends with` | `world$` | | `. Period` | `Any character (except newline character)` | `h..o` | | `\| Pipe` | `Either or` | `stay \| go` | | `? Question Mark` | `Preceding character is optional` | `colou?r` `matches color and colour` | | `* Asterix` | `Zero or more occurrences` | `aix*` | | `+ Plus` | `One or more occurrences` | `aix+` | | `( ) Parenthesis` | `Capture and Group` | | | `[ ] Square Bracket` | `A set of characters` | `[a-m]` | | `{ } Curly brace` | `Exactly the specified number of characters` | `{2,4}` | #### \ Backslash If you want to use any of these special characters as "literal" you will need to escape them with a backslash. For eg if you want to match **`1+1=2`**, You need to escape the + (plus) sign. so the regex becomes **`1\+1=2`** \ The brace **{** and the closed square bracket **]** are seen as a literal characters unless they have a closing brace such as {2,3}, and is part of a **character set \[ ]** ,so don't normally have to escape them. The backslash should not be used to escape any other characters as they could become a regex token like **`\d`** which is used to match a single digit from 0-9 **^ Caret** ### \[ ] Character Classes or Character Sets Here you tell the regex engine to match only **ONE** out of several characters. If you use **`gr[ae]y`** this could match either **'gray'** or **'grey'** \ To specify a **range** we use a hyphen **`[0-9].`** This matches a **SINGLE digit** \ We can also have multiple ranges **`[a-zA-Z0-9]`** where this is a **SINGLE** letter either a lower/upper/ case letter **OR** **SINGLE** digit between 0 and 9. * **\[^ ] Negated Character Classes** * Typing a caret after the opening square bracket matches any character that is **NOT** in the character class.\ Unlike the**`. (period),`** negated character classes **also match (invisible) line break characters**. If you **DON'T** want to negate the line breaks **`[^0-9\r\n]`** matches any character that is **NOT** a digit or line break (new line)\ **`q[^u]`** does NOT mean **`"a q not followed by a u"`** but rather **`" a q followed by a character that is NOT a u"`**. It does not match a string **Iraq**. It **DOES** match the q **AND** the space after the q in **Iraq is a country**. The SPACE becomes part of the overall match because it is the "character that is NOT a u" that is matched by the negated character class. If you want the regex to match the q, and only the q, in both strings, you need to use [negative lookahead](https://www.regular-expressions.info/lookaround.html): **`q(?!u)`**. But we will get to that later. ![](https://2463554094-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LuYJISZ_oyxYSs-KjPX%2F-Mb5y_3NBeDjmGdqxPz8%2F-Mb7-nPcBwj2t4iYakAT%2Fimage.png?alt=media\&token=9bf5ca98-5592-4415-b56c-d1f795c90c46) ![](https://2463554094-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LuYJISZ_oyxYSs-KjPX%2F-Mb5y_3NBeDjmGdqxPz8%2F-Mb8Ar1SnI530RxVgxMO%2Fimage.png?alt=media\&token=995da594-6cbb-4797-bcfb-f6feada32e2c) ![](https://2463554094-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LuYJISZ_oyxYSs-KjPX%2F-Mb5y_3NBeDjmGdqxPz8%2F-Mb89fxzvI2b8SEILtbG%2Fimage.png?alt=media\&token=1d827e00-206a-4bc3-8345-a9e352097d0e) ### Metacharacters Inside Character Classes In most regex flavours the only special characters or metacharacters inside a class are the closing barcket ], the backslash \\, the caret ^, and the hyphen -\ The usual metacharacters (see above) are normal characters inside a character class and do not need to be escaped by a backslash. To search for a star or a plus, use \[\*+]. Your regex will work fine if you escape regular metacharacters INSIDE a class, but doing so will significantly reduce readability.\ If you want to include a backslash as a normal character, you will need to escape it with another \\\ \[\\\x] matches a backslash OR an x\ The closing bracket ], the caret ^, and the hyphen can be included by escaping them with a backslash, or by placing themin a position where they do not take on their special meaning.\ To include an unescaped caret as a literal, place it anywhere excpet right after the opening bracket (which then makes it mean "negate") \[x^] matches either an x or a caret.\ You can include an unescaped closing bracket by placing it right after the opening bracket, or right after the negating caret. This \[ ]x] matches a closing bracket or an x. \[^]x] matches any character not a closing bracket or an x.\ The hyphen can be included right after the opening bracket, or right before the closing bracket or right after the negating caret. Both \[-x] and \[-x] match a hyphen or an x.\ \[^x-] and \[^-x] matches any character NOT a hyphen and NOT an x ### Repeating Character Classes If you repeat a character class by using the ?, \* or + operators, you are repeating the entire character class. YOu are not repearing just the character that it matched. \[0-9]+ can match 837 as well as ### Non Printable Characters You can use special character sequences to put non-printable characters in your regex * **Carriage return** means to return to the beginning of the current line without advancing downward. Commonly escaped with**`\r`** * **Line feed (new line)** means to advance downward to the next line and commonly escaped with a **`\n`** * **Form feed** means advance downward to the next "page". It was commonly used as page separators, but now is also used as section separators. (It's uncommonly used in source code to divide logically independent functions or groups of functions.) Text editors can use this character when you "insert a page break". This is commonly escaped as **`\f`** * **Tab** **`\t`** to match a tab character ## ## ## ### Metacharacters | CHARACTER | DESCRIPTION | | --------- | ------------------------------------------------------------------- | | \[ ] | A set of characters | | \\ | Signals a special sequence (also used to escape special characters) | | . | Any character except new line | | | | ### Special Sequences