Join Scott Simpson for an in-depth discussion in this video Working with text: sed, part of Linux Tips Weekly.
- [Instructor] Sed or a stream editor is a tool that allows us to manipulate text on a Linux system. While we might think of a text editor nowadays as an application where we can move around text on the screen, inserting and deleting as we go, sed is a little bit different. It can be used on its own, and it's also very commonly used as a component of a series of piped commands. That's why it's called a stream editor rather than being an interactive editor. It's more of a way to create a little one shot transformation unit for a text coming into it. And that makes it really helpful for one liners that process data and utilities that transform text in helpful ways.
Sed can delete blank lines in a file for example, or insert text at a given position. It can transpose and substitute characters, replace text and more. While we can pipe text into sed for processing, it can be easier to experiment using a text file. So I'll use a text file that I have here. You can create a file yourself or use an existing one. Let's take a look at the syntax for sed by appending texts to a file. Appending adds text at the end of something, and in the case of sed which processes files line by line, simply saying to append text will add it after each line.
To write a sed statement I'll write sed and then in quotes I'll use the a command for append and then some text to append. I'll write new text and after that I'll give the file that I'm operating on. And there after each line is my new text. It was appended after sed processed each line. We can also insert text with the i command. Inserting adds text before each line rather than appending it after.
I can also add what are called addresses to the beginning of the command here, to instruct sed to only act on a particular line number or some other parameter. To say insert some text before the third line, I'll write sed three i some text, and file name. And up here before the third line in the original file is my text. To append something after the third line I'd write sed three a, add my new text and set the name of my file.
I can use the d command to tell sed to delete particular lines of text as well. We can see that d by itself returns well, nothing, because I've told it for each line that gets processed, delete it. Let's do something a little bit less useless by saying delete the first line, with sed one d and the file. We can also use regular expressions to target lines to work with. For example, I can say sed slash line slash d text file, inside those slashes is my regular expression, in this case a literal match on the word line, and then after that I'm saying to delete it with d.
This will delete the line that contains the word line. Or we can target the line that matches something and append after or insert before. I'll change the d here to a for append and I'll add a couple carets. That's how to use a literal match. And we can use a more advanced match as well, like using lines that start with a capital letter with sed slash a caret for the beginning of the line, and a range of capital letters and then append and let's put in caps.
And there we go, after any line that starts with a capital letter, sed will insert the line that says caps. In addition to working with hold lines, we could use sed to replace the contents of lines as well. A common command for this is s for substitution. To use substitution we use the s and then a slash and a regex, often times a literal match, and then a slash and the stream to replace the match with.
We'll end it with a slash, and this command will take the word lines and replace it with rows. And there's the change right there. In fact you might've seen this syntax in online forums where people use it to note edits or things they might've mistyped. This substitution is helpful for individual words, and you can use it if you know there's a string in a text file you want to replace. For example, an IP address or host name and a log you want to post publicly. And remember we can use a regex here too, to match patterns instead of explicit text.
So we could replace letters that are in the first half of the alphabet with an underscore. With sed s a through m slash underscore slash and the file name. And while we see a response here, we can see that the rule doesn't look as if it was fully implemented. There's still some letters from the first half of the alphabet that didn't get replaced. That's because the way that this is written for each line that processed, sed found a match, did a replacement and then said job done, let's move onto the next line.
In order to make sure the rule matches every character in my range instead of just the first one, we need to add a g at the end of the rule to make the match greedy, and there we go. Now instead of just matching one instance of the range of characters, it keeps going through the line replacing matches as it goes. This substitution replaces whatever value it matches. But we can also use a value that a regex matches in the replacement. Let's write a rule that puts square brackets around vowels. With sed s, a backslash and an opening parenthesis, a range of vowels a backslash and a closing parenthesis, our second slash for the substitution, square brackets with a backslash one for the first match, a slash and g for greedy.
This notation of the parenthesis proceeded with a backslash captures a match, and backslash one here captures that first match. And then we can see the literal square brackets. The output is coming to the standard output here. But in a pipe, it'd be passed along to the next command. We could use basic redirection to save the output to a file as well. So you can see how sed can be useful in processing a stream of text according to particular rules.
It's been a quick look at some of the basics of the sed command. Sed syntax can be cryptic if you've never come across it before. And I hope this episode has helped you to become a little more familiar with what sed's doing if you see it in a set of commands, or even to inspire you to explore using it for your own project. If you want to take a deeper dive to see how to use sed, be sure to check out sed essential training.
Note: Because this is an ongoing series, viewers will not receive a certificate of completion.