Using the sed stream editor¶

1. Why use sed?¶

sed finds text that matches a regular expression and can replace it:

cat macbeth.txt | sed 's/[A-Z][a-z][a-z]*land/Sweden/g'

sed is among the list of ‘Portable Operating System Interface’ (POSIX) commands, which means it is considered a fundamental tool and is likely to be available on your operating system.

sed (short for ‘stream editor’) can do what grep can do and more: it is ‘a non-interactive command-line text editor’ (source)

In this session, we will be manipulating a file from sed (instead of doing so by hand).

2. Overview¶

3. Types of operations¶

3.1. Filtering¶

sed can do what grep can do. For example, in the session aboutgrep, we used the following command:

man grep | grep "^[A-Z]"

The equivalent sed command is this:

man grep | sed --quiet "/^[A-Z]/p"

man grep | sed --quiet "/^[A-Z]/p"
man grep | sed --silent "/^[A-Z]/p"
man grep | sed -n "/^[A-Z]/p"

3.2. Replacing¶

Probably the most used feature of sed is its replacement functionality:

sed 's/[regular_expression]/[replacement]/'

The s is short for ‘substitute’. For example, the command below substitutes ‘morning’ for ‘afternoon’.

echo "Good morning" | sed 's/morning/afternoon/'

Good afternoon

If there may be multiple matches in a sentence, add g at the end:

echo "Good morning, good morning" | sed 's/morning/afternoon/g'

Good afternoon, good afternoon

Good afternoon, good morning

4. Input and output¶

5. Type of regular expressions¶

There are two types of regular expressions present in sed according the sed manual: basic and extended regular expressions.

Here is a quote from the sed manual regarding their differences:

In GNU sed, the only difference between basic and extended regular expressions is in the behavior of a few special characters: ?, +, parentheses, braces ({}), and |.

The difference can be shown using this text:

Bland
England
Gland
Holland

To select the countries, use:

cat lands.txt | sed --quiet '/[A-Z][a-z][a-z]*land/p'
cat lands.txt | sed --quiet --regexp-extended '/[A-Z][a-z]+land/p'

cat lands.txt | sed -n --regexp-extended '/[A-Z][a-z]+land/p'

cat lands.txt | sed --quiet -E '/[A-Z][a-z]+land/p'

Exercises¶

Exercise 1¶

In this exercise, we will work Macbeth, a tale written by William Shakespeare.

In these exercises, we will:

• Replace ‘Weird Sisters’ by ‘witches’: the text will be more clear
• Replace all country names by ‘Sweden’ (or your favorite country): the text may be funnier to read :-)

Exercise 1.1: download Macbeth¶

Download the file from a terminal as such:

wget https://raw.githubusercontent.com/UPPMAX/linux-command-line-201/refs/heads/main/docs/sessions/sed/macbeth.txt

Exercise 1.2: use sed to replace text from standard input¶

Read the ‘Replacing’ section.

In Macbeth, replace Weird Sisters (both words start with an upper-case character) by witches. Do so by using cat on the file macbeth.txt, then piping it to sed.

cat macbeth.txt | sed 's/Weird Sisters/witches/'

cat macbeth.txt | sed 's/Weird Sisters/witches/g'

cat macbeth.txt | grep "Weird Sisters.*Weird Sisters"

Check that your replacement worked.

cat macbeth.txt | sed 's/Weird Sisters/witches/g' | grep witches

The witches, hand in hand,
title, before, these witches saluted me, and referred me to the
I dreamt last night of the three witches:
(And betimes I will) to the witches:
Saw you the witches?

cat macbeth.txt | sed 's/Weird Sisters/witches/g' > macbeth_with_witches.txt

Exercise 1.3: use sed to find text from standard input¶

Read the ‘Filtering’ section.

In Macbeth, there are many place names ending on land.

Search for all place names ending on land using sed. To be precise, search for all matches that:

• (1) start with an uppercase character
• (2) have zero or more lowercase characters
• (3) end on land

Do so by using cat on the file macbeth.txt, then piping it to sed.

cat macbeth.txt | sed --quiet '/[A-Z][a-z]*land/p'

Exercise 1.4: use extended regular expressions¶

Reading your answer in the previous exercise, your non-Swedish non-Finnish colleague comes to you and states that your regular expression makes no sense: your regular expression matches ‘Aland’, ‘Bland’, ‘Cland’ (and ‘Gland’), which can be improved upon.

Read the section ‘Type of regular expressions’.

Search for all place names ending on land using sed. To be precise, search for all matches that:

• (1) start with an uppercase character
• (2) have one or more lowercase characters: use a + in your regular expression
• (3) end on land.

cat macbeth.txt | sed --quiet --regexp-extended '/[A-Z][a-z]+land/p'

cat macbeth.txt | sed --quiet '/[A-Z][a-z][a-z]*land/p'

Exercise 1.5: use sed to replace text in a file¶

Until now, we never have touched the original file. Here we use sed --in-place [commands] [filename] to directly work on the original file.

Using sed directly on the file macbeth.txt, replace Weird Sisters by witches

sed --in-place 's/Weird Sisters/witches/g' macbeth.txt

Using sed directly on the file macbeth.txt, replace ‘lands’ by Sweden.

sed --in-place 's/[A-Z][a-z][a-z]*land/Sweden/' macbeth.txt
sed --in-place 's/[A-Z][a-z][a-z]*land/Sweden/g' macbeth.txt
sed --in-place --regexp-extended 's/[A-Z][a-z]+land/Sweden/' macbeth.txt
sed --in-place --regexp-extended 's/[A-Z][a-z]+land/Sweden/g' macbeth.txt

Using sed directly on the file macbeth.txt, to remove the copyright.

sed --in-place '1,26d;4173,4495d' macbeth.txt

(optional) Exercise 1.6: Do this exercise from a script¶

Do this exercise from a script.

wget https://raw.githubusercontent.com/UPPMAX/linux-command-line-201/refs/heads/main/docs/sessions/sed/macbeth.txt
sed --in-place 's/Weird Sisters/witches/g' macbeth.txt
sed --in-place 's/[A-Z][a-z][a-z]*land/Sweden/' macbeth.txt
sed --in-place '1,26d;4173,4495d' macbeth.txt

#!/bin/bash

rm --force macbeth.txt

wget https://raw.githubusercontent.com/UPPMAX/linux-command-line-201/refs/heads/main/docs/lesson_plans/20251204/script_1.sh

(optional) Exercise 1.7: Do this exercise from a smart script¶

Do this exercise from a script, without hardcoding the lines

#!/bin/bash
rm -f macbeth.txt
wget https://raw.githubusercontent.com/UPPMAX/linux-command-line-201/refs/heads/main/docs/sessions/sed/macbeth.txt
sed --in-place 's/Weird Sisters/witches/g' macbeth.txt
sed --in-place 's/[A-Z][a-z][a-z]*land/Sweden/' macbeth.txt

end_of_first_copyright_line=$(cat --number macbeth.txt | grep "START OF THE PROJECT GUTENBERG EBOOK" | cut --field 1)
begin_of_second_copyright_line=$(cat --number macbeth.txt | grep "START: FULL LICENSE" | cut --field 1)
last_line=$(wc --lines macbeth.txt | cut --field 1 --delimiter " ")

sed --in-place "1,${end_of_first_copyright_line}d;${begin_of_second_copyright_line},${last_line}d" macbeth.txt

wget https://raw.githubusercontent.com/UPPMAX/linux-command-line-201/refs/heads/main/docs/lesson_plans/20251204/script_2.sh