The Grymoire Blog

Dear Grymoire,
I am trying to convert a pipe character into a tab. I tried
sed y/\|/\t/ <in>out
but it did not work. Instead of a tab, I got a ‘t’.  What did I do wrong?

I thought this was a good lesson, as there are several misunderstandings a beginner might have when they write a shell script.

The first lesson concerns meta-characters.

A meta-character is simply a character that has a special meaning.

The character ‘t’ is a normal character. The character ‘\’ is a special character. It has a special meaning. However, the meaning of this character changes depending on how and where it is used.  You have to know the context. ‘|’ is another example. It may or may not be a meta-character.

So what was wrong with ‘\|’ and ‘\t’?

To break down this problem into steps, there are some basic concepts.

In Unix shell scripts. the shell is the interface between you and the computer. When you write a shell script, and that script calls a program like sed, there are two steps

1. The shell reads the shell script.
2. The shell passes the characters it sees to the program.

Let’s example the above script. How does the shell see the script? The easiest way is to insert echo as the first command on the line.

We also have to remove the <in >out so we can see what happens. So the new command is simply

echo sed y/\|/\t/

And this command echos the following to the screen:

sed y/|/t/

So the pipe character is seen by the shell and echoed. But where is the tab character?. What happened?

Normally, the shell sees the ‘|’ character as a special character. It’s a meta-character because of this special meaning. It tells the shell to start a new process, and you take the output of the previous character, and send it to the new process.

Remember that the shell has three kinds of quotations, a single quote character, a double quote character, and a backslash.

So when you quote a special character,it becomes a special kind of special character.

In other words, it’s normal. Crazy rules, but that’s how it works.  Quoting special characters in the shell make them normal and ordinary – just like the letters a-z, etc.

So the ‘\|’ is correct.  So is using single or double quotes. So why didn’t ‘\t’ work? Well, on some systems, sed will output either a ‘t’ or a ‘\t’ instead of a tab. What is happening?

The second lesson is this:

To understand what is a meta-character, you have to be familiar with the program that is looking at the character.

The shell sees ‘\t’ to be the same as ‘\’ followed by ‘t’, so it treats them the same way.

What does sed do with ‘\t’?  Well, it depends.

Some versions of sed treat ‘t’ to be the same as ‘\t’. Others treat ‘\t’ as a tab.

The person who send me this problem was trying to get a tab character into this part of the script. Well, there are two approaches:

1. Let sed convert ‘\t’ into a tab
2. Let the shell convert ‘\t’ into a tab, and pass this into sed.

This causes a problem if neither the shell nor sed interpret ‘\t’ to be a tab character. There are some programs that do treat  ‘\t’ is a tab character. These include

• The C programming language in a string
• The print(1) command, which is based on the C language
• perl
• awk
• The tr(1) command
• Some versions of the echo(1) command.
• … and several other computer languages.

But the shell does not. And sed may (or may not). So what’s the answer?

It’s quite simple.

If you want sed to insert a tab, you may have to include the tab character in the script.

In other words, the answer is to use the following

sed ‘y/|/       /’ <in >out

Where the character between the ‘/…/’ is a tab character. Unfortunately, I cannot show a tab character there, because it looks like zero or more spaces, depending upon the tab stops in your terminal.  However, you are reading this using a browser, which cannot show tabs either.

So how do you get a tab character in the script? If you are using an editor, you can edit a file that contains the script and insert a tab character.

This seemingly simple task can be tricky. Some editors convert tabs to spaces. This can depend upon the settings in your Unix terminal.

To make sure you have a tab character in the script, use ‘od -c’ to display the file. In other words, if you had the script

#!/bin/sh
sed 'y/|/       /'

and typed in od -c script, you would see

0000000   #   !   /   b   i   n   /   s   h  \n   s   e   d       '   y
0000020   /   |   /  \t   /   '  \n
0000027

If this doesn’t work, some of your terminal settings may be wrong.

A second way to get it to work is to enter the script from the command line. The Control-V character tells the terminal handler to “quote” the next character. If it is converting tabs to spaces. Control-V <Tab> may let you type a real tab character.

The third way is to use another program to create the tab character. tr(1) understands ‘\t’ so let it create it for you. ONe way is to set a variable to be equal to a tab character, and then use this variable inside of double quotes:

#!/bin/sh
tab=`echo a | tr a '\t'`
sed "y/|/$tab/' <in >out

This format is useful if you want to publish  a script on the web. That’s because the script does not contain any non-printing characters. This makes it easy to cut and paste a script. Copying a tab character, and pasting it into another window may not work.

There is one more point

Don’t change characters when you don’t need to

In other words, instead of changeing the pipe character ti a tab, just use the pipe character as the field seperator. Awk allows this

Just use

awk -F’|’ ‘{print $3}’

or whatever.