Vivek Chopra

WritingOperational reliability2 min read

Parsing Text Files in Shell

A cleaner way to parse delimited files in bash, and the gotchas that come with it.

This post as MarkdownAll copies

Parsing structured data files is one of those things you end up writing constantly in shell. I spent years reaching for awk or fiddling with cut options before landing on something far simpler, built right into how the shell reads data.

If you want to parse a colon-delimited file like /etc/passwd, all you need is:

Bash
IFS=':'

IFS is the Internal Field Separator. Set it to your delimiter and word splitting does the rest.

In Practice

Bash
line='nobody:foo:-2:-2:Unprivileged User:/var/empty:/usr/bin/false'
IFS=":"
array=($line)

echo "username: ${array[0]}"
echo "uid:      ${array[2]}"
echo "home:     ${array[5]}"
echo "shell:    ${array[6]}"

Note the single quotes around $line. Double quotes would let the shell expand the * wildcard to filenames in the current directory.

Here's a complete script you can invoke as script.sh < /etc/passwd:

Bash
#!/bin/bash

while read line
do
  # skip comment lines
  [[ ${line} == \#* ]] && continue

  set -f  # disable globbing so '*' isn't expanded to filenames

  IFS=':'
  array=(${line})

  username="${array[0]}"
  uid="${array[2]}"
  gid="${array[3]}"
  info="${array[4]}"
  home="${array[5]}"
  shell="${array[6]}"

  echo "$username  uid=$uid  home=$home  shell=$shell"

  set +f  # re-enable globbing
done

Other Useful Array Tricks

Iterate over elements:

Bash
for element in "${array[@]}"; do echo "$element"; done

Iterate with index:

Bash
for index in "${!array[@]}"; do echo "$index ${array[index]}"; done

Count elements:

Bash
echo "${#array[@]}"

Here Be Dragons

Shell arrays have no safety wires. You can unset individual elements or insert at arbitrary indices, which means the array you're working with may be sparse:

Bash
unset "array[5]"    # remove element 5
array[9]="ha ha"    # insert at index 9, skipping 7 and 8

So this will silently give you the wrong answer:

Bash
# breaks on sparse arrays
count=${#array[@]}
last_index=`expr $count - 1`
echo "${array[$last_index]}"

Use this instead:

Bash
# Bash 2.05b+
echo "${array[@]: -1:1}"

# Bash 4.2+
echo "${array[-1]}"

Shell arrays: powerful, footgun-adjacent.