---
altdown: 1
canonical: https://www.vivekchopra.com/posts/parsing-text-files-shell
title: Parsing Text Files in Shell
published: 2015-07-25T00:00:00Z
updated: 2015-07-25T00:00:00Z
lang: en
author: Vivek Chopra
description: A cleaner way to parse delimited files in bash using IFS, and the gotchas that come with it.
signals: search=yes, ai-input=yes, ai-train=no
generator: vivekchopra.com-gen
---
# Parsing Text Files in Shell

A cleaner way to parse delimited files in bash, and the gotchas that come with it.

## Key takeaways

- Setting `IFS` to the delimiter lets bash split a line like an `/etc/passwd` entry into an array without reaching for `awk` or `cut`.
- Disable globbing with `set -f` while splitting, or a `*` in the data expands to filenames.
- Bash arrays can be sparse, so use `${array[-1]}` (Bash 4.2+) for the last element instead of computing the index from the count.

Parsing structured data files is one of those things you end up writing constantly in shell. I spent years reaching for `awk` or fiddling with `cut` options before landing on something far simpler, built right into how the shell reads data.

If you want to parse a colon-delimited file like `/etc/passwd`, all you need is:

```bash
IFS=':'
```

`IFS` is the Internal Field Separator. Set it to your delimiter and word splitting does the rest.

## In Practice

```bash
line='nobody:foo:-2:-2:Unprivileged User:/var/empty:/usr/bin/false'
IFS=":"
array=($line)

echo "username: ${array[0]}"
echo "uid:      ${array[2]}"
echo "home:     ${array[5]}"
echo "shell:    ${array[6]}"
```

Note the single quotes around `$line`. Double quotes would let the shell expand the `*` wildcard to filenames in the current directory.

Here's a complete script you can invoke as `script.sh < /etc/passwd`:

```bash
#!/bin/bash

while read line
do
  # skip comment lines
  [[ ${line} == \#* ]] && continue

  set -f  # disable globbing so '*' isn't expanded to filenames

  IFS=':'
  array=(${line})

  username="${array[0]}"
  uid="${array[2]}"
  gid="${array[3]}"
  info="${array[4]}"
  home="${array[5]}"
  shell="${array[6]}"

  echo "$username  uid=$uid  home=$home  shell=$shell"

  set +f  # re-enable globbing
done
```

## Other Useful Array Tricks

Iterate over elements:

```bash
for element in "${array[@]}"; do echo "$element"; done
```

Iterate with index:

```bash
for index in "${!array[@]}"; do echo "$index ${array[index]}"; done
```

Count elements:

```bash
echo "${#array[@]}"
```

## Here Be Dragons

Shell arrays have no safety wires. You can unset individual elements or insert at arbitrary indices, which means the array you're working with may be [sparse](https://en.wikipedia.org/wiki/Sparse_array):

```bash
unset "array[5]"    # remove element 5
array[9]="ha ha"    # insert at index 9, skipping 7 and 8
```

So this will silently give you the wrong answer:

```bash
# breaks on sparse arrays
count=${#array[@]}
last_index=`expr $count - 1`
echo "${array[$last_index]}"
```

Use this instead:

```bash
# Bash 2.05b+
echo "${array[@]: -1:1}"

# Bash 4.2+
echo "${array[-1]}"
```

Shell arrays: powerful, footgun-adjacent.