Igor's Blog

Awk is a very useful tool for extracting data from text files, all you have to do is tell it which columns you want to extract and you're usually done. However, what if you didn't want to hardcode the column number and needed the ability to define it dynamically? That gets a little be more complicated but luckily there's a way to solve that requirement with bash and awk.

To demonstrate the requirement, lets say we had a data file with 5 columns like so:
 data_file.txt
A1 B1 C1 D1 E1
A2 B2 C2 D2 E2
A3 B3 C3 D3 E3
...


To start lets say you knew you wanted to extract the first and third column. Then your awk command would look something like this:
 Script
awk '{printf "%s\t%s\n", $1, $3}' < data_file.txt




That is very straight forward and works, but lets say now that for various reasons you didn't know which columns you needed to extract ahead of time (or wanted to make your script more flexible). The obvious thing to do here is to define which columns you were interested in as a script variable like so...
 Script
MY_COL1=1
MY_COL2=3
...


How do you use these with awk though? For example the following: `awk '{printf "%s\t%s\n", $$MY_COL1, $$MY_COL2}' < data_file.txt` will fail with this error:
 Error
awk: illegal field $(), name "MY_COL1"
input record number 1, file
source line number 1


The answer is to use awk's -v parameter to assign variables in awk to tell it which column numbers we're interested in. So assuming the above variables have been assigned with the column numbers we want, the following script will do the trick:
 Script
MY_COL1=1
MY_COL2=3
awk -v c1=$MY_COL1 -v c2=$MY_COL2 '{printf "%s\t%s\n", $c1, $c2}' < data_file.txt


Note how the $1 and $3 in the original awk program are now $c1 and $c2 respectively and both of c1 and c2 variables are assigned values from the shell script variables using the -v parameter.

So now you can redefine which columns are extracted by changing the shell script variable while leaving the awk command alone.

-i


, ,
About — I'm an enthusiastic software engineer and consultant interested in many fields including J2EE, programming, electronics, 3D printing, video games, wood working and gardening.
See my Resume for more information.
The views expressed in this blog are my own and not those of my employer.
comments powered by Disqus
My other posts you may like...
Programming, DIY, Games, Hacks, Tech and more.
Follow me on...
Current and Past Projects
See my Resume

Subscribe


RSS Feed

My Other Web Sites

Igor and Elise's Travels
Riverside Expressway Cam
300 George St Blogumentary

My Online Tools

UUID to OID Converter
Guru JSON-RPC Tester
Extrudifier Object Designer
Travel ┬ÁBlog

Recent Blog Posts

How to install Panel Glide Blinds from Blinds Online

Showing whitespace characters in vi

Using Jackson to automatically convert Java POJOs to JSON as output of a Jersey service

Enable quick access to the keyboard and emoji viewer in macOS

How to tell if your site is served via CloudFlare

Googe Cloud Platform doesn't support CloudFlare's free edge certificates

Fixing the invalid private key error in the Google Cloud Platform App Engine app Settings page

Cleaning the sticky PlayStation controller analogue thumb sticks

Invoking interactive shell scripts non-interactively

Using variables as column numbers with Bash and Awk when processing text files

Recent Galleries

Space Food - Chocolate Ice Cream with Chocolate Chips

Legeod Star Wars AT-DP kit

DIY spare parts computer build with a RAIDMAX Anura case

Fake 'Lepin' brand Lego packaging

Hardwood garden bench with clear resin void filler

Fixing a 3D printer extruder that stopped heating up

Easily increase disk space in a Lenovo Ideapad 100S 14" laptop with an M.2 SSD

Making a multi-piece 3D printed solder spool holder stand

DIY indoor apartment grow light wiring

Good Friday Electronics fun Easter Bunny LED PCB Kit IBEABU-01.0

Top Categories

Blogs I follow

Matt Moores Blog
Georgi's FlatPress Guide
Perplexing Permutations
The Security Sleuth

Friends

RAWS Parts Online
Alpha Dimensions Hosting
Kristensen Photography
Ilia Rogatchevski
Travelling Fairy

Blog Activity

Blog Activity
Follow me on... 
     
...or subscribe for updates!

Don't show this again