How to pipe AWK command output to Python as first argument

I have a plain text file input.txt which looks like this:

D000001 D000001 44 1975
D000001 D000408 1 1983
D000001 D000641 1 1977
D000001 D000900 27 1975

I process this file using this simple AWK line:

awk '{if ($4 == 1975) print $1,$2,$3}' input.txt

Then I have a Python script which accept a file as the first command line argument:

#!/usr/bin/env python3
import sys
file_name = sys.argv[1]
print(file_name)

I wonder it is possible to pipe AWK output to Python program as file argument and how to do that?

2 Answers

If you want to use a pipe, then your python script would have to read from stdin. Your script doesn't do that. Instead it expects a file name on the command line. This can be accomplished using a shell feature called process substitution to connect the two together:

script.py <(awk '{if ($4 == 1975) print $1,$2,$3}' input.txt)

<(...) denotes process substitution. What happens here is that the shell creates a file-like object that contains the output of the awk command. This file-like object even has a name. If you run the script, the output will see its name, passed to python as sys.argv[1], is something like:

/dev/fd/63

This is an old question, but if you're on bash and your script has something like

import sys
x = sys.argv[1]
print x

and you wanted to pipe command output from it without putting it in the script, you could do that with

test.py $(some_command -w arguments)

If you're expecting a single string from your output, it will be passed back as argv[1]. If you're expecting multiple strings they will be passed in individually and put into the argv[] array, and if you wanted to check the contents you'd do

print argv[1:] #starting at index 1 so you don't get the script name in the output

If it's the kind of output that'd be returned as an array, you'd have to modify your python script to get all the output put in one callable object. There is the way in the accepted answer, but you can also just call the command directly within your script and save the output to a variable with

import subprocess
x = subprocess.check_output("awk '{if ($4 == 1975) print $1,$2,$3}' ./test.txt",shell=True)
print x

which outputs

D000001 D000001 44
D000001 D000900 27

*print statements are arbitrary and just chosen to show the values