2

Is it possible to capture output from wget and other command line programs that use curses? Here is what I have right now:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=0)
for line in p.stdout:
    print "a"

This works fine for programs that have simple output, but not for wget and other programs that use curses.

Improve this question
4
  • 2
    I don't think wget uses curses. It just prints its messages to stderr rather than stdout. Commented Aug 17, 2009 at 22:57
  • For the case of wget it's probably stderr that you need to intercept (see blog.doughellmann.com/2007/07/pymotw-subprocess.html for some examples) For the intercepting the curses programs - I suspect that would become tedious very quickly when you need to emulate the terminal... Commented Aug 17, 2009 at 22:59
  • My bad, I thought that to update an existing line you had to use curses. Commented Aug 17, 2009 at 23:18
  • A common means of printing progress bars and spinners in terminal applications is to use the backspace/rubout character. It's best to check that your file descriptor is connected to a terminal (avoid cluttering regular files with ASCII control characters). However, this doesn't require curses. In fact the Python termios module has just what you need for this sort of thing. (Usually, even without the termios module you can sys.stdout.write(chr(0x08)) to backspace over a character on the current line). Commented May 18, 2010 at 14:39

1 Answer 1

Reset to default
7

I don't believe that wget is using curses.

Normally when I want to use wget in a script I'd use the -O - option to force its output to stdout. I suspect you're trying to capture the text that you normally see on your console when you're running it, which would be stderr.

From the command line, outside of Python, just run a command like:

wget -O - http://www.somesite.org/ > /tmp/wget.out 2> /tmp/wget.err

Then look at the two output files. If you see any output from wget on your console/terminal then you are running some different flavor of the command than I've seen.

If, as I suspect, you're actually interested in the stderr messages then you have two choices.

  • Change your command to add 2>&1 and add shell=True to your Popen() arguments
  • Alternatively (and preferably) add stderr=subprocess.PIPE to your Popen() arguments

The former is handy if you weren't using stdout anyway (assuming your using wget to fetch the data and write it into files). In the latter case you read from the stderr file option to get your data.

BTW: if you really did need to capture curses data ... you could try to use the standard pty module but I wouldn't recommend that. You'd be far better off fetching the pexpect module from:

And don't be scared off by the age or version numbering, it works on Python 2.5 and 2.6 as well as 2.4 and 2.3.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Definitely with pexpect, as I recommended in other recent answers -- e.g. see stackoverflow.com/questions/1283061
Why would I add shell=True? It is not handy at all - and needlessy invokes a new shell process.
I was pointing out that for redirections of the 2>&1 sort you'd either have to do shell=True or you'd have to do your own os.fork(), os.dup2() and os.execve() functions.
You should be getting wget's output before it's "done" (that is before the process has completed). However, it's likely that you're seeing some buffering effects. Try setting the bufsize=0 on the subprocess list.
@Jim Dennis: you're wrong. subprocess.Popen can do 2>&1 just fine, you don't need to fork anything and it works in all platforms. Just use subproces.Popen(cmd, stderr=subprocess.STDOUT)
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.