Extract file names from a string

Question

i have a long string in each line like

1000 AS34_59329 RICwdsRSYHSD11-2-IPAAPEK-93 /ifshk5/BC_IP/PROJECT/T11073/T11073_RICekkR/Fq/AS34_59329/111220_I631_FCC0E5EACXX_L4_RICwdsRSYHSD11-2-IPAAPEK-93_1.fq.gz /ifshk5/BC_IP/PROJECT/T11073/T11073_RICekkR/Fq/AS34_59329/111220_I631_FCC0E5EACXX_L4_RICwdsRSYHSD11-2-IPAAPEK-93_2.fq.gz /ifshk5/BC_IP/PROJECT/T11073/T11073_RICekkR/Fq/AS34_59329/clean_111220_I631_FCC0E5EACXX_L4_RICwdsRSYHSD11-2-IPAAPEK-93_1.fq.gz.total.info 11.824 0.981393 43.8283 95.7401 OK

This line contains three file locations (in bold). I need to scp those files to another location like /sample.

How can I use shell to scp files in each line?

 while read myline  
  do  
    echo "LINE:"$myline 
    scp .......

  done < datafile.list

I think you might want to check out awk.

Some programmer dude
– Some programmer dude

2013-02-01 08:14:10 +00:00
Commented Feb 1, 2013 at 8:14 — Some programmer dude
– Some programmer dude, Commented Feb 1, 2013 at 8:14

Chris Seymour · Accepted Answer · 2013-02-01 08:44:37Z

1

Using grep:

# grep each file name
$ grep -o "/[^ ]*" datafile.list
/ifshk5/BC_IP/PROJECT/T11073/T11073_RICekkR/Fq/AS34_59329/111220_I631_FCC0E5EACXX_L4_RICwdsRSYHSD11-2-IPAAPEK-93_1.fq.gz
/ifshk5/BC_IP/PROJECT/T11073/T11073_RICekkR/Fq/AS34_59329/111220_I631_FCC0E5EACXX_L4_RICwdsRSYHSD11-2-IPAAPEK-93_2.fq.gz
/ifshk5/BC_IP/PROJECT/T11073/T11073_RICekkR/Fq/AS34_59329/clean_111220_I631_FCC0E5EACXX_L4_RICwdsRSYHSD11-2-IPAAPEK-93_1.fq.gz.total.info

# Pipe files to scp 
$ grep -o "/[^ ]*" datafile.list | xargs -i% scp % host:/sample

edited Feb 1, 2013 at 8:44

answered Feb 1, 2013 at 8:37

Chris Seymour

86.4k32 gold badges166 silver badges209 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Aaron Digulla Over a year ago

+1 Nice solution. It will try to collect as many files as possible into one scp call.

that other guy · Accepted Answer · 2013-02-01 08:15:10Z

1

If these are columns 4, 5 and 6, you can do:

while read _ _ _ file1 file2 file3 _ 
do  
    scp "$file1" "$file2" "$file3" "user@host:path"
done < datafile.list

answered Feb 1, 2013 at 8:15

that other guy

125k12 gold badges188 silver badges214 bronze badges

1 Comment

AntiGMO Over a year ago

thanks, but i don't know you answer. each line contains 3 file locations and other information, i need scp all lines files.

Aaron Digulla · Accepted Answer · 2013-02-01 08:38:30Z

0

You can check if the part of your string is a file with -f and afterwards do your scp.

#!/bin/bash
str="1000 AS34_59329 RICwdsRSYHSD11-2-IPAAPEK-93 /ifshk5/BC_IP/PROJECT/T11073/T11073_RICekkR/Fq/AS34_59329/111220_I631_FCC0E5EACXX_L4_RICwdsRSYHSD11-2-IPAAPEK-93_1.fq.gz /ifshk5/BC_IP/PROJECT/T11073/T11073_RICekkR/Fq/AS34_59329/111220_I631_FCC0E5EACXX_L4_RICwdsRSYHSD11-2-IPAAPEK-93_2.fq.gz /ifshk5/BC_IP/PROJECT/T11073/T11073_RICekkR/Fq/AS34_59329/clean_111220_I631_FCC0E5EACXX_L4_RICwdsRSYHSD11-2-IPAAPEK-93_1.fq.gz.total.info 11.824 0.981393 43.8283 95.7401 OK"

for i in $str
do
  if [ -f $i ]; then
    echo $i
    # scp here
  fi
done

edited Feb 1, 2013 at 8:38

Aaron Digulla

330k111 gold badges626 silver badges840 bronze badges

answered Feb 1, 2013 at 8:26

jens-na

2,2741 gold badge18 silver badges22 bronze badges

2 Comments

AntiGMO Over a year ago

thanks, i just start learn shell. the i in echo str, i means? it can means any part in str?

Aaron Digulla Over a year ago

The echo is completely useless. Fixed. Also note that this will fail when the file names contain spaces.

michaelmeyer · Accepted Answer · 2013-02-01 09:18:47Z

0

If your goal is to catch the filenames, that the number of words in a record is variable, and that the filenames don't comprehend spaces (as in your example), you could try this :

while read -rd ' ' word || [ "$word" ] ; do
    if [ "$word" != "${word#\/}" ] ; then
        echo '$word is the name of a file'
    fi
done < datafile.list

answered Feb 1, 2013 at 9:18

michaelmeyer

8,2657 gold badges33 silver badges38 bronze badges

Collectives™ on Stack Overflow

Extract file names from a string

4 Answers 4

1 Comment

1 Comment

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

1 Comment

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related