Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -77,5 +77,9 @@ RUN cd /MG-RAST/conf && ln -s /api-server-conf/Conf.pm
# m5nr blast files in mounted dir
RUN mkdir -p /m5nr

ARG GIT_DESCRIBE
ENV GIT_DESCRIBE=$GIT_DESCRIBE
RUN echo $GIT_DESCRIBE > /VERSION.txt

# Execute:
# /usr/local/apache2/bin/httpd -DFOREGROUND -f /api-server-conf/httpd.conf
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Systems-Software
7. NGINX

For the bioinformatics software and databases used in MG-RAST please see our manual:
ftp://ftp.metagenomics.anl.gov/manual.pdf
ftp://ftp.mg-rast.org/manual.pdf


### INSTRUCTIONS
Expand All @@ -50,10 +50,9 @@ Build image and push to dockerhub:

git clone https://github.com/MG-RAST/MG-RAST.git
cd MG-RAST
docker build -t mgrast/api-server .
docker build -t mgrast/api-server:dev .
docker build --build-arg GIT_DESCRIBE=$(git describe --tags) -t mgrast/api-server:dev .

docker push mgrast/api-server[:dev]
docker push mgrast/api-server:dev

```

Expand Down
4 changes: 2 additions & 2 deletions bin/download_m5nr_blast.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ TARGET="/m5nr"
for i in $M5NR_VERSIONS; do
M5NR_VERSION=$i
VERSION_DIR=$TARGET/$M5NR_VERSION
URL=ftp://ftp.metagenomics.anl.gov/data/MD5nr/${M5NR_VERSION}/md5nr.blast.tgz
URL=ftp://ftp.mg-rast.org/data/MD5nr/${M5NR_VERSION}/md5nr.blast.tgz

echo ""
echo "TARGET = $TARGET"
Expand All @@ -24,4 +24,4 @@ for i in $M5NR_VERSIONS; do
curl -s "${URL}" | tar -zxvf - -C ${VERSION_DIR}
fi
done
exit 0
exit 0
8 changes: 4 additions & 4 deletions src/Babel/WebPage/Download.pm
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ sub output {
my $content = qq~
<h2>Download source data and code for the M5nr</h2>
<p><ul>
<li><a target=_blank href='ftp://ftp.metagenomics.anl.gov/data/M5nr'>M5nr FTP site</a></li>
<li><a target=_blank href='ftp://ftp.metagenomics.anl.gov/data/M5nr/current/M5nr.gz'>Current M5nr FASTA</a></li>
<li><a target=_blank href='ftp://ftp.metagenomics.anl.gov/data/M5nr/current/M5nr_blast.tar'>Current M5nr NCBI-BLAST db</a></li>
<li><a target=_blank href='ftp://ftp.metagenomics.anl.gov/data/M5nr/sources'>M5nr Source Data</a></li>
<li><a target=_blank href='ftp://ftp.mg-rast.org/data/M5nr'>M5nr FTP site</a></li>
<li><a target=_blank href='ftp://ftp.mg-rast.org/data/M5nr/current/M5nr.gz'>Current M5nr FASTA</a></li>
<li><a target=_blank href='ftp://ftp.mg-rast.org/data/M5nr/current/M5nr_blast.tar'>Current M5nr NCBI-BLAST db</a></li>
<li><a target=_blank href='ftp://ftp.mg-rast.org/data/M5nr/sources'>M5nr Source Data</a></li>
<li><a target=_blank href='https://github.com/MG-RAST/M5nr'>M5nr GitHub Repository</a></li>
</ul></p>
~;
Expand Down
2 changes: 1 addition & 1 deletion src/Babel/bin/load_memcache.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ sudo chown ${USER}:${USER} $DEPLOY
for T in "${TYPES[@]}"; do
echo `date`
echo "downloading $T ..."
wget -q -O ${DEPLOY}/${T}_map.gz ftp://ftp.metagenomics.anl.gov/data/MD5nr/memcached/v${VACH}/${T}_map.gz
wget -q -O ${DEPLOY}/${T}_map.gz ftp://ftp.mg-rast.org/data/MD5nr/memcached/v${VACH}/${T}_map.gz
echo "gunzipping $T ..."
gunzip -v ${DEPLOY}/${T}_map.gz
echo "loading $T ..."
Expand Down
4 changes: 2 additions & 2 deletions src/MGRAST/Templates/pipeline.xml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
</stage>
<stage num='450'>
<input>The input sequence data for this step is in the file 440.cluster.rna97.fna.</input>
<description>Some of the annotated (.expand) files contain index numbers for md5s and annotations. Mapping files for these numbers can be found &lt;a href='ftp://ftp.metagenomics.anl.gov/data/M5nr/annotations'&gt;here&lt;/a&gt;.</description>
<description>Some of the annotated (.expand) files contain index numbers for md5s and annotations. Mapping files for these numbers can be found &lt;a href='ftp://ftp.mg-rast.org/data/M5nr/annotations'&gt;here&lt;/a&gt;.</description>
<output>
<file>The similarity output from BLAT against the RNA databases (Greengenes, LSU, SSU, and RDP) is the file 450.rna.sims in BLAST m8 format. This includes the identifier for the query which is either the FASTA id or the cluster ID, and the internal identifier for the sequence that it hits.</file>
<column>Query id (either fasta ID or cluster ID), e.g. 11847922</column>
Expand Down Expand Up @@ -72,7 +72,7 @@
</stage>
<stage num='650'>
<input>The input sequence data for this step is in the file 550.cluster.aa90.faa.</input>
<description>Some of the annotated (.expand) files contain index numbers for md5s and annotations. Mapping files for these numbers can be found &lt;a href='ftp://ftp.metagenomics.anl.gov/data/M5nr/annotations'&gt;here&lt;/a&gt;.</description>
<description>Some of the annotated (.expand) files contain index numbers for md5s and annotations. Mapping files for these numbers can be found &lt;a href='ftp://ftp.mg-rast.org/data/M5nr/annotations'&gt;here&lt;/a&gt;.</description>
<output>
<file>The similarity output from BLAT against the M5NR protein database is the file 650.superblat.sims in BLAST m8 format. This includes the identifier for the query which is either the FASTA id or the cluster ID, and the internal identifier for the sequence that it hits.</file>
<column>Query id (either fasta ID or cluster ID), e.g. 11847922_1_108_-</column>
Expand Down
4 changes: 2 additions & 2 deletions src/MGRAST/lib/WebPage/Contact.pm
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,10 @@ sub output {
my $content = "";

if ($self->application->session->user) {
$content .= "<p width=800px align=justify >If you have any questions, comments or concerns about the metagenomics analysis server, please direct them to our <a href='mailto:help\@mg-rast.org'>help desk</a>. Please take a look at the <a href='ftp://ftp.metagenomics.anl.gov/data/manual/mg-rast-manual.pdf' target=_blank>manual</a> before submitting your help desk email. Emails to the help-desk will reach the entire team, please do not send email to individuals.</p>";
$content .= "<p width=800px align=justify >If you have any questions, comments or concerns about the metagenomics analysis server, please direct them to our <a href='mailto:help\@mg-rast.org'>help desk</a>. Please take a look at the <a href='ftp://ftp.mg-rast.org/data/manual/mg-rast-manual.pdf' target=_blank>manual</a> before submitting your help desk email. Emails to the help-desk will reach the entire team, please do not send email to individuals.</p>";
} else {
my $c = Captcha::reCAPTCHA->new;
$content .= "<p width=800px align=justify >If you have any questions, comments or concerns about the metagenomics analysis server, use the form below to reach our help desk. Please take a look at the <a href='ftp://ftp.metagenomics.anl.gov/data/manual/mg-rast-manual.pdf' target=_blank>manual</a> before submitting your help desk email. Emails to the help-desk will reach the entire team, please do not send email to individuals.</p>";
$content .= "<p width=800px align=justify >If you have any questions, comments or concerns about the metagenomics analysis server, use the form below to reach our help desk. Please take a look at the <a href='ftp://ftp.mg-rast.org/data/manual/mg-rast-manual.pdf' target=_blank>manual</a> before submitting your help desk email. Emails to the help-desk will reach the entire team, please do not send email to individuals.</p>";

$content .= $self->start_form('contact_form', { action => 'try_contact' });
$content .= "<br><table><tr><td><b>your email</b></td><td><input type='text' size='50' name='email'></td></tr>";
Expand Down
2 changes: 1 addition & 1 deletion src/MGRAST/lib/WebPage/Home.pm
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ function forward_to_search (e) {
$content .= "<div class='sidebar_subitem' style='font-size: 13px; padding: 1 0;'># of sequences<span class='sidebar_stat' style='font-size: 11px; padding-top:2px;' id='seqcount'></span></div>";
$content .= "<div class='sidebar_subitem' style='font-size: 13px; padding: 1 0;'># of public metagenomes<span class='sidebar_stat' style='font-size: 11px; padding-top:2px;' id='publiccount'></span></div>";
$content .= "</div>";
$content .= "<div style='float: left; width: 410px; line-height: 17px; margin: 10 0 0 10;'>The server primarily provides upload, quality control, automated annotation and analysis for prokaryotic metagenomic shotgun samples. MG-RAST was launched in 2007 and has over 12,000 registered users and <span id='jobcount2'></span> data sets. The current server version is ".$Conf::server_version.". We suggest users take a look at <a href='http://blog.metagenomics.anl.gov/mg-rast-for-the-impatient'>MG-RAST for the impatient</a>. Also available for download is the <a href='ftp://ftp.metagenomics.anl.gov/data/manual/mg-rast-manual.pdf' target=_blank>MG-RAST manual</a>.</div>";
$content .= "<div style='float: left; width: 410px; line-height: 17px; margin: 10 0 0 10;'>The server primarily provides upload, quality control, automated annotation and analysis for prokaryotic metagenomic shotgun samples. MG-RAST was launched in 2007 and has over 12,000 registered users and <span id='jobcount2'></span> data sets. The current server version is ".$Conf::server_version.". We suggest users take a look at <a href='http://blog.metagenomics.anl.gov/mg-rast-for-the-impatient'>MG-RAST for the impatient</a>. Also available for download is the <a href='ftp://ftp.mg-rast.org/data/manual/mg-rast-manual.pdf' target=_blank>MG-RAST manual</a>.</div>";

$content .= "<div class='clear'></div>";
$content .= qq~
Expand Down
77 changes: 77 additions & 0 deletions src/MGRAST/lib/resources/metadata.pm
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,11 @@ sub new {
"validate_get" => {
'is_valid' => [ 'boolean', 'the inputted value is valid for the given category and label' ],
'message' => [ 'string', 'if not valid, reason why' ]
},
"patch" => {
'metagenome' => [ 'string', 'metagenome_id' ],
'key' => [ 'string', 'key' ],
'value' => [ 'string', 'value' ]
}
};
return $self;
Expand All @@ -103,6 +108,16 @@ sub info {
'required' => {},
'body' => {} }
},
{ 'name' => "patch",
'request' => $self->url."/".$self->name,
'description' => "",
'method' => "GET",
'type' => "synchronous",
'attributes' => $self->attributes->{validate_post} ,
'parameters' => { 'options' => {},
'required' => {},
'body' => {} }
},
{ 'name' => "template",
'request' => $self->url."/".$self->name."/template",
'description' => "Returns static template for metadata object relationships and types",
Expand Down Expand Up @@ -292,6 +307,8 @@ sub request {
$self->validate_value();
} elsif (($self->rest->[0] =~ /^(validate|import|update)$/) && ($self->method eq 'POST')) {
$self->process_file($self->rest->[0]);
} elsif (($self->rest->[0] =~ /^(patch)$/) && ($self->method eq 'POST')) {
$self->patch($self->rest->[0]);
} elsif ($self->rest->[0] eq 'google') {
$self->google($self->rest->[1]);
} else {
Expand Down Expand Up @@ -847,6 +864,66 @@ sub process_file {

$self->return_data($data);
}
sub patch{ # Resource to update a single field at a time in the metadata database
my ($self, $type) = @_;
my $master = $self->connect_to_datasource();
my $mddb = MGRAST::Metadata->new();
my $post = $self->get_post_data(["tag", "value", "metagenome", "ctype"]);
my $mgid = $post->{metagenome};
my $tag = $post->{tag};
my $value = $post->{value};
my $ctype = exists ($post->{ctype}) ? $post->{ctype} : "sample";
if (not $mgid =~ /mgm\d*.\d/)
{die ("metagenome not in mgm.... format");} # This is the protection against attacks
$mgid =~ s/mgm//;
unless ($self->user && ($self->user->has_right(undef, 'edit', 'metagenome', $mgid) || $self->user && $self->user->has_star_right('edit', 'metagenome') ) ) {
$self->return_data( {"ERROR" => "Insufficient permissions to edit mgm".$mgid}, 401 );
}
# my $jobs = $master->Job->get_objects( {metagenome_id => $mgid} );

my $query = "";
my $collection = undef;
if ($ctype =~ "sample")
{
$query = "SELECT c._id, c.ID, c.type, c.name, c.parent FROM Job j, MetaDataCollection c WHERE j.metagenome_id=".$mgid." AND j.sample=c._id";
}
# my $jobs = $master->Job->get_objects( {metagenome_id => $mgid} );
# die ( "Too many jobs found!". scalar(@$jobs)) unless scalar(@$jobs) == 1;
# my $collection_id = @$jobs[0]->{sample}; # This did not work because "sample" and "library" fields in Jobs table
# refer to _id field in MetaDataCollection but for some reason MetaDataCollection->init({_id => $collection}) fails with
# "There must be a unique index on the combination of attributes passed." Going around this using ID field ...
# The $query = ... selectall_arrayref syntax is copied from add_valid_metadata
elsif ($ctype =~ "library" )
{
$query = "SELECT c._id, c.ID, c.type, c.name, c.parent FROM Job j, MetaDataCollection c WHERE j.metagenome_id=".$mgid." AND j.library=c._id";
}
elsif ($ctype =~ "ep" )
{
$query = "SELECT c._id, c.ID, c.type, c.name, c.parent FROM Job j, MetaDataCollection c WHERE j.metagenome_id=".$mgid." AND j.sample=c.parent AND c.type='ep'";
}
else {die ("Can't handle collection type ". $ctype); }
my $result = $mddb->{_handle}->db_handle->selectall_arrayref($query);
die ( "Wrong number of collections found:". scalar(@$result).". This resource does not create collections.") unless scalar(@$result) == 1;
$collection = @$result[0]->[1];
{
{
# I can get a samp_coll object if I have collection ID:
my $samp_coll = $mddb->{_handle}->MetaDataCollection->init({ID => $collection}); # collection of samples...

# retrieving sample collection via MetaDataCollection->init and assigning into it with MetDataEntry-> create inadvertently
# 1) wipes all existing data and
# 2) adds duplicate fields
# retrieving sample collection via MetaDataCollection->init and assigning into it with mddb->add_entries
# does not have the above problems, even though $append = 0
my $append = 0;
my $data = [ [ $tag, $value ] ] ;
$mddb->add_entries ($samp_coll, $data, $append);
}
}
# update elasticsearch
$self->upsert_to_elasticsearch_metadata($mgid);
$self->return_data({ "OK" => "update attempted on mgm".$mgid }, 200);
} ## WLT

sub google {
my ($self, $id) = @_;
Expand Down
2 changes: 1 addition & 1 deletion src/MGRAST/lib/resources/metagenome.pm
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ sub new {
$self->{name} = "metagenome";
$self->{rights} = \%rights;
$self->{cv} = {
verbosity => {'minimal' => 1, 'mixs' => 1, 'metadata' => 1, 'stats' => 1, 'full' => 1, 'seqstats' => 1},
verbosity => {'minimal' => 1, 'mixs' => 1, 'metadata' => 1, 'stats' => 1, 'full' => 1, 'seqstats' => 1, 'pipeline' => 1},
direction => {'asc' => 1, 'desc' => 1},
status => {'both' => 1, 'public' => 1, 'private' => 1},
match => {'any' => 1, 'all' => 1},
Expand Down