Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
1a011a4
Merge pull request #597 from MG-RAST/develop
teharrison Sep 4, 2014
38aa092
Merge pull request #608 from MG-RAST/develop
teharrison Sep 16, 2014
7304bed
Merge pull request #612 from MG-RAST/develop
teharrison Sep 19, 2014
00e5d91
Merge pull request #614 from MG-RAST/develop
teharrison Sep 23, 2014
c3ae3ff
Merge pull request #617 from MG-RAST/develop
teharrison Sep 24, 2014
d149b04
Merge pull request #622 from MG-RAST/develop
teharrison Sep 26, 2014
7d052c4
Merge pull request #634 from MG-RAST/develop
teharrison Oct 20, 2014
273f70a
Merge pull request #639 from MG-RAST/develop
teharrison Nov 3, 2014
262cefc
Merge pull request #644 from MG-RAST/develop
teharrison Nov 12, 2014
2a2a0dd
made email2 non mandatory
Nov 17, 2014
d5bd9a1
Merge pull request #646 from paczian/develop
Nov 17, 2014
5f8aa61
fixed example in auth key usage
Dec 1, 2014
e64bc79
Merge pull request #647 from paczian/develop
Dec 1, 2014
b5e08b7
Merge branch 'develop' of https://github.com/MG-RAST/MG-RAST into dev…
jaredbischof Dec 4, 2014
53804a2
added job moving for admins
Dec 5, 2014
0dca9f4
Merge pull request #648 from paczian/develop
Dec 5, 2014
bfea12d
fixed bugs in metagenome move function
Dec 5, 2014
1ffc5e4
Merge pull request #649 from paczian/develop
Dec 5, 2014
6ef907b
removing optional mgm in move metagenomes function
Dec 5, 2014
8432cee
Merge branch 'master' of https://github.com/MG-RAST/MG-RAST into develop
jaredbischof Dec 5, 2014
870bf0e
Merge pull request #650 from paczian/develop
Dec 5, 2014
5c8bbca
Redirected annotation resource call to use m5nr solr database to retr…
jaredbischof Dec 9, 2014
05624f5
Merge pull request #651 from jaredbischof/develop
teharrison Dec 9, 2014
496da0f
Checking for key exists.
jaredbischof Dec 12, 2014
d8dd358
Merge pull request #653 from jaredbischof/develop
jaredbischof Dec 12, 2014
e160ee0
Reverting last edit.
jaredbischof Dec 12, 2014
8b17302
Merge pull request #654 from jaredbischof/develop
jaredbischof Dec 12, 2014
8981597
added option to change sequence type
Dec 17, 2014
76c290a
Merge pull request #655 from paczian/develop
Dec 17, 2014
ecc30bc
only add external metadata ref if IDs exist
teharrison Dec 22, 2014
8d7b840
Merge pull request #656 from teharrison/develop
teharrison Dec 22, 2014
242c07e
change solr return filed for ontology lookup
teharrison Dec 22, 2014
98dd55f
Merge pull request #658 from teharrison/develop
teharrison Dec 22, 2014
0581825
second fix for annotations through solr
teharrison Dec 22, 2014
8006392
Merge pull request #660 from teharrison/develop
teharrison Dec 22, 2014
724d169
Edited Metagenome Overview page to display DRISEE error when drisee r…
jaredbischof Dec 22, 2014
d8ff02f
Merge pull request #662 from jaredbischof/develop
jaredbischof Dec 22, 2014
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/MGRAST/cgi/api.cgi
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ else {
service => 'MG-RAST',
url => ($Conf::url_base || $cgi->url),
documentation => $cgi_url.'/api.html',
description => "<p>The MG-RAST API covers most of the functionality available through the MG-RAST website, with access to annotations, analyses, metadata and access to the MG-RAST user inbox to view contents as well as upload files. All sequence data and data products from intermediate stages in the analysis pipeline are available for download. Other resources provide services not available through the website, e.g. the m5nr resource lets you query the m5nr database.</p><p>Each query to the API is represented as a URI beginning with \"http://api.metagenomics.anl.gov/\" and has a defined structure to pass the requests and parameters to the API server.</p><p>The URI queries can be used from the command line, e.g. using curl, in a browser, or incorporated in a shell script or program.</p><p>Each URI has the form</p><pre>\"http://api.metagenomics.anl.gov/{version}/{resourcepath}?{querystring}\"</pre><p>The {version} value (currently '1') explicitly directs the request to a specific version of the API, if it is omitted the latest API version will be used.</p><p>The resource path is constructed from the path parameters listed below to define a specific resource and the optional query string is used to filter the results obtained for the resource. For example:</p><pre>http://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt</pre><p>In this example the resource path \"annotation/sequence/mgm4447943.3\" defines a request for the annotated sequences for the MG-RAST job with ID 4447943.3.</p><p>The optional query string \"evalue=10&type=organism&source=SwissProt\" modifies the results by setting an evalue cutoff, annotation type and database source.</p><p>The API provides an authentication mechanism for access to private MG-RAST jobs and users' inbox. The 'auth_key' (or 'webkey') is a 25 character long string (e.g. 'j6FNL61ekNarTgqupMma6eMx5') which is used by the API to identify an MG-RAST user account and determine access rights to metagenomes. Note that the auth_key is valid for a limited time after which queries using the key will be rejected. You can create a new auth_key or view the expiration date and time of an existing auth_key on the MG-RAST website. An account can have only one valid auth_key and creating a new key will invalidate an existing key.</p><p>All public data in MG-RAST is available without an auth_key. All API queries for private data which either do not have an auth_key or use an invalid or expired auth_key will get a \"insufficient permissions to view this data\" response.</p><p>The auth_key can be included in the query string like:</p><pre>\nhttp://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt&auth_key=j6FNL61ekNarTgqupMma6eMx5</pre><p>or in a request using curl like:</p><pre>curl -X GET -H \"auth: j6FNL61ekNarTgqupMma6eMx5\" \"http://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt\"</pre><p>Note that for the curl command the quotes are necessary for the query to be passed to the API correctly.</p><p>If an optional parameter passed through the query string has a list of values only the first will be used. When multiple values are required, e.g. for multiple md5 checksum values, they can be passed to the API like:</p><pre>curl -X POST -d '{\"data\":[\"000821a2e2f63df1a3873e4b280002a8\",\"15bf1950bd9867099e72ea6516e3d602\"]}' \"http://api.metagenomics.anl.gov/m5nr/md5\"</pre><p>In some cases, the data requested is in the form of a list with a large number of entries. In these cases the limit and offset parameters can be used to step through the list, for example:</p><pre>http://api.metagenomics.anl.gov/1/project?order=name&limit=20&offset=100</pre><p>will limit the number of entries returned to 20 with an offset of 100. If these parameters are not provided default values of limit=10 and offset=0 are used. The returned JSON structure will contain the 'next' and 'prev' (previous) URIs to simplify stepping through the list.</p><p>The data returned may be plain text, compressed gzipped files or a JSON structure.</p><p>Most API queries are 'synchronous' and results are returned immediately. Some queries may require a substantial time to compute results, in these cases you can select the asynchronous option by adding '&asynchronous=1' to the end of the query string. This query will then return a URL which will return the query results when they are ready.</p>",
description => "<p>The MG-RAST API covers most of the functionality available through the MG-RAST website, with access to annotations, analyses, metadata and access to the MG-RAST user inbox to view contents as well as upload files. All sequence data and data products from intermediate stages in the analysis pipeline are available for download. Other resources provide services not available through the website, e.g. the m5nr resource lets you query the m5nr database.</p><p>Each query to the API is represented as a URI beginning with \"http://api.metagenomics.anl.gov/\" and has a defined structure to pass the requests and parameters to the API server.</p><p>The URI queries can be used from the command line, e.g. using curl, in a browser, or incorporated in a shell script or program.</p><p>Each URI has the form</p><pre>\"http://api.metagenomics.anl.gov/{version}/{resourcepath}?{querystring}\"</pre><p>The {version} value (currently '1') explicitly directs the request to a specific version of the API, if it is omitted the latest API version will be used.</p><p>The resource path is constructed from the path parameters listed below to define a specific resource and the optional query string is used to filter the results obtained for the resource. For example:</p><pre>http://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt</pre><p>In this example the resource path \"annotation/sequence/mgm4447943.3\" defines a request for the annotated sequences for the MG-RAST job with ID 4447943.3.</p><p>The optional query string \"evalue=10&type=organism&source=SwissProt\" modifies the results by setting an evalue cutoff, annotation type and database source.</p><p>The API provides an authentication mechanism for access to private MG-RAST jobs and users' inbox. The 'auth_key' (or 'webkey') is a 25 character long string (e.g. 'j6FNL61ekNarTgqupMma6eMx5') which is used by the API to identify an MG-RAST user account and determine access rights to metagenomes. Note that the auth_key is valid for a limited time after which queries using the key will be rejected. You can create a new auth_key or view the expiration date and time of an existing auth_key on the MG-RAST website. An account can have only one valid auth_key and creating a new key will invalidate an existing key.</p><p>All public data in MG-RAST is available without an auth_key. All API queries for private data which either do not have an auth_key or use an invalid or expired auth_key will get a \"insufficient permissions to view this data\" response.</p><p>The auth_key can be included in the query string like:</p><pre>\nhttp://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt&auth=j6FNL61ekNarTgqupMma6eMx5</pre><p>or in a request using curl like:</p><pre>curl -X GET -H \"auth: j6FNL61ekNarTgqupMma6eMx5\" \"http://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt\"</pre><p>Note that for the curl command the quotes are necessary for the query to be passed to the API correctly.</p><p>If an optional parameter passed through the query string has a list of values only the first will be used. When multiple values are required, e.g. for multiple md5 checksum values, they can be passed to the API like:</p><pre>curl -X POST -d '{\"data\":[\"000821a2e2f63df1a3873e4b280002a8\",\"15bf1950bd9867099e72ea6516e3d602\"]}' \"http://api.metagenomics.anl.gov/m5nr/md5\"</pre><p>In some cases, the data requested is in the form of a list with a large number of entries. In these cases the limit and offset parameters can be used to step through the list, for example:</p><pre>http://api.metagenomics.anl.gov/1/project?order=name&limit=20&offset=100</pre><p>will limit the number of entries returned to 20 with an offset of 100. If these parameters are not provided default values of limit=10 and offset=0 are used. The returned JSON structure will contain the 'next' and 'prev' (previous) URIs to simplify stepping through the list.</p><p>The data returned may be plain text, compressed gzipped files or a JSON structure.</p><p>Most API queries are 'synchronous' and results are returned immediately. Some queries may require a substantial time to compute results, in these cases you can select the asynchronous option by adding '&asynchronous=1' to the end of the query string. This query will then return a URL which will return the query results when they are ready.</p>",
contact => 'mg-rast@mcs.anl.gov',
resources => \@res };
print $cgi->header(-type => 'application/json',
Expand Down
2 changes: 1 addition & 1 deletion src/MGRAST/lib/WebPage/MetagenomeOverview.pm
Original file line number Diff line number Diff line change
Expand Up @@ -536,7 +536,7 @@ sub output {
<p>DRISEE is a tool that utilizes artificial duplicate reads (ADRs) to provide a platform independent assessment of sequencing error in metagenomic (or genomic) sequencing data. DRISEE is designed to consider shotgun data. Currently, it is not appropriate for amplicon data.</p>
<p>Note that DRISEE is designed to examine sequencing error in raw whole genome shotgun sequence data. It assumes that adapter and/or barcode sequences have been removed, but that the sequence data have not been modified in any additional way. (e.g.) Assembly or merging, QC based triage or trimming will both reduce DRISEE's ability to provide an accurate assessment of error by removing error before it is analyzed.</p>~;

if (exists($job_stats->{drisee_score_raw}) && ($drisee_num == 0) && (! $is_rna)) {
if (($drisee_num == 0) && (! $is_rna)) {
$html .= qq~<a name='drisee_ref'></a>
<h3>DRISEE
<a target=_blank href='http://blog.metagenomics.anl.gov/glossary-of-mg-rast-terms-and-concepts/#drisee' style='font-size:14px;padding-left:5px;'>[?]</a></h3>
Expand Down
163 changes: 107 additions & 56 deletions src/MGRAST/lib/resources/annotation.pm
Original file line number Diff line number Diff line change
Expand Up @@ -279,75 +279,126 @@ sub prepare_data {
print join("\t", @head)."\n";

# start query
my $hs = HTML::Strip->new();
my $sth = $mgdb->_dbh->prepare($query);
$sth->execute() or die "Couldn't execute statement: ".$sth->errstr;

# loop through indexes and print data
my $srcid = $mgdb->_src_id->{$source};
my $count = 0;
my @md5s = ();
my @seeks = ();
my @lens = ();
my $batch_count = 0;
while (my @row = $sth->fetchrow_array()) {
my ($md5, $seek, $len, $mmd5) = @row;
my $sql = "";
my $ann = [];
if ($type eq 'organism') {
$sql = "SELECT DISTINCT o.name FROM md5_annotation a, organisms_ncbi o WHERE a.md5=$md5 AND a.source=$srcid AND a.organism=o._id";
if ($filter && $flevel) {
$sql .= " AND o.tax_".$flevel."=".$mgdb->_dbh->quote($filter);
}
} elsif ($type eq 'function') {
$sql = "SELECT DISTINCT f.name FROM md5_annotation a, functions f WHERE a.md5=$md5 AND a.source=$srcid AND a.function=f._id";
} elsif ($type eq 'ontology') {
$sql = "SELECT DISTINCT o.name FROM md5_annotation a, ontologies o WHERE a.md5=$md5 AND a.source=$srcid AND a.id=o.name";
if ($filter && $flevel) {
$sql .= " AND o.".$flevel."=".$mgdb->_dbh->quote($filter);
}
} elsif ($type eq 'feature') {
$sql = "SELECT DISTINCT id FROM md5_annotation WHERE md5=$md5 AND source=$srcid";
}

if ($type ne 'md5') {
$ann = $mgdb->_dbh->selectcol_arrayref($sql);
# remove non-matching annotations if using filter without hierarchal level
if ($filter && (! $flevel)) {
my @matches = grep {/$filter/} @$ann;
@$ann = @matches;
}
if (@$ann == 0) { next; }
}

# pull data from indexed shock file
my $rec = $self->get_shock_file($node_id, undef, $self->mgrast_token, 'seek='.$seek.'&length='.$len);
chomp $rec;
foreach my $line ( split(/\n/, $rec) ) {
my @tabs = split(/\t/, $line);
if ($tabs[0]) {
my @out = ();
my $rid = $hs->parse($tabs[0]);
unless ($mgid && $rid) {
next;
}
$hs->eof;
if (($format eq 'sequence') && (@tabs == 13)) {
@out = ('mgm'.$mgid."|".$rid, $tabs[1], $tabs[12], join(";", @$ann));
} elsif ($format eq 'similarity') {
@out = ('mgm'.$mgid."|".$rid, @tabs[1..11], join(";", @$ann));
$count += 1;
}
if ($type eq 'md5') {
pop @out;
}
print join("\t", map {$_ || ''} @out)."\n";
$count += 1;
}
my ($md5, $seek, $len) = @row;
push @md5s, $md5;
push @seeks, $seek;
push @lens, $len;
$batch_count++;
if($batch_count == 1000) {
my $solr_query_str = "(source_id:$srcid) AND (md5_id:(".join(" OR ", @md5s)."))";
$count = $self->print_batch($count, $node_id, $format, $mgid, $solr_query_str, $mgdb->_version, \@md5s, \@seeks, \@lens);
@md5s = ();
@seeks = ();
@lens = ();
$batch_count = 0;
}
}

if($batch_count > 0) {
my $solr_query_str = "(source_id:$srcid) AND (md5_id:(".join(" OR ", @md5s)."))";
$count = $self->print_batch($count, $node_id, $format, $mgid, $solr_query_str, \@md5s, \@seeks, \@lens);
}

# cleanup
$sth->finish;
$mgdb->_dbh->commit;
print "Download complete. $count rows retrieved\n";
exit 0;
}

sub print_batch {
my ($self, $count, $node_id, $format, $mgid, $solr_query_str, $ann_ver, $md5s, $seeks, $lens) = @_;

my $cgi = $self->cgi;
my $type = $cgi->param('type') ? $cgi->param('type') : 'organism';
my $filter = $cgi->param('filter') || undef;
my $flevel = $cgi->param('filter_level') || undef;
my $fields = ();
my $solr_key = "";
if ($type eq 'organism') {
$solr_key = 'organism';
if ($filter && $flevel) {
$fields = [$solr_key, 'md5_id', $flevel];
$solr_query_str .= " AND ($flevel:$filter)";
} else {
$fields = [$solr_key, 'md5_id'];
}
} elsif ($type eq 'function') {
$solr_key = 'function';
$fields = [$solr_key, 'md5_id'];
} elsif ($type eq 'ontology') {
$solr_key = 'accession';
if ($filter && $flevel) {
$fields = [$solr_key, 'md5_id', $flevel];
$solr_query_str .= " AND ($flevel:$filter)";
} else {
$fields = [$solr_key, 'md5_id'];
}
} elsif ($type eq 'feature') {
$fields = ['md5_id'];
}

my ($data, $row_count) = $self->get_solr_query("POST", $Conf::m5nr_solr, $Conf::m5nr_collect.'_'.$ann_ver, $solr_query_str, "", 0, 1000000000, $fields);
my %md5s_to_annot = ();
if ($type ne 'md5') {
foreach my $result (@$data) {
$md5s_to_annot{$result->{md5_id}}{$result->{$solr_key}} = 1;
}
}

my $hs = HTML::Strip->new();
for(my $i=0; $i<@{$md5s}; $i++) {
if ($type ne 'md5' && !exists $md5s_to_annot{$md5s->[$i]}) {
next;
}
# pull data from indexed shock file
my $rec = $self->get_shock_file($node_id, undef, $self->mgrast_token, 'seek='.$seeks->[$i].'&length='.$lens->[$i]);
chomp $rec;
foreach my $line (split(/\n/, $rec)) {
my @tabs = split(/\t/, $line);
if ($tabs[0]) {
my @out = ();
my $rid = $hs->parse($tabs[0]);
unless ($mgid && $rid) {
next;
}
$hs->eof;
my $ann = [];
foreach my $key (keys %{$md5s_to_annot{$md5s->[$i]}}) {
push @$ann, $key;
}
if ($type ne 'md5') {
if ($filter && (! $flevel)) {
my @matches = grep {/$filter/} @$ann;
@$ann = @matches;
}
if (@$ann == 0) { next; }
}
if (($format eq 'sequence') && (@tabs == 13)) {
@out = ('mgm'.$mgid."|".$rid, $tabs[1], $tabs[12], join(";", @$ann));
} elsif ($format eq 'similarity') {
@out = ('mgm'.$mgid."|".$rid, @tabs[1..11], join(";", @$ann));
$count += 1;
}
if ($type eq 'md5') {
pop @out;
}
print join("\t", map {$_ || ''} @out)."\n";
$count += 1;
}
}
}
return $count;
}

1;
33 changes: 30 additions & 3 deletions src/MGRAST/lib/resources/metagenome.pm
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,27 @@ sub instance {
$self->return_data( {"ERROR" => "id $id does not exist"}, 404 );
}
$job = $job->[0];

# check if we are changing the sequence type
if (scalar(@$rest) == 3 && $rest->[1] eq 'changesequencetype') {

# check if the user is allowed to change the data
if ($self->user && ($self->user->has_right(undef, 'edit', 'metagenome', $id) || $self->user->has_star_right('edit', 'user'))) {

# check if the passed type is valid
my $valid_types = { "Amplicon" => 1,
"MT" => 1,
"WGS" => 1,
"Unknown" => 1 };
if ($valid_types->{$rest->[2]}) {
$job->sequence_type($rest->[2]);
} else {
$self->return_data({"ERROR" => "Invalid sequence type passed (".$rest->[2].")."}, 404);
}
} else {
$self->return_data( {"ERROR" => "insufficient permissions to edit this data"}, 401 );
}
}

# job is in pipeline, just view minimal info
unless ($job->viewable) {
Expand Down Expand Up @@ -395,15 +416,21 @@ sub prepare_data {
# add metadata pointers
eval {
my $proj = $job->primary_project;
$obj->{project} = ["mgp".$proj->{id}, $url."/project/mgp".$proj->{id}];
if ($proj->{id}) {
$obj->{project} = ["mgp".$proj->{id}, $url."/project/mgp".$proj->{id}];
}
};
eval {
my $samp = $job->sample;
$obj->{sample} = ["mgs".$samp->{ID}, $url."/sample/mgs".$samp->{ID}];
if ($samp->{ID}) {
$obj->{sample} = ["mgs".$samp->{ID}, $url."/sample/mgs".$samp->{ID}];
}
};
eval {
my $lib = $job->library;
$obj->{library} = ["mgl".$lib->{ID}, $url."/library/mgl".$lib->{ID}];
if ($lib->{ID}) {
$obj->{library} = ["mgl".$lib->{ID}, $url."/library/mgl".$lib->{ID}];
}
};
# get job info
my $jstats = $job->stats();
Expand Down
Loading