Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/MGRAST/cgi/api.cgi
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ else {
service => 'MG-RAST',
url => ($Conf::url_base || $cgi->url),
documentation => $cgi_url.'/api.html',
description => "<p>The MG-RAST API covers most of the functionality available through the MG-RAST website, with access to annotations, analyses, metadata and access to the MG-RAST user inbox to view contents as well as upload files. All sequence data and data products from intermediate stages in the analysis pipeline are available for download. Other resources provide services not available through the website, e.g. the m5nr resource lets you query the m5nr database.</p><p>Each query to the API is represented as a URI beginning with \"http://api.metagenomics.anl.gov/\" and has a defined structure to pass the requests and parameters to the API server.</p><p>The URI queries can be used from the command line, e.g. using curl, in a browser, or incorporated in a shell script or program.</p><p>Each URI has the form</p><pre>\"http://api.metagenomics.anl.gov/{version}/{resourcepath}?{querystring}\"</pre><p>The {version} value (currently '1') explicitly directs the request to a specific version of the API, if it is omitted the latest API version will be used.</p><p>The resource path is constructed from the path parameters listed below to define a specific resource and the optional query string is used to filter the results obtained for the resource. For example:</p><pre>http://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt</pre><p>In this example the resource path \"annotation/sequence/mgm4447943.3\" defines a request for the annotated sequences for the MG-RAST job with ID 4447943.3.</p><p>The optional query string \"evalue=10&type=organism&source=SwissProt\" modifies the results by setting an evalue cutoff, annotation type and database source.</p><p>The API provides an authentication mechanism for access to private MG-RAST jobs and users' inbox. The 'auth_key' (or 'webkey') is a 25 character long string (e.g. 'j6FNL61ekNarTgqupMma6eMx5') which is used by the API to identify an MG-RAST user account and determine access rights to metagenomes. Note that the auth_key is valid for a limited time after which queries using the key will be rejected. You can create a new auth_key or view the expiration date and time of an existing auth_key on the MG-RAST website. An account can have only one valid auth_key and creating a new key will invalidate an existing key.</p><p>All public data in MG-RAST is available without an auth_key. All API queries for private data which either do not have an auth_key or use an invalid or expired auth_key will get a \"insufficient permissions to view this data\" response.</p><p>The auth_key can be included in the query string like:</p><pre>\nhttp://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt&auth_key=j6FNL61ekNarTgqupMma6eMx5</pre><p>or in a request using curl like:</p><pre>curl -X GET -H \"auth: j6FNL61ekNarTgqupMma6eMx5\" \"http://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt\"</pre><p>Note that for the curl command the quotes are necessary for the query to be passed to the API correctly.</p><p>If an optional parameter passed through the query string has a list of values only the first will be used. When multiple values are required, e.g. for multiple md5 checksum values, they can be passed to the API like:</p><pre>curl -X POST -d '{\"data\":[\"000821a2e2f63df1a3873e4b280002a8\",\"15bf1950bd9867099e72ea6516e3d602\"]}' \"http://api.metagenomics.anl.gov/m5nr/md5\"</pre><p>In some cases, the data requested is in the form of a list with a large number of entries. In these cases the limit and offset parameters can be used to step through the list, for example:</p><pre>http://api.metagenomics.anl.gov/1/project?order=name&limit=20&offset=100</pre><p>will limit the number of entries returned to 20 with an offset of 100. If these parameters are not provided default values of limit=10 and offset=0 are used. The returned JSON structure will contain the 'next' and 'prev' (previous) URIs to simplify stepping through the list.</p><p>The data returned may be plain text, compressed gzipped files or a JSON structure.</p><p>Most API queries are 'synchronous' and results are returned immediately. Some queries may require a substantial time to compute results, in these cases you can select the asynchronous option by adding '&asynchronous=1' to the end of the query string. This query will then return a URL which will return the query results when they are ready.</p>",
description => "<p>The MG-RAST API covers most of the functionality available through the MG-RAST website, with access to annotations, analyses, metadata and access to the MG-RAST user inbox to view contents as well as upload files. All sequence data and data products from intermediate stages in the analysis pipeline are available for download. Other resources provide services not available through the website, e.g. the m5nr resource lets you query the m5nr database.</p><p>Each query to the API is represented as a URI beginning with \"http://api.metagenomics.anl.gov/\" and has a defined structure to pass the requests and parameters to the API server.</p><p>The URI queries can be used from the command line, e.g. using curl, in a browser, or incorporated in a shell script or program.</p><p>Each URI has the form</p><pre>\"http://api.metagenomics.anl.gov/{version}/{resourcepath}?{querystring}\"</pre><p>The {version} value (currently '1') explicitly directs the request to a specific version of the API, if it is omitted the latest API version will be used.</p><p>The resource path is constructed from the path parameters listed below to define a specific resource and the optional query string is used to filter the results obtained for the resource. For example:</p><pre>http://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt</pre><p>In this example the resource path \"annotation/sequence/mgm4447943.3\" defines a request for the annotated sequences for the MG-RAST job with ID 4447943.3.</p><p>The optional query string \"evalue=10&type=organism&source=SwissProt\" modifies the results by setting an evalue cutoff, annotation type and database source.</p><p>The API provides an authentication mechanism for access to private MG-RAST jobs and users' inbox. The 'auth_key' (or 'webkey') is a 25 character long string (e.g. 'j6FNL61ekNarTgqupMma6eMx5') which is used by the API to identify an MG-RAST user account and determine access rights to metagenomes. Note that the auth_key is valid for a limited time after which queries using the key will be rejected. You can create a new auth_key or view the expiration date and time of an existing auth_key on the MG-RAST website. An account can have only one valid auth_key and creating a new key will invalidate an existing key.</p><p>All public data in MG-RAST is available without an auth_key. All API queries for private data which either do not have an auth_key or use an invalid or expired auth_key will get a \"insufficient permissions to view this data\" response.</p><p>The auth_key can be included in the query string like:</p><pre>\nhttp://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt&auth=j6FNL61ekNarTgqupMma6eMx5</pre><p>or in a request using curl like:</p><pre>curl -X GET -H \"auth: j6FNL61ekNarTgqupMma6eMx5\" \"http://api.metagenomics.anl.gov/1/annotation/sequence/mgm4447943.3?evalue=10&type=organism&source=SwissProt\"</pre><p>Note that for the curl command the quotes are necessary for the query to be passed to the API correctly.</p><p>If an optional parameter passed through the query string has a list of values only the first will be used. When multiple values are required, e.g. for multiple md5 checksum values, they can be passed to the API like:</p><pre>curl -X POST -d '{\"data\":[\"000821a2e2f63df1a3873e4b280002a8\",\"15bf1950bd9867099e72ea6516e3d602\"]}' \"http://api.metagenomics.anl.gov/m5nr/md5\"</pre><p>In some cases, the data requested is in the form of a list with a large number of entries. In these cases the limit and offset parameters can be used to step through the list, for example:</p><pre>http://api.metagenomics.anl.gov/1/project?order=name&limit=20&offset=100</pre><p>will limit the number of entries returned to 20 with an offset of 100. If these parameters are not provided default values of limit=10 and offset=0 are used. The returned JSON structure will contain the 'next' and 'prev' (previous) URIs to simplify stepping through the list.</p><p>The data returned may be plain text, compressed gzipped files or a JSON structure.</p><p>Most API queries are 'synchronous' and results are returned immediately. Some queries may require a substantial time to compute results, in these cases you can select the asynchronous option by adding '&asynchronous=1' to the end of the query string. This query will then return a URL which will return the query results when they are ready.</p>",
contact => 'mg-rast@mcs.anl.gov',
resources => \@res };
print $cgi->header(-type => 'application/json',
Expand Down
159 changes: 103 additions & 56 deletions src/MGRAST/lib/resources/annotation.pm
Original file line number Diff line number Diff line change
Expand Up @@ -279,75 +279,122 @@ sub prepare_data {
print join("\t", @head)."\n";

# start query
my $hs = HTML::Strip->new();
my $sth = $mgdb->_dbh->prepare($query);
$sth->execute() or die "Couldn't execute statement: ".$sth->errstr;

# loop through indexes and print data
my $srcid = $mgdb->_src_id->{$source};
my $count = 0;
my @md5s = ();
my @seeks = ();
my @lens = ();
my $batch_count = 0;
while (my @row = $sth->fetchrow_array()) {
my ($md5, $seek, $len, $mmd5) = @row;
my $sql = "";
my $ann = [];
if ($type eq 'organism') {
$sql = "SELECT DISTINCT o.name FROM md5_annotation a, organisms_ncbi o WHERE a.md5=$md5 AND a.source=$srcid AND a.organism=o._id";
if ($filter && $flevel) {
$sql .= " AND o.tax_".$flevel."=".$mgdb->_dbh->quote($filter);
}
} elsif ($type eq 'function') {
$sql = "SELECT DISTINCT f.name FROM md5_annotation a, functions f WHERE a.md5=$md5 AND a.source=$srcid AND a.function=f._id";
} elsif ($type eq 'ontology') {
$sql = "SELECT DISTINCT o.name FROM md5_annotation a, ontologies o WHERE a.md5=$md5 AND a.source=$srcid AND a.id=o.name";
if ($filter && $flevel) {
$sql .= " AND o.".$flevel."=".$mgdb->_dbh->quote($filter);
}
} elsif ($type eq 'feature') {
$sql = "SELECT DISTINCT id FROM md5_annotation WHERE md5=$md5 AND source=$srcid";
}

if ($type ne 'md5') {
$ann = $mgdb->_dbh->selectcol_arrayref($sql);
# remove non-matching annotations if using filter without hierarchal level
if ($filter && (! $flevel)) {
my @matches = grep {/$filter/} @$ann;
@$ann = @matches;
}
if (@$ann == 0) { next; }
}

# pull data from indexed shock file
my $rec = $self->get_shock_file($node_id, undef, $self->mgrast_token, 'seek='.$seek.'&length='.$len);
chomp $rec;
foreach my $line ( split(/\n/, $rec) ) {
my @tabs = split(/\t/, $line);
if ($tabs[0]) {
my @out = ();
my $rid = $hs->parse($tabs[0]);
unless ($mgid && $rid) {
next;
}
$hs->eof;
if (($format eq 'sequence') && (@tabs == 13)) {
@out = ('mgm'.$mgid."|".$rid, $tabs[1], $tabs[12], join(";", @$ann));
} elsif ($format eq 'similarity') {
@out = ('mgm'.$mgid."|".$rid, @tabs[1..11], join(";", @$ann));
$count += 1;
}
if ($type eq 'md5') {
pop @out;
}
print join("\t", map {$_ || ''} @out)."\n";
$count += 1;
}
my ($md5, $seek, $len) = @row;
push @md5s, $md5;
push @seeks, $seek;
push @lens, $len;
$batch_count++;
if($batch_count == 1000) {
my $solr_query_str = "(source_id:$srcid) AND (md5_id:(".join(" OR ", @md5s)."))";
$count = $self->print_batch($count, $node_id, $format, $mgid, $solr_query_str, $mgdb->_version, \@md5s, \@seeks, \@lens);
@md5s = ();
@seeks = ();
@lens = ();
$batch_count = 0;
}
}

if($batch_count > 0) {
my $solr_query_str = "(source_id:$srcid) AND (md5_id:(".join(" OR ", @md5s)."))";
$count = $self->print_batch($count, $node_id, $format, $mgid, $solr_query_str, \@md5s, \@seeks, \@lens);
}

# cleanup
$sth->finish;
$mgdb->_dbh->commit;
print "Download complete. $count rows retrieved\n";
exit 0;
}

sub print_batch {
my ($self, $count, $node_id, $format, $mgid, $solr_query_str, $ann_ver, $md5s, $seeks, $lens) = @_;

my $cgi = $self->cgi;
my $type = $cgi->param('type') ? $cgi->param('type') : 'organism';
my $filter = $cgi->param('filter') || undef;
my $flevel = $cgi->param('filter_level') || undef;
my $fields = ();
if ($type eq 'organism') {
if ($filter && $flevel) {
$fields = ['organism', 'md5_id', $flevel];
$solr_query_str .= " AND ($flevel:$filter)";
} else {
$fields = ['organism', 'md5_id'];
}
} elsif ($type eq 'function') {
$fields = ['function', 'md5_id'];
} elsif ($type eq 'ontology') {
if ($filter && $flevel) {
$fields = ['organism', 'md5_id', $flevel];
$solr_query_str .= " AND ($flevel:$filter)";
} else {
$fields = ['organism', 'md5_id'];
}
} elsif ($type eq 'feature') {
$fields = ['md5_id'];
}

my ($data, $row_count) = $self->get_solr_query("POST", $Conf::m5nr_solr, $Conf::m5nr_collect.'_'.$ann_ver, $solr_query_str, "", 0, 1000000000, $fields);
my %md5s_to_annot = ();
if ($type ne 'md5') {
foreach my $result (@$data) {
$md5s_to_annot{$result->{md5_id}}{$result->{$type}} = 1;
}
}

my $hs = HTML::Strip->new();
for(my $i=0; $i<@{$md5s}; $i++) {
if ($type ne 'md5' && !exists $md5s_to_annot{$md5s->[$i]}) {
next;
}
# pull data from indexed shock file
my $rec = $self->get_shock_file($node_id, undef, $self->mgrast_token, 'seek='.$seeks->[$i].'&length='.$lens->[$i]);
chomp $rec;
foreach my $line (split(/\n/, $rec)) {
my @tabs = split(/\t/, $line);
if ($tabs[0]) {
my @out = ();
my $rid = $hs->parse($tabs[0]);
unless ($mgid && $rid) {
next;
}
$hs->eof;
my $ann = [];
foreach my $key (keys %{$md5s_to_annot{$md5s->[$i]}}) {
push @$ann, $key;
}
if ($type ne 'md5') {
if ($filter && (! $flevel)) {
my @matches = grep {/$filter/} @$ann;
@$ann = @matches;
}
if (@$ann == 0) { next; }
}
if (($format eq 'sequence') && (@tabs == 13)) {
@out = ('mgm'.$mgid."|".$rid, $tabs[1], $tabs[12], join(";", @$ann));
} elsif ($format eq 'similarity') {
@out = ('mgm'.$mgid."|".$rid, @tabs[1..11], join(";", @$ann));
$count += 1;
}
if ($type eq 'md5') {
pop @out;
}
print join("\t", map {$_ || ''} @out)."\n";
$count += 1;
}
}
}
return $count;
}

1;
45 changes: 45 additions & 0 deletions src/MGRAST/lib/resources/project.pm
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,51 @@ sub instance {

# get database
my $master = $self->connect_to_datasource();

# check if this is an action request
my $requests = { 'movemetagenomes' => 1 };
if (scalar(@$rest) > 1 && $requests->{$rest->[1]}) {
# move metagenomes to a different project
if ($rest->[1] eq 'movemetagenomes') {
unless ($self->user->has_star_right('edit', 'user')) {
$self->return_data( {"ERROR" => "insufficient permissions for this user call"}, 401 );
}
my ($id2) = $self->cgi->param('target') =~ /^mgp(\d+)$/;
if (! $id2) {
$self->return_data( {"ERROR" => "invalid id format: " . $self->cgi->param('target')}, 400 );
}
my $project_a = $master->Project->init( {id => $id} );
my $project_b = $master->Project->init( {id => $id2} );

unless (ref($project_a) && ref($project_b)) {
$self->return_data( {"ERROR" => "id $id or $id2 does not exists"}, 404 );
}

my $proj_job = $master->ProjectJob->get_objects({ project => $project_a });
my $job_a_hash = {};
foreach my $j (@$proj_job) {
$job_a_hash->{$j->job->{metagenome_id}} = $j->job;
}
my @move_over = $self->cgi->param("move");
foreach my $m (@move_over) {
$m =~ s/^mgm//;
unless ($job_a_hash->{$m}) {
$self->return_data( {"ERROR" => "metagenome not part of source project: " . $m}, 400 );
}
}

foreach my $m (@move_over) {
my $job = $master->Job->get_objects( { metagenome_id => $m });
if (scalar(@$job)) {
$job = $job->[0];
$project_a->remove_job($job);
$project_b->add_job($job);
}
}

$self->return_data( {"OK" => "metagenomes moved"}, 200 );
}
}

# get data
my $project = $master->Project->init( {id => $id} );
Expand Down
2 changes: 1 addition & 1 deletion src/WebApplication/WebServerBackend/ObjectBase.pm
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ sub attributes {
return {
firstname => [ DB_SCALAR, undef, 0, undef ],
email => [ DB_SCALAR, undef, 1, undef ],
email2 => [ DB_SCALAR, undef, 1, undef ],
email2 => [ DB_SCALAR, undef, 0, undef ],
password => [ DB_SCALAR, undef, 0, undef ],
comment => [ DB_SCALAR, undef, 0, undef ],
entry_date => [ DB_SCALAR, undef, 0, undef ],
Expand Down