Avoid List copy between R and Rcpp

Question

Let's say I have a List with a few elements (< 20) but each of them is a vector (IntegerVector, Numericvector or CharacterVector) of several GB. I therefor want to avoid any copy.

To remove element from my List, I write the following Rcpp code :

void list_remove_element (List x, int i) {
  Rcout << "Size before : " << x.size() << endl;
  x.erase(i);
  Rcout << "Size after : " <<x.size() << endl;
}

Internaly, this code effectively erase the corresponding. Sadly after the return of this function no change appears in R :

> u = list(a=1:5, b=3:4, c=5:6)
> list_remove_elements (u, 1)
Size before : 3
Size after : 2
> str(u)
List of 3
 $ a: int [1:5] 1 2 3 4 5
 $ b: int [1:2] 3 4
 $ c: int [1:2] 5 6

As far as I have understood, using a function to grow or shrink an Rcpp object results of a data copy from the original object into a new object. Is there any solution to avoid this?

EDIT :

I also tried to do the following :

void list_remove_elements (SEXP x) {
  SET_VECTOR_ELT(x, 1, R_NilValue);
}

It almost works since I get :

> str(u)
List of 3
 $ a: int [1:5] 1 2 3 4 5
 $ b: NULL
 $ c: int [1:2] 5 6

But I still have the element 'b' and not sure this is the correct way to do ...

Not with SEXP types which are always contiguous vectors. Now, a list of vectors does not copy its vectors. Only the top-level list is re-created. — Dirk is no longer here
– Dirk is no longer here, Commented Sep 19, 2017 at 14:08
See the Writing R Extensions manual and its hints about memory profiling. That is probably what you want. — Dirk is no longer here
– Dirk is no longer here, Commented Sep 19, 2017 at 14:09
The point is that I have a lot of other functions that directly modify the argument. I then do not use return value and I would like for this function the same behavior (for consistency purpose). Let's say I have a my list A. I use my other functions like this : fct1(A); fct2(A) ; And I don't really want to do the following : fct1(A); A = fct2 (A) ; But maybe I misunderstand your answers ... — Wil Mon
– Wil Mon, Commented Sep 19, 2017 at 15:34

Romain Francois · Accepted Answer · 2017-09-23 07:42:04Z

1

Creating a new list with some elements from an existing list is not making copies of the content, i.e.

> data <- list( x = rnorm(1e6), y = rnorm(1e6), z = rnorm(1e6) ).   
> pryr::object_size(data)
24 MB
> data2 <- data[ c("x", "y") ]
> pryr::object_size(data2)
16 MB

But data and data2 share their memory

> pryr::object_size(data, data2)
24 MB

The memory would only be copied if your make changes to say data2$x

> data2$x[1] <- 12
> pryr::object_size( data, data2 )
32 MB

In the original list_remove_element function you just have to return x this will be a new list:

#include <Rcpp.h>
using namespace Rcpp;

// [[Rcpp::export]]
List list_remove_element (List x, int i) {
  x.erase(i);
  return x ;
}


/*** R
  data <- list( x = rnorm(1e6), y = rnorm(1e6), z = rnorm(1e6) )
  data2 <- list_remove_element(data, 1)

  pryr::object_size(data)
  pryr::object_size(data2)
  pryr::object_size(data, data2)
*/

which gives:

> Rcpp::sourceCpp('~/Desktop/test.cpp')
>   data <- list( x = rnorm(1e6), y = rnorm(1e6), z = rnorm(1e6) )
>   data2 <- list_remove_element(data, 1)
>   pryr::object_size(data)
24 MB
>   pryr::object_size(data2)
16 MB
>   pryr::object_size(data, data2)
24 MB

answered Sep 23, 2017 at 7:42

Romain Francois

17.7k3 gold badges54 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

thc Over a year ago

Nice answer! Did not think that memory sharing would be preserved after a return from Rcpp.

Stefanos · Accepted Answer · 2017-09-24 00:07:01Z

-1

If you rename your parameter to 'List& x', then you should probably get the desired result.

Like below:

void list_remove_element (List& x, int i) {
  Rcout << "Size before : " << x.size() << endl;
  x.erase(i);
  Rcout << "Size after : " <<x.size() << endl;
}

answered Sep 24, 2017 at 0:07

Stefanos

92912 silver badges21 bronze badges

Collectives™ on Stack Overflow

Avoid List copy between R and Rcpp

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related