Iterate over vec of iterators in parallel

Question

Say I've got a Vec that looks something like [[1,2,3],[4,5,6],[7,8,9]]. How would I iterate over all the inner vecs at once? That is,

for k in some_func([[1,2,3],[4,5,6],[7,8,9]]) {
    println!("{k:?}")
}
=> [1,4,7],[2,5,8],[3,6,9]

I can manually construct the transpose, but that only works if the inner iterators are finite. How would I do this for any iterator?

Edit: more scientifically, I want a version of

fn parallel_iterate<T>(items: Vec<Vec<T>>) -> Vec<Vec<T>>
where
    T: Clone,
{
    let size = items.iter().map(|row| row.len()).min().unwrap_or(0);
    let mut result = Vec::with_capacity(size);
    for index in 0..size {
        result.push(Vec::new());
        for inner in items.iter() {
            result[index].push(inner[index].clone())
        }
    }
    return result;
}

where items is of type Vec<I> and I: IntoIterator<Item = T>

What do you expect to be returned by the iterator?

fdan
– fdan

2025-02-05 21:08:12 +00:00
Commented Feb 5 at 21:08 — fdan
– fdan, Commented Feb 5 at 21:08

prog-fh · Accepted Answer · 2025-02-07 09:51:36Z

3

After the question has been edited, it seems that the number of rows and columns could be arbitrary.
The previous attempt (kept under the below line) considered only a fixed number of rows.

The example provided in the question makes extensive use of vectors, but we can generalise to any kind of iterator.
The main advantage is that we do not need to allocate the transposed result as a whole; we can just consume the partial results on the fly if the whole storage is not necessary.

The idea here is to build our own TransposeIterator structure, keeping the provided row-iterators, in order to consume all of them at each invocation of .next().
The use of IntoIterator instead of Iterator is just a convenience, so that we can directly pass structured data instead of explicitly making iterators at the call site.

mod transpose {
    pub struct TransposeIterator<I: IntoIterator> {
        iters: Vec<I::IntoIter>,
    }
    impl<I: IntoIterator> TransposeIterator<I> {
        pub fn new(iters: impl IntoIterator<Item = I>) -> Self {
            let iters = iters.into_iter().map(|i| i.into_iter());
            Self {
                iters: Vec::from_iter(iters),
            }
        }
    }
    impl<I: IntoIterator> Iterator for TransposeIterator<I> {
        type Item = Vec<I::Item>;

        fn next(&mut self) -> Option<Self::Item> {
            let mut result = Vec::with_capacity(self.iters.len());
            for it in self.iters.iter_mut() {
                if let Some(elem) = it.next() {
                    result.push(elem);
                } else {
                    return None;
                }
            }
            Some(result)
        }
    }
}

fn main() {
    use transpose::TransposeIterator;
    let mut data =
        vec![[11, 12, 13], [21, 22, 23], [31, 32, 33], [41, 42, 43]];
    for row in data.iter() {
        println!("{:?}", row);
    }
    println!("~~~~ consumes original data ~~~~");
    for row in TransposeIterator::new(data.clone()) {
        println!("{:?}", row);
    }
    println!("~~~~ references original data ~~~~");
    for row in TransposeIterator::new(&data) {
        println!("{:?}", row);
    }
    println!("~~~~ mutably references original data ~~~~");
    for mut row in TransposeIterator::new(&mut data) {
        for elem in row.iter_mut() {
            **elem += 900;
        }
        println!("{:?}", row);
    }
}
/*
[11, 12, 13]
[21, 22, 23]
[31, 32, 33]
[41, 42, 43]
~~~~ consumes original data ~~~~
[11, 21, 31, 41]
[12, 22, 32, 42]
[13, 23, 33, 43]
~~~~ references original data ~~~~
[11, 21, 31, 41]
[12, 22, 32, 42]
[13, 23, 33, 43]
~~~~ mutably references original data ~~~~
[911, 921, 931, 941]
[912, 922, 932, 942]
[913, 923, 933, 943]
*/

Initial answer

I would do it with two nested invocations of std::iter::zip().

The best solution is certainly itertools::multizip() (it yields tuples, not arrays, I guess this is not important).

fn some_func<I: IntoIterator>(
    i0: I,
    i1: I,
    i2: I,
) -> impl Iterator<Item = [I::Item; 3]> {
    std::iter::zip(i0, std::iter::zip(i1, i2))
        .map(|(e1, (e2, e3))| [e1, e2, e3])
}

fn main() {
    for k in std::iter::zip([1, 2, 3], std::iter::zip([4, 5, 6], [7, 8, 9])) {
        println!("{k:?}")
    }
    println!("~~~~ consumes original data ~~~~~~~~~~~~");
    for k in some_func([1, 2, 3], [4, 5, 6], [7, 8, 9]) {
        println!("{k:?}")
    }
    println!("~~~~ references original data ~~~~~~~~~~~~");
    for k in some_func(&[1, 2], &[4, 5], &[7, 8]) {
        println!("{k:?}")
    }
    println!("~~~~ consumes original data ~~~~~~~~~~~~");
    for k in itertools::multizip(([1, 2, 3], [4, 5, 6], [7, 8, 9])) {
        println!("{k:?}")
    }
    println!("~~~~ references original data ~~~~~~~~~~~~");
    for k in itertools::multizip((&[1, 2], &[4, 5], &[7, 8])) {
        println!("{k:?}")
    }
}
/*
(1, (4, 7))
(2, (5, 8))
(3, (6, 9))
~~~~ consumes original data ~~~~~~~~~~~~
[1, 4, 7]
[2, 5, 8]
[3, 6, 9]
~~~~ references original data ~~~~~~~~~~~~
[1, 4, 7]
[2, 5, 8]
~~~~ consumes original data ~~~~~~~~~~~~
(1, 4, 7)
(2, 5, 8)
(3, 6, 9)
~~~~ references original data ~~~~~~~~~~~~
(1, 4, 7)
(2, 5, 8)
*/

edited Feb 7 at 9:51

answered Feb 5 at 11:40

prog-fh

17.3k1 gold badge21 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Vessel Feb 5 at 11:51

That only works for vecs where the inner iterators are slices of exactly three elements - I want a general solution that works on any kind of inner iterator

prog-fh Feb 5 at 13:58

@Vessel Could you edit your question and explain exactly what is expected. The provided example shows arrays, but now you want vectors, that's not clear at all.

user4815162342 Feb 7 at 12:34

@prog-fh Thanks for the edit. It's my view that Vec is so central to Rust that using it is a reasonable default rather than a mere implementation detail.

user4815162342 Feb 7 at 12:35

@prog-fh Reusing an internally stored vector would indeed require a lending iterator. While those are possible to implement using GATs (e.g. here), you get zero interoperability with real iterators. Additionally, third-party crates that define lending iterators never gained much traction for reasons I don't fully understand, but strongly suspect boil down to some limitations when you start using them. In summary, I unfortunately cannot provide actionable advice that would improve the answer in that direction.

user4815162342 Feb 7 at 12:50

@prog-fh Here is a proof of concept that uses gat-lending-iterator to define a lending version of TransposeIterator. (It won't run on the playground because it requires the external crate, just using playground for sharing.) I couldn't get the referencing versions to work, not sure if that's due to a fundamental limitation of lending iterators as defined by that crate or it can be fixed.

|

user4815162342 · Accepted Answer · 2025-02-07 07:25:33Z

1

This satisfies the signature of the edited question:

pub fn parallel_iterate<T>(into_iters: Vec<impl IntoIterator<Item = T>>) -> Vec<Vec<T>> {
    let mut iters: Vec<_> = into_iters
        .into_iter()
        .map(IntoIterator::into_iter)
        .collect();
    let mut result = vec![];
    'outer: loop {
        let mut inner_result = Vec::with_capacity(iters.len());
        for inner_iter in &mut iters {
            let Some(item) = inner_iter.next() else {
                break 'outer;
            };
            inner_result.push(item);
        }
        result.push(inner_result);
    }
    result
}

Playground

edited Feb 7 at 7:25

answered Feb 6 at 9:11

user4815162342

159k22 gold badges351 silver badges419 bronze badges

Comments

BallpointBen · Accepted Answer · 2025-02-05 15:39:50Z

0

Something like this should work.

fn zip_iters<T>(
    v: &mut [impl Iterator<Item = T>],
) -> impl '_ + Iterator<Item = Vec<T>> {
    std::iter::repeat(()).map_while(|()| {
        v.iter_mut().map(|it| it.next()).collect::<Option<Vec<_>>>()
    })
}

fn main() {
    let mut v = [
        // squares of even numbers
        Box::new((0_i32..).filter_map(|i| (i % 2 == 0).then_some(i.pow(2))))
            as Box<dyn Iterator<Item = i32>>,
        // 0..=4
        Box::new((0_i32..).take_while(|&i| i < 5)) as _,
    ];

    println!("{:?}", zip_iters(&mut v).collect::<Vec<_>>());
    // [[0, 0], [4, 1], [16, 2], [36, 3], [64, 4]]
}

answered Feb 5 at 15:39

BallpointBen

15.7k2 gold badges46 silver badges81 bronze badges

Comments

Ron Slosberg · Accepted Answer · 2025-02-05 20:00:56Z

0

How about this?

use core::panic;

fn flatten_vectors<T: Copy>(vectors: Vec<Vec<T>>) -> Vec<Vec<T>> {
    let mut result: Vec<Vec<T>> = Vec::new();
    let max_len = vectors.iter().map(|v| v.len()).max().unwrap_or(0);

    for i in 0..max_len {
        let mut inner = Vec::<T>::new();
        for vec in &vectors {
            if let Some(&val) = vec.get(i) {
                inner.push(val);
            }
            else {
                panic!("Index out of bounds");
            }
        }
        result.push(inner);
    }

    result
}

fn main() {
    let a = [[1, 2, 3], [4, 5, 6], [7, 8, 9]];
    let v: Vec<Vec<i32>> = a.iter().map(|a| a.to_vec()).collect();
    let flattened = flatten_vectors(v);
    println!("{:?}", flattened);
}

-> [[1, 4, 7], [2, 5, 8], [3, 6, 9]]

Playground

edited Feb 5 at 20:00

answered Feb 5 at 19:29

Ron Slosberg

275 bronze badges

1 Comment

Jeremy Caney Feb 7 at 20:06

Thank you for contributing to the Stack Overflow community. This may be a correct answer, but it’d be really useful to provide additional explanation of your code so developers can understand your reasoning. This is especially useful for new developers who aren’t as familiar with the syntax or struggling to understand the concepts. Would you kindly edit your answer to include additional details for the benefit of the community?

Collectives™ on Stack Overflow

Iterate over vec of iterators in parallel

4 Answers 4

9 Comments

Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

9 Comments

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related