2

I am trying to implement a kind of simple processing pipeline in Go, where each processor has a determined input and output type, and a list of successor processors that take current processor output type as input, and may have successor processors of their own.

I am running into issues on how to add successor processors to the current one, regardless of their output type. I tried using any as a wildcard type like I would do with ? in Java, but Go is not having it.

What I have in Go is this:

type Processor[InputType any, OutputType any] struct {
    nextProcessors     []*Processor[OutputType, any]
    ProcessingFunction func(InputType) OutputType
}

func (b *Processor[InputType, OutputType]) Process(input InputType) {
    result := b.ProcessingFunction(input)
    for _, nextProcessor := range b.nextProcessors {
        nextProcessor.Process(result)
    }
}

func (b *Processor[InputType, OutputType]) AddOutputProcessor(p *Processor[OutputType, any]) {
    b.nextProcessors = append(b.nextProcessors, p)
}

func main() {
    outputer := Processor[int, string]{ProcessingFunction: func(input int) string {
        print(input)
        return string(input)
    }}
    doubler := Processor[int, int]{ProcessingFunction: func(input int) int { return input * 2 }}
    rng := Processor[int, int]{ProcessingFunction: func(input int) int { return rand.Intn(input) }}
    rng.AddOutputProcessor(&doubler)
    doubler.AddOutputProcessor(&outputer)
    rng.Process(20)
}

Which gives me a compilation error:

cannot use &doubler (value of type *Processor[int, int]) as type *Processor[int, any]

Is there a way to ignore the output type of the successor processor? Or should I maybe go a different way about it? I would just like to make sure that successor processors can accept the right type of input.

For reference, here is the interface definition in Java that works the way I intend it to.

public interface Processor<InputType, OutputType> {
    void addOutputProcessor(Processor<OutputType, ?> outputProcessor);
    void process(InputType input);
}

public class Pipeline {

    private abstract class BaseProcessor<InputType, OutputType>  implements Processor<InputType, OutputType> {

        List<Processor<OutputType, ?>> nextProcessors = new ArrayList<>();

        abstract OutputType processHelper(InputType input);

        @Override
        public void addOutputProcessor(Processor<OutputType, ?> outputProcessor) {
            nextProcessors.add(outputProcessor);
        }

        @Override
        public void process(InputType input) {
            OutputType result = processHelper(input);
            for (Processor<OutputType, ?> nextProcessor : nextProcessors) {
                nextProcessor.process(result);
            }
        }
    }

    private class RandomNumberGenerator extends BaseProcessor<Integer, Integer> {

        @Override
        Integer processHelper(Integer input) {
            int generatedNumber = new Random().nextInt(input);
            return generatedNumber;
        }
    }

    private class IncrementProcessor extends BaseProcessor<Integer, Integer> {

        @Override
        Integer processHelper(Integer input) {
            return input + 1;
        }
    }

    private class DoubleProcessor extends BaseProcessor<Integer, Integer> {

        @Override
        Integer processHelper(Integer input) {
            return input * 2;
        }
    }

    private class Outputer extends BaseProcessor<Integer, String> {

        String name;

        public Outputer(String name) {
            this.name = name;
        }

        @Override
        String processHelper(Integer input) {
            String output = String.format("Pipeline %s result: %d", name, input);
            System.out.println(output);
            return output;
        }
    }

    public void buildAndRunPipeline() {
        Processor<Integer, String> doublingOutputter = new Outputer("Doubling");
        Processor<Integer, String> incrementingOutputter = new Outputer("Incrementing");
        Processor<Integer, String> rngOutputter = new Outputer("Generating number");
        Processor<Integer, Integer> doubler = new DoubleProcessor();
        doubler.addOutputProcessor(doublingOutputter);
        Processor<Integer, Integer> incrementer = new IncrementProcessor();
        incrementer.addOutputProcessor(incrementingOutputter);
        Processor<Integer, Integer> starter = new RandomNumberGenerator();
        starter.addOutputProcessor(rngOutputter);
        starter.addOutputProcessor(doubler);
        starter.addOutputProcessor(incrementer);
        starter.process(20);
    }

    public static void main(String[] args) {
        Pipeline p = new Pipeline();
        p.buildAndRunPipeline();
    }
}

2 Answers 2

2

Is there a way to ignore the output type of the successor processor?

No.

In Go any is just a static type (alias of interface{}. It can never be a replacement for Java's unbounded wildcard ?. So *Processor[int, any] is just not the same type as *Processor[int, int] and you can't assign one to the other, as reported by your error message.

In order to construct an arbitrarily long chain you would need to parametrize the Process method itself, but this is not possible in Go 1.18. You must declare all type parameters on the type itself. Though, even if you do this, you will keep incurring in the same issue of not knowing the output type of the next processor.

Generally speaking, using a for loop can't work because the static types of the in/out values keep changing.

I believe the closest you can get without reflection is to implement some sort of composition operator — like . in haskell, via a top-level function. But you would have to manually nest calls.

A simplified example (the type Processor is redundant, but keeping it closer to your code):

package main

import (
    "fmt"
    "strconv"
)

type Processor[In, Out any] func(In) Out

func Process[In, Out any](input In, processor Processor[In, Out]) Out {
    return processor(input)
}

func main() {
    parser := Processor[string, int](func(input string) int { s, _ := strconv.Atoi(input); return s })
    doubler := Processor[int, int](func(input int) int { return input * 2 })
    outputer := Processor[int, string](func(input int) string { return fmt.Sprintf("%d", input) })

    out := Process(Process(Process("20", parser), doubler), outputer)
    fmt.Println(out)
}

Playground: https://go.dev/play/p/Iv-virKATyb

Sign up to request clarification or add additional context in comments.

Comments

0

You can't use any keyword to instantiate the value of generic type.

nextProcessors     []*Processor[OutputType, any] // keyword any is not a valid type here

You can actually, but the second parameter always should be interface{}. But it's not a part of answer to your question.

To solve your issue you can use generic interface instead

type IProcess[InputType any] interface {
    Process(input InputType)
}

type Processor[InputType any, OutputType any] struct {
    nextProcessors     []IProcess[OutputType]
    ProcessingFunction func(InputType) OutputType
}

func (b *Processor[InputType, OutputType]) AddOutputProcessor(p IProcess[OutputType]) {
    b.nextProcessors = append(b.nextProcessors, p)
}

https://go.dev/play/p/B1wlOvSbb0I

4 Comments

sorry, this is not a good answer: 1) any is not a keyword 2) you didn't explain the error the OP is facing 3) the interface doesn't solve the issue because it doesn't return anything. How would the function Process work?
@blackgreen 1. Yes, any it's a type alias any = interface{} but it doesn't matter in this case. 2. I explain that you can't use any in instantiation, you should use OutputType of previous processor 3. In the question Process function doesn't return anything too. So, the only problem in the question with type instantiation
Thank you for your answer! As I understand it though, that would only allow for two layers of processing, right? I can call the Process function of the IProcessors, but I wouldn't be able to have them have their own successor processors in turn?
@woooosh blackgreen Sorry, my fault. Now I got my mistake. Will check later but looks like for now answer of blackgreen is only way

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.