…ytorch#166846)

**Summary**
This PR adds an AVX512-VNNI-based micro kernel for u8s8s32 in CPP GEMM template. It can be chosen to construct a GEMM kernel for hardware platforms that support AVX512_VNNI but not AMX. Without this feature, only the aten `qlinear` op is available to compute u8s8s32 GEMM (or fall back to reference implementation, which is super slow). The new microkernel brings performance gain over the aten `qlinear` op when M is small (see performance data below).
On platforms that support both AVX512_VNNI and AMX, AMX is preferred regardless of input shapes. This ensures there won't be performance regression on such platforms. We can add heuristics to select from AVX512_VNNI and AMX in the future if we need.
Note that this PR only adds a new microkernel. It does not change the outer loops and blockings of CPP GEMM template.

We found block_m=6 and block_n=64 is the best by experiments. OneDNN also uses such blocking strategy.

**Performance benchmark**
We collected performance data on an Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with the following script.
```python
import torch
import torchao
import copy
import os
import itertools

import torch._inductor.config as config
config.freezing = True
config.max_autotune_gemm_backends = "CPP"
config.cpp_wrapper = True
config.cpp.enable_kernel_profile = True

from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e
import torchao.quantization.pt2e.quantizer.x86_inductor_quantizer as xiq
from torchao.quantization.pt2e.quantizer.x86_inductor_quantizer import X86InductorQuantizer
from torchao.quantization.pt2e import move_exported_model_to_eval

def pt2e_ptq(m, example_inputs):
    m = m.eval()
    exported_model = torch.export.export(m, example_inputs, strict=True).module()
    quantizer = X86InductorQuantizer()
    quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())

    with torch.no_grad():
        prepared_model = prepare_pt2e(exported_model, quantizer)
        _ = prepared_model(*example_inputs)
        converted_model = convert_pt2e(prepared_model)
        move_exported_model_to_eval(converted_model)
        optimized_model = torch.compile(converted_model)
        optimized_model(*example_inputs)
        return optimized_model

def benchmark(model, inputs):
    import time
    warmup, active = 100, 1000
    with torch.no_grad():
        for i in range(warmup):
            model(*inputs)
        t0 = time.time()
        for i in range(active):
            model(*inputs)
        te = time.time() - t0
        print("Time per iteration:", round(te * 1000 / active, 3), "ms")

in1, out1 = 1024, 1024
in2, out2 = 1024, 1024

class Mod(torch.nn.Module):
    def __init__(self, bias=True):
        super().__init__()
        self.linear1 = torch.nn.Linear(in1, out1, bias=bias)
        self.relu = torch.nn.ReLU()
        self.linear2 = torch.nn.Linear(in2, out2, bias=(not bias))

    def forward(self, x):
        return self.linear2(self.relu(self.linear1(x)))

    def get_example_input(self, M=1):
        return torch.randn(M, in1)

if __name__ == "__main__":
    use_max_autotune_list = [True, False]
    M_list = [1, 4, 32, 128, 256]
    cases = itertools.product(use_max_autotune_list, M_list)
    for use_max_autotune, M in cases:
        config.max_autotune = use_max_autotune
        model_fp = Mod().eval()
        data = model_fp.get_example_input(M)
        inputs = (data,)
        m = pt2e_ptq(copy.deepcopy(model_fp), inputs)
        print("[TEST INFO] Using GEMM template:", use_max_autotune, ", M:", M, ", num of cores:", len(os.sched_getaffinity(0)))
        benchmark(m, inputs)
```

Command to run:
```
# 1 core
numactl -C0 python benchmark_vnni_microkernel.py
# 4 cores
numactl -C0-3 python benchmark_vnni_microkernel.py
```

Results:

Num of Cores | M | ATEN (ms) | CPP (ms) | Improve
-- | -- | -- | -- | --
1 | 1 | 0.2 | 0.128 | 36.00%
1 | 4 | 0.202 | 0.132 | 34.65%
1 | 32 | 0.358 | 0.298 | 16.76%
1 | 128 | 0.898 | 0.882 | 1.78%
1 | 256 | 1.613 | 1.656 | -2.67%
4 | 1 | 0.108 | 0.058 | 46.30%
4 | 4 | 0.112 | 0.054 | 51.79%
4 | 32 | 0.165 | 0.109 | 33.94%
4 | 128 | 0.352 | 0.306 | 13.07%
4 | 256 | 0.59 | 0.573 | 2.88%

**Test plan**
```
python -m pytest  -sv test/inductor/test_cpu_select_algorithm.py -k test_quantized_linear_with_pointwise
```

Pull Request resolved: pytorch#166846
Approved by: https://github.com/CaoE, https://github.com/jansel

…166846)

**Summary**
This PR adds an AVX512-VNNI-based micro kernel for u8s8s32 in CPP GEMM template. It can be chosen to construct a GEMM kernel for hardware platforms that support AVX512_VNNI but not AMX. Without this feature, only the aten `qlinear` op is available to compute u8s8s32 GEMM (or fall back to reference implementation, which is super slow). The new microkernel brings performance gain over the aten `qlinear` op when M is small (see performance data below).
On platforms that support both AVX512_VNNI and AMX, AMX is preferred regardless of input shapes. This ensures there won't be performance regression on such platforms. We can add heuristics to select from AVX512_VNNI and AMX in the future if we need.
Note that this PR only adds a new microkernel. It does not change the outer loops and blockings of CPP GEMM template.

We found block_m=6 and block_n=64 is the best by experiments. OneDNN also uses such blocking strategy.

**Performance benchmark**
We collected performance data on an Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with the following script.
```python
import torch
import torchao
import copy
import os
import itertools

import torch._inductor.config as config
config.freezing = True
config.max_autotune_gemm_backends = "CPP"
config.cpp_wrapper = True
config.cpp.enable_kernel_profile = True

from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e
import torchao.quantization.pt2e.quantizer.x86_inductor_quantizer as xiq
from torchao.quantization.pt2e.quantizer.x86_inductor_quantizer import X86InductorQuantizer
from torchao.quantization.pt2e import move_exported_model_to_eval

def pt2e_ptq(m, example_inputs):
    m = m.eval()
    exported_model = torch.export.export(m, example_inputs, strict=True).module()
    quantizer = X86InductorQuantizer()
    quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())

    with torch.no_grad():
        prepared_model = prepare_pt2e(exported_model, quantizer)
        _ = prepared_model(*example_inputs)
        converted_model = convert_pt2e(prepared_model)
        move_exported_model_to_eval(converted_model)
        optimized_model = torch.compile(converted_model)
        optimized_model(*example_inputs)
        return optimized_model

def benchmark(model, inputs):
    import time
    warmup, active = 100, 1000
    with torch.no_grad():
        for i in range(warmup):
            model(*inputs)
        t0 = time.time()
        for i in range(active):
            model(*inputs)
        te = time.time() - t0
        print("Time per iteration:", round(te * 1000 / active, 3), "ms")

in1, out1 = 1024, 1024
in2, out2 = 1024, 1024

class Mod(torch.nn.Module):
    def __init__(self, bias=True):
        super().__init__()
        self.linear1 = torch.nn.Linear(in1, out1, bias=bias)
        self.relu = torch.nn.ReLU()
        self.linear2 = torch.nn.Linear(in2, out2, bias=(not bias))

    def forward(self, x):
        return self.linear2(self.relu(self.linear1(x)))

    def get_example_input(self, M=1):
        return torch.randn(M, in1)

if __name__ == "__main__":
    use_max_autotune_list = [True, False]
    M_list = [1, 4, 32, 128, 256]
    cases = itertools.product(use_max_autotune_list, M_list)
    for use_max_autotune, M in cases:
        config.max_autotune = use_max_autotune
        model_fp = Mod().eval()
        data = model_fp.get_example_input(M)
        inputs = (data,)
        m = pt2e_ptq(copy.deepcopy(model_fp), inputs)
        print("[TEST INFO] Using GEMM template:", use_max_autotune, ", M:", M, ", num of cores:", len(os.sched_getaffinity(0)))
        benchmark(m, inputs)
```

Command to run:
```
# 1 core
numactl -C0 python benchmark_vnni_microkernel.py
# 4 cores
numactl -C0-3 python benchmark_vnni_microkernel.py
```

Results:

Num of Cores | M | ATEN (ms) | CPP (ms) | Improve
-- | -- | -- | -- | --
1 | 1 | 0.2 | 0.128 | 36.00%
1 | 4 | 0.202 | 0.132 | 34.65%
1 | 32 | 0.358 | 0.298 | 16.76%
1 | 128 | 0.898 | 0.882 | 1.78%
1 | 256 | 1.613 | 1.656 | -2.67%
4 | 1 | 0.108 | 0.058 | 46.30%
4 | 4 | 0.112 | 0.054 | 51.79%
4 | 32 | 0.165 | 0.109 | 33.94%
4 | 128 | 0.352 | 0.306 | 13.07%
4 | 256 | 0.59 | 0.573 | 2.88%

**Test plan**
```
python -m pytest  -sv test/inductor/test_cpu_select_algorithm.py -k test_quantized_linear_with_pointwise
```

Pull Request resolved: #166846
Approved by: https://github.com/CaoE, https://github.com/jansel

…ytorch#166846)

**Summary**
This PR adds an AVX512-VNNI-based micro kernel for u8s8s32 in CPP GEMM template. It can be chosen to construct a GEMM kernel for hardware platforms that support AVX512_VNNI but not AMX. Without this feature, only the aten `qlinear` op is available to compute u8s8s32 GEMM (or fall back to reference implementation, which is super slow). The new microkernel brings performance gain over the aten `qlinear` op when M is small (see performance data below).
On platforms that support both AVX512_VNNI and AMX, AMX is preferred regardless of input shapes. This ensures there won't be performance regression on such platforms. We can add heuristics to select from AVX512_VNNI and AMX in the future if we need.
Note that this PR only adds a new microkernel. It does not change the outer loops and blockings of CPP GEMM template.

We found block_m=6 and block_n=64 is the best by experiments. OneDNN also uses such blocking strategy.

**Performance benchmark**
We collected performance data on an Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with the following script.
```python
import torch
import torchao
import copy
import os
import itertools

import torch._inductor.config as config
config.freezing = True
config.max_autotune_gemm_backends = "CPP"
config.cpp_wrapper = True
config.cpp.enable_kernel_profile = True

from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e
import torchao.quantization.pt2e.quantizer.x86_inductor_quantizer as xiq
from torchao.quantization.pt2e.quantizer.x86_inductor_quantizer import X86InductorQuantizer
from torchao.quantization.pt2e import move_exported_model_to_eval

def pt2e_ptq(m, example_inputs):
    m = m.eval()
    exported_model = torch.export.export(m, example_inputs, strict=True).module()
    quantizer = X86InductorQuantizer()
    quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())

    with torch.no_grad():
        prepared_model = prepare_pt2e(exported_model, quantizer)
        _ = prepared_model(*example_inputs)
        converted_model = convert_pt2e(prepared_model)
        move_exported_model_to_eval(converted_model)
        optimized_model = torch.compile(converted_model)
        optimized_model(*example_inputs)
        return optimized_model

def benchmark(model, inputs):
    import time
    warmup, active = 100, 1000
    with torch.no_grad():
        for i in range(warmup):
            model(*inputs)
        t0 = time.time()
        for i in range(active):
            model(*inputs)
        te = time.time() - t0
        print("Time per iteration:", round(te * 1000 / active, 3), "ms")

in1, out1 = 1024, 1024
in2, out2 = 1024, 1024

class Mod(torch.nn.Module):
    def __init__(self, bias=True):
        super().__init__()
        self.linear1 = torch.nn.Linear(in1, out1, bias=bias)
        self.relu = torch.nn.ReLU()
        self.linear2 = torch.nn.Linear(in2, out2, bias=(not bias))

    def forward(self, x):
        return self.linear2(self.relu(self.linear1(x)))

    def get_example_input(self, M=1):
        return torch.randn(M, in1)

if __name__ == "__main__":
    use_max_autotune_list = [True, False]
    M_list = [1, 4, 32, 128, 256]
    cases = itertools.product(use_max_autotune_list, M_list)
    for use_max_autotune, M in cases:
        config.max_autotune = use_max_autotune
        model_fp = Mod().eval()
        data = model_fp.get_example_input(M)
        inputs = (data,)
        m = pt2e_ptq(copy.deepcopy(model_fp), inputs)
        print("[TEST INFO] Using GEMM template:", use_max_autotune, ", M:", M, ", num of cores:", len(os.sched_getaffinity(0)))
        benchmark(m, inputs)
```

Command to run:
```
# 1 core
numactl -C0 python benchmark_vnni_microkernel.py
# 4 cores
numactl -C0-3 python benchmark_vnni_microkernel.py
```

Results:

Num of Cores | M | ATEN (ms) | CPP (ms) | Improve
-- | -- | -- | -- | --
1 | 1 | 0.2 | 0.128 | 36.00%
1 | 4 | 0.202 | 0.132 | 34.65%
1 | 32 | 0.358 | 0.298 | 16.76%
1 | 128 | 0.898 | 0.882 | 1.78%
1 | 256 | 1.613 | 1.656 | -2.67%
4 | 1 | 0.108 | 0.058 | 46.30%
4 | 4 | 0.112 | 0.054 | 51.79%
4 | 32 | 0.165 | 0.109 | 33.94%
4 | 128 | 0.352 | 0.306 | 13.07%
4 | 256 | 0.59 | 0.573 | 2.88%

**Test plan**
```
python -m pytest  -sv test/inductor/test_cpu_select_algorithm.py -k test_quantized_linear_with_pointwise
```

Pull Request resolved: pytorch#166846
Approved by: https://github.com/CaoE, https://github.com/jansel