Your C++ function implementation does not match the expectations of the C# function declaration.
The C# byte array is marshalled into the function as a pinned pointer to the array's raw memory. The syntax you are using in the C++ code for the parameter (std::byte buf[256]) is just syntax sugar, the compiler actually treats it as a pointer (std::byte* buf). Which is fine in this situation. However, your C++ function is not actually copying anything into the memory that the pointer is pointing at. You are simply changing the pointer itself to point at a different memory address. And the pointer itself is a local variable, so when the function exits, the pointer will no longer exist, and it won't matter what it is pointing at.
Also, the C# declaration is expecting the function to return something that can be marshalled to a .NET string, but the C++ function is not actually return'ing anything at all. The default marshalling behavior for a string return value is as UnmanagedType.LPStr, so the C++ function needs to return a pointer to a null-terminated char* string. The memory for that string must be allocated with CoTaskMemAlloc(), as the marshaller will take ownership of the memory and free it with CoTaskMemFree() after converting the char data to a string.
Also, the C++ function has no calling convention defined, so it is going to use the compiler's default (which is usually __cdecl, unless you change your compiler's settings). However, the default calling convention that .NET's DllImport expects the C++ function to use is __stdcall instead (for compatibility with Win32 APIs). This mismatch won't matter in 64bit, but it matters alot in 32bit.
Without changing your C# declaration, try this on the C++ side:
char* __stdcall Name(std::byte buf[256])
{
std::string s{ "test" };
size_t size = s.size() + 1;
memcpy(buf, s.c_str(), size);
void *ptr = CoTaskMemAlloc(size);
memcpy(ptr, s.c_str(), size);
return ptr;
}
That being said, it is weird to have a function that returns a string in both a byte array output parameter and in a return value. Are you sure the byte array is not meant to be used as an input parameter instead, where the function parses its content to extract amd a string? That would make more sense. It would really help if you would update your question to show how your .NET code is actually calling the C++ function and using the byte array.
