Repository navigation
Make sure all backends are thread safe #907
Description
Hi team,
Currently I'm saturating my GPU, which is cool. But on the same server I can have more GPU's - eg. an Amazon g2.8xlarge instance has 4 Nvidia K520's. I want to be able to use them too.
I want to be able to launch 4 threads and allocate each one a GPU for them to work on. I was reading the documentation for cudaSetDevice() and I believe it is possible to do this, but I think it requires some work on the ArrayFire memory manager and device manager. I really want to be able to do something like this:
Main thread
(launch # of gpu's of std::threads, eg. 4 threads)
Worker thread
af::setDevice(threadID); // threadID is 0-3, passed in as a parameter by the main thread
Then the rest of the worker thread uses the GPU for the device id it set.
The traditional solution of a cluster of GPU hosts would be ideal, but I'm processing huge amounts of data very fast and distributing the work to multiple hosts adds too much latency. There's just too much data to serialise, transfer and deserialise to make the calculation worthwhile on a cluster. So it needs to be done on the same server.
Currently my only alternative is creating multiple processes o the server each having their dedicated GPU and having 1 coordinator process managing the work and shared memory for the data structures. It can be massively simplified if ArrayFire could support multiple GPU's used through dedicated threads.
Please could you add this to the library?
Activity
pavanky commentedon Jul 14, 2015
@munnybearz We are trying to make the library thread safe. Once this is done, we'll have an example demonstrating what you request.
munnybearz commentedon Jul 14, 2015
Awesome I'm really looking forward to that then!!
[-]Add support for processing on multiple GPU's simultaneously[/-][+]Make sure all backends are thread safe[/+]15 remaining items
vakopian commentedon Nov 23, 2016
@pavanky So if I'm not mistaken the
queue,async_queueand the related code is basically trying to implement a thread pool, right? If that's indeed the case, perhaps this (single header) ThreadPool library could be used instead? I think it's the same ideas and even similar implementation, but in a cleaner and more self-contained class.Another thing I wanted to note here (since you mentioned static variables), I just learned recently that the scoped static variables are actually thread-safe in C++11 (as opposed to older versions of the standard). AF has lots of those in many places, but if we compile with C++11, those don't need to be fixed anymore.
pavanky commentedon Nov 23, 2016
@umar456 is in charge of the threadpool. Umar any thoughts ?
@vakopian
The idea was to have a multi threaded cpu backend eventually. But for now it is single threaded.
As for scoped static variables being thread safe, that will make things a lot easier. However I think we still have a couple of places which use global statics. Though these can be moved into a function that returns the static variable.
@arrayfire/core-devel What are your thoughts on moving all backends to C++-11 (including CUDA). I think CUDA supports C++11 since CUDA 7.0 which makes things easier. Alternatively we can have it so just the
.cppfiles have the static variables and just compile the.cppfiles with C++11 (like we do right now).pavanky commentedon Nov 23, 2016
stackoverflow post talking about scoped static variables:
http://stackoverflow.com/questions/8102125/is-local-static-variable-initialization-thread-safe-in-c11
pavanky commentedon Nov 23, 2016
btw VS 2013 does not support this feature: https://msdn.microsoft.com/en-us/library/hh567368(v=vs.120).aspx. Look under the section called "magic static".
GCC and CLANG (unless Apple dropped it) seem to support it.
VS 2015 also supports it: https://msdn.microsoft.com/en-us/library/hh409293.aspx
Also this seems useful: http://en.cppreference.com/w/cpp/compiler_support
umar456 commentedon Nov 23, 2016
The C++11 standard says that the initialization of magic statics are thread safe. The access and modifications are not.
I fail to see the advantages of the ThreadPool library over the async_queue class. It seems to be a reimplementation of the std::async class. Just like std::async the ThreadPool returns std::futures every time something is queued. This would mean that we will need to keep track of these futures otherwise the operation will be synchronous. If we are keeping track of these futures then we will have another singleton container that will need to be locked every time something is enqueued. Also it looks like it is processing each command on the first available thread. This would mean that we would have to implement a method to define dependencies between operations.
The async_queue class was created to mimic the behavior of command queues in OpenCL and streams in CUDA. We can add concurrency by creating multiple async_queues and queueing tasks based on the thread_id or some other mechanism.
This is also a difficult problem because ArrayFire is essentially a parallelism library meaning that we want all of the CPU resources to be working on one task then another; similar to the OpenCL and CUDA backends. Users using ArrayFire in concurrent threads may hurt performance because each task assumes it has full control of the hardware.
We need ArrayFire to be thread safe in that you should be able to access the arrays from multiple threads. Users should not try to use threads in their code to improve performance by running the same operation on two threads. Tasks should be parallelized to use all of the compute resources of the CPU to speed computations. I have some ideas on how to do something like this but that is not related to this issue.
WilliamTambellini commentedon Nov 24, 2016
Good morning gentlemen,
Is this issue/item likely to be in 3.4.2 or is it really for 3.5 ?
Kindly
pavanky commentedon Nov 24, 2016
@WilliamTambellini I am not sure if this is possible in the timeframe of 3.4.2
vakopian commentedon Dec 6, 2016
I understand what motivates the statement above, but I think it might be missing 2 important scenarios:
Hybrid applications that are very "parallelizable". By "hybrid", I mean applications that do lots of CPU calculations, then lots of GPU (ArrayFire), then more CPU, then more GPU, etc. And by "parallelizable", I mean it can benefit from concurrency not just from GPU, but at CPU level as well. A simple example would be an image processing algorithm that can easily be parallelized by splitting it into tiles and each tile needs both CPU and GPU processing. For such an application, not using threads means not taking advantage of the CPU for some tiles while the GPU is busy processing the other tiles.
Machines with multiple GPU devices. If the application is not designed to be multi-threaded, it would have no way of taking advantage of multiple GPUs.
@umar456, do you agree that for these use cases it's important to write the application as multi-threaded. And if so, shouldn't we make sure that ArrayFire does not prevent, nor create unnecessary bottlenecks for, such usage?
pavanky commentedon Dec 6, 2016
@vakopian
FYI
Should not require multi threading support in individual backends if you are using only one thread per backend. We may need to fix the stuff in
src/api/unified.ArrayFire is asynchronous (for the most part).You can hand off
function 1to GPU 1,function 2to GPU 2,function 3to GPU 3 in a for loop beforefunction 1is completed. Granted this is a bit ugly but what you are requesting for can be mostly achieved using existing functionality.pavanky commentedon Dec 6, 2016
Having said that, thread safety is still a very important feature to have.
vakopian commentedon Dec 7, 2016
@pavanky I'm not sure if there is a recommendation in your (1). I was just trying to say that @umar456's recommendation of "not relying on threads to improve performance" is not really viable in some important scenarios. The way I understood that recommendation was this:
"since the GPU operations are pushed into a single queue anyway, there is no benefit nor need for multiple threads, and so designing the application as if it's a single threaded program is the recommended way to go".
Is this understanding correct? Do you really recommend making the application quasi-single-threaded? If yes, then I still think that following this recommendation would result in a sub-optimal application for the 2 scenarios I described.
pavanky commentedon Dec 7, 2016
@vakopian No I don't mean that at all :)
I was just letting you know the workarounds for now if this is a blocking feature on your end.
WilliamTambellini commentedon Feb 20, 2017
Good morning,
What is the current status of this one: still early WIP ? halfly done ? mostly done ready for 3.5 ? done ?
Kind
pavanky commentedon Feb 20, 2017
@WilliamTambellini check #1706 for the progress being made.