GPUs have been becoming an indispensable computing platform in deep learning. Co-locating multiple DNN servings to share GPU resource is widely used to improve resource utilization while guaranteeing user QoS. However, despite recent efforts toward designing multi-DNN co-running, existing solutions still encounter difficulties in effectively handling fluctuating and heterogeneous requests. This is because the resource management mechanisms they employ to co-running multiple DNN requests lack reconfigurability, co-location performance isolation, and QoS-awareness.

In this paper, we propose a novel resource management paradigm called dynamic spatial multikernel that enables fine-grained management of GPU resources at kernel-wise, specifically for concurrent DNN inferences. We design a multi-DNN serving system, FineCo, support this resource management without relying on hardware or operating system assistance. FineCo enables resource allocation in kernel-wise by giving DNN compiler the ability to generate multiple candidates implementations that have different resource demand sensitivities for DNN kernels. Base on multi-candidate implementations of kernels, FineCo utilizes two-level scheduler to dynamically coordinate the scheduling of concurrent requests and the candidate scheduling of among co-located kernels for fully tapping the available GPU resources and avoiding resource conflict. Our prototype implementation demonstrates that FineCo achieves up to 40% throughput improvement and reduces real-time latency by 19% over the state-of-the-art work.