A wireless network operator typically divides its radio spectrum into a number of subbands and reuse them to serve traffic in many cells. To mitigate co-channel interference, allocation of spectrum and power resources needs to be adapted to time-varying channel and traffic conditions throughout the network. Standard model-based network utility maximization is severely limited by the computational complexity and the difficulty of acquiring instantaneous global channel state information. In this paper, a learning-based method is proposed to optimize discrete subband allocations and continuous power allocations using (generally delayed and inaccurate) channel state information in local and nearby cells. For these two types of allocations, two complementary deep reinforcement learning algorithms are designed to be executed and trained simultaneously to maximize a joint objective. Simulation results show that the proposed method outperforms a state-of-the-art fractional programming algorithm as well as a previous solution based on deep reinforcement learning.