Space-time coded massive (STCM) multiple-input multiple-output (MIMO) system provides superior bit error rate (BER) performance compared with the conventional space-time coding and massive MIMO techniques. The transmitter of the STCM-MIMO system consists of a large antenna array... In a practical system, the self-interference created by the signals transmitted by the elements of this antenna array, known as mutual coupling (MC), degrades the performance of the system. The MC effect is pronounced in communication systems with a large antenna array. On the other hand, increasing the number of transmitting antennas results in improved BER performance. Hence, there is a trade off in selecting the optimum number of transmitting antennas in an STCM-MIMO system. In order to take the impact of MC into account, we have derived an analytical expression for the received signal to accurately model the STCM-MIMO system under the existence of the MC effect. We present an algorithm to select the optimal number of antennas to minimize mutual coupling and the system bit error rate (BER). Through computer simulations, we investigate the BER performance of the STCM-MIMO system for different numbers of array elements. (read more)