This paper addresses the relation between the grain size distribution, the elastic stress field fluctuations and the description of incipient plasticity in polycrystals with local elastic anisotropy. We propose a numerical approach based on full-field computations on polycrystalline microstructures. The generation of microstructures with prescribed grain size distribution and grain shape is made by combining dense-sphere packing with Power diagrams. By using a local plastic criterion based on the stress field statistics, it is shown that the macroscopic Hall-Petch relation significantly depends on the grain size distribution and the stress heterogeneity. Previous results of the literature obtained with mean-field approaches are confirmed by our study.