We propose a system of conservation laws with relaxation source terms (i.e. balance laws) for non-isothermal viscoelastic flows of Maxwell fluids. The system is an extension of the polyconvex elastodynamics of hyperelastic bodies using additional structure variables. It is obtained by writing the Helmholtz free energy as the sum of a volumetric energy density (function of the determinant of the deformation gradient det F and the temperature θ like the standard perfect-gas law or Noble-Abel stiffened-gas law) plus a polyconvex strain energy density function of F, θ and of symmetric positive-definite structure tensors that relax at a characteristic time scale. One feature of our model is that it unifies various ideal materials ranging from hyperelastic solids to perfect fluids, encompassing fluids with memory like Maxwell fluids. We establish a strictly convex mathematical entropy to show that the system is symmetric-hyperbolic. Another feature of the proposed model is therefore the short-time existence and uniqueness of smooth solutions, which define genuinely causal viscoelastic flows with waves propagating at finite speed. In heat-conductors, we complement the system by a Maxwell-Cattaneo equation for an energy-flux variable. The system is still symmetric-hyperbolic, and smooth evolutions with finite-speed waves remain well-defined.