Physics-informed neural networks are a powerful approach that combines deep learning with physical principles to solve complex problems. However, like any method, they do have some drawbacks. The first one is hyperparameter sensitivity such as learning rates, network architectures, and activation functions. Many researchers have devoted their time and energy to design efficient neural network models by searching optimal hyperparameters. In this article, we follow another path by showing that the mathematical formulation of the problem to be solved, has a critical influence on the performance of the model. Electrostatic examples illustrate this.