​ The field of in-hand robot manipulation of deformable objects is an open and key issue for the next-coming robots. The complexity of physics in a soft object and the huge variety of the deformations of its deformable shape, which can be caused when a robot grasps and manipulates it, makes necessary to develop an adaptable and agile framework for these tasks. Many research works have been proposed to control the manipulation tasks using a model of the manipulated object. Despite these techniques are precise to model the deformations, they are time consuming and, using them in real environments is almost impossible because of the large amount of objects which the robot could find. In this paper, we propose a model-independent framework to control the movements of the fingers of the hands while the robot executes manipulation tasks with deformable objects. This technique is based on tactile images which are obtained as a common interface for different tactile sensors, and uses a servo-tactile control to stabilize the grasping points, avoid sliding and adapt the contacts' configuration regarding to position and magnitude of the applied force. Tactile images are obtained using a combination of dynamic Gaussians, which allows the creation of a common representation for tactile data given by different sensors with different technologies and resolutions. The framework was tested on different manipulation tasks which include complex deformations of the objects, and without using a model of them.