For high-accuracy applications, accurate electromagnetic modeling tools are required. One commonly used technique to model the magnetic field of permanent magnets is the magnetic surface charge method. In principle, this method assumes a volume without boundaries, i.e., a relative permeability equal to unity everywhere in the volume. In previous publications, material with a relative permeability unequal to unity in the vicinity of the permanent magnet was modeled using the method of images, however, commonly, the relative permeability of the permanent magnet material itself was neglected. This paper presents a method to take the permeability of the magnetic material into account.