Despite his success, Skidmore's perfectionism lost him money. Employees described how he destroyed work or materials which didn't meet his exacting standards. In 1872 he had to sell the Art Manufacturers company, setting up a smaller business in Meriden. His work still in demand, Skidmore continued until 1883 when he merged with Winfield's of Birmingham.

Disabled by a carriage accident, and having neglected the commercial side of his business, Skidmore was relatively poor by the time he returned to Coventry in 1893. To help make his last years more comfortable, local dignitaries raised a special pension for him. Skidmore died at his house in Eagle Street in 1896 and was buried at London Road Cemetery.

The Herbert has several examples of Skidmore's decorative metalwork in its collections. Additionally, the Hereford Screen can be seen at the Victoria and Albert Museum in London.