Once a star has evolved to become a white dwarf, it no longer has an internal source of heat, and is shining only because it is still hot. Like something taken from the oven, left alone a white dwarf will cool down until it is the same temperature as its surroundings. Unlike tonight's dinner, which cools by convection, conduction, and radiation, a white dwarf cools only by radiation.
Because it's electron degeneracy pressure that stops it from collapsing to become a black hole, a white dwarf is a fantastic conductor of heat (in fact, the physics of Fermi gasses explains the conductivity of both white dwarfs and metals!). How fast a white dwarf cools is thus easy to work out … it depends on only its initial temperature, mass, and composition (most are carbon plus oxygen; some maybe predominantly oxygen, neon and magnesium; others helium). Oh, and as at least part of the core of a white dwarf may crystallize, the cooling curve will have a bit of a bump around then.
The universe is only 13.7 billion years old, so even a white dwarf formed 13 billion years ago (unlikely; the stars which become white dwarfs take a billion years, or so, to do so) it would still have a temperature of a few thousand degrees. The coolest white dwarf observed to date has a temperature of a little less than 3,000 K. A long way to go before it becomes a black dwarf.
Working out how long it would take for a white dwarf to cool to the temperature of the CMB is actually quite tricky. Why? Because there are lots of interesting effects that may be important, effects we cannot model yet. For example, a white dwarf will contain some dark matter, and at least some of that may decay, over timespans of quadrillions of years, generating heat. Perhaps diamonds are not forever (protons too may decay); more heat. And the CMB is getting cooler all the time too, as the universe continues to expand.
Anyway, if we say, arbitrarily, that at 5 K a white dwarf becomes a black dwarf, then it'll take at least 10^15 years for one to form.
However, if weakly interacting massive particles exist, it is possible that interactions with these particles will keep some white dwarfs much warmer than this for approximately 10^25 years. If protons are not stable, white dwarfs will also be kept warm by energy released from proton decay. For a hypothetical proton lifetime of 10^37 years, Adams and Laughlin calculate that proton decay will raise the effective surface temperature of an old one-solar mass white dwarf to approximately 0.06 K. Although cold, this is thought to be hotter than the temperature that the cosmic background radiation will have 10^37 years in the future
One more thing: no white dwarf is totally alone; some have binary companions, others may wander through a dust cloud … the infalling mass generates heat too, and if enough hydrogen builds up on the surface, it may go off like a hydrogen bomb (that's what novae are!), warming the white dwarf quite a bit.
If black dwarfs were to exist, they would be extremely difficult to detect, since, by definition, they would emit very little radiation. One theory is that they might be detectable through their gravitational influence.
Source: universetoday and wikipedia