pasar de castaño oscuro

to be beyond a joke; to go too far

Literally, “to surpass the dark chestnut color”.