It does take far longer to charge the controller than the Phantoms battery, I'd leave it on overnight if need be.
The lights should all go out when it's done