The mechanical portion of the latch broke. When this happens, its a pain to get the door open. Since you were able to get the door panel off, you have to destroy the mechanical portion of the lock.
Try using a prybar and a hammer. It takes a lot of work, but once you get the back cover of the mechanical latch off by breaking it, you will be able to manually release the latch when it becomes visible.
You will need a new door lock actuator, and latch assembly once you get it apart.
You can get it, just be patient.