I needed to have a dynamic action that fires when a modal dialog is cancelled or the escape key is pressed. To do this in APEX 5, on the parent page, you can create a dynamic action based on a custom event called dialogclose.
Custom Dynamic Action
Custom Event: dialogclose
The True/False actions can be set as usual, such as refreshing a report or setting some value.