The most basic difference is the association
Listener is associated with Event Source (Ex: key board)
Handler is associated with an Event (Ex: keydown)

A listener watches for an event to be fired. For example, a KeyListener waits for KeyEvents, a MessageListener waits for messages to arrive on a queue and so on.The handler is responsible for dealing with the event.