Is it possible to set up a custom engagement workflow?

This is not possible. A custom workflow can be defined for requests, resource requests, projects and opportunities.

For engagements, it is only possible to define the role that is required to set each status.