The ticket key will be set when creating a ticket and is mandatory for later use.
If a user does send a mail to your monitored (imap) mailbox, without ticket key information, a new ticket will be created. (where the user (sender) will be set as ticket requestor).
If not disabled, the user will then receive a notification that his ticket was created; ticket key reference is included. When the user does now reply to it, the new mail will not generate a new ticket but attached as worklog entry to the existing one.
The same procedure if a agent does send to user(s) from within the ticket system.
Each ticket system i know does work in a way like this. You need some reference between the ticket and the imported mail.