In optimistic detection, you wait to get to the commit point before you do the conflict detection. To do conflict detection, you compare the committing transaction’s write set with the read set of outstanding transactions. You also compare the write set of the committing transaction with the write set of outstanding transactions. This is checking that the above 2 set intersection are empty. If they are empty, you can proceed with the commit. If not, then I think you have to sort out the conflicts in such a way that gives priority to the committing transaction.

I think giving priority to the committing transaction means that the other transaction will abort and restart. Are there other ways in which priority could be given to the committing transaction?

I think commiting transaction would succeed and the rest would abort in optimistic detection.

Does the decision to do optimistic vs pessimistic detection happen at the hardware level, or could a programmer ever signal to the hardware that they want one or the other to happen?

Pessimistic detection always expects conflicts so we check after each operation. Optimistic detection only expects conflicts after commits so we check then.