Yes, if you’re installing them from the Microsoft Store. Only the add-ins have code (JavaScript) to do anything, and Microsoft validates all add-ins before allowing them to be uploaded to the store. You can see the privacy statements and terms and conditions for any add-in its consent form in the Microsoft store.
Whether the add-in uses a button or uses content highlighting, it only has access to the data in the email message that you’re working in.
If you trust Microsoft to validate the add-in, then you should be able to trust that the add-in is not malicious. If you’re interested, Microsoft provides detailed information about the validation they do, what developers are and are not allowed to do, and the sandboxing process in which all add-ins are run.
While add-ins do something with information within Outlook, a connector is really just a way of giving a third-party service permission to display notifications and alerts in Outlook. Microsoft validates connectors, but really you need to trust the 3rd-party service. If you’ve got an account with that service though, presumably you’re already comfortable using them.
As always, we recommend that you only ever install add-ins and connectors from the Microsoft Store.