Tried your suggestion.
Removing the entry in ostum_plugins and lines from ostum_config where namespace=2 (originaly plugin ID in ostum_plugins) removed the entry from the UI. Attempting to reinstall produces this again:
Here's the ostum_plugins table after all this:
Table ostum_config contains no lines where namespace=plugin.3