Resources owned by a project/tenant are not cleaned up after that project is deleted from keystone

This is actually tracked in the blueprint tenant-deletion (https://blueprints.launchpad.net/horizon/+spec/tenant-deletion), but I'll leave the bug as a reminder.

Needs some discussion about how resources should be treated from a UX point of view at the Folsom design summit. My inclination would be to destroy all resources owned by a tenant when it's being removed, but that may be too harsh, and doesn't take into account future states where trust may be delegated.

Regardless, that also implies an inversion of control where Keystone would potentially need to reach into other systems with administrative access to impact resources, which I'm not sure I like.

For the record, I agree that the action should be to terminate and delete all resources attached to a tenant when the tenant is deleted...

I agree that, when a tenant is deleted, the running instances associated with it should be terminated.

The easier and non destructive approach for this problem may be to check the resources (such as images, instances, volumes) under the tenant and make sure there is not anything left to be hanging around without anyone can access them when tenant get deleted. When the detects any such resource, the delete tenant should not proceed and should tell the user that it can not be deleted due to the resource issues. It also forces the operator to handle these resources first. In many cases, this approach might be a safer way to deal with deletion.

In regards to #5, for what it's worth, I would like a choice... let me delete all resources but confirm with me first.

I like a verification here where there is both a 'are you sure' along with an optional password, or other validated check, will prevent the last user from being deleted if resources are still active... however if the override is approved then the instances should be blown away with the last user

Closing this as a bug as it is superseded by the blueprint https://blueprints.launchpad.net/horizon/+spec/tenant-deletion

This won't be fixed in Nova in Grizzly, and is something that is cross project and needs to be hashed out at the Summit.

Wouldn't a good way be to have keystone broadcast user/tenant/role deletion/creation and let downstream systems consume said messages? Then nova can react however it wants, glance to, and any other downstream system that desires to take some kind of action when users/tenants/roles are deleted/added...

<dolphm> that's being worked in bp notifications
<dolphm> https://blueprints.launchpad.net/keystone/+spec/notifications
<ttx> so for havana-2 ?
<dolphm> which is blocked by https://blueprints.launchpad.net/keystone/+spec/unified-logging-in-keystone ... which we just added to m1 and is nearly complete
<dolphm> ttx: yes, looks to be on pace for m2

The RPC mechanism assumes eventlet. Many people are running with Keystone in HTTPD and without eventlet. Is it possible that we can do the notification mechanism using straight AMQP and punt on 0Mq for a first implementation? We just need to be able to publish notifications, not accept them, so Keystone does not need the full RPC.

Removed m2 target because bp notifications has some blockers; it looks like it won't land during Havana at this point.

keystone now emits notifications when projects/tenants are delete as part of https://blueprints.launchpad.net/keystone/+spec/notifications

Added glance as a result of https://bugs.launchpad.net/keystone/+bug/1221732

I feel it is the responsibility of something external which does clean tenant deletion like horizon(https://blueprints.launchpad.net/horizon/+spec/tenant-deletion) should be doing the image members clean up

I don't think it makes sense to look at this as a glance issue.

I agree some work may be needed in glance as in other projects to respond to tenant deletion events in a sensible way. But glance shouldn't be innovating that independently. Sounds like a cross-cutting-concern for all projects.

This was brought up again (specifically for nova) in bug https://bugs.launchpad.net/nova/+bug/1288230

Fix proposed to branch: master
Review: https://review.openstack.org/92599

Fix proposed to branch: master
Review: https://review.openstack.org/92600

Not just the instances, but likely the subnet, router, gateway etc. Once
the tenant is gone it is very difficult to get the id's you need. If you
do a quantum (nova) port-list you will likely see lots of floating and
fixed ips related to it as well.

On Wed, May 7, 2014 at 9:37 AM, Openstack Gerrit
<email address hidden>wrote:

> Fix proposed to branch: master
> Review: https://review.openstack.org/92599
>
> ** Changed in: neutron
> Status: Confirmed => In Progress
>
> --
> You received this bug notification because you are subscribed to
> neutron.
> https://bugs.launchpad.net/bugs/967832
>
> Title:
> Resources owned by a project/tenant are not cleaned up after that
> project is deleted from keystone
>
> Status in OpenStack Image Registry and Delivery Service (Glance):
> Won't Fix
> Status in OpenStack Dashboard (Horizon):
> Won't Fix
> Status in OpenStack Identity (Keystone):
> Fix Released
> Status in OpenStack Neutron (virtual network service):
> In Progress
> Status in OpenStack Compute (Nova):
> Confirmed
>
> Bug description:
> If you have running instances in a tenant, then remove all the users,
> and finally delete the tenant, the instances are still running.
> Causing serious trouble, since nobody has access to delete them. Also
> affects the "instances" page in horizon. It will break if this
> scenario occurs.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/glance/+bug/967832/+subscriptions
>

Neutron spec:
https://review.openstack.org/#/c/98097/

Ryan, I *just* resumed working on this issue today (In Neutron). I think we should have a summit design session about an OpenStack wide solution. I hope you're coming to Paris :)

Hey Assaf,
I also picked this up again only about a week ago.
That would be awesome if we could do a summit design session about this solution!
I sumbitted a presentation related to openstack and selinux, so hopefully
I will get accepted and be headed to Paris!

Fix proposed to branch: master
Review: https://review.openstack.org/115964

Fix proposed to branch: master
Review: https://review.openstack.org/115965

Fix proposed to branch: master
Review: https://review.openstack.org/115966

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/92600
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/115964
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/115965
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/115966
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.

Change abandoned by Assaf Muller (<email address hidden>) on branch: master
Review: https://review.openstack.org/92599

Did anything happen at the Kilo summit in Paris about this? Are there any mailing list threads on this because if it hasn't come to a summit yet we should talk about it in the cross-project sessions in Vancouver.

A simple approach w/o listening on event notifications (which would be the slickest way) would be to have a periodic task running which is checking resource tenants to see if the tenant exists and if not, reap them (like we have for orphaned instances).

Started a mailing list thread: http://lists.openstack.org/pipermail/openstack-dev/2015-February/055801.html

I agree with Mark's assertion in comment #17 that Glance shouldn't be going about a solution alone, but this certainly affects Glance.

This also certainly affects Horizon

The Tokyo Summit solution here was that this should be done via an osc plugin. There are really dramatic issues with auto delete from keystone deletes. Many sites need an archive process. Nova itself soft deletes many resources, and even has the ability to set an undo time on some of them.

This shouldn't be an automatic process in the cloud, it should be deliberate. Just like not deleting all the files on your system owned by a user if you remove that user from /etc/passwd.