[OSSA 2013-032] Keystone trust circumvention through EC2-style tokens (CVE-2013-6391)

So, looking into this a bit more, it seems that we need to implement something similar to https://review.openstack.org/#/c/40444/ to fix this.

Currently the token in the ec2 controller is generated with the user_id and no knowledge of trusts. There are some issues with that patch atm though, looks like it still needs significant work.

Proposed fix attached - this currently has no tests, as there is no coverage at all of the Ec2Token controller, can I just propose adding the coverage via the normal gerrit process?

Also, this renders https://review.openstack.org/#/c/40444/11 (and the related blueprint) obsolete, so we may want to warn the submitter of that patch not to bother doing any rework and resubmitting?

Oops, my bad, I didn't spot test_keystoneclient_sql.py - I'll add some tests and propose another patch

Update patch, one minor fix and added a test.

There are some limitations in what we can test atm, because the v2 token trust-scoping happens at the controller level (on an existing token) rather than when issuing it. This doesn't seem to enable a representative response to be generated (in terms of the token contents) due to the way the fixtures/mocking works in these tests.

I think probably what needs to happen (under a different bug) is the trust-scoping code needs to move from the controller to where the token is generated behind the identity_api? (comments welcome on this..)

i.e this code is never hit in these tests, but it is when you make a real request:
https://github.com/openstack/keystone/blob/master/keystone/token/controllers.py#L130

So I guess that really needs to be happening here instead?
https://github.com/openstack/keystone/blob/master/keystone/token/providers/uuid.py#L359

Anyway, even with the somewhat limited test, this patch should fix the vulnerability and the test does provide coverage of the new code.

Note, I still need to test this with a MD5 hashed v3 token as I've only tested with pure v2 interfaces so far (other than the v3 call to create the trust). I'd like to ensure that creds_ref still returns the trust_id correctly in that case.

Keystone folks, please review proposed patch. Feel free to subscribe more appropriate keystone-core reviewers.

I tested with MD5 hashed v3 tokens, and all works OK, except for a client bug #1244675 which I've reported and have proposed a fix for. The API level behaviour is OK though, so I think this patch is OK to be proposed for merging, unless keystone folks have any review feedback (I've not had any comments at all so far)

Adding Matthieu Huin to help review, who proposed a related patch in https://review.openstack.org/#/c/40444/

Your patch looks good to me, Steven.

TO explain the patch: the metadata on the frist token will havea trust_id on it. When creating the EC2 Keyspair, this needs to be carried over. That is done by earlier code. What was missing is the call that appends the trust_id on to the token created from the ec2 keypair. That is what is done at line 106. From this point forward, the trust_id will be maintained.

In the future, we probably want to change from "if we can't find the trust id, assume that there is none." To an approach of "we need to see an avid disavowal of a trust_id." Something like metadata.trust_id = None. We need to ensure that we carry over proof of non-delegation from mechanism to mechanism or we are going to continue to have these forms of errors.

The patch as written looks good.

Mattieu, does you patch take any security issues into account that this one does not? It seems to be a larger patch, but I can't tell if that means it is covering a wider array of issues.

Ayoung: as far as I can tell this patch is more appropriate than mine regarding the problem at hand. What I was doing was serving the same trust token (ie the one used to create the ec2 creds) whenever there was an ec2 authentication. This wasn't a good idea, due to the short lifespan of tokens.
I also added a reference to the trustee id when creating the credentials, so that it'd be possible to differenciate impersonation.

The patchset seems to be a sane approach to solving the concern. I'm still making sure I'm not seeing any gaps in it, but the hole seems to be covered with the patchset in #6

@keystone-core: still need a formal second +2 on this patch... so far only Adam clearly expressed it.
I'll be working on an impact description.

Proposed impact description, please doublecheck that I got it right (included affected versions):

-------------------------------------------
Title: Keystone trust circumvention through EC2-style tokens
Reporter: Steven Hardy (Red Hat)
Products: Keystone
Affects: Grizzly and later

Description:
Steven Hardy from Red Hat reported a vulnerability in Keystone trusts when used in conjunction with the ec2tokens API. By generating EC2 tokens from a trust-scoped token, a trustee may retrieve a token not scoped to the trust, therefore elevtaing privileges to all of the trustor's roles. Only Keystone setups enabling EC2-style authentication are affected.
---------------------------------------------

elevtaing -> elevating

I'd say "By generating ec2 credentials using a trust-scoped token"

Also typo s/elevtaing/elevating

Otherwise lgtm

Impact description looks good to me aside from the recommendations above.

New version including recommendations:

-------------------------------------------
Title: Keystone trust circumvention through EC2-style tokens
Reporter: Steven Hardy (Red Hat)
Products: Keystone
Affects: Grizzly and later

Description:
Steven Hardy from Red Hat reported a vulnerability in Keystone trusts when used in conjunction with the ec2tokens API. By generating EC2 credentials using a trust-scoped token, a trustee may retrieve a token not scoped to the trust, therefore elevating privileges to all of the trustor's roles. Only Keystone setups enabling EC2-style authentication are affected.
---------------------------------------------

Thierry's updated impact description in comment #20 looks good to me.

The impact description looks good to me (#20).

This patchset only seems to apply (#6) to stable/havana. For use in stable/havana +2 for me.

We will need slightly different patchsets for grizzly and master as there was enough shuffling in havana to move code around fairly significantly. Master has now seen some changes that eliminate the identity_api -> assignment_api proxy that this patchset relies on.

I'll work on posting the "fixed" patchsets for grizzly and master today to this thread.

Patch Rebased for Master

For Grizzly, there is no good way to correlate (as far as I can tell) the Trust to the EC2 Credential. EC2 Credential was refactored in Havana to have a lot more information available to it (generic credentials backend that is not limited to the very limited EC2 data set).

The easiest solution is to either
1: Disable EC2 credentials for trust scoped tokens
2: State that it is recommended that trusts and EC2 contrib not be enabled at the same time.

I am not sure what the alternatives for getting more data into the EC2 credential would be in the Grizzly code base. It might be possible to add a migration to grizzly that encompasses the new data and the first migration of Havana for the DB. I think that a new migration is a sub-optimal choice.

@Morgan Fainberg - my vote is for (1), since there may be users of both trusts and ec2tokens on grizzly (who won't want this hole to be exploitable), but they probably don't care about ec2 credentials derived from a trust (well they can't because it doesn't currently work..)

AFAIK Heat is the only thing which wants to make use of ec2 credentials derived from a trust ID, which will be possible after this is fixed, and we only care about that functionality from Havana onwards, so (1) wfm and shouldn't impact existing users (whereas the second option will, potentially)

Note, I'm aware of https://blueprints.launchpad.net/keystone/+spec/ec2-keypairs-from-tokens which Matthieu has been working on, but since that functionality doesn't exist yet, I'm assuming potential users can't care about it in Grizzly. Heat on the other hand might make use of this to fix bug #1089261, which could be a candidate for backporting to Havana.

@Morgan: go for (1)

I should probably add a grizzly-specific note explaining that we took this radical action in the patch.
Something like "Note: the patch for this vulnerability in Grizzly disables the ability to derive ec2 credentials from a trust, as there is no clean way in Grizzly to properly correlate the two." -- Feel free to suggest better wording to add to the description.

@Thierry,

Sounds good. I'll get going on that change for grizzly. Hopefully have the patchset in order today.

Ok a couple bits of added information.

Grizzly is not affected by this bug to an issue with how impersonation is checked for. As seen here: http://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/controllers.py?h=stable/grizzly#n204
we check for the string "True" not the singleton True. This means that impersonation will never work within grizzly and mitigates this problem. This issue was fixed in Havana but never backported.

I have not finalized testing in this grizzly patchset, I will work on finalizing tests once I am back at home this evening. This fix can be validated by using Steven Hardy's script from #1. I have confirmed this re-enables Impersonation Trusts and fixes the security flaw.

In theory the patchset in #29 can go through public gerrit after the Havana / master patchset go through if desired.

OK, amending description based on that info, please reconfirm:

-------------------------------------------
Title: Keystone trust circumvention through EC2-style tokens
Reporter: Steven Hardy (Red Hat)
Products: Keystone
Affects: Havana and later

Description:
Steven Hardy from Red Hat reported a vulnerability in Keystone trusts when used in conjunction with the ec2tokens API. By generating EC2 credentials using a trust-scoped token, a trustee may retrieve a token not scoped to the trust, therefore elevating privileges to all of the trustor's roles. Only Keystone setups enabling EC2-style authentication are affected.
---------------------------------------------

(Yes, we can push the grizzly fix publicly once this is opened)

I am good with the revised message, +2

I am also happy to push the grizzly "fix" publicly afterwards.

CVE requested

We should block https://review.openstack.org/#/c/51973/2 until after we get the "fix" into stable/grizzly.

Re: comment on #34, this would open grizzly to the exploit here.

Can anyone provide a status update on this - will the fix be landing in Icehouse-1?

The heat instance-users work (related to bug #1089261) has been blocked since this was reported, so I'm really hoping we can get the fix in soon.. :)

Scheduled advisory publication date is Wednesday, December 11, 2013 at 1500UTC (so as to fall before the 2013.2.1 point release). Patches will be pushed to review.openstack.org for last-minute approval shortly prior to that time.

Fix proposed to branch: master
Review: https://review.openstack.org/61419

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/61425

Reviewed: https://review.openstack.org/61419
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=fa16882507d05f3bf2bbfbd62bb1b5bc0a6fb2b4
Submitter: Jenkins
Branch: master

commit fa16882507d05f3bf2bbfbd62bb1b5bc0a6fb2b4
Author: Steven Hardy <email address hidden>
Date: Mon Oct 21 19:49:01 2013 +0100

    Fix issues handling trust tokens via ec2tokens API

    Trust scoped tokens are handled incorectly when making requests
    via the ec2tokens API, meaning that the restrictions enforced
    by trust-scoped tokens are not respected when obtaining a token
    via ec2token signature validation.

    Storing the trust_id in the blob associated with the ec2 keypair,
    and passing that id in the metadata when requesting a v2 token
    solves the issue.

    Change-Id: I52566384d7813ef0e2f20fb94a5076386457ff02
    Closes-Bug: #1242597

Reviewed: https://review.openstack.org/61425
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=bd0e3e4cf0bda19ba9f45a60be7e4d6de196f224
Submitter: Jenkins
Branch: stable/havana

commit bd0e3e4cf0bda19ba9f45a60be7e4d6de196f224
Author: Steven Hardy <email address hidden>
Date: Mon Oct 21 19:49:01 2013 +0100

    Fix issues handling trust tokens via ec2tokens API

    Trust scoped tokens are handled incorectly when making requests
    via the ec2tokens API, meaning that the restrictions enforced
    by trust-scoped tokens are not respected when obtaining a token
    via ec2token signature validation.

    Storing the trust_id in the blob associated with the ec2 keypair,
    and passing that id in the metadata when requesting a v2 token
    solves the issue.

    Change-Id: I52566384d7813ef0e2f20fb94a5076386457ff02
    Closes-Bug: #1242597

I'm getting a testsuite failures on Ubuntu 13.10 with the patch from https://review.openstack.org/61425:

======================================================================
ERROR: keystone.tests.test_keystoneclient_sql.KcMasterSqlTestCase.test_ec2_auth_success_trust
----------------------------------------------------------------------
_StringException: Traceback (most recent call last):
  File "/«PKGBUILDDIR»/keystone/tests/test_keystoneclient_sql.py", line 140, in test_ec2_auth_success_trust
    tenant_id=self.tenant_bar['id'])
  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 458, in authenticate
    resp, body = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 160, in get_raw_token_from_identity_service
    token=token)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 189, in _base_authN
    resp, body = self.request(url, 'POST', body=params, headers=headers)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 609, in request
    **request_kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 114, in request
    raise exceptions.from_response(resp)
Unauthorized: The request you have made requires authentication. (HTTP 401)

-------------------- >> begin captured logging << --------------------
...
keystone.token.controllers: WARNING: User two is unauthorized for tenant bar
keystone.common.wsgi: WARNING: Authorization failed. The request you have made requires authentication. from 127.0.0.1
access: INFO: 127.0.0.1 - - [17/Dec/2013:19:30:18 +0000] "POST http://localhost:49237/v2.0/tokens HTTP/1.0" 401 114
eventlet.wsgi.server: INFO: 127.0.0.1 - - [17/Dec/2013 13:30:18] "POST /v2.0/tokens HTTP/1.1" 401 253 0.026661

urllib3.connectionpool: DEBUG: "POST /v2.0/tokens HTTP/1.1" 401 114
keystoneclient.httpclient: DEBUG: Request returned failure status: 401
keystoneclient.v2_0.client: DEBUG: Authorization Failed.
--------------------- >> end captured logging << ---------------------

Our package is 1:2013.2 with only this fix and the one for CVE-2013-4477 (bug #1242855). Am I missing other required patches?

> /usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py

Are you using the system installed keystoneclient rather than a venv? If so what version?

I'm hitting the same as Jamie trying to prepare Ubuntu package builds of the stable/havana 2013.2.1 release. Neither of us are using venv, we're attempting to build system packages using distro provided dependencies. In this case, we're using the python-keystoneclient package version 0.3.2-0ubuntu1, which should satisfy keystones requirement (python-keystoneclient>=0.3.2).

I know that this fix was blocked from merging into stable/havana until a corresponding update to global requirements got merged. The requirements update for keystone is @ https://review.openstack.org/#/c/61534/. However, its difficult to know what changes to the requirements are relevant to this fix and what changes came in as part of the general sync.

Any hints appreciated.

Did a bit more digging... Steven's right, looks like an issue using older version of keystoneclient. With https://review.openstack.org/#/c/48462/ applied against our 0.3.2 package, tests run again. The patch didn't release with keystoneclient until 0.4.1

@Steven
Is the keystones requirements' "python-keystoneclient>=0.3.2" entry correct, or does it need to be updated to >=0.4.1? In other words, is this just an issue for the test suite or is keystone still vulnerable if using an older client library?

Thanks,
Adam

I confirm it was a limitation of python-keystoneclient 0.3.2. If I upgrade to 0.4.1 the test passes.

Reviewed: https://review.openstack.org/51973
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=8fcc18c42bde2db34e4b29236dc2e971d40f146b
Submitter: Jenkins
Branch: stable/grizzly

commit 8fcc18c42bde2db34e4b29236dc2e971d40f146b
Author: Steven Hardy <email address hidden>
Date: Sun Oct 13 10:44:52 2013 +0100

    Fix v2 token user ref with trust impersonation=True

    The v2 token controller incorrectly checks for a string instead
    of a boolean, which results in the wrong user ID (trustee, when
    it should be the trustor) when impersonation=True. So fix the
    comparison and tests, adding a test which illustrates the issue.

    This patchset also closes the gap that allows EC2 credentials to
    be issued from trust-scoped tokens, allowing privilege escalation
    since EC2 tokens have no concept of trust-scoping/role
    restrictions in the Grizzly release.

    Change-Id: Ic94f30f2354c9fda20531bb598387368fde8a096
    Closes-Bug: #1239303
    Related-Bug: #1242597