README.md 10.6 KB

This folder contains OASIS XACML Committee's 2.0 version of conformance tests upgraded to conform to the XACML 3.0 standard, including new tests for new features introducted in XACML 3.0. Most of them have been submitted to the OASIS XACML Committee in April 2014 by AT&T. The original files are available on the xacml-comment mailing list: https://lists.oasis-open.org/archives/xacml-comment/201404/msg00001.html and on AT&T's Github repository (MIT License): https://github.com/att/XACML/wiki/XACML-TEST-Project-Information

For a description of the tests, see file ConformanceTests.html which is the original HTML description published on the OASIS xacml-comments mailing list.

WARNING: There are several issues with these original conformance tests (as of 26 September 2015) and therefore have been fixed to adapt to our PDP implementation:

  1. For all tests testing the validation of XACML policy syntax, our PDP implementation is expected to reject the policy at initialization time, before receiving any Request. For these tests, the original Request.xml and Response.xml have been renamed to Request.xml.ignore and Response.xml.ignore to indicate to our test framework that an invalid policy syntax is expected.
  2. For tests testing the validation of XACML Request syntax, our PDP implementation is expected to reject the request before evaluation. For these tests, the original Policy.xml and Response.xml have been renamed to Policy.xml.ignore and Response.xml.ignore to indicate to our test framework that an invalid Request syntax is expected.
  3. Invalid schemaLocation in many XXXXPolicy.xml have xsi:schemaLocation="urn:oasis:names🇹🇨xacml:3.0:policy:schema:os access_control-xacml-2.0-policy-schema-os.xsd".
  4. IIA002Special.txt missing. We put it back from original XACML 2.0 conformance tests. We also consider this test to be optional because it is not mandatory in the spec that the PDP must be able to retrieve specific attribute not provided in the request. We consider it an optional feature, therefore moved to 'optional' subfolder.
  5. IIA006Policy.xml uses SubjectCategory attribute, which is not valid in XACML 3.0 schema (errata in section 8). We fixed it here (replaced with Category).
  6. IIA016, IIA018, IIA020 tests are using fixed value of standard current-time attribute in the Request. However, XACML 3.0 spec says: "In practice it is the time at which the request context was created." We consider the request context is created when the ContextHandler handles the request, so it is not the same time (but at a later time). And our implementation overrides the value of current-time (if Issuer undefined), no matter what. Therefore, for this test to work, we specified an issuer ("pep") to prevent override by our context handler.
  7. IIA022 and IIA023 Request.xml use wrong attributeId "urn:oasis:names🇹🇨xacml:1.0:subject:subject-rfc822Name" on "http://www.w3.org/2001/XMLSchema#base64Binary" value: c3VyZS4=. Fixed here by replacing AttributeId with urn:oasis:names🇹🇨xacml:1.0:subject:subject-base64Binary.
  8. IIA022 and IIA023 Request.xml use xpathExpression datatype which is optional feature, therefore in the wrong section. Should be renamed IIA300. So we replaced with a copy without xpathExpressions (renamed IIA023_FIXED) and moved the original test files to optional directory.
  9. IIA022Request.xml missing RequestDefaults although it has xpathExpressions, so fixed in the version located in optional directory
  10. IIA023Request.xml uses RequestDefaults element which is optional feature, therefore should be moved to Optional testset. We fixed it by commenting out RequestDefaults for the mandatory tests, and moving the original version to 'optional' folder.
  11. IIA023Request.xml using timezone -14:30 in AttributeId="urn:oasis:names🇹🇨xacml:1.0:subject:subject-dateTime" and "urn:oasis:names🇹🇨xacml:1.0:environment:current-time" with datatype "http://www.w3.org/2001/XMLSchema#dateTime" and "http://www.w3.org/2001/XMLSchema#time" respectively. This is not valid per XML schema dateTime and time definitions. We fixed it here replacing them with timezone -14:00.
  12. IIA023Request.xml contains c_clown@NOSE_MEDICO.COM in one of the urn:oasis:names🇹🇨xacml:1.0:subject:subject-rfc822Name attributes, which is not valid (underscore is illegal in Domain name). Fixed here by replacing with "c_clown@NOSE.MEDICO.COM".
  13. IIB010Policy.xml and IIB011Policy.xml mention SubjectCategory in Description, which is not valid in XACML 3.0 schema (errata in section 8)
  14. IID312Policy.xml not valid because of duplicate Rule with RuleId = ...rule-5.
  15. IID321-329 missing.
  16. IID334-339 missing.
  17. IIE tests concern Policy(Set)IdReference, therefore require configuration of the referenced policies in a separate repository. In the original conformance tests provided by AT&T, this is done in the ATT-specific way with a IIEXXXRepository.properties file. We use a directory named 'IIEXXXRepository' containing all the referenced policies instead. For more advanced tests on Policy references, see the 'others' directory.
  18. IIF300Policy.xml is using wrong XPath version URI: "http://www.w3.org/TR/1999/Rec-xpath-19991116". Fixed: http://www.w3.org/TR/1999/REC-xpath-19991116. And test exclusively on xpathExpression (optional), so moved to 'optional' folder.
  19. IIF301Policy.xml uses xpathExpression datatype which is optional feature, therefore in the wrong section. So we replaced with a copy without xpathExpressions in directory (renamed IIF301_FIXED_NO_XPATH) and moved the original to optional directory. Also it is using invalid PolicyDefaults/XPathVersion: "http://www.w3.org/TR/1999/Rec-xpath-19991116". Fixed: http://www.w3.org/TR/1999/REC-xpath-19991116
  20. IIF310 uses xpathExpression datatype which is optional feature, therefore in the wrong section. So we replaced with a copy without xpathExpressions in directory (renamed IIF310_FIXED_NO_XPATH) and moved the original to optional directory. Also it is using invalid PolicyDefaults/XPathVersion: "http://www.w3.org/TR/1999/Rec-xpath-19991116". Fixed: http://www.w3.org/TR/1999/REC-xpath-19991116
  21. IIIA: The tests on Obligations are wrongly part of the Optional section (III). Indeed, they are now mandatory in XACML 3.0 whereas they were not in XACML 2.0. This is why we added a subdirectory "mandatory" in the conformance tests, where there are the "III" tests on Obligations; except III.A.30 which uses xpathExpressions. As xpathExpression datatype is optional feature, we moved it to a subdirectory "optional". Note that it should be numbered above 300 as it is XACML 3.0 only.
  22. IIIA002Response.xml, IIIA005Response.xml, IIIA006Response.xml, IIIA009Response.xml, IIIA010Response.xml,IIIA013Response.xml,IIIA014Response.xml,IIIA017Response.xml,IIIA018Response.xml,IIIA021Response.xml,IIIA022Response.xml,IIIA025Response.xml,IIIA026Response.xml are not schema-valid: use of invalid FulfillOn attribute on Obligation
  23. IIIA029 is missing although IIIA028 and IIA030 are there.
  24. IIIA030Policy.xml and IIIA330Policy.xml use xpathExpression but no PolicyDefaults XPathVersion specified, whereas it is done for others, e.g. IIF300Policy.xml. We fixed it here by adding the same PolicyDefaults element as in IIF300Policy.xml. Besides, IIIA330Policy.xml uses xpathExpression with md prefix but md prefix-namespace mapping undefined, whereas it is done for others, e.g. IIF300Policy.xml so considered not clearly schema compliant. We fixed it here by adding '... xmlns:md="http://www.medico.com/schemas/record" '.
  25. III.B tests missing.
  26. IIIC*Response.xml are not XACML 3.0 schema-compliant: ResourceId attribute not allowed on Result.
  27. IIIE302Response.xml have the subject and resource Attributes elements in reverse order of the Request, whereas our PDP implementation returns them in same order as in the Request. FIXED by putting back the Attributes in the same order.
  28. IIIF001 is using invalid PolicyDefaults/XPathVersion: "http://www.w3.org/TR/1999/Rec-xpath-19991116". Fixed: http://www.w3.org/TR/1999/REC-xpath-19991116
  29. IIIG001 is using invalid PolicyDefaults/XPathVersion: "http://www.w3.org/TR/1999/Rec-xpath-19991116". Fixed: http://www.w3.org/TR/1999/REC-xpath-19991116
  30. IIIG302 does not exist, but IIIG300 does. Fix: re-numbered 301 to 302, and 300 to 301.
  31. IIIG301 and IIIG302: tests for ReturnPolicyIdList: the XACML specification is ambiguous about what is considered an "applicable" policy, and therefore what should be included in the PolicyIdentifierList. See the discussion here for more info: https://lists.oasis-open.org/archives/xacml-comment/201605/msg00004.html. In our (Authzforce definition), for instance, even if a policy evaluates to Indeterminate, it may still be considered applicable, which makes it different from what was intended in the original conformance tests. More generally, we define here an "applicable" policy as follows: a policy is "applicable" if and only if its evaluation result is different from NotApplicable (not NotApplicable means Applicable, shouldn't it?), and one of these two conditions is met:
    • The policy/policy reference has no enclosing policy, i.e. it is the root policy in PDP's evaluation.
    • The policy has an enclosing policy and the enclosing policy is "applicable". (This definition is recursive.)
  32. These conformance tests do not include any test on VariableDefinitions/VariableReferences, such as VariableId uniqueness. We added our own in the parent directory.
  33. These conformance tests do not include any test Policy(Set)Id/RuleId uniqueness. We added our own in the parent directory.

WARNING: There are conformance tests which are intentionally not supported (in unsupported directory):

  1. IIA010, IIA012, IIA024: the test Requests contain attributes with same AttributeId in same Category but different Datatypes. We don't support different Datatypes for the same Attribute meta-data (Category/AttributeId), i.e. the PDP replies with INDETERMINATE, as we consider that a bad practice.
  2. IID029, IID030: for PDP using multiple root/initial policies. Does not apply to our implementation.
  3. III.C: Except for 001 (original is not XACML schema-compliant, because of ResourceId attribute on Result, so we fixed it by removing ResourceId), the tests are using Hierarchical Resource Profile which is not supported (except Immediate scope).
  4. IIIE - Multiple Decision Profile: only IIIE301 is supported, i.e. scheme 2.3 (repetition of Attributes elements) is supported. Other schemes of Multiple Decision Profile not supported (MultiRequests, use of multiple:content-selector attribute in Attributes), so IIIE302 and IIIE303 not supported.
  5. IIIG: IIIG001 - xpath-node-count is the only XPath function supported, so IIIG002-6 not supported.