usnistgov/macos_security
1
0
Fork 0
mirror of https://github.com/usnistgov/macos_security.git synced 2026-02-03 05:53:24 +00:00
Code Issues 21 Actions Packages Projects Releases Wiki Activity

Compliance percentage incorrect when exempted rules pass #170

New Issue
Closed
opened 2026-01-19 18:29:29 +00:00 by michael · 8 comments
michael commented 2026-01-19 18:29:29 +00:00
Owner
Copy Link

Originally created by @grismemj on GitHub.

Summary

When rules that are exempted actually do not fail the test, the reported compliance percentage is incorrect.

Steps to reproduce

Create some rules that will pass, but exempt them, and then run a compliance check. It is easiest to see when things are 100% compliant, and then number of exempt rules does not match the number of failed rules.

Operating System version

I'm testing on 12.6.6, but I'm sure this will happen on all

Intel or Apple Silicon

I'm testing on Apple Silicon, but should happen on Intel too

What is the current bug behavior?

Looks like it is calculating the number of tests passed divided by the number of non-exempt rules, instead of the number of non-exempt rules passed divided by the number of non-exempt rules.

What is the expected correct behavior?

Should calculate the number of non-exempt rules passed divided by the number of non-exempt rules.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)

Output of checks

Last compliance scan: Fri Jun 2 15:18:25 EDT 2023

  1. View Last Compliance Report
  2. Run New Compliance Scan
  3. Run Commands to remediate non-compliant settings
  4. Exit
    Enter choice [ 1 - 4 ] 1

Number of tests passed: 115
Number of test FAILED: 16
Number of exempt rules: 18
You are 101.77% percent compliant!

Possible fixes

Change the calculation to number of non-exempt rules passed/number of non-exempt rules

Originally created by @grismemj on GitHub. <!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate. ---> ### Summary When rules that are exempted actually do not fail the test, the reported compliance percentage is incorrect. ### Steps to reproduce Create some rules that will pass, but exempt them, and then run a compliance check. It is easiest to see when things are 100% compliant, and then number of exempt rules does not match the number of failed rules. ### Operating System version I'm testing on 12.6.6, but I'm sure this will happen on all ### Intel or Apple Silicon I'm testing on Apple Silicon, but should happen on Intel too ### What is the current *bug* behavior? Looks like it is calculating the number of tests passed divided by the number of non-exempt rules, instead of the number of non-exempt rules passed divided by the number of non-exempt rules. ### What is the expected *correct* behavior? Should calculate the number of non-exempt rules passed divided by the number of non-exempt rules. ### Relevant logs and/or screenshots (Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.) ### Output of checks Last compliance scan: Fri Jun 2 15:18:25 EDT 2023 1. View Last Compliance Report 2. Run New Compliance Scan 3. Run Commands to remediate non-compliant settings 4. Exit Enter choice [ 1 - 4 ] 1 Number of tests passed: 115 Number of test FAILED: 16 Number of exempt rules: 18 You are 101.77% percent compliant! ### Possible fixes Change the calculation to number of non-exempt rules passed/number of non-exempt rules
michael commented 2026-01-19 18:29:29 +00:00
Author
Owner
Copy Link

@ZebGris commented on GitHub:

Looks good, thanks!

@ZebGris commented on GitHub: Looks good, thanks!
michael commented 2026-01-19 18:29:29 +00:00
Author
Owner
Copy Link

@bernstei commented on GitHub:

On a closely related note, when there are exempt rules, the output from the remediate script's report is something like

Number of tests passed: 55
Number of test FAILED: 69
Number of exempt rules: 13
You are 49.55% percent compliant!

This breakdown is then essentially useless, because it does not separate the failed rules into ones that are exempt (and presumably OK by local policy, which is why they are exempt) and ones that are not.

The report would be much more useful in the presence of exempt rules if it gave a but more information on which rules failed.

@bernstei commented on GitHub: On a closely related note, when there are exempt rules, the output from the remediate script's report is something like ``` Number of tests passed: 55 Number of test FAILED: 69 Number of exempt rules: 13 You are 49.55% percent compliant! ``` This breakdown is then essentially useless, because it does not separate the failed rules into ones that are exempt (and presumably OK by local policy, which is why they are exempt) and ones that are not. The report would be much more useful in the presence of exempt rules if it gave a but more information on which rules failed.
michael commented 2026-01-19 18:29:29 +00:00
Author
Owner
Copy Link

@brodjieski commented on GitHub:

There were adjustments to the compliance calculations made available on a branch tied to this issue (branch is named dev_ventura_issue267). Have you performed testing using this new method? It should account for exemptions on those checks that fail, and give a more accurate percentage.

As for the reporting using the script interactively, it's meant to give a quick snapshot of the status on the system when run locally (typically used during development of your baseline on a test system) Additional details about exempt rules and status of each individual check are available in the logs and corresponding .plist file. These files are available for additional processing to build your own reporting based on the data generated from the scans...whether it's in the form of a Jamf EA, Splunk analysis, or custom script.

@brodjieski commented on GitHub: There were adjustments to the compliance calculations made available on a branch tied to this issue (branch is named dev_ventura_issue267). Have you performed testing using this new method? It should account for exemptions on those checks that fail, and give a more accurate percentage. As for the reporting using the script interactively, it's meant to give a quick snapshot of the status on the system when run locally (typically used during development of your baseline on a test system) Additional details about exempt rules and status of each individual check are available in the logs and corresponding .plist file. These files are available for additional processing to build your own reporting based on the data generated from the scans...whether it's in the form of a Jamf EA, Splunk analysis, or custom script.
michael commented 2026-01-19 18:29:29 +00:00
Author
Owner
Copy Link

@brodjieski commented on GitHub:

@grismemj can you checkout the branch and test the adjustments?

@brodjieski commented on GitHub: @grismemj can you checkout the branch and test the adjustments?
michael commented 2026-01-19 18:29:29 +00:00
Author
Owner
Copy Link

@BrendaHubbell commented on GitHub:

Answer:)
To calculate the compliance percentage, we need to determine the total number of tests attempted. This can be done by adding the number of tests passed and the number of tests failed:

Total tests attempted = Number of tests passed + Number of tests failed
= 55 + 69
= 124

Next, we need to calculate the number of compliant tests. This can be done by subtracting the number of exempt rules from the number of tests passed:

Number of compliant tests = Number of tests passed - Number of exempt rules
= 55 - 13
= 42

Now, we can calculate the compliance percentage:

Compliance percentage = (Number of compliant tests / Total tests attempted) * 100
= (42 / 124) * 100
≈ 33.87%

Therefore, the correct compliance percentage is approximately 33.87%.

@BrendaHubbell commented on GitHub: Answer:) To calculate the compliance percentage, we need to determine the total number of tests attempted. This can be done by adding the number of tests passed and the number of tests failed: Total tests attempted = Number of tests passed + Number of tests failed = 55 + 69 = 124 Next, we need to calculate the number of compliant tests. This can be done by subtracting the number of exempt rules from the number of tests passed: Number of compliant tests = Number of tests passed - Number of exempt rules = 55 - 13 = 42 Now, we can calculate the compliance percentage: Compliance percentage = (Number of compliant tests / Total tests attempted) * 100 = (42 / 124) * 100 ≈ 33.87% Therefore, the correct compliance percentage is approximately 33.87%.
michael commented 2026-01-19 18:29:29 +00:00
Author
Owner
Copy Link

@BrendaHubbell commented on GitHub:

This issue is related to the usnistgov/macos_security repository and specifically to the calculation of compliance percentages when exempted rules pass.

The problem is that the compliance percentage is currently calculated based on the total number of rules, regardless of whether they are exempted or not. This means that if exempted rules pass, they are still counted as failed rules in the compliance calculation, resulting in an incorrect percentage.

To fix this issue, the compliance calculation should exclude exempted rules from the total count. Only non-exempted rules should be considered when calculating the compliance percentage.

This issue has been reported and is being tracked under Issue #267 in the usnistgov/macos_security repository. Developers are working on a solution to accurately calculate the compliance percentage by considering exempted rules correctly.

@BrendaHubbell commented on GitHub: This issue is related to the usnistgov/macos_security repository and specifically to the calculation of compliance percentages when exempted rules pass. The problem is that the compliance percentage is currently calculated based on the total number of rules, regardless of whether they are exempted or not. This means that if exempted rules pass, they are still counted as failed rules in the compliance calculation, resulting in an incorrect percentage. To fix this issue, the compliance calculation should exclude exempted rules from the total count. Only non-exempted rules should be considered when calculating the compliance percentage. This issue has been reported and is being tracked under Issue #267 in the usnistgov/macos_security repository. Developers are working on a solution to accurately calculate the compliance percentage by considering exempted rules correctly.
michael commented 2026-01-19 18:29:29 +00:00
Author
Owner
Copy Link

@bernstei commented on GitHub:

Thanks - I'll take a look at that branch.

@bernstei commented on GitHub: Thanks - I'll take a look at that branch.
michael commented 2026-01-19 18:29:29 +00:00
Author
Owner
Copy Link

@BrendaHubbell commented on GitHub:

Me va a tener que pagar!! Lol😜

@BrendaHubbell commented on GitHub: Me va a tener que pagar!! Lol😜
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
michael
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: usnistgov/macos_security#170
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?