mirror of
https://github.com/usnistgov/macos_security.git
synced 2026-02-03 05:53:24 +00:00
generate_scap.py is not functioning as expected #54
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @north-star-strategies on GitHub.
Originally assigned to: @robertgendler on GitHub.
Summary
When following the wiki to use generate_scap.py, after having walked through each of the previous steps in the wiki which did work successfuly, I encounter a host of errors when trying to generate_scap with or without flags.
Steps to reproduce
Start at the beginning of the wiki and continue going through each default step - no requirement to do otherwise - and make your way to the generate_scap.py step in the wiki. In my case I've tried and failed using NIST-800-171 and CMMClvl2.
Operating System version
Sequoia 15.5
Intel or Apple Silicon
Silicon, M1
What is the current bug behavior?
A host of errors related to the syntax or escape sequences within the generate_scap.py file
What is the expected correct behavior?
An xml file output with SCAP data
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)
Output of checks
(Paste any output that occurs with the bug)
/Users/USER/macos_security/scripts/generate_scap.py:586: SyntaxWarning: invalid escape sequence '\s'
^[^\s].*
/Users/USER/macos_security/scripts/generate_scap.py:1651: SyntaxWarning: invalid escape sequence '\s'
^[^\s].
/Users/USER/macos_security/scripts/generate_scap.py:1959: SyntaxWarning: invalid escape sequence '\s'
^[^_\s].
/Users/USER/macos_security/scripts/generate_scap.py:2010: SyntaxWarning: invalid escape sequence '['
oval_definition = re.sub('(?=\n[NOTE])(?s)(.)=\n<', '<', oval_definition)
/Users/USER/macos_security/scripts/generate_scap.py:2106: SyntaxWarning: invalid escape sequence '\s'
^[^_\s].
/Users/USER/macos_security/scripts/generate_scap.py:2201: SyntaxWarning: invalid escape sequence '\s'
^[^\s].*
/Users/USER/macos_security/scripts/generate_scap.py:2275: SyntaxWarning: invalid escape sequence '\s'
^[^\s].
/Users/USER/macos_security/scripts/generate_scap.py:2332: SyntaxWarning: invalid escape sequence '['
oval_definition = re.sub('(?=\n[NOTE])(?s)(.)=\n<', '<', oval_definition)
/Users/USER/macos_security/scripts/generate_scap.py:2544: SyntaxWarning: invalid escape sequence '('
grep_search = re.search('((.?))', s).group(1)
/Users/USER/macos_security/scripts/generate_scap.py:2559: SyntaxWarning: invalid escape sequence '\s'
{}:\s(.)$
/Users/USER/macos_security/scripts/generate_scap.py:2896: SyntaxWarning: invalid escape sequence '.'
matches = text_to_find.replace(".",".").replace(")",")").replace("(","(").replace("","*")
/Users/USER/macos_security/scripts/generate_scap.py:2896: SyntaxWarning: invalid escape sequence ')'
matches = text_to_find.replace(".",".").replace(")",")").replace("(","(").replace("","*")
/Users/USER/macos_security/scripts/generate_scap.py:2896: SyntaxWarning: invalid escape sequence '('
matches = text_to_find.replace(".",".").replace(")",")").replace("(","(").replace("","*")
/Users/USER/macos_security/scripts/generate_scap.py:2896: SyntaxWarning: invalid escape sequence '*'
matches = text_to_find.replace(".",".").replace(")",")").replace("(","(").replace("","*")
/Users/USER/macos_security/scripts/generate_scap.py:3119: SyntaxWarning: invalid escape sequence '\s'
^[^_\s].
/Users/USER/macos_security/scripts/generate_scap.py:3194: SyntaxWarning: invalid escape sequence '\s'
^[^_\s].
/Users/USER/macos_security/scripts/generate_scap.py:3247: SyntaxWarning: invalid escape sequence '['
oval_definition = re.sub('(?=\n[NOTE])(?s)(.)=\n<', '<', oval_definition)
/Users/USER/macos_security/scripts/generate_scap.py:3343: SyntaxWarning: invalid escape sequence '['
final_oval = re.sub('(?=\n[NOTE])(?s)(.)=\n$.', '<', total_oval)
/Users/USER/macos_security/scripts/generate_scap.py:3647: SyntaxWarning: invalid escape sequence '|'
all_rules.append(MacSecurityRule(rule_yaml['title'].replace('|', '|'),
/Users/USER/macos_security/scripts/generate_scap.py:3648: SyntaxWarning: invalid escape sequence '|'
rule_yaml['id'].replace('|', '|'),
/Users/USER/macos_security/scripts/generate_scap.py:3650: SyntaxWarning: invalid escape sequence '|'
rule_yaml['discussion'].replace('|', '|'),
/Users/USER/macos_security/scripts/generate_scap.py:3651: SyntaxWarning: invalid escape sequence '|'
rule_yaml['check'].replace('|', '|'),
/Users/USER/macos_security/scripts/generate_scap.py:3652: SyntaxWarning: invalid escape sequence '|'
rule_yaml['fix'].replace('|', '|'),
system_settings_time_machine_encrypted_configure - Manual Check Required
Traceback (most recent call last):
:2: SyntaxWarning: invalid escape sequence '['
:1: SyntaxWarning: invalid escape sequence '['
File "/Users/USER/macos_security/scripts/generate_scap.py", line 3734, in
main()
~~~~^^
File "/Users/USER/macos_security/scripts/generate_scap.py", line 3729, in main
generate_scap(all_rules_pruned, all_baselines, args)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/USER/macos_security/scripts/generate_scap.py", line 2332, in generate_scap
oval_definition = re.sub('(?=\n[NOTE])(?s)(.*)=\n<', '<', oval_definition)
File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/re/init.py", line 208, in sub
return _compile(pattern, flags).sub(repl, string, count)
~~~~~~~~^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/re/init.py", line 350, in _compile
p = _compiler.compile(pattern, flags)
File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/re/_compiler.py", line 748, in compile
p = _parser.parse(p, flags)
File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/re/_parser.py", line 980, in parse
p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/re/_parser.py", line 459, in _parse_sub
itemsappend(_parse(source, state, verbose, nested + 1,
~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
not nested and not items))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/re/_parser.py", line 841, in _parse
raise source.error('global flags not at the start '
'of the expression',
source.tell() - start)
re.PatternError: global flags not at the start of the expression at position 13 (line 2, column 10)
(If you can, link to the line of code that might be responsible for the problem)
@north-star-strategies commented on GitHub:
Yes, generate guidance and generate baseline both work without issue.
I initially installed python directly from python.org. More recently I attempted an installation from Homebrew to try to do a few versions back thinking that maybe the issue I was running into is because the script was written for an older version of python. Neither solution attempt worked for me.
I'm using terminal, and following the instructions step by step starting here: https://github.com/usnistgov/macos_security/wiki/Getting-Started
Everything works for me until I reach the steps laid out here: https://github.com/usnistgov/macos_security/wiki/Generate-SCAP
That's where I begin getting those errors I posted earlier.
@north-star-strategies commented on GitHub:
When I run the script I am not seeing the file you reference in the following quote get generated: "You may get an error but rename the file that gets put in build from macOS_15.0_Security_Compliance_Benchmark-Revision_1.1.xmltemp to macOS_15.0_Security_Compliance_Benchmark-Revision_1.1.xml"
@north-star-strategies commented on GitHub:
Thanks for the suggestion - unfortunately received the same result.
@robertgendler commented on GitHub:
I've never seen that error before. Something is odd. Also I just noticed it's referencing Python 3.13.
But anyway try this
./scripts/generate_scap -b cmmc_lvl2You may get an error but rename the file that gets put in build from macOS_15.0_Security_Compliance_Benchmark-Revision_1.1.xmltemp to macOS_15.0_Security_Compliance_Benchmark-Revision_1.1.xml
@robertgendler commented on GitHub:
Let me try to re create the issue.
How are you installing python?
Do the other scripts work? Generate guidance and generate baseline?
@robertgendler commented on GitHub:
if you run
which python3
rmg2@PN130628 macos_security % which python3
/usr/bin/python3
Does it come back like this? We build everything mostly using the Apple provided python from the apple command line developer tools. I'm going to install it from Python.org and see what happens.
@north-star-strategies commented on GitHub:
We have liftoff!!! Thank you very much for your help getting this working for me.
@north-star-strategies commented on GitHub:
OK thank you for testing and confirming! I will do your suggestions and report back.
@robertgendler commented on GitHub:
So I re-created the issue with python 3.13.
There are a lot of changes between python 3.9.6 and python 3.13.4.
My suggestion, remove python 3.13 and install the Apple provided one (if you type
xcode-select --installit'll prompt to install the apple command line developer tools which includes git and python and a few other things.If you're removing 3.13, delete the folder /Library/Frameworks/Python.framework and the App in /Applications. After removing that python3 will point to the Apple supplied one. You'll need to reinstall the requirements
pip3 install -r requirements.txt --user@north-star-strategies commented on GitHub:
Here's what I get: /Library/Frameworks/Python.framework/Versions/3.13/bin/python3
@robertgendler commented on GitHub:
You're welcome.
What tool will you be using the content with?
@robertgendler commented on GitHub:
Scap workbench doesn’t support macOS. Openscap tool doesn't support macOS. Tenable, bigfix, rapid7 and other major commercial vendors do not support scap on macOS as far as I know.
The SCC app as far as I know is the only freely available tool that does scap for macOS. However the old builds do not work on the most up to date macOS. They are working on it, but need funding. If your org is willing to help I connect you to that group
The other tool for macOS is JOVAL from Arctic Wolfe. No idea the cost but I've heard it's costly.
@north-star-strategies commented on GitHub:
Hey @robertgendler - thank you! Yes, I dug through the compliance script section and am on a good path forward. Re: SCAP results, was just looking for a way to validate results in a universal language for future auditors. Was primarily looking for an effective tool to visualize progress on a one-to-one relationship with controls from frameworks as opposed to the text output in the existing compliance.sh scripts that are generated. Certainly open to suggestions from your experience here - I really appreciate your help getting things rolling on this for us. We're miles ahead of when we started this conversation.
@robertgendler commented on GitHub:
@north-star-strategies What "visualizer element" are you meaning?
I also suggest reading this https://github.com/usnistgov/macos_security/wiki/Compliance-Script.
Do you have an actual purpose for SCAP results? Like actually know and understand what scap is? Or just looking for a way to scan and get results?
@north-star-strategies commented on GitHub:
I'm actually glad you kept the conversation going because...that's becoming a headache on MacOS. I see that there's the visualizer script, but it's expecting a result file, which from what I can tell is not being generated with any of the existing tools provided as a part of the project. Am I missing something about the output of the compliance scan tool and the SCAP tool that should help with the visualizer element? Tried using scap workbench locally on Mac but it's having an issue with streams.