Python | System Gain and Compliance Reports Using Lynis

| | | | | | | | | | | |

👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!

Because Lynis is flexible, it is used for several different purposes. Typical use cases for Lynis include:

  • Security Audit
  • Compliance Testing (e.g. PCI, HIPAA, SOx)
  • Penetration Testing
  • Vulnerability detection
  • System hardening

System protection — it protects your system from potential threats and vulnerabilities. Lynis can be used to generate a detailed report on various threats and vulnerabilities on your system. The user or system administrator can then take the necessary action to secure the system.

Lynis reports are difficult to read and usually contain a lot of information. Therefore, we use Bash and Python scripts to parse the report, extract relevant information from the report, such as warnings, suggestions, and save them to an Excel file as a report.

Prerequisites for Lynis —

    You must be running a Linux / Unix based OS such as Ubuntu, Mac Os or any other Linux distribution.

  • Install Lynis on your system by cloning github repository: https://github.com/CISOfy/lynis
  • Install the pandas library using the command sudo pip3 install pandas .
  • After you have installed Lynis on your system, navigate to the Lynis directory where you will find a set of files along with an executable named Lynis.
  • Use a bash script (code below) to extract relevant information such as the warning and suggestions given in the lynis report. create a file called run.sh and copy, paste the bash code into that file and type: sudo ./run.sh to run the bash script.
  • Run the Python script (the code is given below) to clean and parse the extracted data and output the relevant information as an Excel file.

Below are Bash and Python scripts —

Script Bash:

#! / bin / bash


# script to clean / parse the report file and
# extract the relevant details and run
# Python script to display details on the server.

echo "running ......"

echo ""

sudo . / lynis audit system --quick


# execute warnings. sudo ./warnings.sh

echo "Generating warnings"

echo ""

echo "warnings are:"

echo ""

sudo cat / var / log / lynis-report . dat | grep warning | sed - e "s / warning [] = // g"

sudo cat / var / log / lynis-report . dat | grep warning | sed - e "s / warning [] = // g" | cat & gt; warnings.txt

echo ""

echo "warnings generated"

echo "output file: warnings.txt"

sudo chmod 755 warnings .txt


# execute suggestions. sudo ./suggestions.sh

echo "Generating suggestions"

echo ""

echo "suggestions are:"

echo ""

sudo cat / var / log / lynis-report . dat | grep suggestion | sed - e "s / suggestion [] = // g"

sudo cat / var / log / lynis-report . dat | grep suggestion | sed - e "s / suggestion [] = // g" | cat & gt; suggestions.txt

echo ""

echo "suggestions generated"

echo "output file: suggestions.txt"

sudo chmod 755 suggestions .txt


# execute packages. sudo ./packages.sh

echo "Generating packages"

echo ""

echo "packages are:"

echo ""

sudo cat / var / log / lynis-report . dat | grep installed_package | sed - e "s / installed_package [] = // g"

sudo cat / var / log / lynis-report . dat | grep installed_package | sed - e "s / installed_package [] = // g" | cat & gt; packages.txt

echo ""

echo "packages generated"

sudo chmod 755 packages.txt


# execute shells. sudo ./shells.sh

echo "Generating avaliable shells"

echo ""

echo "shells are:"

echo ""

sudo cat / var / log / lynis-report . dat | grep available_shell | sed - e "s / available_shell [] = // g"

sudo cat / var / log / lynis-report . dat | grep available_shell | sed - e "s / available_shell [] = // g" | cat & gt; shells.txt

echo ""

echo "shells generated"

echo "output file: shells .txt "

sudo chmod 755 shells.txt

Python script:

# import libraries

import pandas as pd

from pandas import ExcelWriter

import os


# function to get data.

def get_data ():

warnings = open ( ’ warnings.txt’ , ’r’ )

suggestions = open ( ’suggestions.txt’ , ’r’ )

packages = open ( ’packages.txt’ , ’ r’ )

shells = open ( ’shells.txt’ , ’ r’ )

warn_data = warnings.readlines ()

sugg_data = suggestions.readlines ()

pack_dat a = packages.read ()

shell_data = shells.readlines ()

return warn_data, sugg_data, pack_data, shell_data

def clean_data ():

warn, sugg, pack, shell = get_data ()

warn_clean = []

for line in warn:

warn_clean.append (line.split ( ’|’ ))

for i in range ( len (warn_clean)):

warn_clean [i] = warn_clean [i] [: 2 ]

# print (warn_clean [i])

sugg_clean = []

for line in sugg:

sugg_clean.append (line.split ( ’|’ ))

for i in range ( len (sugg_clean)):

sugg_clean [i] = sugg_clean [i] [: 2 ]

# print (sugg_clean [i])

pack_clean = []

pack = pack.split ( ’|’ )

pack_clean = pack

del pack_clean [ 0 ]

shell_clean = []

for i in range ( len (shell)):

shell_clean.append (shell [i] .rstrip ( ’’ ))

# print (shell_clean [i])

return warn_clean, sugg_clean, pack_clean, shell_clean

def convert_to_excel ():

warnings, suggestions, packages , shells = clean_data ()

try :

os.mkdir ( ’outputs’ )

except (Exception):

pass

os.chdir ( ’ outputs’ )

warn_packages = []

warn_text = []

for i in range ( len (w arnings)):

warn_packages.append (warnings [i] [ 0 ])

for i in range ( len (warnings)):

warn_text.append (warnings [i] [ 1 ])

print (warn_packages, warn_text)

warn = pd.DataFrame ()

warn [ ’Packages’ ] = warn_packages

warn [ ’warnings’ ] = warn_text

# warn.to_excel (& # 39; warnings.xlsx & # 39 ;, index = False)

writer = ExcelWriter ( ’warnings.xlsx’ )

warn.to_excel (writer, ’ report1’ , index = False )

workbook = writer.book

worksheet = writer. sheets [ ’report1’ ]

# Invoice information columns

worksheet.set_column ( ’A: A’ , 15 )

# State column

worksheet.set_column ( ’B: B’ , 45 )

# Zip code

# worksheet.set_column (& # 39; F: F & # 39 ;, 10)

writer.save ()

sugg_packages = []

sugg_text = []

for i in range ( len (suggestions)):

sugg_packages.append (suggestions [i] [ 0 ])

for i in range ( len (suggestions)):

sugg_text.append (suggestions [i] [ 1 ])

# print (sugg_packages, sugg_text)

suggestive = pd.DataFrame ()

sugg [ ’Packages’ ] = sugg_packages

[ ’suggestions’ ] = sugg_text

writer1 = ExcelWriter ( ’suggestions.xlsx’ )

sugg.to_excel (writer1, ’report2’ , index = False )

workbook = writer1.book

worksheet = writer1.sheets [ ’report2’ ]

# Invoice information columns

worksheet.set_column ( ’A: A’ , 25 )

# State column

worksheet.set_column ( ’B: B’ , 120 )

# Postal code

# worksheet.set_column (& # 39; F: F & # 39 ;, 10)

writer1.save ()

pack_data = pd.DataFrame ()

pack_data [ ’ Packages’ ] = packages

writer1 = ExcelWriter ( ’ packages.xlsx’ )

pack_data.to_excel (writer1, ’report3’ , index = False )

workbook = writer1.book

worksheet = writer1 .sheets [ ’report2’ ]

# Invoice information columns

worksheet.set_column ( ’A: A’ , 25 )

# State column

worksheet.set_column ( ’B: B’ , 120 )

# Zip code

# worksheet.set_column (& # 39; F: F & # 39 ;, 10)

writer1.save ()

👻 Read also: what is the best laptop for engineering students?

Python | System Gain and Compliance Reports Using Lynis __del__: Questions

How can I make a time delay in Python?

5 answers

I would like to know how to put a time delay in a Python script.

2973

Answer #1

import time
time.sleep(5)   # Delays for 5 seconds. You can also use a float value.

Here is another example where something is run approximately once a minute:

import time
while True:
    print("This prints once a minute.")
    time.sleep(60) # Delay for 1 minute (60 seconds).

2973

Answer #2

You can use the sleep() function in the time module. It can take a float argument for sub-second resolution.

from time import sleep
sleep(0.1) # Time in seconds

Python | System Gain and Compliance Reports Using Lynis __del__: Questions

How to delete a file or folder in Python?

5 answers

How do I delete a file or folder in Python?

2639

Answer #1


Path objects from the Python 3.4+ pathlib module also expose these instance methods:

How do I list all files of a directory?

5 answers

How can I list all files of a directory in Python and add them to a list?

3467

Answer #1

os.listdir() will get you everything that"s in a directory - files and directories.

If you want just files, you could either filter this down using os.path:

from os import listdir
from os.path import isfile, join
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

or you could use os.walk() which will yield two lists for each directory it visits - splitting into files and dirs for you. If you only want the top directory you can break the first time it yields

from os import walk

f = []
for (dirpath, dirnames, filenames) in walk(mypath):
    f.extend(filenames)
    break

or, shorter:

from os import walk

filenames = next(walk(mypath), (None, None, []))[2]  # [] if no file

3467

Answer #2

I prefer using the glob module, as it does pattern matching and expansion.

import glob
print(glob.glob("/home/adam/*"))

It does pattern matching intuitively

import glob
# All files ending with .txt
print(glob.glob("/home/adam/*.txt")) 
# All files ending with .txt with depth of 2 folder
print(glob.glob("/home/adam/*/*.txt")) 

It will return a list with the queried files:

["/home/adam/file1.txt", "/home/adam/file2.txt", .... ]

3467

Answer #3

os.listdir() - list in the current directory

With listdir in os module you get the files and the folders in the current dir

 import os
 arr = os.listdir()
 print(arr)
 
 >>> ["$RECYCLE.BIN", "work.txt", "3ebooks.txt", "documents"]

Looking in a directory

arr = os.listdir("c:\files")

glob from glob

with glob you can specify a type of file to list like this

import glob

txtfiles = []
for file in glob.glob("*.txt"):
    txtfiles.append(file)

glob in a list comprehension

mylist = [f for f in glob.glob("*.txt")]

get the full path of only files in the current directory

import os
from os import listdir
from os.path import isfile, join

cwd = os.getcwd()
onlyfiles = [os.path.join(cwd, f) for f in os.listdir(cwd) if 
os.path.isfile(os.path.join(cwd, f))]
print(onlyfiles) 

["G:\getfilesname\getfilesname.py", "G:\getfilesname\example.txt"]

Getting the full path name with os.path.abspath

You get the full path in return

 import os
 files_path = [os.path.abspath(x) for x in os.listdir()]
 print(files_path)
 
 ["F:\documentiapplications.txt", "F:\documenticollections.txt"]

Walk: going through sub directories

os.walk returns the root, the directories list and the files list, that is why I unpacked them in r, d, f in the for loop; it, then, looks for other files and directories in the subfolders of the root and so on until there are no subfolders.

import os

# Getting the current work directory (cwd)
thisdir = os.getcwd()

# r=root, d=directories, f = files
for r, d, f in os.walk(thisdir):
    for file in f:
        if file.endswith(".docx"):
            print(os.path.join(r, file))

os.listdir(): get files in the current directory (Python 2)

In Python 2, if you want the list of the files in the current directory, you have to give the argument as "." or os.getcwd() in the os.listdir method.

 import os
 arr = os.listdir(".")
 print(arr)
 
 >>> ["$RECYCLE.BIN", "work.txt", "3ebooks.txt", "documents"]

To go up in the directory tree

# Method 1
x = os.listdir("..")

# Method 2
x= os.listdir("/")

Get files: os.listdir() in a particular directory (Python 2 and 3)

 import os
 arr = os.listdir("F:\python")
 print(arr)
 
 >>> ["$RECYCLE.BIN", "work.txt", "3ebooks.txt", "documents"]

Get files of a particular subdirectory with os.listdir()

import os

x = os.listdir("./content")

os.walk(".") - current directory

 import os
 arr = next(os.walk("."))[2]
 print(arr)
 
 >>> ["5bs_Turismo1.pdf", "5bs_Turismo1.pptx", "esperienza.txt"]

next(os.walk(".")) and os.path.join("dir", "file")

 import os
 arr = []
 for d,r,f in next(os.walk("F:\_python")):
     for file in f:
         arr.append(os.path.join(r,file))

 for f in arr:
     print(files)

>>> F:\_python\dict_class.py
>>> F:\_python\programmi.txt

next(os.walk("F:\") - get the full path - list comprehension

 [os.path.join(r,file) for r,d,f in next(os.walk("F:\_python")) for file in f]
 
 >>> ["F:\_python\dict_class.py", "F:\_python\programmi.txt"]

os.walk - get full path - all files in sub dirs**

x = [os.path.join(r,file) for r,d,f in os.walk("F:\_python") for file in f]
print(x)

>>> ["F:\_python\dict.py", "F:\_python\progr.txt", "F:\_python\readl.py"]

os.listdir() - get only txt files

 arr_txt = [x for x in os.listdir() if x.endswith(".txt")]
 print(arr_txt)
 
 >>> ["work.txt", "3ebooks.txt"]

Using glob to get the full path of the files

If I should need the absolute path of the files:

from path import path
from glob import glob
x = [path(f).abspath() for f in glob("F:\*.txt")]
for f in x:
    print(f)

>>> F:acquistionline.txt
>>> F:acquisti_2018.txt
>>> F:ootstrap_jquery_ecc.txt

Using os.path.isfile to avoid directories in the list

import os.path
listOfFiles = [f for f in os.listdir() if os.path.isfile(f)]
print(listOfFiles)

>>> ["a simple game.py", "data.txt", "decorator.py"]

Using pathlib from Python 3.4

import pathlib

flist = []
for p in pathlib.Path(".").iterdir():
    if p.is_file():
        print(p)
        flist.append(p)

 >>> error.PNG
 >>> exemaker.bat
 >>> guiprova.mp3
 >>> setup.py
 >>> speak_gui2.py
 >>> thumb.PNG

With list comprehension:

flist = [p for p in pathlib.Path(".").iterdir() if p.is_file()]

Alternatively, use pathlib.Path() instead of pathlib.Path(".")

Use glob method in pathlib.Path()

import pathlib

py = pathlib.Path().glob("*.py")
for file in py:
    print(file)

>>> stack_overflow_list.py
>>> stack_overflow_list_tkinter.py

Get all and only files with os.walk

import os
x = [i[2] for i in os.walk(".")]
y=[]
for t in x:
    for f in t:
        y.append(f)
print(y)

>>> ["append_to_list.py", "data.txt", "data1.txt", "data2.txt", "data_180617", "os_walk.py", "READ2.py", "read_data.py", "somma_defaltdic.py", "substitute_words.py", "sum_data.py", "data.txt", "data1.txt", "data_180617"]

Get only files with next and walk in a directory

 import os
 x = next(os.walk("F://python"))[2]
 print(x)
 
 >>> ["calculator.bat","calculator.py"]

Get only directories with next and walk in a directory

 import os
 next(os.walk("F://python"))[1] # for the current dir use (".")
 
 >>> ["python3","others"]

Get all the subdir names with walk

for r,d,f in os.walk("F:\_python"):
    for dirs in d:
        print(dirs)

>>> .vscode
>>> pyexcel
>>> pyschool.py
>>> subtitles
>>> _metaprogramming
>>> .ipynb_checkpoints

os.scandir() from Python 3.5 and greater

import os
x = [f.name for f in os.scandir() if f.is_file()]
print(x)

>>> ["calculator.bat","calculator.py"]

# Another example with scandir (a little variation from docs.python.org)
# This one is more efficient than os.listdir.
# In this case, it shows the files only in the current directory
# where the script is executed.

import os
with os.scandir() as i:
    for entry in i:
        if entry.is_file():
            print(entry.name)

>>> ebookmaker.py
>>> error.PNG
>>> exemaker.bat
>>> guiprova.mp3
>>> setup.py
>>> speakgui4.py
>>> speak_gui2.py
>>> speak_gui3.py
>>> thumb.PNG

Examples:

Ex. 1: How many files are there in the subdirectories?

In this example, we look for the number of files that are included in all the directory and its subdirectories.

import os

def count(dir, counter=0):
    "returns number of files in dir and subdirs"
    for pack in os.walk(dir):
        for f in pack[2]:
            counter += 1
    return dir + " : " + str(counter) + "files"

print(count("F:\python"))

>>> "F:\python" : 12057 files"

Ex.2: How to copy all files from a directory to another?

A script to make order in your computer finding all files of a type (default: pptx) and copying them in a new folder.

import os
import shutil
from path import path

destination = "F:\file_copied"
# os.makedirs(destination)

def copyfile(dir, filetype="pptx", counter=0):
    "Searches for pptx (or other - pptx is the default) files and copies them"
    for pack in os.walk(dir):
        for f in pack[2]:
            if f.endswith(filetype):
                fullpath = pack[0] + "\" + f
                print(fullpath)
                shutil.copy(fullpath, destination)
                counter += 1
    if counter > 0:
        print("-" * 30)
        print("	==> Found in: `" + dir + "` : " + str(counter) + " files
")

for dir in os.listdir():
    "searches for folders that starts with `_`"
    if dir[0] == "_":
        # copyfile(dir, filetype="pdf")
        copyfile(dir, filetype="txt")


>>> _compiti18Compito Contabilità 1conti.txt
>>> _compiti18Compito Contabilità 1modula4.txt
>>> _compiti18Compito Contabilità 1moduloa4.txt
>>> ------------------------
>>> ==> Found in: `_compiti18` : 3 files

Ex. 3: How to get all the files in a txt file

In case you want to create a txt file with all the file names:

import os
mylist = ""
with open("filelist.txt", "w", encoding="utf-8") as file:
    for eachfile in os.listdir():
        mylist += eachfile + "
"
    file.write(mylist)

Example: txt with all the files of an hard drive

"""
We are going to save a txt file with all the files in your directory.
We will use the function walk()
"""

import os

# see all the methods of os
# print(*dir(os), sep=", ")
listafile = []
percorso = []
with open("lista_file.txt", "w", encoding="utf-8") as testo:
    for root, dirs, files in os.walk("D:\"):
        for file in files:
            listafile.append(file)
            percorso.append(root + "\" + file)
            testo.write(file + "
")
listafile.sort()
print("N. of files", len(listafile))
with open("lista_file_ordinata.txt", "w", encoding="utf-8") as testo_ordinato:
    for file in listafile:
        testo_ordinato.write(file + "
")

with open("percorso.txt", "w", encoding="utf-8") as file_percorso:
    for file in percorso:
        file_percorso.write(file + "
")

os.system("lista_file.txt")
os.system("lista_file_ordinata.txt")
os.system("percorso.txt")

All the file of C: in one text file

This is a shorter version of the previous code. Change the folder where to start finding the files if you need to start from another position. This code generate a 50 mb on text file on my computer with something less then 500.000 lines with files with the complete path.

import os

with open("file.txt", "w", encoding="utf-8") as filewrite:
    for r, d, f in os.walk("C:\"):
        for file in f:
            filewrite.write(f"{r + file}
")

How to write a file with all paths in a folder of a type

With this function you can create a txt file that will have the name of a type of file that you look for (ex. pngfile.txt) with all the full path of all the files of that type. It can be useful sometimes, I think.

import os

def searchfiles(extension=".ttf", folder="H:\"):
    "Create a txt file with all the file of a type"
    with open(extension[1:] + "file.txt", "w", encoding="utf-8") as filewrite:
        for r, d, f in os.walk(folder):
            for file in f:
                if file.endswith(extension):
                    filewrite.write(f"{r + file}
")

# looking for png file (fonts) in the hard disk H:
searchfiles(".png", "H:\")

>>> H:4bs_18Dolphins5.png
>>> H:4bs_18Dolphins6.png
>>> H:4bs_18Dolphins7.png
>>> H:5_18marketing htmlassetsimageslogo2.png
>>> H:7z001.png
>>> H:7z002.png

(New) Find all files and open them with tkinter GUI

I just wanted to add in this 2019 a little app to search for all files in a dir and be able to open them by doubleclicking on the name of the file in the list. enter image description here

import tkinter as tk
import os

def searchfiles(extension=".txt", folder="H:\"):
    "insert all files in the listbox"
    for r, d, f in os.walk(folder):
        for file in f:
            if file.endswith(extension):
                lb.insert(0, r + "\" + file)

def open_file():
    os.startfile(lb.get(lb.curselection()[0]))

root = tk.Tk()
root.geometry("400x400")
bt = tk.Button(root, text="Search", command=lambda:searchfiles(".png", "H:\"))
bt.pack()
lb = tk.Listbox(root)
lb.pack(fill="both", expand=1)
lb.bind("<Double-Button>", lambda x: open_file())
root.mainloop()

Shop

Learn programming in R: courses

$

Best Python online courses for 2022

$

Best laptop for Fortnite

$

Best laptop for Excel

$

Best laptop for Solidworks

$

Best laptop for Roblox

$

Best computer for crypto mining

$

Best laptop for Sims 4

$

Latest questions

NUMPYNUMPY

psycopg2: insert multiple rows with one query

12 answers

NUMPYNUMPY

How to convert Nonetype to int or string?

12 answers

NUMPYNUMPY

How to specify multiple return types using type-hints

12 answers

NUMPYNUMPY

Javascript Error: IPython is not defined in JupyterLab

12 answers


Wiki

Python OpenCV | cv2.putText () method

numpy.arctan2 () in Python

Python | os.path.realpath () method

Python OpenCV | cv2.circle () method

Python OpenCV cv2.cvtColor () method

Python - Move item to the end of the list

time.perf_counter () function in Python

Check if one list is a subset of another in Python

Python os.path.join () method