Python | os.chmod method: StackOverflow Questions

How do you do a simple "chmod +x" from within python?

I want to create a file from within a python script that is executable.

import os
import stat
os.chmod("somefile", stat.S_IEXEC)

it appears os.chmod doesn"t "add" permissions the way unix chmod does. With the last line commented out, the file has the filemode -rw-r--r--, with it not commented out, the file mode is ---x------. How can I just add the u+x flag while keeping the rest of the modes intact?

Python module os.chmod(file, 664) does not change the permission to rw-rw-r-- but -w--wx----

Recently I am using Python module os, when I tried to change the permission of a file, I did not get the expected result. For example, I intended to change the permission to rw-rw-r--,

os.chmod("/tmp/test_file", 664)

The ownership permission is actually -w--wx--- (230)

--w--wx--- 1 ag ag 0 Mar 25 05:45 test_file

However, if I change 664 to 0664 in the code, the result is just what I need, e.g.

os.chmod("/tmp/test_file", 0664)

The result is:

-rw-rw-r-- 1 ag ag 0 Mar 25 05:55 test_file

Could anybody help explaining why does that leading 0 is so important to get the correct result?

Answer #1

1. It"s faster, os.system and subprocess.call create new processes which is unnecessary for something this simple. In fact, os.system and subprocess.call with the shell argument usually create at least two new processes: the first one being the shell, and the second one being the command that you"re running (if it"s not a shell built-in like test).

2. Some commands are useless in a separate process. For example, if you run os.spawn("cd dir/"), it will change the current working directory of the child process, but not of the Python process. You need to use os.chdir for that.

3. You don"t have to worry about special characters interpreted by the shell. os.chmod(path, mode) will work no matter what the filename is, whereas os.spawn("chmod 777 " + path) will fail horribly if the filename is something like ; rm -rf ~. (Note that you can work around this if you use subprocess.call without the shell argument.)

4. You don"t have to worry about filenames that begin with a dash. os.chmod("--quiet", mode) will change the permissions of the file named --quiet, but os.spawn("chmod 777 --quiet") will fail, as --quiet is interpreted as an argument. This is true even for subprocess.call(["chmod", "777", "--quiet"]).

5. You have fewer cross-platform and cross-shell concerns, as Python"s standard library is supposed to deal with that for you. Does your system have chmod command? Is it installed? Does it support the parameters that you expect it to support? The os module will try to be as cross-platform as possible and documents when that it"s not possible.

6. If the command you"re running has output that you care about, you need to parse it, which is trickier than it sounds, as you may forget about corner-cases (filenames with spaces, tabs and newlines in them), even when you don"t care about portability.

Answer #2

It is safer. To give you an idea here is an example script

import os
file = raw_input("Please enter a file: ")
os.system("chmod 777 " + file)

If the input from the user was test; rm -rf ~ this would then delete the home directory.

This is why it is safer to use the built in function.

Hence why you should use subprocess instead of system too.

Answer #3

The exec system call of the Linux kernel understands shebangs (#!) natively

When you do on bash:

./something

on Linux, this calls the exec system call with the path ./something.

This line of the kernel gets called on the file passed to exec: https://github.com/torvalds/linux/blob/v4.8/fs/binfmt_script.c#L25

if ((bprm->buf[0] != "#") || (bprm->buf[1] != "!"))

It reads the very first bytes of the file, and compares them to #!.

If the comparison is true, then the rest of the line is parsed by the Linux kernel, which makes another exec call with:

• executable: /usr/bin/env
• first argument: python
• second argument: script path

therefore equivalent to:

/usr/bin/env python /path/to/script.py

env is an executable that searches PATH to e.g. find /usr/bin/python, and then finally calls:

/usr/bin/python /path/to/script.py

The Python interpreter does see the #! line in the file, but # is the comment character in Python, so that line just gets ignored as a regular comment.

And yes, you can make an infinite loop with:

printf "#!/a
" | sudo tee /a
sudo chmod +x /a
/a

Bash recognizes the error:

-bash: /a: /a: bad interpreter: Too many levels of symbolic links

#! just happens to be human readable, but that is not required.

If the file started with different bytes, then the exec system call would use a different handler. The other most important built-in handler is for ELF executable files: https://github.com/torvalds/linux/blob/v4.8/fs/binfmt_elf.c#L1305 which checks for bytes 7f 45 4c 46 (which also happens to be human readable for .ELF). Let"s confirm that by reading the 4 first bytes of /bin/ls, which is an ELF executable:

head -c 4 "$(which ls)" | hd 

output:

00000000  7f 45 4c 46                                       |.ELF|
00000004                                                                 

So when the kernel sees those bytes, it takes the ELF file, puts it into memory correctly, and starts a new process with it. See also: How does kernel get an executable binary file running under linux?

Finally, you can add your own shebang handlers with the binfmt_misc mechanism. For example, you can add a custom handler for .jar files. This mechanism even supports handlers by file extension. Another application is to transparently run executables of a different architecture with QEMU.

I don"t think POSIX specifies shebangs however: https://unix.stackexchange.com/a/346214/32558 , although it does mention in on rationale sections, and in the form "if executable scripts are supported by the system something may happen". macOS and FreeBSD also seem to implement it however.

PATH search motivation

Likely, one big motivation for the existence of shebangs is the fact that in Linux, we often want to run commands from PATH just as:

basename-of-command

instead of:

/full/path/to/basename-of-command

But then, without the shebang mechanism, how would Linux know how to launch each type of file?

Hardcoding the extension in commands:

 basename-of-command.py

or implementing PATH search on every interpreter:

python basename-of-command

would be a possibility, but this has the major problem that everything breaks if we ever decide to refactor the command into another language.

Shebangs solve this problem beautifully.

Major use case of env: pyenv and other version managers

One major use case of why you should use #!/usr/bin/env python instead of just /usr/bin/python is that of version managers with pyenv.

pyenv allows you to easily install multiple python versions on a single machine, to be able to better reproduce other projects without virtualization.

Then, it manages the "current" python version by setting its order in the PATH: e.g. as shown at apt-get install for different python versions a pyenv managed python could be located at:

/home/ciro/.pyenv/shims/python

so nowhere close to /usr/bin/python, which some systems might deal with via update-alternatives symlinks.

Answer #4

There are four strong cases for preferring Python"s more-specific methods in the os module over using os.system or the subprocess module when executing a command:

• Redundancy - spawning another process is redundant and wastes time and resources.
• Portability - Many of the methods in the os module are available in multiple platforms while many shell commands are os-specific.
• Understanding the results - Spawning a process to execute arbitrary commands forces you to parse the results from the output and understand if and why a command has done something wrong.
• Safety - A process can potentially execute any command it"s given. This is a weak design and it can be avoided by using specific methods in the os module.

Redundancy (see redundant code):

You"re actually executing a redundant "middle-man" on your way to the eventual system calls (chmod in your example). This middle man is a new process or sub-shell.

From os.system:

Execute the command (a string) in a subshell ...

And subprocess is just a module to spawn new processes.

You can do what you need without spawning these processes.

Portability (see source code portability):

The os module"s aim is to provide generic operating-system services and it"s description starts with:

This module provides a portable way of using operating system dependent functionality.

You can use os.listdir on both windows and unix. Trying to use os.system / subprocess for this functionality will force you to maintain two calls (for ls / dir) and check what operating system you"re on. This is not as portable and will cause even more frustration later on (see Handling Output).

Understanding the command"s results:

Suppose you want to list the files in a directory.

If you"re using os.system("ls") / subprocess.call(["ls"]), you can only get the process"s output back, which is basically a big string with the file names.

How can you tell a file with a space in it"s name from two files?

What if you have no permission to list the files?

How should you map the data to python objects?

These are only off the top of my head, and while there are solutions to these problems - why solve again a problem that was solved for you?

This is an example of following the Don"t Repeat Yourself principle (Often reffered to as "DRY") by not repeating an implementation that already exists and is freely available for you.

Safety:

os.system and subprocess are powerful. It"s good when you need this power, but it"s dangerous when you don"t. When you use os.listdir, you know it can not do anything else other then list files or raise an error. When you use os.system or subprocess to achieve the same behaviour you can potentially end up doing something you did not mean to do.

Injection Safety (see shell injection examples):

If you use input from the user as a new command you"ve basically given him a shell. This is much like SQL injection providing a shell in the DB for the user.

An example would be a command of the form:

# ... read some user input
os.system(user_input + " some continutation")

This can be easily exploited to run any arbitrary code using the input: NASTY COMMAND;# to create the eventual:

os.system("NASTY COMMAND; # some continuation")

There are many such commands that can put your system at risk.

Answer #5

#!/usr/bin/python3 is a shebang line.

A shebang line defines where the interpreter is located. In this case, the python3 interpreter is located in /usr/bin/python3. A shebang line could also be a bash, ruby, perl or any other scripting languages" interpreter, for example: #!/bin/bash.

Without the shebang line, the operating system does not know it"s a python script, even if you set the execution flag (chmod +x script.py) on the script and run it like ./script.py. To make the script run by default in python3, either invoke it as python3 script.py or set the shebang line.

You can use #!/usr/bin/env python3 for portability across different systems in case they have the language interpreter installed in different locations.

Answer #6

os.makedirs is what you need. For chmod or chown you"ll have to use os.walk and use it on every file/dir yourself.

Answer #7

Use os.stat() to get the current permissions, use | to or the bits together, and use os.chmod() to set the updated permissions.

Example:

import os
import stat

st = os.stat("somefile")
os.chmod("somefile", st.st_mode | stat.S_IEXEC)

Answer #8

Just make sure the python executable is in your PATH environment variable then add in your script

python path/to/the/python_script.py

Details:

• In the file job.sh, put this

#!/bin/sh
python python_script.py

• Execute this command to make the script runnable for you : chmod u+x job.sh
• Run it : ./job.sh

Answer #9

create a bash script with the following:

#!/bin/bash
exec ./manage.py runserver 0.0.0.0:<your_port>

save it as runserver in the same dir as manage.py

chmod +x runserver

and run it as

./runserver

Answer #10

I think you"re a little confused. PYTHONPATH sets the search path for importing python modules, not for executing them like you"re trying.

PYTHONPATH Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored.

In addition to normal directories, individual PYTHONPATH entries may refer to zipfiles containing pure Python modules (in either source or compiled form). Extension modules cannot be imported from zipfiles.

The default search path is installation dependent, but generally begins with prefix/lib/pythonversion (see PYTHONHOME above). It is always appended to PYTHONPATH.

An additional directory will be inserted in the search path in front of PYTHONPATH as described above under Interface options. The search path can be manipulated from within a Python program as the variable sys.path.

http://docs.python.org/2/using/cmdline.html#envvar-PYTHONPATH

What you"re looking for is PATH.

export PATH=$PATH:/home/randy/lib/python 

However, to run your python script as a program, you also need to set a shebang for Python in the first line. Something like this should work:

#!/usr/bin/env python

And give execution privileges to it:

chmod +x /home/randy/lib/python/gbmx.py

Then you should be able to simply run gmbx.py from anywhere.