Syntax :
dictionary_name.values()
Parameters:
No parameters
Returns :
returns a list of all the values available in a given dictionary. the values have been stored in a reversed manner.
Error :
As we are not passing any parameters there is no scope for any error.
Code # 1:
|
|
Exit:
dict_values ([2, 3, 4]) dict_values (['5 ',' 3'])
Practical Application:
Given the name and salary, return the total salary to all workers.
Code # 2:
|
|
Exit:
115000
I have two Python dictionaries, and I want to write a single expression that returns these two dictionaries, merged (i.e. taking the union). The update() method would be what I need, if it returned its result instead of modifying a dictionary in-place.
>>> x = {"a": 1, "b": 2}
>>> y = {"b": 10, "c": 11}
>>> z = x.update(y)
>>> print(z)
None
>>> x
{"a": 1, "b": 10, "c": 11}
How can I get that final merged dictionary in z, not x?
(To be extra-clear, the last-one-wins conflict-handling of dict.update() is what I"m looking for as well.)
I am a bit puzzled by the following code:
d = {"x": 1, "y": 2, "z": 3}
for key in d:
print (key, "corresponds to", d[key])
What I don"t understand is the key portion. How does Python recognize that it needs only to read the key from the dictionary? Is key a special word in Python? Or is it simply a variable?
I have a dictionary of values read from two fields in a database: a string field and a numeric field. The string field is unique, so that is the key of the dictionary.
I can sort on the keys, but how can I sort based on the values?
Note: I have read Stack Overflow question here How do I sort a list of dictionaries by a value of the dictionary? and probably could change my code to have a list of dictionaries, but since I do not really need a list of dictionaries I wanted to know if there is a simpler solution to sort either in ascending or descending order.
Is it possible to add a key to a Python dictionary after it has been created?
It doesn"t seem to have an .add() method.
I wanted to test if a key exists in a dictionary before updating the value for the key. I wrote the following code:
if "key1" in dict.keys():
print "blah"
else:
print "boo"
I think this is not the best way to accomplish this task. Is there a better way to test for a key in the dictionary?
I have a list of dictionaries and want each item to be sorted by a specific value.
Take into consideration the list:
[{"name":"Homer", "age":39}, {"name":"Bart", "age":10}]
When sorted by name, it should become:
[{"name":"Bart", "age":10}, {"name":"Homer", "age":39}]
When deleting a key from a dictionary, I use:
if "key" in my_dict:
del my_dict["key"]
Is there a one line way of doing this?
Is there a way to delete an item from a dictionary in Python?
Additionally, how can I delete an item from a dictionary to return a copy (i.e., not modifying the original)?
Imagine that you have the following list.
keys = ["name", "age", "food"]
values = ["Monty", 42, "spam"]
What is the simplest way to produce the following dictionary?
a_dict = {"name": "Monty", "age": 42, "food": "spam"}
I like the Python list comprehension syntax.
Can it be used to create dictionaries too? For example, by iterating over pairs of keys and values:
mydict = {(k,v) for (k,v) in blah blah blah} # doesn"t work
In Python, what is the purpose of
__slots__and what are the cases one should avoid this?
The special attribute __slots__ allows you to explicitly state which instance attributes you expect your object instances to have, with the expected results:
The space savings is from
__dict__.__dict__ and __weakref__ creation if parent classes deny them and you declare __slots__.Small caveat, you should only declare a particular slot one time in an inheritance tree. For example:
class Base:
__slots__ = "foo", "bar"
class Right(Base):
__slots__ = "baz",
class Wrong(Base):
__slots__ = "foo", "bar", "baz" # redundant foo and bar
Python doesn"t object when you get this wrong (it probably should), problems might not otherwise manifest, but your objects will take up more space than they otherwise should. Python 3.8:
>>> from sys import getsizeof
>>> getsizeof(Right()), getsizeof(Wrong())
(56, 72)
This is because the Base"s slot descriptor has a slot separate from the Wrong"s. This shouldn"t usually come up, but it could:
>>> w = Wrong()
>>> w.foo = "foo"
>>> Base.foo.__get__(w)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: foo
>>> Wrong.foo.__get__(w)
"foo"
The biggest caveat is for multiple inheritance - multiple "parent classes with nonempty slots" cannot be combined.
To accommodate this restriction, follow best practices: Factor out all but one or all parents" abstraction which their concrete class respectively and your new concrete class collectively will inherit from - giving the abstraction(s) empty slots (just like abstract base classes in the standard library).
See section on multiple inheritance below for an example.
To have attributes named in __slots__ to actually be stored in slots instead of a __dict__, a class must inherit from object (automatic in Python 3, but must be explicit in Python 2).
To prevent the creation of a __dict__, you must inherit from object and all classes in the inheritance must declare __slots__ and none of them can have a "__dict__" entry.
There are a lot of details if you wish to keep reading.
__slots__: Faster attribute access.The creator of Python, Guido van Rossum, states that he actually created __slots__ for faster attribute access.
It is trivial to demonstrate measurably significant faster access:
import timeit
class Foo(object): __slots__ = "foo",
class Bar(object): pass
slotted = Foo()
not_slotted = Bar()
def get_set_delete_fn(obj):
def get_set_delete():
obj.foo = "foo"
obj.foo
del obj.foo
return get_set_delete
and
>>> min(timeit.repeat(get_set_delete_fn(slotted)))
0.2846834529991611
>>> min(timeit.repeat(get_set_delete_fn(not_slotted)))
0.3664822799983085
The slotted access is almost 30% faster in Python 3.5 on Ubuntu.
>>> 0.3664822799983085 / 0.2846834529991611
1.2873325658284342
In Python 2 on Windows I have measured it about 15% faster.
__slots__: Memory SavingsAnother purpose of __slots__ is to reduce the space in memory that each object instance takes up.
My own contribution to the documentation clearly states the reasons behind this:
The space saved over using
__dict__can be significant.
SQLAlchemy attributes a lot of memory savings to __slots__.
To verify this, using the Anaconda distribution of Python 2.7 on Ubuntu Linux, with guppy.hpy (aka heapy) and sys.getsizeof, the size of a class instance without __slots__ declared, and nothing else, is 64 bytes. That does not include the __dict__. Thank you Python for lazy evaluation again, the __dict__ is apparently not called into existence until it is referenced, but classes without data are usually useless. When called into existence, the __dict__ attribute is a minimum of 280 bytes additionally.
In contrast, a class instance with __slots__ declared to be () (no data) is only 16 bytes, and 56 total bytes with one item in slots, 64 with two.
For 64 bit Python, I illustrate the memory consumption in bytes in Python 2.7 and 3.6, for __slots__ and __dict__ (no slots defined) for each point where the dict grows in 3.6 (except for 0, 1, and 2 attributes):
Python 2.7 Python 3.6
attrs __slots__ __dict__* __slots__ __dict__* | *(no slots defined)
none 16 56 + 272† 16 56 + 112† | †if __dict__ referenced
one 48 56 + 272 48 56 + 112
two 56 56 + 272 56 56 + 112
six 88 56 + 1040 88 56 + 152
11 128 56 + 1040 128 56 + 240
22 216 56 + 3344 216 56 + 408
43 384 56 + 3344 384 56 + 752
So, in spite of smaller dicts in Python 3, we see how nicely __slots__ scale for instances to save us memory, and that is a major reason you would want to use __slots__.
Just for completeness of my notes, note that there is a one-time cost per slot in the class"s namespace of 64 bytes in Python 2, and 72 bytes in Python 3, because slots use data descriptors like properties, called "members".
>>> Foo.foo
<member "foo" of "Foo" objects>
>>> type(Foo.foo)
<class "member_descriptor">
>>> getsizeof(Foo.foo)
72
__slots__:To deny the creation of a __dict__, you must subclass object. Everything subclasses object in Python 3, but in Python 2 you had to be explicit:
class Base(object):
__slots__ = ()
now:
>>> b = Base()
>>> b.a = "a"
Traceback (most recent call last):
File "<pyshell#38>", line 1, in <module>
b.a = "a"
AttributeError: "Base" object has no attribute "a"
Or subclass another class that defines __slots__
class Child(Base):
__slots__ = ("a",)
and now:
c = Child()
c.a = "a"
but:
>>> c.b = "b"
Traceback (most recent call last):
File "<pyshell#42>", line 1, in <module>
c.b = "b"
AttributeError: "Child" object has no attribute "b"
To allow __dict__ creation while subclassing slotted objects, just add "__dict__" to the __slots__ (note that slots are ordered, and you shouldn"t repeat slots that are already in parent classes):
class SlottedWithDict(Child):
__slots__ = ("__dict__", "b")
swd = SlottedWithDict()
swd.a = "a"
swd.b = "b"
swd.c = "c"
and
>>> swd.__dict__
{"c": "c"}
Or you don"t even need to declare __slots__ in your subclass, and you will still use slots from the parents, but not restrict the creation of a __dict__:
class NoSlots(Child): pass
ns = NoSlots()
ns.a = "a"
ns.b = "b"
And:
>>> ns.__dict__
{"b": "b"}
However, __slots__ may cause problems for multiple inheritance:
class BaseA(object):
__slots__ = ("a",)
class BaseB(object):
__slots__ = ("b",)
Because creating a child class from parents with both non-empty slots fails:
>>> class Child(BaseA, BaseB): __slots__ = ()
Traceback (most recent call last):
File "<pyshell#68>", line 1, in <module>
class Child(BaseA, BaseB): __slots__ = ()
TypeError: Error when calling the metaclass bases
multiple bases have instance lay-out conflict
If you run into this problem, You could just remove __slots__ from the parents, or if you have control of the parents, give them empty slots, or refactor to abstractions:
from abc import ABC
class AbstractA(ABC):
__slots__ = ()
class BaseA(AbstractA):
__slots__ = ("a",)
class AbstractB(ABC):
__slots__ = ()
class BaseB(AbstractB):
__slots__ = ("b",)
class Child(AbstractA, AbstractB):
__slots__ = ("a", "b")
c = Child() # no problem!
"__dict__" to __slots__ to get dynamic assignment:class Foo(object):
__slots__ = "bar", "baz", "__dict__"
and now:
>>> foo = Foo()
>>> foo.boink = "boink"
So with "__dict__" in slots we lose some of the size benefits with the upside of having dynamic assignment and still having slots for the names we do expect.
When you inherit from an object that isn"t slotted, you get the same sort of semantics when you use __slots__ - names that are in __slots__ point to slotted values, while any other values are put in the instance"s __dict__.
Avoiding __slots__ because you want to be able to add attributes on the fly is actually not a good reason - just add "__dict__" to your __slots__ if this is required.
You can similarly add __weakref__ to __slots__ explicitly if you need that feature.
The namedtuple builtin make immutable instances that are very lightweight (essentially, the size of tuples) but to get the benefits, you need to do it yourself if you subclass them:
from collections import namedtuple
class MyNT(namedtuple("MyNT", "bar baz")):
"""MyNT is an immutable and lightweight object"""
__slots__ = ()
usage:
>>> nt = MyNT("bar", "baz")
>>> nt.bar
"bar"
>>> nt.baz
"baz"
And trying to assign an unexpected attribute raises an AttributeError because we have prevented the creation of __dict__:
>>> nt.quux = "quux"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: "MyNT" object has no attribute "quux"
You can allow __dict__ creation by leaving off __slots__ = (), but you can"t use non-empty __slots__ with subtypes of tuple.
Even when non-empty slots are the same for multiple parents, they cannot be used together:
class Foo(object):
__slots__ = "foo", "bar"
class Bar(object):
__slots__ = "foo", "bar" # alas, would work if empty, i.e. ()
>>> class Baz(Foo, Bar): pass
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
multiple bases have instance lay-out conflict
Using an empty __slots__ in the parent seems to provide the most flexibility, allowing the child to choose to prevent or allow (by adding "__dict__" to get dynamic assignment, see section above) the creation of a __dict__:
class Foo(object): __slots__ = ()
class Bar(object): __slots__ = ()
class Baz(Foo, Bar): __slots__ = ("foo", "bar")
b = Baz()
b.foo, b.bar = "foo", "bar"
You don"t have to have slots - so if you add them, and remove them later, it shouldn"t cause any problems.
Going out on a limb here: If you"re composing mixins or using abstract base classes, which aren"t intended to be instantiated, an empty __slots__ in those parents seems to be the best way to go in terms of flexibility for subclassers.
To demonstrate, first, let"s create a class with code we"d like to use under multiple inheritance
class AbstractBase:
__slots__ = ()
def __init__(self, a, b):
self.a = a
self.b = b
def __repr__(self):
return f"{type(self).__name__}({repr(self.a)}, {repr(self.b)})"
We could use the above directly by inheriting and declaring the expected slots:
class Foo(AbstractBase):
__slots__ = "a", "b"
But we don"t care about that, that"s trivial single inheritance, we need another class we might also inherit from, maybe with a noisy attribute:
class AbstractBaseC:
__slots__ = ()
@property
def c(self):
print("getting c!")
return self._c
@c.setter
def c(self, arg):
print("setting c!")
self._c = arg
Now if both bases had nonempty slots, we couldn"t do the below. (In fact, if we wanted, we could have given AbstractBase nonempty slots a and b, and left them out of the below declaration - leaving them in would be wrong):
class Concretion(AbstractBase, AbstractBaseC):
__slots__ = "a b _c".split()
And now we have functionality from both via multiple inheritance, and can still deny __dict__ and __weakref__ instantiation:
>>> c = Concretion("a", "b")
>>> c.c = c
setting c!
>>> c.c
getting c!
Concretion("a", "b")
>>> c.d = "d"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: "Concretion" object has no attribute "d"
__class__ assignment with another class that doesn"t have them (and you can"t add them) unless the slot layouts are identical. (I am very interested in learning who is doing this and why.)You may be able to tease out further caveats from the rest of the __slots__ documentation (the 3.7 dev docs are the most current), which I have made significant recent contributions to.
The current top answers cite outdated information and are quite hand-wavy and miss the mark in some important ways.
__slots__ when instantiating lots of objects"I quote:
"You would want to use
__slots__if you are going to instantiate a lot (hundreds, thousands) of objects of the same class."
Abstract Base Classes, for example, from the collections module, are not instantiated, yet __slots__ are declared for them.
Why?
If a user wishes to deny __dict__ or __weakref__ creation, those things must not be available in the parent classes.
__slots__ contributes to reusability when creating interfaces or mixins.
It is true that many Python users aren"t writing for reusability, but when you are, having the option to deny unnecessary space usage is valuable.
__slots__ doesn"t break picklingWhen pickling a slotted object, you may find it complains with a misleading TypeError:
>>> pickle.loads(pickle.dumps(f))
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled
This is actually incorrect. This message comes from the oldest protocol, which is the default. You can select the latest protocol with the -1 argument. In Python 2.7 this would be 2 (which was introduced in 2.3), and in 3.6 it is 4.
>>> pickle.loads(pickle.dumps(f, -1))
<__main__.Foo object at 0x1129C770>
in Python 2.7:
>>> pickle.loads(pickle.dumps(f, 2))
<__main__.Foo object at 0x1129C770>
in Python 3.6
>>> pickle.loads(pickle.dumps(f, 4))
<__main__.Foo object at 0x1129C770>
So I would keep this in mind, as it is a solved problem.
The first paragraph is half short explanation, half predictive. Here"s the only part that actually answers the question
The proper use of
__slots__is to save space in objects. Instead of having a dynamic dict that allows adding attributes to objects at anytime, there is a static structure which does not allow additions after creation. This saves the overhead of one dict for every object that uses slots
The second half is wishful thinking, and off the mark:
While this is sometimes a useful optimization, it would be completely unnecessary if the Python interpreter was dynamic enough so that it would only require the dict when there actually were additions to the object.
Python actually does something similar to this, only creating the __dict__ when it is accessed, but creating lots of objects with no data is fairly ridiculous.
The second paragraph oversimplifies and misses actual reasons to avoid __slots__. The below is not a real reason to avoid slots (for actual reasons, see the rest of my answer above.):
They change the behavior of the objects that have slots in a way that can be abused by control freaks and static typing weenies.
It then goes on to discuss other ways of accomplishing that perverse goal with Python, not discussing anything to do with __slots__.
The third paragraph is more wishful thinking. Together it is mostly off-the-mark content that the answerer didn"t even author and contributes to ammunition for critics of the site.
Create some normal objects and slotted objects:
>>> class Foo(object): pass
>>> class Bar(object): __slots__ = ()
Instantiate a million of them:
>>> foos = [Foo() for f in xrange(1000000)]
>>> bars = [Bar() for b in xrange(1000000)]
Inspect with guppy.hpy().heap():
>>> guppy.hpy().heap()
Partition of a set of 2028259 objects. Total size = 99763360 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1000000 49 64000000 64 64000000 64 __main__.Foo
1 169 0 16281480 16 80281480 80 list
2 1000000 49 16000000 16 96281480 97 __main__.Bar
3 12284 1 987472 1 97268952 97 str
...
Access the regular objects and their __dict__ and inspect again:
>>> for f in foos:
... f.__dict__
>>> guppy.hpy().heap()
Partition of a set of 3028258 objects. Total size = 379763480 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1000000 33 280000000 74 280000000 74 dict of __main__.Foo
1 1000000 33 64000000 17 344000000 91 __main__.Foo
2 169 0 16281480 4 360281480 95 list
3 1000000 33 16000000 4 376281480 99 __main__.Bar
4 12284 0 987472 0 377268952 99 str
...
This is consistent with the history of Python, from Unifying types and classes in Python 2.2
If you subclass a built-in type, extra space is automatically added to the instances to accomodate
__dict__and__weakrefs__. (The__dict__is not initialized until you use it though, so you shouldn"t worry about the space occupied by an empty dictionary for each instance you create.) If you don"t need this extra space, you can add the phrase "__slots__ = []" to your class.
os.listdir()- list in the current directory
With listdir in os module you get the files and the folders in the current dir
import os
arr = os.listdir()
print(arr)
>>> ["$RECYCLE.BIN", "work.txt", "3ebooks.txt", "documents"]
arr = os.listdir("c:\files")
globfrom glob
with glob you can specify a type of file to list like this
import glob
txtfiles = []
for file in glob.glob("*.txt"):
txtfiles.append(file)
glob in a list comprehensionmylist = [f for f in glob.glob("*.txt")]
import os
from os import listdir
from os.path import isfile, join
cwd = os.getcwd()
onlyfiles = [os.path.join(cwd, f) for f in os.listdir(cwd) if
os.path.isfile(os.path.join(cwd, f))]
print(onlyfiles)
["G:\getfilesname\getfilesname.py", "G:\getfilesname\example.txt"]
Getting the full path name with
os.path.abspath
You get the full path in return
import os
files_path = [os.path.abspath(x) for x in os.listdir()]
print(files_path)
["F:\documentiapplications.txt", "F:\documenticollections.txt"]
Walk: going through sub directories
os.walk returns the root, the directories list and the files list, that is why I unpacked them in r, d, f in the for loop; it, then, looks for other files and directories in the subfolders of the root and so on until there are no subfolders.
import os
# Getting the current work directory (cwd)
thisdir = os.getcwd()
# r=root, d=directories, f = files
for r, d, f in os.walk(thisdir):
for file in f:
if file.endswith(".docx"):
print(os.path.join(r, file))
os.listdir(): get files in the current directory (Python 2)
In Python 2, if you want the list of the files in the current directory, you have to give the argument as "." or os.getcwd() in the os.listdir method.
import os
arr = os.listdir(".")
print(arr)
>>> ["$RECYCLE.BIN", "work.txt", "3ebooks.txt", "documents"]
To go up in the directory tree
# Method 1
x = os.listdir("..")
# Method 2
x= os.listdir("/")
Get files:
os.listdir()in a particular directory (Python 2 and 3)
import os
arr = os.listdir("F:\python")
print(arr)
>>> ["$RECYCLE.BIN", "work.txt", "3ebooks.txt", "documents"]
Get files of a particular subdirectory with
os.listdir()
import os
x = os.listdir("./content")
os.walk(".")- current directory
import os
arr = next(os.walk("."))[2]
print(arr)
>>> ["5bs_Turismo1.pdf", "5bs_Turismo1.pptx", "esperienza.txt"]
next(os.walk("."))andos.path.join("dir", "file")
import os
arr = []
for d,r,f in next(os.walk("F:\_python")):
for file in f:
arr.append(os.path.join(r,file))
for f in arr:
print(files)
>>> F:\_python\dict_class.py
>>> F:\_python\programmi.txt
next(os.walk("F:\")- get the full path - list comprehension
[os.path.join(r,file) for r,d,f in next(os.walk("F:\_python")) for file in f]
>>> ["F:\_python\dict_class.py", "F:\_python\programmi.txt"]
os.walk- get full path - all files in sub dirs**
x = [os.path.join(r,file) for r,d,f in os.walk("F:\_python") for file in f]
print(x)
>>> ["F:\_python\dict.py", "F:\_python\progr.txt", "F:\_python\readl.py"]
os.listdir()- get only txt files
arr_txt = [x for x in os.listdir() if x.endswith(".txt")]
print(arr_txt)
>>> ["work.txt", "3ebooks.txt"]
Using
globto get the full path of the files
If I should need the absolute path of the files:
from path import path
from glob import glob
x = [path(f).abspath() for f in glob("F:\*.txt")]
for f in x:
print(f)
>>> F:acquistionline.txt
>>> F:acquisti_2018.txt
>>> F:ootstrap_jquery_ecc.txt
Using
os.path.isfileto avoid directories in the list
import os.path
listOfFiles = [f for f in os.listdir() if os.path.isfile(f)]
print(listOfFiles)
>>> ["a simple game.py", "data.txt", "decorator.py"]
Using
pathlibfrom Python 3.4
import pathlib
flist = []
for p in pathlib.Path(".").iterdir():
if p.is_file():
print(p)
flist.append(p)
>>> error.PNG
>>> exemaker.bat
>>> guiprova.mp3
>>> setup.py
>>> speak_gui2.py
>>> thumb.PNG
With list comprehension:
flist = [p for p in pathlib.Path(".").iterdir() if p.is_file()]
Alternatively, use pathlib.Path() instead of pathlib.Path(".")
Use glob method in pathlib.Path()
import pathlib
py = pathlib.Path().glob("*.py")
for file in py:
print(file)
>>> stack_overflow_list.py
>>> stack_overflow_list_tkinter.py
Get all and only files with os.walk
import os
x = [i[2] for i in os.walk(".")]
y=[]
for t in x:
for f in t:
y.append(f)
print(y)
>>> ["append_to_list.py", "data.txt", "data1.txt", "data2.txt", "data_180617", "os_walk.py", "READ2.py", "read_data.py", "somma_defaltdic.py", "substitute_words.py", "sum_data.py", "data.txt", "data1.txt", "data_180617"]
Get only files with next and walk in a directory
import os
x = next(os.walk("F://python"))[2]
print(x)
>>> ["calculator.bat","calculator.py"]
Get only directories with next and walk in a directory
import os
next(os.walk("F://python"))[1] # for the current dir use (".")
>>> ["python3","others"]
Get all the subdir names with
walk
for r,d,f in os.walk("F:\_python"):
for dirs in d:
print(dirs)
>>> .vscode
>>> pyexcel
>>> pyschool.py
>>> subtitles
>>> _metaprogramming
>>> .ipynb_checkpoints
os.scandir()from Python 3.5 and greater
import os
x = [f.name for f in os.scandir() if f.is_file()]
print(x)
>>> ["calculator.bat","calculator.py"]
# Another example with scandir (a little variation from docs.python.org)
# This one is more efficient than os.listdir.
# In this case, it shows the files only in the current directory
# where the script is executed.
import os
with os.scandir() as i:
for entry in i:
if entry.is_file():
print(entry.name)
>>> ebookmaker.py
>>> error.PNG
>>> exemaker.bat
>>> guiprova.mp3
>>> setup.py
>>> speakgui4.py
>>> speak_gui2.py
>>> speak_gui3.py
>>> thumb.PNG
Examples:
Ex. 1: How many files are there in the subdirectories?
In this example, we look for the number of files that are included in all the directory and its subdirectories.
import os
def count(dir, counter=0):
"returns number of files in dir and subdirs"
for pack in os.walk(dir):
for f in pack[2]:
counter += 1
return dir + " : " + str(counter) + "files"
print(count("F:\python"))
>>> "F:\python" : 12057 files"
Ex.2: How to copy all files from a directory to another?
A script to make order in your computer finding all files of a type (default: pptx) and copying them in a new folder.
import os
import shutil
from path import path
destination = "F:\file_copied"
# os.makedirs(destination)
def copyfile(dir, filetype="pptx", counter=0):
"Searches for pptx (or other - pptx is the default) files and copies them"
for pack in os.walk(dir):
for f in pack[2]:
if f.endswith(filetype):
fullpath = pack[0] + "\" + f
print(fullpath)
shutil.copy(fullpath, destination)
counter += 1
if counter > 0:
print("-" * 30)
print(" ==> Found in: `" + dir + "` : " + str(counter) + " files
")
for dir in os.listdir():
"searches for folders that starts with `_`"
if dir[0] == "_":
# copyfile(dir, filetype="pdf")
copyfile(dir, filetype="txt")
>>> _compiti18Compito Contabilità 1conti.txt
>>> _compiti18Compito Contabilità 1modula4.txt
>>> _compiti18Compito Contabilità 1moduloa4.txt
>>> ------------------------
>>> ==> Found in: `_compiti18` : 3 files
Ex. 3: How to get all the files in a txt file
In case you want to create a txt file with all the file names:
import os
mylist = ""
with open("filelist.txt", "w", encoding="utf-8") as file:
for eachfile in os.listdir():
mylist += eachfile + "
"
file.write(mylist)
Example: txt with all the files of an hard drive
"""
We are going to save a txt file with all the files in your directory.
We will use the function walk()
"""
import os
# see all the methods of os
# print(*dir(os), sep=", ")
listafile = []
percorso = []
with open("lista_file.txt", "w", encoding="utf-8") as testo:
for root, dirs, files in os.walk("D:\"):
for file in files:
listafile.append(file)
percorso.append(root + "\" + file)
testo.write(file + "
")
listafile.sort()
print("N. of files", len(listafile))
with open("lista_file_ordinata.txt", "w", encoding="utf-8") as testo_ordinato:
for file in listafile:
testo_ordinato.write(file + "
")
with open("percorso.txt", "w", encoding="utf-8") as file_percorso:
for file in percorso:
file_percorso.write(file + "
")
os.system("lista_file.txt")
os.system("lista_file_ordinata.txt")
os.system("percorso.txt")
All the file of C: in one text file
This is a shorter version of the previous code. Change the folder where to start finding the files if you need to start from another position. This code generate a 50 mb on text file on my computer with something less then 500.000 lines with files with the complete path.
import os
with open("file.txt", "w", encoding="utf-8") as filewrite:
for r, d, f in os.walk("C:\"):
for file in f:
filewrite.write(f"{r + file}
")
How to write a file with all paths in a folder of a type
With this function you can create a txt file that will have the name of a type of file that you look for (ex. pngfile.txt) with all the full path of all the files of that type. It can be useful sometimes, I think.
import os
def searchfiles(extension=".ttf", folder="H:\"):
"Create a txt file with all the file of a type"
with open(extension[1:] + "file.txt", "w", encoding="utf-8") as filewrite:
for r, d, f in os.walk(folder):
for file in f:
if file.endswith(extension):
filewrite.write(f"{r + file}
")
# looking for png file (fonts) in the hard disk H:
searchfiles(".png", "H:\")
>>> H:4bs_18Dolphins5.png
>>> H:4bs_18Dolphins6.png
>>> H:4bs_18Dolphins7.png
>>> H:5_18marketing htmlassetsimageslogo2.png
>>> H:7z001.png
>>> H:7z002.png
(New) Find all files and open them with tkinter GUI
I just wanted to add in this 2019 a little app to search for all files in a dir and be able to open them by doubleclicking on the name of the file in the list.
import tkinter as tk
import os
def searchfiles(extension=".txt", folder="H:\"):
"insert all files in the listbox"
for r, d, f in os.walk(folder):
for file in f:
if file.endswith(extension):
lb.insert(0, r + "\" + file)
def open_file():
os.startfile(lb.get(lb.curselection()[0]))
root = tk.Tk()
root.geometry("400x400")
bt = tk.Button(root, text="Search", command=lambda:searchfiles(".png", "H:\"))
bt.pack()
lb = tk.Listbox(root)
lb.pack(fill="both", expand=1)
lb.bind("<Double-Button>", lambda x: open_file())
root.mainloop()
TL;DR: If you are using Python 3.10 or later, it just works. As of today (2019), in 3.7+ you must turn this feature on using a future statement (from __future__ import annotations). In Python 3.6 or below, use a string.
I guess you got this exception:
NameError: name "Position" is not defined
This is because Position must be defined before you can use it in an annotation unless you are using Python 3.10 or later.
from __future__ import annotationsPython 3.7 introduces PEP 563: postponed evaluation of annotations. A module that uses the future statement from __future__ import annotations will store annotations as strings automatically:
from __future__ import annotations
class Position:
def __add__(self, other: Position) -> Position:
...
This is scheduled to become the default in Python 3.10. Since Python still is a dynamically typed language so no type checking is done at runtime, typing annotations should have no performance impact, right? Wrong! Before python 3.7 the typing module used to be one of the slowest python modules in core so if you import typing you will see up to 7 times increase in performance when you upgrade to 3.7.
According to PEP 484, you should use a string instead of the class itself:
class Position:
...
def __add__(self, other: "Position") -> "Position":
...
If you use the Django framework this may be familiar as Django models also use strings for forward references (foreign key definitions where the foreign model is self or is not declared yet). This should work with Pycharm and other tools.
The relevant parts of PEP 484 and PEP 563, to spare you the trip:
Forward references
When a type hint contains names that have not been defined yet, that definition may be expressed as a string literal, to be resolved later.
A situation where this occurs commonly is the definition of a container class, where the class being defined occurs in the signature of some of the methods. For example, the following code (the start of a simple binary tree implementation) does not work:
class Tree: def __init__(self, left: Tree, right: Tree): self.left = left self.right = rightTo address this, we write:
class Tree: def __init__(self, left: "Tree", right: "Tree"): self.left = left self.right = rightThe string literal should contain a valid Python expression (i.e., compile(lit, "", "eval") should be a valid code object) and it should evaluate without errors once the module has been fully loaded. The local and global namespace in which it is evaluated should be the same namespaces in which default arguments to the same function would be evaluated.
and PEP 563:
Implementation
In Python 3.10, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective 
__annotations__ dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation....
Enabling the future behavior in Python 3.7
The functionality described above can be enabled starting from Python 3.7 using the following special import:
from __future__ import annotations
PositionBefore the class definition, place a dummy definition:
class Position(object):
pass
class Position(object):
...
This will get rid of the NameError and may even look OK:
>>> Position.__add__.__annotations__
{"other": __main__.Position, "return": __main__.Position}
But is it?
>>> for k, v in Position.__add__.__annotations__.items():
... print(k, "is Position:", v is Position)
return is Position: False
other is Position: False
You may want to try some Python meta programming magic and write a decorator to monkey-patch the class definition in order to add annotations:
class Position:
...
def __add__(self, other):
return self.__class__(self.x + other.x, self.y + other.y)
The decorator should be responsible for the equivalent of this:
Position.__add__.__annotations__["return"] = Position
Position.__add__.__annotations__["other"] = Position
At least it seems right:
>>> for k, v in Position.__add__.__annotations__.items():
... print(k, "is Position:", v is Position)
return is Position: True
other is Position: True
Probably too much trouble.
As you are in python3 , use dict.items() instead of dict.iteritems()
iteritems() was removed in python3, so you can"t use this method anymore.
Take a look at Python 3.0 Wiki Built-in Changes section, where it is stated:
Removed
dict.iteritems(),dict.iterkeys(), anddict.itervalues().Instead: use
dict.items(),dict.keys(), anddict.values()respectively.
My quick & dirty JSON dump that eats dates and everything:
json.dumps(my_dictionary, indent=4, sort_keys=True, default=str)
default is a function applied to objects that aren"t serializable.
In this case it"s str, so it just converts everything it doesn"t know to strings. Which is great for serialization but not so great when deserializing (hence the "quick & dirty") as anything might have been string-ified without warning, e.g. a function or numpy array.
a = numpy.array([0, 3, 0, 1, 0, 1, 2, 1, 0, 0, 0, 0, 1, 3, 4])
unique, counts = numpy.unique(a, return_counts=True)
dict(zip(unique, counts))
# {0: 7, 1: 4, 2: 1, 3: 2, 4: 1}
Non-numpy way:
Use collections.Counter;
import collections, numpy
a = numpy.array([0, 3, 0, 1, 0, 1, 2, 1, 0, 0, 0, 0, 1, 3, 4])
collections.Counter(a)
# Counter({0: 7, 1: 4, 3: 2, 2: 1, 4: 1})
Because [] and {} are literal syntax. Python can create bytecode just to create the list or dictionary objects:
>>> import dis
>>> dis.dis(compile("[]", "", "eval"))
1 0 BUILD_LIST 0
3 RETURN_VALUE
>>> dis.dis(compile("{}", "", "eval"))
1 0 BUILD_MAP 0
3 RETURN_VALUE
list() and dict() are separate objects. Their names need to be resolved, the stack has to be involved to push the arguments, the frame has to be stored to retrieve later, and a call has to be made. That all takes more time.
For the empty case, that means you have at the very least a LOAD_NAME (which has to search through the global namespace as well as the builtins module) followed by a CALL_FUNCTION, which has to preserve the current frame:
>>> dis.dis(compile("list()", "", "eval"))
1 0 LOAD_NAME 0 (list)
3 CALL_FUNCTION 0
6 RETURN_VALUE
>>> dis.dis(compile("dict()", "", "eval"))
1 0 LOAD_NAME 0 (dict)
3 CALL_FUNCTION 0
6 RETURN_VALUE
You can time the name lookup separately with timeit:
>>> import timeit
>>> timeit.timeit("list", number=10**7)
0.30749011039733887
>>> timeit.timeit("dict", number=10**7)
0.4215109348297119
The time discrepancy there is probably a dictionary hash collision. Subtract those times from the times for calling those objects, and compare the result against the times for using literals:
>>> timeit.timeit("[]", number=10**7)
0.30478692054748535
>>> timeit.timeit("{}", number=10**7)
0.31482696533203125
>>> timeit.timeit("list()", number=10**7)
0.9991960525512695
>>> timeit.timeit("dict()", number=10**7)
1.0200958251953125
So having to call the object takes an additional 1.00 - 0.31 - 0.30 == 0.39 seconds per 10 million calls.
You can avoid the global lookup cost by aliasing the global names as locals (using a timeit setup, everything you bind to a name is a local):
>>> timeit.timeit("_list", "_list = list", number=10**7)
0.1866450309753418
>>> timeit.timeit("_dict", "_dict = dict", number=10**7)
0.19016098976135254
>>> timeit.timeit("_list()", "_list = list", number=10**7)
0.841480016708374
>>> timeit.timeit("_dict()", "_dict = dict", number=10**7)
0.7233691215515137
but you never can overcome that CALL_FUNCTION cost.
There are many ways to convert an instance to a dictionary, with varying degrees of corner case handling and closeness to the desired result.
instance.__dict__instance.__dict__
which returns
{"_foreign_key_cache": <OtherModel: OtherModel object>,
"_state": <django.db.models.base.ModelState at 0x7ff0993f6908>,
"auto_now_add": datetime.datetime(2018, 12, 20, 21, 34, 29, 494827, tzinfo=<UTC>),
"foreign_key_id": 2,
"id": 1,
"normal_value": 1,
"readonly_value": 2}
This is by far the simplest, but is missing many_to_many, foreign_key is misnamed, and it has two unwanted extra things in it.
model_to_dictfrom django.forms.models import model_to_dict
model_to_dict(instance)
which returns
{"foreign_key": 2,
"id": 1,
"many_to_many": [<OtherModel: OtherModel object>],
"normal_value": 1}
This is the only one with many_to_many, but is missing the uneditable fields.
model_to_dict(..., fields=...)from django.forms.models import model_to_dict
model_to_dict(instance, fields=[field.name for field in instance._meta.fields])
which returns
{"foreign_key": 2, "id": 1, "normal_value": 1}
This is strictly worse than the standard model_to_dict invocation.
query_set.values()SomeModel.objects.filter(id=instance.id).values()[0]
which returns
{"auto_now_add": datetime.datetime(2018, 12, 20, 21, 34, 29, 494827, tzinfo=<UTC>),
"foreign_key_id": 2,
"id": 1,
"normal_value": 1,
"readonly_value": 2}
This is the same output as instance.__dict__ but without the extra fields.
foreign_key_id is still wrong and many_to_many is still missing.
The code for django"s model_to_dict had most of the answer. It explicitly removed non-editable fields, so removing that check and getting the ids of foreign keys for many to many fields results in the following code which behaves as desired:
from itertools import chain
def to_dict(instance):
opts = instance._meta
data = {}
for f in chain(opts.concrete_fields, opts.private_fields):
data[f.name] = f.value_from_object(instance)
for f in opts.many_to_many:
data[f.name] = [i.id for i in f.value_from_object(instance)]
return data
While this is the most complicated option, calling to_dict(instance) gives us exactly the desired result:
{"auto_now_add": datetime.datetime(2018, 12, 20, 21, 34, 29, 494827, tzinfo=<UTC>),
"foreign_key": 2,
"id": 1,
"many_to_many": [2],
"normal_value": 1,
"readonly_value": 2}
Django Rest Framework"s ModelSerialzer allows you to build a serializer automatically from a model.
from rest_framework import serializers
class SomeModelSerializer(serializers.ModelSerializer):
class Meta:
model = SomeModel
fields = "__all__"
SomeModelSerializer(instance).data
returns
{"auto_now_add": "2018-12-20T21:34:29.494827Z",
"foreign_key": 2,
"id": 1,
"many_to_many": [2],
"normal_value": 1,
"readonly_value": 2}
This is almost as good as the custom function, but auto_now_add is a string instead of a datetime object.
If you want a django model that has a better python command-line display, have your models child-class the following:
from django.db import models
from itertools import chain
class PrintableModel(models.Model):
def __repr__(self):
return str(self.to_dict())
def to_dict(instance):
opts = instance._meta
data = {}
for f in chain(opts.concrete_fields, opts.private_fields):
data[f.name] = f.value_from_object(instance)
for f in opts.many_to_many:
data[f.name] = [i.id for i in f.value_from_object(instance)]
return data
class Meta:
abstract = True
So, for example, if we define our models as such:
class OtherModel(PrintableModel): pass
class SomeModel(PrintableModel):
normal_value = models.IntegerField()
readonly_value = models.IntegerField(editable=False)
auto_now_add = models.DateTimeField(auto_now_add=True)
foreign_key = models.ForeignKey(OtherModel, related_name="ref1")
many_to_many = models.ManyToManyField(OtherModel, related_name="ref2")
Calling SomeModel.objects.first() now gives output like this:
{"auto_now_add": datetime.datetime(2018, 12, 20, 21, 34, 29, 494827, tzinfo=<UTC>),
"foreign_key": 2,
"id": 1,
"many_to_many": [2],
"normal_value": 1,
"readonly_value": 2}
Are dictionaries ordered in Python 3.6+?
They are insertion ordered[1]. As of Python 3.6, for the CPython implementation of Python, dictionaries remember the order of items inserted. This is considered an implementation detail in Python 3.6; you need to use OrderedDict if you want insertion ordering that"s guaranteed across other implementations of Python (and other ordered behavior[1]).
As of Python 3.7, this is no longer an implementation detail and instead becomes a language feature. From a python-dev message by GvR:
Make it so. "Dict keeps insertion order" is the ruling. Thanks!
This simply means that you can depend on it. Other implementations of Python must also offer an insertion ordered dictionary if they wish to be a conforming implementation of Python 3.7.
How does the Python
3.6dictionary implementation perform better[2] than the older one while preserving element order?
Essentially, by keeping two arrays.
The first array, dk_entries, holds the entries (of type PyDictKeyEntry) for the dictionary in the order that they were inserted. Preserving order is achieved by this being an append only array where new items are always inserted at the end (insertion order).
The second, dk_indices, holds the indices for the dk_entries array (that is, values that indicate the position of the corresponding entry in dk_entries). This array acts as the hash table. When a key is hashed it leads to one of the indices stored in dk_indices and the corresponding entry is fetched by indexing dk_entries. Since only indices are kept, the type of this array depends on the overall size of the dictionary (ranging from type int8_t(1 byte) to int32_t/int64_t (4/8 bytes) on 32/64 bit builds)
In the previous implementation, a sparse array of type PyDictKeyEntry and size dk_size had to be allocated; unfortunately, it also resulted in a lot of empty space since that array was not allowed to be more than 2/3 * dk_size full for performance reasons. (and the empty space still had PyDictKeyEntry size!).
This is not the case now since only the required entries are stored (those that have been inserted) and a sparse array of type intX_t (X depending on dict size) 2/3 * dk_sizes full is kept. The empty space changed from type PyDictKeyEntry to intX_t.
So, obviously, creating a sparse array of type PyDictKeyEntry is much more memory demanding than a sparse array for storing ints.
You can see the full conversation on Python-Dev regarding this feature if interested, it is a good read.
In the original proposal made by Raymond Hettinger, a visualization of the data structures used can be seen which captures the gist of the idea.
For example, the dictionary:
d = {"timmy": "red", "barry": "green", "guido": "blue"}is currently stored as [keyhash, key, value]:
entries = [["--", "--", "--"], [-8522787127447073495, "barry", "green"], ["--", "--", "--"], ["--", "--", "--"], ["--", "--", "--"], [-9092791511155847987, "timmy", "red"], ["--", "--", "--"], [-6480567542315338377, "guido", "blue"]]Instead, the data should be organized as follows:
indices = [None, 1, None, None, None, 0, None, 2] entries = [[-9092791511155847987, "timmy", "red"], [-8522787127447073495, "barry", "green"], [-6480567542315338377, "guido", "blue"]]
As you can visually now see, in the original proposal, a lot of space is essentially empty to reduce collisions and make look-ups faster. With the new approach, you reduce the memory required by moving the sparseness where it"s really required, in the indices.
json.dumps() converts a dictionary to str object, not a json(dict) object! So you have to load your str into a dict to use it by using json.loads() method
See json.dumps() as a save method and json.loads() as a retrieve method.
This is the code sample which might help you understand it more:
import json
r = {"is_claimed": "True", "rating": 3.5}
r = json.dumps(r)
loaded_r = json.loads(r)
loaded_r["rating"] #Output 3.5
type(r) #Output str
type(loaded_r) #Output dict