How to import a text file on AWS S3 into pandas without writing to disk

Michael Zippo

👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!

I have a text file saved on S3 which is a tab delimited table. I want to load it into pandas but cannot save it first because I am running on a heroku server. Here is what I have so far.

import io
import boto3
import os
import pandas as pd

os.environ["AWS_ACCESS_KEY_ID"] = "xxxxxxxx"
os.environ["AWS_SECRET_ACCESS_KEY"] = "xxxxxxxx"

s3_client = boto3.client("s3")
response = s3_client.get_object(Bucket="my_bucket",Key="filename.txt")
file = response["Body"]


pd.read_csv(file, header=14, delimiter="	", low_memory=False)

the error is

OSError: Expected file path name or file-like object, got <class "bytes"> type

How do I convert the response body into a format pandas will accept?

pd.read_csv(io.StringIO(file), header=14, delimiter="	", low_memory=False)

returns

TypeError: initial_value must be str or None, not StreamingBody

pd.read_csv(io.BytesIO(file), header=14, delimiter="	", low_memory=False)

returns

TypeError: "StreamingBody" does not support the buffer interface

UPDATE - Using the following worked

file = response["Body"].read()

and

pd.read_csv(io.BytesIO(file), header=14, delimiter="	", low_memory=False)

👻 Read also: what is the best laptop for engineering students?

How to import a text file on AWS S3 into pandas without writing to disk del: Questions

__del__

How can I make a time delay in Python?

5 answers

I would like to know how to put a time delay in a Python script.

2973

Answer #1

import time
time.sleep(5)   # Delays for 5 seconds. You can also use a float value.

Here is another example where something is run approximately once a minute:

import time
while True:
    print("This prints once a minute.")
    time.sleep(60) # Delay for 1 minute (60 seconds).

2973

Answer #2

You can use the sleep() function in the time module. It can take a float argument for sub-second resolution.

from time import sleep
sleep(0.1) # Time in seconds

How to import a text file on AWS S3 into pandas without writing to disk del: Questions

__del__

How to delete a file or folder in Python?

5 answers

How do I delete a file or folder in Python?

2639

Answer #1

os.remove() removes a file.
os.rmdir() removes an empty directory.
shutil.rmtree() deletes a directory and all its contents.

Path objects from the Python 3.4+ pathlib module also expose these instance methods:

pathlib.Path.unlink() removes a file or symbolic link.
pathlib.Path.rmdir() removes an empty directory.

We hope this article has helped you to resolve the problem. Apart from How to import a text file on AWS S3 into pandas without writing to disk, check other __del__-related topics.

Want to excel in Python? See our review of the best Python online courses 2023. If you are interested in Data Science, check also how to learn programming in R.

By the way, this material is also available in other languages:

Xu Porretti

Warsaw | 2023-03-24

Simply put and clear. Thank you for sharing. How to import a text file on AWS S3 into pandas without writing to disk and other issues with StackOverflow was always my weak point 😁. Will get back tomorrow with feedback

Cornwall Williams

California | 2023-03-24

Thanks for explaining! I was stuck with How to import a text file on AWS S3 into pandas without writing to disk for some hours, finally got it done 🤗. Checked yesterday, it works!

Boris Lehnman

Boston | 2023-03-24

Thanks for explaining! I was stuck with How to import a text file on AWS S3 into pandas without writing to disk for some hours, finally got it done 🤗. Will get back tomorrow with feedback