{ "componentChunkName": "component---src-templates-blog-post-js", "path": "/memory-optimization-in-pandas-dataframes/", "result": {"data":{"markdownRemark":{"html":"

Let’s read an Excel file into a DataFrame:

\n
import pandas as pd\n\nnsv = pd.read_excel("./NSV/PA/August/PA-AUGUST-19.xlsx", sheet_name = "Data")\n\nnsv.info()
\n

Output:

\n

\n \n \n \n \n

\n

We have 18429 rows in our DataFrame and the memory usage is 3.4+ MB.

\n

We can employ a method to optimize memory usage.

\n

We will have to find the columns where there are a few unique values and then convert those columns to categorical data type.

\n

The following columns have a few unique values:

\n\n

We can check the unique values in a column using the nunique() method.

\n
Learn how to count unique values in a column in my blog post here.
\n
nsv["ARM"].nunique()\n7\n\nnsv["LINE"].nunique()\n7\n\nnsv["SEASON"].nunique()\n14
\n

Let’s convert these columns to categorical data types.

\n
nsv["ARM"] = nsv["ARM"].astype("category")\n\nnsv["LINE"] = nsv["LINE"].astype("category")\n\nnsv["SEASON"] = nsv["SEASON"].astype("category")
\n

Let’s check the memory usage now:

\n
nsv.info()
\n

Output:

\n

\n \n \n \n \n

\n

The new memory usage is 2.4MB, which is a reduction of approximately 30%.

\n","frontmatter":{"title":"Memory Optimization in Pandas DataFrames","date":"2021-08-09"}}},"pageContext":{"slug":"/memory-optimization-in-pandas-dataframes/","prev":{"fields":{"slug":"/creating-spreadsheet-style-pivot-tables-in-pandas/"},"frontmatter":{"modules":null}},"next":{"fields":{"slug":"/count-the-unique-values-in-a-column-in-a-pandas-dataframe/"},"frontmatter":{"modules":null}}}}, "staticQueryHashes": ["3159585216"]}