In previous editions of The Analyst, we covered Python’s data types and string manipulation with regular expressions. This article delves into data referencing and copying in Python, emphasizing the importance of distinguishing between the two to avoid unintended code outcomes. Additionally, we provide a brief overview of computer memory.

Consider the below snippet, where we initialize a list of tickers, list_a. Suppose we are interested in making changes to the list but also want to make a copy in case we need to revert the list back to the original state. Does the below create a copy?

Figure 1

At first glance, through the line list_b = list_a, list_b seems to have taken on the value of list_a. As such, printing both list_a and list_b gets us the same list of [“MSFT”, “AAPL”, “ZSP”].

Now let’s make some changes to list_b.

Figure 2

After adding “GOOGL” to list_b, it naturally contains “MSFT”, “AAPL”, “ZSP”, and “GOOGL”. The curious part is what happens to list_a.

Figure 3

Clearly, the copy did not work as intended, and now we are left without the original list of [“MSFT”, “AAPL”, “ZSP”]. Why was “GOOGL” added to list_a when the modification was done to list_b and not to list_a?

The tricky part is the seemingly innocent line list_b = list_a. Many new programmers might implicitly assume that it means list_b is equal to list_a, or that it takes on the value of list_a (i.e., as a copy), and that the two should exist independently going forward. This is not the case.

This can be confusing to a lot of new programmers, who likely started Python with some variation of the below lines of code:

Figure 4

That the above lines lead to b = 4 and a = 3 is so logical that suggesting the contrary goes against basic algebra. Indeed, adding a “1” to b should have no impact on a, and there was no arithmetic operation after b = b + 1 that concerns a. Why, then, can we not extrapolate such a basic train of logic to list_b and list_a?

The key to understanding this lies in the concept of memory addresses.

Computer architecture and memory

We will start by exploring the basics of the memory hierarchy in modern computers, providing context for memory addresses and their relevance to Python programming. In computer systems, memory encompasses physical devices for temporary or permanent data storage. Below is a hierarchy of the different types.

Figure 5

At the top of the pyramid, the central processing unit (CPU) registers are the fastest and are responsible for performing computations. However, they are small and costly. As such, the typical process involves the CPU loading data from the main memory (RAM), which can store much more data, before performing computations. The RAM is much faster than the SSD and hard drives but cannot replace disk storage, as it only temporarily stores data and, unlike disks, loses data when the computer powers off.

The RAM will be the topic of focus. When a Python program is executed, the CPU interacts with RAM for data access and storage in contiguous memory addresses. Memory addresses are relied upon as identifiers to refer to specific locations on RAM. A good way to visualize this is to think of RAM as a chest with multiple drawers, each identified by a unique memory address.

Memory addresses are represented using bits, where a bit is a binary number that can be 1 or 0. Binary is at the heart of computer science and is the language of machine code. Below are several 8-bit binary numbers and their decimal equivalents (in computing, the fundamental storage unit is one byte. One byte equals to 8 bits).

Figure 6

Binary numbers become rapidly less readable as the number of bits grows. Software engineers often represent binary numbers using hexadecimals (also shown above), a system that uses sixteen unique symbols, where “0-9” represents their decimal counterparts, and “A-F” represents 10-15. Hexadecimals are the standard format for memory addresses.

Why does all this matter? Because the equal sign in Python does not assign values, it assigns memory addresses!

Returning to the earlier example, a = 3 does not mean that a is equal to 3. Rather, the integer 3’s memory address (reference) was assigned to the variable a.

In the line b = a, b references the memory address of 3, the same way that a references the memory address of 3. Because integers are immutable (refer to “Hashing in Python” in the June 2023 edition of The Analyst), Python shares the memory address to optimize memory usage rather than create another integer 3 on RAM. In b = b + 1, b‘s reference is updated to the memory address of a newly created integer 4. The variable a, meanwhile, continues to point to integer 3’s memory address. This can be verified using Python’s id() method. See the diagrams below for clarification.

Figure 7

Figure 8

Figure 9

If we keep incrementing the value of b, Python will keep creating new integer objects and reassigning new memory addresses to b, as shown below. The binary equivalent is included for illustration purposes. 

Figure 10

Astute readers may observe the memory addresses incrementing by 0x20 in hex, equivalent to 32 bits. This is because the size of an integer in standard Python implementation is 4 bytes (32 bits).

Revisiting our example

With a stronger understanding of memory addresses and referencing, let’s revisit the list example. In the line list_b = list_a, we were in fact pointing list_b to the memory address of list_a. In this case, we can see that list_a and list_b have the same memory address even after adding “GOOGL” to the list.

Figure 11

When we added “GOOGL” to list_b, we were modifying the list that was stored in the memory address pointed to by list_b. And since we did not reassign the reference for list_a, both list_a and list_b continued to point to the same memory address.

The takeaway is that a simple assignment does not create copies. This is a critical concept and will save a lot of time debugging code as programs get more complex.

To create copies in Python, we can import the copy module and call its copy() method, which creates a new object behind the scenes, and assign its memory address to list_b. We can see that changes to list_b no longer impact list_a.

Figure 12

Shallow copy and deep copy

The above is called a shallow copy, which is very useful but might not work as intended when dealing with nested objects, where the copied object contains inner references to mutable objects (i.e., a list of lists). For example, what happens if list_a contains two lists of tickers instead, and we want to change one of the “inner lists” by adding the ticker “NVDA”? A shallow copy would copy the outer reference, but the inner references remain the same.

Figure 13

A deep copy can be invoked through the copy module’s deepcopy() method. As the name suggests, it recursively creates copies of an object and any nested references. Below shows the same code but replaces cp.copy() with cp.deepcopy().

Figure 14

Deep copy is often considered a safer way to copy if the programmer is working with complex data structures and is unsure of any possible underlying references. However, it is generally slower than shallow copy and consumes more memory by making copies of the entire object hierarchy, which might not always be necessary. The choice between the two depends on memory and performance considerations and the programmer’s understanding of whether the internal objects need to be changed.