Introduction

I come to you with an interesting open-source project I came across, albeit after a lot of misdirected efforts, that solved a problem in my workflow: dropboxignore.

For context, I'm running Ubuntu 22.04LTS and using bash as my shell. Therefore, the following will be with this setup.

Besides using Git and remote repositories on my GitHub (check it out, shameless plug!), I also rely on Dropbox to back up and sync my code files between machines. It's like having a spare parachute, but also a safety net! So if you ever feel like peeking at my code, you know where to find it 😏.

node_modules... cue the memes

As I increasingly started using npm to manage my React project dependencies, node_modules took up the majority of my limited 2 GB Dropbox cloud storage.

It seemed pointless to keep my node_modules backed up as well since I could simply install all the dependencies mentioned in the package.json file of my npm project at once using the npm install command.

Attempt 1

I started looking for ways to relocate my node_modules folder to a local directory in my system and somehow reference it by changing the paths that npm checks when trying to find locally installed packages.

It turns out, this changing of the installation directory of the global node_modules directory is possible, but not so for packages installed locally in a particular npm project. Read about it in the npm docs here.

So there was no official way of doing this.

Attempt 2

I started thinking of hacks that could somehow make this work.

The solution I thought of to make this work for locally installed packages was to use symbolic links to a folder in my local file system, outside of my Dropbox directory. To test this, I created 2 npm projects in 2 different directories actual-project and symbolic-project, installed a package in one of the projects and tried to make the other project recognize it.

I did this by creating a symbolic soft link to the node_modules folder in the symbolic-project inside of the actual-project directory, using:

ln -s [Source Directory] [Destination Directory]

This gave me a bit of hope as it was able to recognize installed packages in the symlinked node_modules folder of the second project, so I concluded that dependencies would also be resolved as usual.

Here is the output to my little thought experiment:

rohan-verma@G3-3500:~/tmp/symbolic-project$ npm init -y
Wrote to /home/rohan-verma/tmp/symbolic-project/package.json:

{
  "name": "symbolic-project",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "Rohan Verma <rohan.verma@mail.org>",
  "license": "ISC"
}

rohan-verma@G3-3500:~/tmp/symbolic-project$ npm install lodash

added 1 package, and audited 2 packages in 796ms

found 0 vulnerabilities
rohan-verma@G3-3500:~/tmp/symbolic-project$ npm list
symbolic-project@1.0.0 /home/rohan-verma/tmp/symbolic-project
`-- lodash@4.17.21

rohan-verma@G3-3500:~/tmp/symbolic-project$ cd ../actual-project
rohan-verma@G3-3500:~/tmp/actual-project$ npm init -y
Wrote to /home/rohan-verma/tmp/actual-project/package.json:

{
  "name": "actual-project",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "Rohan Verma <rohan.verma@mail.org>",
  "license": "ISC"
}


rohan-verma@G3-3500:~/tmp/actual-project$ ln -s ~/tmp/symbolic-project/node_modules ~/tmp/actual-project/node_modules
rohan-verma@G3-3500:~/tmp/actual-project$ npm list
actual-project@1.0.0 /home/rohan-verma/tmp/actual-project
`-- lodash@4.17.21 extraneous

rohan-verma@G3-3500:~/tmp/actual-project$

Code Summary

In this code snippet, we have two directories: symbolic-project and actual-project.

In the symbolic-project directory, we run npm init -y to create a new npm project with default settings. Then we install the lodash package using npm install lodash. Finally, we run npm list to see the installed packages. It shows that lodash is installed in symbolic-project.

In the actual-project directory, we run npm init -y to create a new npm project with default settings. Then, we create a symbolic link between the node_modules directory of symbolic-project and actual-project using ln -s ~/tmp/symbolic-project/node_modules ~/tmp/actual-project/node_modules. This allows actual-project to access the packages installed in symbolic-project.

Finally, we run npm list to see the installed packages. It shows that lodash is installed, but it's listed as extraneous because it's not included in actual-project's package.json file. This happened because lodash was only added to the package.json file of the symbolic-project directory, where it was installed.

If only life were that simple. When I tried to install a package using npm install inside of actual-project, npm simply identified that the node_modules folder wasn't an actual directory but rather a symbolic link. It simply removed the symbolic link, created a new node_modules folder and installed the package as usual.

I looked around on GitHub and other devs were facing the same problem. Check out the GitHub issue here.

This is the intended behaviour introduced in version 2.8.2 of @npmcli/arborist.

Fun Fact

Arbor is the word for tree in Latin and an Arborist is a professional who specializes in taking care of trees and shrubs.

The @npmcli/arborist library is used by the npm CLI for managing the dependency TREES (😂 🤣😂 🤣) of npm projects.

This patch in @npmcli/arborist corresponded to the version 7.20.7 of npm.

This was due to a known security vulnerability that is described in detail here; which I won't get into since it goes beyond the scope of this article.

So, at this point, I had given up on moving the node_modules folder out of the npm project it belonged to.

Aha moment! Removing node_modules folders from sync in Dropbox?

I felt stupid about not thinking of this before.

What immediately came to mind was something similar to gitignore files in the case of git, which allow specifying of files that shouldn't be tracked.

I tried finding documentation on how to modify the .dropbox config file, usually present in the root of the Dropbox folder, but to no avail.

What I found was an official guide on how to exclude certain files or directories from syncing to Dropbox manually. Here is a link to the guide.

It took advantage of a feature called Extended Attributes in the case of both MacOS and Linux. Visit this link for a more detailed guide on Extended File Attributes, but in layman's terms, it is a method for specifying additional information related to files/directories that are not processed by the file system.

The way of noobs

I was able to use the attr utility, as specified in the official guide, to set the extended attribute com.dropbox.ignored; that led to the exclusion of the node_modules directory of a particular npm project from syncing to Dropbox.

Here is the exact set of commands I used:

rohan-verma@G3-3500:~/Dropbox$ cd ./testing-webpack-babel
rohan-verma@G3-3500:~/Dropbox/testing-webpack-babel$ attr -s com.dropbox.ignored -V 1 ./node_modules
Attribute "com.dropbox.ignored" set to a 1 byte value for ./node_modules: 
1
rohan-verma@G3-3500:~/Dropbox/testing-webpack-babel$

Refer to this official list of all the extended attributes that Dropbox supports.

However, I found this method to be extremely unintuitive and I felt it was extremely easy to lose track of which files I had excluded.

The smart way 😎, using dropboxignore

So, I googled a bit more and stumbled across this repository. This was what I was looking for all along!

This is a command line utility that simplifies the process of excluding files in your Dropbox directory from syncing.

After examining the source code, I discovered that the script executes the same command repeatedly, which is similar to the one I manually inputted earlier (Not to mention that the codebase contains numerous features that enhance the user interface). To accomplish this, the script uses the call() method from the subprocess module in Python.

About the subprocess module

The subprocess module has a lot of useful functionalities but it is relevant here since it allows for the execution of external commands or programs from within a Python script, and can be used to control their input, output, and behaviour using a variety of options and arguments.

By using the call() method, the script can execute the specified command in the shell environment, and capture its return code. The command is executed repeatedly for each specified file or directory until all files have been processed.

Here is a link to the documentation for your perusal.

In an attempt not to overcomplicate things any further; I'll go over the one command that is perfect for my application.

Since I already have gitignore files in my npm projects in which node_modules is mentioned, running dropboxignore genupi . (a shorthand command for generate, update and ignore functionalities of dropboxignore) performs the following actions sequentially:

• generate - Generates dropboxignore files in each folder containing a gitignore file, if one doesn't already exist.

• update - Updates the dropboxignore files that already exist, adding missing patterns for the .gitignore file.

• ignore - This is where the automation happens. The content of the dropboxignore files are used to run the attr commands to ignore newly specified file/folder patterns.

At last...

That was it. All node_modules directories taking up space in my Dropbox cloud storage, vanished within 10 minutes as they unsynced one-by-one, allowing me to continue my work uninterrupted; albeit after a quite significant 12-hour interruption... 🤫