Overleaf Git Integration: Difference between revisions

From Stat DS Computing Wiki
Created page with "<!-- TODO: set the category (and sort key) to match where this article lives in the wiki --> XX If your experiment code lives in a GitHub repository and your paper lives on Overleaf, you have probably done '''the upload dance''': rerun an experiment, hunt down the regenerated figure on disk, open Overleaf, delete the old file, upload the new one, and repeat for every figure and table that changed. It is slow, it is easy to le..."
 
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
<!-- TODO: set the category (and sort key) to match where this article lives in the wiki -->
<!-- TODO: set the category (and sort key) to match where this article lives in the wiki -->
[[Category:Beginner's_Guide_Implementation|XX]]
[[Category:Typesetting]]


If your experiment code lives in a GitHub repository and your paper lives on Overleaf, you have probably done '''the upload dance''': rerun an experiment, hunt down the regenerated figure on disk, open Overleaf, delete the old file, upload the new one, and repeat for every figure and table that changed. It is slow, it is easy to leave a stale figure behind, and it leaves no record of which version of your code produced which figure in the paper.
If your experiment code lives in a Git repository and your paper lives on Overleaf, you have probably done '''the upload dance''': rerun an experiment, hunt down the regenerated figure on disk, open Overleaf, delete the old file, upload the new one, and repeat for every figure and table that changed. It is slow, it is easy to leave a stale figure behind, and it leaves no record of which version of your code produced which figure in the paper.


Overleaf Premium's '''GitHub Synchronization''' feature removes most of this busywork. This article describes one particular way to use it: cloning the paper's repository '''inside''' your experiment repository, in a directory that the experiment repository ignores. Your analysis scripts then write figures and tables straight into the paper, and a single push sends them to Overleaf.
Overleaf Premium's '''Git integration''' removes most of this busywork. Every Overleaf project can be cloned as an ordinary Git repository, with no GitHub account or extra repository required. This article describes one way to use it: cloning the Overleaf project '''inside''' your experiment repository, in a directory that the experiment repository ignores. Your analysis scripts then write figures and tables straight into the paper, and a single <code>git push</code> puts them on Overleaf.


<div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
<div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
'''Important: Overleaf Premium is required'''<br>
'''Important: Overleaf Premium is required'''<br>
GitHub Synchronization is a paid Overleaf feature. Check whether you already have access through an institutional license before paying for it yourself. If you don't use Git yet, read up on the basics first, since this workflow assumes you are comfortable with <code>clone</code>, <code>commit</code>, <code>push</code>, and <code>pull</code>.
Git integration is a paid Overleaf feature. Check whether you already have access through an institutional license before paying for it yourself. If you don't use Git yet, read up on the basics first, since this workflow assumes you are comfortable with <code>clone</code>, <code>commit</code>, <code>push</code>, and <code>pull</code>.
</div>
</div>


== How It Works ==
== How It Works ==
There are several ways to connect a code repository to an Overleaf project. In this one, the two repositories stay '''completely independent'''. The paper repository is cloned into a subdirectory (here called <code>paper/</code>), and that subdirectory is listed in the experiment repository's <code>.gitignore</code>. The experiment repository never records anything about the paper, and the paper repository never sees your experimental code. Your scripts simply write files into a folder that happens to be a different Git repository.
There are several ways to connect a code repository to an Overleaf project. In this one, the two repositories stay '''completely independent'''. The Overleaf project is cloned into a subdirectory (here called <code>paper/</code>), and that subdirectory is listed in the experiment repository's <code>.gitignore</code>. The experiment repository never records anything about the paper, and the paper never sees your experimental code. Your scripts simply write files into a folder that happens to be a different Git repository, one whose remote is Overleaf itself.


The final layout looks like this:
The final layout looks like this:


<pre>
<pre>
my-experiments/                ← experiment repo (github.com/you/my-experiments)
my-experiments/                ← experiment repo
├── .gitignore                  ← contains "paper/"
├── .gitignore                  ← contains "paper/"
├── src/
├── src/
Line 24: Line 24:
├── Makefile
├── Makefile
└── paper/                      ← SEPARATE git repo, ignored by the outer repo
└── paper/                      ← SEPARATE git repo, ignored by the outer repo
     ├── .git/                  ← points to github.com/you/my-paper
     ├── .git/                  ← remote: https://git.overleaf.com/<project-id>
     ├── main.tex
     ├── main.tex
     ├── refs.bib
     ├── refs.bib
     ├── figures/                ← written by code
     ├── figures/                ← written by code
     │  └── coverage.pdf
     │  ├── coverage.pdf
    │  └── PROVENANCE.txt      ← which code version made these
     └── tables/                ← written by code
     └── tables/                ← written by code
         └── results.tex
         └── results.tex
</pre>
</pre>


Generated figures and tables flow from your code to Overleaf, and prose written on Overleaf flows back to your local copy:
Changes flow directly in both directions, with nothing in between:


<pre>
<pre>
scripts/make_figures.py ──writes──▶ paper/figures, paper/tables
scripts/make_figures.py ──writes──▶ paper/figures, paper/tables
paper/  ──git push──▶ GitHub (my-paper)  ──"Pull GitHub changes"──▶ Overleaf
paper/  ──git push──▶  Overleaf
Overleaf ──"Push Overleaf changes"──▶  GitHub (my-paper)  ──git pull──▶  paper/
Overleaf ──git pull──▶  paper/
</pre>
</pre>


Line 48: Line 49:
| style="padding: 15px; border-right: 2px solid white;" |
| style="padding: 15px; border-right: 2px solid white;" |
* '''No More Uploading:''' Regenerated figures and tables reach the paper with one commit and one push, no manual file shuffling required.
* '''No More Uploading:''' Regenerated figures and tables reach the paper with one commit and one push, no manual file shuffling required.
* '''Clean Histories:''' Paper edits never clutter your code history, and code changes never clutter your paper history.
* '''Instant Updates:''' A <code>git push</code> shows up in the Overleaf editor right away. There is no sync button to remember.
* '''Minimal Setup:''' No GitHub account, no extra repository, and no linking accounts. You need a token and a clone URL, and that's it.
* '''No Coupled Tracking:''' Unlike git submodules, you never need an extra "update the paper pointer" commit in your experiment repository every time the paper changes.
* '''No Coupled Tracking:''' Unlike git submodules, you never need an extra "update the paper pointer" commit in your experiment repository every time the paper changes.
* '''Collaborator Friendly:''' Co-authors can keep writing on Overleaf as usual. Nothing about their workflow changes.
* '''Collaborator Friendly:''' Co-authors keep writing on Overleaf as usual. Nothing about their workflow changes.
| style="padding: 15px;" |
| style="padding: 15px;" |
* '''Sync Is Manual:''' Overleaf does not watch GitHub. Someone has to click a button in Overleaf to pull new changes in (and to push Overleaf edits out).
* '''Provenance Is Up to You:''' Because the repositories are not linked, you need to record which code produced which figure yourself. A small script (below) handles this.
* '''Provenance Is Up to You:''' Because the repositories are not linked, the only record of which code produced a figure is whatever you write in the paper's commit messages. A small script (below) handles this for you.
* '''No Off-Site Copy:''' The paper lives only on Overleaf (and your local clones). If you want GitHub issues, pull-request reviews, or an independent backup, Overleaf's GitHub Synchronization is the better fit.
* '''One More Thing to Clone:''' Anyone who wants to regenerate the paper's figures needs to clone the paper repository separately.
* '''Token Management:''' You need an Overleaf Git token on every machine you push from, including clusters.
* '''One More Thing to Clone:''' Anyone who wants to regenerate the paper's figures needs to clone the Overleaf project separately.
|-
|-
| colspan="2" style="background-color: #e9ecef; text-align: center; padding: 10px;" | 🔗 '''[https://www.overleaf.com/learn/how-to/GitHub_Synchronization Overleaf Documentation: GitHub Synchronization]'''
| colspan="2" style="background-color: #e9ecef; text-align: center; padding: 10px;" | 🔗 '''[https://www.overleaf.com/learn/how-to/Git_integration Overleaf Documentation: Git Integration]'''
|}
|}


== Setting It Up ==
== Setting It Up ==


=== Step 1: Link Overleaf to GitHub ===
=== Step 1: Generate a Git authentication token ===
In Overleaf, open '''Account Settings''' and find the GitHub entry under the project synchronization section. Click '''Link''' and authorize Overleaf in the GitHub dialog that pops up.
Overleaf doesn't accept your account password over Git. In Overleaf, open '''Account Settings''', find the '''Git Integration''' section, and generate a token. Copy it somewhere safe right away, because Overleaf will only show it once.


<!-- SCREENSHOT: Overleaf Account Settings page with the GitHub "Link" button visible. -->
<!-- SCREENSHOT: Overleaf Account Settings, Git Integration section, with the "Generate token" button visible (blur any existing token). -->
[[File:Overleaf-account-settings-github-link.png|thumb|center|600px|Linking a GitHub account from Overleaf's account settings.]]
[[File:Overleaf-git-token.png|thumb|center|600px|Generating an Overleaf Git authentication token.]]


=== Step 2: Create a GitHub repository from your Overleaf project ===
=== Step 2: Find your project's clone URL ===
Open your paper in Overleaf, click '''Menu''' in the top left, and choose '''GitHub''' under the ''Sync'' section. Select '''Create a GitHub repository''', give it a name (e.g. <code>my-paper</code>), and make it private. Overleaf then pushes the current contents of your project to the new repository.
Open your paper in Overleaf, click '''Menu''' in the top left, and choose '''Git''' under the ''Sync'' section. The dialog shows a <code>git clone</code> command with your project's URL, which looks like <code>https://git.overleaf.com/<project-id></code>. The project ID is the same string that appears in your browser's address bar when the project is open.


<!-- SCREENSHOT: Overleaf project Menu, with the "Sync → GitHub" entry highlighted. -->
<!-- SCREENSHOT: Overleaf project Menu, with the "Sync → Git" entry highlighted. -->
[[File:Overleaf-menu-sync-github.png|thumb|center|300px|The GitHub entry in the Overleaf project menu.]]
[[File:Overleaf-menu-sync-git.png|thumb|center|300px|The Git entry in the Overleaf project menu.]]
 
<!-- SCREENSHOT: The "Create a GitHub repository" dialog filled in with name and private visibility. -->
[[File:Overleaf-create-github-repo.png|thumb|center|500px|Creating the paper repository from Overleaf.]]
 
<div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
'''Tip: Already have the paper on GitHub?'''<br>
You can go the other direction instead. In Overleaf, choose '''New Project → Import from GitHub''' and select your existing repository.
</div>


=== Step 3: Clone the paper inside your experiment repository ===
=== Step 3: Clone the paper inside your experiment repository ===
Line 90: Line 85:
git commit -m "Ignore nested paper clone"
git commit -m "Ignore nested paper clone"


git clone git@github.com:you/my-paper.git paper
git clone https://git.overleaf.com/<project-id> paper
</syntaxhighlight>
</syntaxhighlight>
When Git asks for credentials, enter <code>git</code> as the username and your '''token''' as the password. To avoid typing the token every time, let Git remember it with your system's credential helper (e.g. <code>osxkeychain</code> on macOS, <code>manager</code> on Windows). On shared machines, avoid the plain-text <code>store</code> helper.


Then check that the two repositories don't see each other:
Then check that the two repositories don't see each other:
Line 97: Line 94:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
git status          # should NOT list paper/
git status          # should NOT list paper/
git -C paper status # shows the paper repo, on its own branch
git -C paper status # shows the Overleaf project, on branch master
</syntaxhighlight>
</syntaxhighlight>


<!-- SCREENSHOT: Terminal showing both `git status` outputs (outer repo clean, inner repo on its own branch). -->
[[File:Nested-clone-git-status.png|thumb|center|600px|The outer repository ignores <code>paper/</code>; the inner repository is independent.]]


<div style="background-color: #f8d7da; border-left: 5px solid #721c24; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
<div style="background-color: #f8d7da; border-left: 5px solid #721c24; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
Line 178: Line 173:


cd paper
cd paper
git pull                        # pick up any text edits pushed from Overleaf first
git pull                        # pick up any text edits made on Overleaf first
git add figures tables
git add figures tables
git commit -m "Regenerate figures from my-experiments@$(git -C .. rev-parse --short HEAD)"
git commit -m "Regenerate figures"
git push
git push
</syntaxhighlight>
</syntaxhighlight>


Then, in Overleaf, open '''Menu → GitHub''' and click '''Pull GitHub changes into Overleaf'''.
That's it. Refresh the Overleaf editor and recompile, and the new figures are there.


<!-- SCREENSHOT: Overleaf GitHub sync dialog showing the "Pull GitHub changes into Overleaf" button, ideally with a pending commit listed. -->
[[File:Overleaf-pull-github-changes.png|thumb|center|500px|Bringing the regenerated figures into Overleaf.]]


<div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
<div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
'''Important: Overleaf does not sync automatically'''<br>
'''Important: Always pull before you push'''<br>
Overleaf only picks up changes from GitHub when someone clicks the pull button, and only sends its own changes to GitHub when someone clicks the push button. Forgetting the pull is by far the most common reason the paper still shows old figures.
Your co-authors are editing the same repository through the Overleaf editor. If anything changed on Overleaf since your last pull, your push will be rejected until you <code>git pull</code> and merge. As long as your code only touches <code>figures/</code> and <code>tables/</code>, these merges are automatic.
</div>
</div>


The commit message above includes the short hash of your experiment repository. Since the two repositories aren't linked, this message is your record of which code produced which figure. Running <code>git log -- figures/coverage.pdf</code> inside <code>paper/</code> will answer "where did this figure come from?" months later.
=== Recording provenance ===
Since the two repositories aren't linked, have your script record which code produced the figures. Overleaf's Git integration does not reliably preserve your local commit messages, so write the information into a '''file''' in the paper instead:
 
<syntaxhighlight lang="bash">
SHA=$(git rev-parse --short HEAD)
git diff --quiet || SHA="$SHA-dirty"
echo "Generated from my-experiments@$SHA on $(date -u +%Y-%m-%dT%H:%MZ)" > paper/figures/PROVENANCE.txt
</syntaxhighlight>
 
The <code>-dirty</code> suffix flags figures made from uncommitted code. Months later, opening <code>PROVENANCE.txt</code> (or its history in Overleaf) answers "where did these figures come from?"


=== Automating it with a Makefile ===
=== Automating it with a Makefile ===
Line 210: Line 212:
paper-sync: figures
paper-sync: figures
git -C $(PAPER) pull --rebase
git -C $(PAPER) pull --rebase
git -C $(PAPER) diff --quiet -- figures tables || \
  echo "Generated from my-experiments@$(SHA) on $$(date -u +%Y-%m-%dT%H:%MZ)" > $(PAPER)/figures/PROVENANCE.txt
git -C $(PAPER) add figures tables
git -C $(PAPER) add figures tables
git -C $(PAPER) diff --cached --quiet || \
git -C $(PAPER) diff --cached --quiet || \
  git -C $(PAPER) commit -m "Regenerate figures from my-experiments@$(SHA)"
  git -C $(PAPER) commit -m "Regenerate figures from my-experiments@$(SHA)"
git -C $(PAPER) push
git -C $(PAPER) push
@echo "Now click 'Pull GitHub changes into Overleaf' in the Overleaf GitHub menu."
</syntaxhighlight>
</syntaxhighlight>


Now <code>make paper-sync</code> regenerates everything, skips the commit if nothing actually changed, marks the commit <code>-dirty</code> if your experiment code had uncommitted changes, and reminds you to pull in Overleaf.
Now <code>make paper-sync</code> regenerates everything, pulls in your co-authors' edits, updates the provenance file only when a figure or table actually changed, skips the commit entirely if nothing changed, and pushes straight to Overleaf.
 
=== When co-authors edit on Overleaf ===
Before regenerating figures locally, push any Overleaf edits out to GitHub ('''Menu → GitHub → Push Overleaf changes to GitHub''') so that the <code>git pull</code> in <code>paper/</code> picks them up.
 
<!-- SCREENSHOT: The same GitHub sync dialog, this time highlighting the "Push Overleaf changes to GitHub" button. -->
[[File:Overleaf-push-to-github.png|thumb|center|500px|Sending Overleaf edits back to GitHub.]]


=== Avoiding conflicts ===
=== Avoiding conflicts ===
Line 233: Line 230:


<div style="background-color: #f8f9fa; border-left: 4px solid #228B22; padding: 15px; margin-bottom: 10px;">
<div style="background-color: #f8f9fa; border-left: 4px solid #228B22; padding: 15px; margin-bottom: 10px;">
'''<code>.tex</code> and <code>.bib</code> files belong to people.''' Write them on Overleaf (or locally, then push), and always pull before you push figures.
'''<code>.tex</code> and <code>.bib</code> files belong to people.''' Write them in the Overleaf editor (or locally, then push), and always pull before you push figures.
</div>
</div>


Line 244: Line 241:
! style="background-color: #e9ecef; padding: 10px; text-align: left;" | Gitignored Clone
! style="background-color: #e9ecef; padding: 10px; text-align: left;" | Gitignored Clone
! style="background-color: #e9ecef; padding: 10px; text-align: left;" | Git Submodule
! style="background-color: #e9ecef; padding: 10px; text-align: left;" | Git Submodule
! style="background-color: #e9ecef; padding: 10px; text-align: left;" | Git Subtree
|- style="background-color: #f8f9fa;"
|- style="background-color: #f8f9fa;"
| style="padding: 10px;" | '''Experiment repo records the paper version''' || style="padding: 10px;" | No || style="padding: 10px;" | Yes (a pointer) || style="padding: 10px;" | Yes (full history)
| style="padding: 10px;" | '''Experiment repo records the paper version''' || style="padding: 10px;" | No || style="padding: 10px;" | Yes (a pointer)
|-
|-
| style="padding: 10px;" | '''Extra commit in experiment repo per paper change''' || style="padding: 10px;" | No || style="padding: 10px;" | Yes || style="padding: 10px;" | Yes
| style="padding: 10px;" | '''Extra commit in experiment repo per paper change''' || style="padding: 10px;" | No || style="padding: 10px;" | Yes
|- style="background-color: #f8f9fa;"
|- style="background-color: #f8f9fa;"
| style="padding: 10px;" | '''Paper history mixed into code history''' || style="padding: 10px;" | No || style="padding: 10px;" | No || style="padding: 10px;" | Yes
| style="padding: 10px;" | '''Collaborators need Overleaf credentials to clone the code''' || style="padding: 10px;" | No || style="padding: 10px;" | Yes, for a full clone
|-
|-
| style="padding: 10px;" | '''Special clone instructions for collaborators''' || style="padding: 10px;" | No || style="padding: 10px;" | <code>--recurse-submodules</code> || style="padding: 10px;" | No
| style="padding: 10px;" | '''Link between figure and code version''' || style="padding: 10px;" | Provenance file || style="padding: 10px;" | Automatic
|- style="background-color: #f8f9fa;"
| style="padding: 10px;" | '''Link between figure and code version''' || style="padding: 10px;" | Commit message || style="padding: 10px;" | Automatic || style="padding: 10px;" | Automatic
|}
|}


A submodule gives you exact, automatic provenance, but at a price: every change to the paper, including every typo fix a co-author pushes from Overleaf, needs a matching "bump the submodule" commit in your experiment repository. For a paper that is edited daily, that noise usually isn't worth it. The gitignored clone trades automatic provenance for a scripted commit message, which is plenty in practice.
A submodule gives you exact, automatic provenance, but at a price. Every change to the paper, including every typo fix a co-author makes on Overleaf, needs a matching "bump the submodule" commit in your experiment repository. On top of that, anyone cloning your code with submodules would need an Overleaf token. For a paper that is edited daily, this usually isn't worth it. The gitignored clone trades automatic provenance for a one-line provenance file, which is plenty in practice.


== Variations ==
== Variations ==


<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''Skip GitHub with Overleaf's Git integration:''' Overleaf Premium also lets you clone a project directly (<code>git clone https://git.overleaf.com/<project-id></code>, using a Git token generated in Account Settings). Clone that into the gitignored directory instead, and a <code>git push</code> lands in Overleaf immediately with no pull button. The catch is that you no longer get a GitHub copy of the paper for issues, reviews, or backups.
'''Want a GitHub copy of the paper?''' Overleaf's GitHub Synchronization keeps the project mirrored in a GitHub repository, which you can clone into the gitignored directory in exactly the same way. You gain issues, pull-request reviews, and an off-site backup. The cost is a manual sync step, since Overleaf only exchanges changes with GitHub when someone clicks a button in the Overleaf menu.
</div>
</div>


<div style="background-color: #f8f9fa; border-left: 4px solid #FFD43B; padding: 15px; margin-bottom: 10px;">
<div style="background-color: #f8f9fa; border-left: 4px solid #FFD43B; padding: 15px; margin-bottom: 10px;">
'''Generating figures on a cluster:''' Clone the paper repository on the cluster as well and run <code>make paper-sync</code> there. Alternatively, point <code>PAPER_DIR</code> at a scratch directory on the cluster, copy the outputs back to your laptop, and push from there.
'''Generating figures on a cluster:''' Clone the Overleaf project on the cluster as well, set up your token with a credential helper, and run <code>make paper-sync</code> there. Alternatively, point <code>PAPER_DIR</code> at a scratch directory on the cluster, copy the outputs back to your laptop, and push from there, so your token never leaves your own machine.
</div>
</div>


Line 275: Line 269:
|- style="vertical-align: top; background-color: #f8f9fa;"
|- style="vertical-align: top; background-color: #f8f9fa;"
| style="padding: 15px;" |
| style="padding: 15px;" |
* '''Forgetting the Overleaf pull:''' Your figures are on GitHub, but Overleaf still shows the old ones. The reminder in the Makefile helps.
* '''Rejected pushes:''' Someone edited the paper on Overleaf since your last pull. Run <code>git pull</code>, then push again. Never force-push to Overleaf.
* '''Pushing from both sides at once:''' If Overleaf has unpushed edits and GitHub has new figures, Overleaf has to merge them. Push from Overleaf first, pull locally, then push your figures.
* '''Authentication failures:''' The username is literally <code>git</code>, and the password is your token, not your Overleaf password. If a token stops working, generate a new one and update your credential helper.
* '''Bloating the paper repository:''' PNGs and timestamped PDFs change on every run. Prefer vector PDFs with the timestamp removed, and only commit figures that actually changed.
* '''Working on branches:''' Overleaf only knows about <code>master</code>. You can branch locally, but only commits merged into <code>master</code> can be pushed.
* '''Bloating the project:''' PNGs and timestamped PDFs change on every run. Prefer vector PDFs with the timestamp removed, and only commit figures that actually changed.
* '''Overleaf project limits:''' Overleaf limits project size and file counts. Keep raw outputs, checkpoints, and intermediate data in the experiment repository (or out of Git entirely); only put paper-ready figures and tables in <code>paper/</code>.
* '''Overleaf project limits:''' Overleaf limits project size and file counts. Keep raw outputs, checkpoints, and intermediate data in the experiment repository (or out of Git entirely); only put paper-ready figures and tables in <code>paper/</code>.
* '''Working on branches:''' Overleaf only syncs one branch. Stick to the default branch inside <code>paper/</code>; anything on another branch won't show up in Overleaf.
* '''Leaking tokens:''' Never paste the token into the clone URL in a shared script or commit it anywhere. Let a credential helper store it.
|}
|}


<div style="background-color: #e8f4f8; padding: 15px; text-align: center; margin-top: 20px; font-style: italic;">
<div style="background-color: #e8f4f8; padding: 15px; text-align: center; margin-top: 20px; font-style: italic;">
'''Note:''' This is just one of several ways to connect your code and your paper. If the manual pull button bothers you, try the Git integration variation above; if you want airtight provenance, a submodule may be worth the extra commits.
'''Note:''' This is just one of several ways to connect your code and your paper. If you want your paper on GitHub, try GitHub Synchronization instead; if you want airtight provenance, a submodule may be worth the extra commits.
</div>
</div>

Latest revision as of 23:31, 18 October 2026


If your experiment code lives in a Git repository and your paper lives on Overleaf, you have probably done the upload dance: rerun an experiment, hunt down the regenerated figure on disk, open Overleaf, delete the old file, upload the new one, and repeat for every figure and table that changed. It is slow, it is easy to leave a stale figure behind, and it leaves no record of which version of your code produced which figure in the paper.

Overleaf Premium's Git integration removes most of this busywork. Every Overleaf project can be cloned as an ordinary Git repository, with no GitHub account or extra repository required. This article describes one way to use it: cloning the Overleaf project inside your experiment repository, in a directory that the experiment repository ignores. Your analysis scripts then write figures and tables straight into the paper, and a single git push puts them on Overleaf.

Important: Overleaf Premium is required
Git integration is a paid Overleaf feature. Check whether you already have access through an institutional license before paying for it yourself. If you don't use Git yet, read up on the basics first, since this workflow assumes you are comfortable with clone, commit, push, and pull.

How It Works

There are several ways to connect a code repository to an Overleaf project. In this one, the two repositories stay completely independent. The Overleaf project is cloned into a subdirectory (here called paper/), and that subdirectory is listed in the experiment repository's .gitignore. The experiment repository never records anything about the paper, and the paper never sees your experimental code. Your scripts simply write files into a folder that happens to be a different Git repository, one whose remote is Overleaf itself.

The final layout looks like this:

my-experiments/                 ← experiment repo
├── .gitignore                  ← contains "paper/"
├── src/
├── scripts/
│   └── make_figures.py         ← writes into paper/figures and paper/tables
├── Makefile
└── paper/                      ← SEPARATE git repo, ignored by the outer repo
    ├── .git/                   ← remote: https://git.overleaf.com/<project-id>
    ├── main.tex
    ├── refs.bib
    ├── figures/                ← written by code
    │   ├── coverage.pdf
    │   └── PROVENANCE.txt      ← which code version made these
    └── tables/                 ← written by code
        └── results.tex

Changes flow directly in both directions, with nothing in between:

scripts/make_figures.py ──writes──▶ paper/figures, paper/tables
paper/   ──git push──▶  Overleaf
Overleaf ──git pull──▶  paper/
Strengths Weaknesses
  • No More Uploading: Regenerated figures and tables reach the paper with one commit and one push, no manual file shuffling required.
  • Instant Updates: A git push shows up in the Overleaf editor right away. There is no sync button to remember.
  • Minimal Setup: No GitHub account, no extra repository, and no linking accounts. You need a token and a clone URL, and that's it.
  • No Coupled Tracking: Unlike git submodules, you never need an extra "update the paper pointer" commit in your experiment repository every time the paper changes.
  • Collaborator Friendly: Co-authors keep writing on Overleaf as usual. Nothing about their workflow changes.
  • Provenance Is Up to You: Because the repositories are not linked, you need to record which code produced which figure yourself. A small script (below) handles this.
  • No Off-Site Copy: The paper lives only on Overleaf (and your local clones). If you want GitHub issues, pull-request reviews, or an independent backup, Overleaf's GitHub Synchronization is the better fit.
  • Token Management: You need an Overleaf Git token on every machine you push from, including clusters.
  • One More Thing to Clone: Anyone who wants to regenerate the paper's figures needs to clone the Overleaf project separately.
🔗 Overleaf Documentation: Git Integration

Setting It Up

Step 1: Generate a Git authentication token

Overleaf doesn't accept your account password over Git. In Overleaf, open Account Settings, find the Git Integration section, and generate a token. Copy it somewhere safe right away, because Overleaf will only show it once.

Generating an Overleaf Git authentication token.

Step 2: Find your project's clone URL

Open your paper in Overleaf, click Menu in the top left, and choose Git under the Sync section. The dialog shows a git clone command with your project's URL, which looks like https://git.overleaf.com/<project-id>. The project ID is the same string that appears in your browser's address bar when the project is open.

The Git entry in the Overleaf project menu.

Step 3: Clone the paper inside your experiment repository

From the root of your experiment repository, tell Git to ignore the paper/ directory before you clone into it:

cd my-experiments
echo "paper/" >> .gitignore
git add .gitignore
git commit -m "Ignore nested paper clone"

git clone https://git.overleaf.com/<project-id> paper

When Git asks for credentials, enter git as the username and your token as the password. To avoid typing the token every time, let Git remember it with your system's credential helper (e.g. osxkeychain on macOS, manager on Windows). On shared machines, avoid the plain-text store helper.

Then check that the two repositories don't see each other:

git status          # should NOT list paper/
git -C paper status # shows the Overleaf project, on branch master


Warning: Don't skip the .gitignore entry
Without it, a careless git add . in the experiment repository will register paper/ as an "embedded repository," which confuses collaborators and anything else that clones your code. If this happens, git rm --cached paper undoes it.

Step 4: Point your code at the paper directory

Rather than hard-coding paths in every script, define the paper location once. Using an environment variable with a sensible default means the code still runs for people who haven't cloned the paper (their figures just land in an ignored folder).

# src/paths.py
import os
from pathlib import Path

REPO_ROOT = Path(__file__).resolve().parents[1]
PAPER_DIR = Path(os.environ.get("PAPER_DIR", REPO_ROOT / "paper"))
FIG_DIR = PAPER_DIR / "figures"
TAB_DIR = PAPER_DIR / "tables"

FIG_DIR.mkdir(parents=True, exist_ok=True)
TAB_DIR.mkdir(parents=True, exist_ok=True)

For figures, save vector PDFs directly into the paper. Removing the creation timestamp means a figure that didn't actually change produces an identical file, so Git won't report a pointless change:

import matplotlib.pyplot as plt
from src.paths import FIG_DIR

fig, ax = plt.subplots(figsize=(3.25, 2.4))   # match your template's column width
...
fig.savefig(FIG_DIR / "coverage.pdf", bbox_inches="tight",
            metadata={"CreationDate": None})

For tables, write a LaTeX fragment rather than a full document:

from src.paths import TAB_DIR

tex = df.to_latex(index=False, escape=True, float_format="%.3f")
(TAB_DIR / "results.tex").write_text(tex)

and pull both into the paper as usual:

\includegraphics[width=\linewidth]{figures/coverage}
...
\input{tables/results}

For numbers quoted in the text, the same idea works. Have your code write LaTeX macros so the prose never goes stale:

(TAB_DIR / "numbers.tex").write_text(
    f"\\newcommand{{\\coverageNinetyFive}}{{{cov95:.1f}\\%}}\n"
)
% preamble
\input{tables/numbers}
% body
Our intervals achieve \coverageNinetyFive{} empirical coverage.

Day-to-Day Workflow

Sending new results to the paper

After rerunning your experiments, commit and push from inside paper/:

python scripts/make_figures.py

cd paper
git pull                         # pick up any text edits made on Overleaf first
git add figures tables
git commit -m "Regenerate figures"
git push

That's it. Refresh the Overleaf editor and recompile, and the new figures are there.


Important: Always pull before you push
Your co-authors are editing the same repository through the Overleaf editor. If anything changed on Overleaf since your last pull, your push will be rejected until you git pull and merge. As long as your code only touches figures/ and tables/, these merges are automatic.

Recording provenance

Since the two repositories aren't linked, have your script record which code produced the figures. Overleaf's Git integration does not reliably preserve your local commit messages, so write the information into a file in the paper instead:

SHA=$(git rev-parse --short HEAD)
git diff --quiet || SHA="$SHA-dirty"
echo "Generated from my-experiments@$SHA on $(date -u +%Y-%m-%dT%H:%MZ)" > paper/figures/PROVENANCE.txt

The -dirty suffix flags figures made from uncommitted code. Months later, opening PROVENANCE.txt (or its history in Overleaf) answers "where did these figures come from?"

Automating it with a Makefile

Once the steps above feel familiar, wrap them in a single command:

PAPER := paper
SHA   := $(shell git rev-parse --short HEAD)$(shell git diff --quiet || echo -dirty)

.PHONY: figures paper-sync

figures:
	python scripts/make_figures.py

paper-sync: figures
	git -C $(PAPER) pull --rebase
	git -C $(PAPER) diff --quiet -- figures tables || \
	  echo "Generated from my-experiments@$(SHA) on $$(date -u +%Y-%m-%dT%H:%MZ)" > $(PAPER)/figures/PROVENANCE.txt
	git -C $(PAPER) add figures tables
	git -C $(PAPER) diff --cached --quiet || \
	  git -C $(PAPER) commit -m "Regenerate figures from my-experiments@$(SHA)"
	git -C $(PAPER) push

Now make paper-sync regenerates everything, pulls in your co-authors' edits, updates the provenance file only when a figure or table actually changed, skips the commit entirely if nothing changed, and pushes straight to Overleaf.

Avoiding conflicts

Conflicts are rare if every file has exactly one "owner":

figures/ and tables/ belong to your code. Never edit them by hand on Overleaf; the next regeneration will silently overwrite your changes. Change the formatting in the code that generates them instead.

.tex and .bib files belong to people. Write them in the Overleaf editor (or locally, then push), and always pull before you push figures.

Why Not a Git Submodule?

Git submodules are the "official" way to put one repository inside another, so it is natural to ask why this workflow avoids them.

Gitignored Clone Git Submodule
Experiment repo records the paper version No Yes (a pointer)
Extra commit in experiment repo per paper change No Yes
Collaborators need Overleaf credentials to clone the code No Yes, for a full clone
Link between figure and code version Provenance file Automatic

A submodule gives you exact, automatic provenance, but at a price. Every change to the paper, including every typo fix a co-author makes on Overleaf, needs a matching "bump the submodule" commit in your experiment repository. On top of that, anyone cloning your code with submodules would need an Overleaf token. For a paper that is edited daily, this usually isn't worth it. The gitignored clone trades automatic provenance for a one-line provenance file, which is plenty in practice.

Variations

Want a GitHub copy of the paper? Overleaf's GitHub Synchronization keeps the project mirrored in a GitHub repository, which you can clone into the gitignored directory in exactly the same way. You gain issues, pull-request reviews, and an off-site backup. The cost is a manual sync step, since Overleaf only exchanges changes with GitHub when someone clicks a button in the Overleaf menu.

Generating figures on a cluster: Clone the Overleaf project on the cluster as well, set up your token with a credential helper, and run make paper-sync there. Alternatively, point PAPER_DIR at a scratch directory on the cluster, copy the outputs back to your laptop, and push from there, so your token never leaves your own machine.

Common Pitfalls

Things to Watch Out For
  • Rejected pushes: Someone edited the paper on Overleaf since your last pull. Run git pull, then push again. Never force-push to Overleaf.
  • Authentication failures: The username is literally git, and the password is your token, not your Overleaf password. If a token stops working, generate a new one and update your credential helper.
  • Working on branches: Overleaf only knows about master. You can branch locally, but only commits merged into master can be pushed.
  • Bloating the project: PNGs and timestamped PDFs change on every run. Prefer vector PDFs with the timestamp removed, and only commit figures that actually changed.
  • Overleaf project limits: Overleaf limits project size and file counts. Keep raw outputs, checkpoints, and intermediate data in the experiment repository (or out of Git entirely); only put paper-ready figures and tables in paper/.
  • Leaking tokens: Never paste the token into the clone URL in a shared script or commit it anywhere. Let a credential helper store it.

Note: This is just one of several ways to connect your code and your paper. If you want your paper on GitHub, try GitHub Synchronization instead; if you want airtight provenance, a submodule may be worth the extra commits.