Overleaf Git Integration: Difference between revisions
No edit summary |
|||
| Line 1: | Line 1: | ||
<!-- TODO: set the category (and sort key) to match where this article lives in the wiki --> | <!-- TODO: set the category (and sort key) to match where this article lives in the wiki --> | ||
[[Category: | [[Category:Typesetting]] | ||
If your experiment code lives in a | If your experiment code lives in a Git repository and your paper lives on Overleaf, you have probably done '''the upload dance''': rerun an experiment, hunt down the regenerated figure on disk, open Overleaf, delete the old file, upload the new one, and repeat for every figure and table that changed. It is slow, it is easy to leave a stale figure behind, and it leaves no record of which version of your code produced which figure in the paper. | ||
Overleaf Premium's ''' | Overleaf Premium's '''Git integration''' removes most of this busywork. Every Overleaf project can be cloned as an ordinary Git repository, with no GitHub account or extra repository required. This article describes one way to use it: cloning the Overleaf project '''inside''' your experiment repository, in a directory that the experiment repository ignores. Your analysis scripts then write figures and tables straight into the paper, and a single <code>git push</code> puts them on Overleaf. | ||
<div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;"> | <div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;"> | ||
'''Important: Overleaf Premium is required'''<br> | '''Important: Overleaf Premium is required'''<br> | ||
Git integration is a paid Overleaf feature. Check whether you already have access through an institutional license before paying for it yourself. If you don't use Git yet, read up on the basics first, since this workflow assumes you are comfortable with <code>clone</code>, <code>commit</code>, <code>push</code>, and <code>pull</code>. | |||
</div> | </div> | ||
== How It Works == | == How It Works == | ||
There are several ways to connect a code repository to an Overleaf project. In this one, the two repositories stay '''completely independent'''. The | There are several ways to connect a code repository to an Overleaf project. In this one, the two repositories stay '''completely independent'''. The Overleaf project is cloned into a subdirectory (here called <code>paper/</code>), and that subdirectory is listed in the experiment repository's <code>.gitignore</code>. The experiment repository never records anything about the paper, and the paper never sees your experimental code. Your scripts simply write files into a folder that happens to be a different Git repository, one whose remote is Overleaf itself. | ||
The final layout looks like this: | The final layout looks like this: | ||
<pre> | <pre> | ||
my-experiments/ ← experiment repo | my-experiments/ ← experiment repo | ||
├── .gitignore ← contains "paper/" | ├── .gitignore ← contains "paper/" | ||
├── src/ | ├── src/ | ||
| Line 24: | Line 24: | ||
├── Makefile | ├── Makefile | ||
└── paper/ ← SEPARATE git repo, ignored by the outer repo | └── paper/ ← SEPARATE git repo, ignored by the outer repo | ||
├── .git/ ← | ├── .git/ ← remote: https://git.overleaf.com/<project-id> | ||
├── main.tex | ├── main.tex | ||
├── refs.bib | ├── refs.bib | ||
├── figures/ ← written by code | ├── figures/ ← written by code | ||
│ | │ ├── coverage.pdf | ||
│ └── PROVENANCE.txt ← which code version made these | |||
└── tables/ ← written by code | └── tables/ ← written by code | ||
└── results.tex | └── results.tex | ||
</pre> | </pre> | ||
Changes flow directly in both directions, with nothing in between: | |||
<pre> | <pre> | ||
scripts/make_figures.py ──writes──▶ paper/figures, paper/tables | scripts/make_figures.py ──writes──▶ paper/figures, paper/tables | ||
paper/ ──git push──▶ | paper/ ──git push──▶ Overleaf | ||
Overleaf | Overleaf ──git pull──▶ paper/ | ||
</pre> | </pre> | ||
| Line 48: | Line 49: | ||
| style="padding: 15px; border-right: 2px solid white;" | | | style="padding: 15px; border-right: 2px solid white;" | | ||
* '''No More Uploading:''' Regenerated figures and tables reach the paper with one commit and one push, no manual file shuffling required. | * '''No More Uploading:''' Regenerated figures and tables reach the paper with one commit and one push, no manual file shuffling required. | ||
* ''' | * '''Instant Updates:''' A <code>git push</code> shows up in the Overleaf editor right away. There is no sync button to remember. | ||
* '''Minimal Setup:''' No GitHub account, no extra repository, and no linking accounts. You need a token and a clone URL, and that's it. | |||
* '''No Coupled Tracking:''' Unlike git submodules, you never need an extra "update the paper pointer" commit in your experiment repository every time the paper changes. | * '''No Coupled Tracking:''' Unlike git submodules, you never need an extra "update the paper pointer" commit in your experiment repository every time the paper changes. | ||
* '''Collaborator Friendly:''' Co-authors | * '''Collaborator Friendly:''' Co-authors keep writing on Overleaf as usual. Nothing about their workflow changes. | ||
| style="padding: 15px;" | | | style="padding: 15px;" | | ||
* '''Provenance Is Up to You:''' Because the repositories are not linked, you need to record which code produced which figure yourself. A small script (below) handles this. | |||
* '''Provenance Is Up to You:''' Because the repositories are not linked, | * '''No Off-Site Copy:''' The paper lives only on Overleaf (and your local clones). If you want GitHub issues, pull-request reviews, or an independent backup, Overleaf's GitHub Synchronization is the better fit. | ||
* '''One More Thing to Clone:''' Anyone who wants to regenerate the paper's figures needs to clone the | * '''Token Management:''' You need an Overleaf Git token on every machine you push from, including clusters. | ||
* '''One More Thing to Clone:''' Anyone who wants to regenerate the paper's figures needs to clone the Overleaf project separately. | |||
|- | |- | ||
| colspan="2" style="background-color: #e9ecef; text-align: center; padding: 10px;" | 🔗 '''[https://www.overleaf.com/learn/how-to/ | | colspan="2" style="background-color: #e9ecef; text-align: center; padding: 10px;" | 🔗 '''[https://www.overleaf.com/learn/how-to/Git_integration Overleaf Documentation: Git Integration]''' | ||
|} | |} | ||
== Setting It Up == | == Setting It Up == | ||
=== Step 1: | === Step 1: Generate a Git authentication token === | ||
In Overleaf, open '''Account Settings''' | Overleaf doesn't accept your account password over Git. In Overleaf, open '''Account Settings''', find the '''Git Integration''' section, and generate a token. Copy it somewhere safe right away, because Overleaf will only show it once. | ||
<!-- SCREENSHOT: Overleaf Account Settings | <!-- SCREENSHOT: Overleaf Account Settings, Git Integration section, with the "Generate token" button visible (blur any existing token). --> | ||
[[File:Overleaf- | [[File:Overleaf-git-token.png|thumb|center|600px|Generating an Overleaf Git authentication token.]] | ||
=== Step 2: | === Step 2: Find your project's clone URL === | ||
Open your paper in Overleaf, click '''Menu''' in the top left, and choose ''' | Open your paper in Overleaf, click '''Menu''' in the top left, and choose '''Git''' under the ''Sync'' section. The dialog shows a <code>git clone</code> command with your project's URL, which looks like <code>https://git.overleaf.com/<project-id></code>. The project ID is the same string that appears in your browser's address bar when the project is open. | ||
<!-- SCREENSHOT: Overleaf project Menu, with the "Sync → | <!-- SCREENSHOT: Overleaf project Menu, with the "Sync → Git" entry highlighted. --> | ||
[[File:Overleaf-menu-sync- | [[File:Overleaf-menu-sync-git.png|thumb|center|300px|The Git entry in the Overleaf project menu.]] | ||
<!-- SCREENSHOT: The | <!-- SCREENSHOT: The Git dialog showing the clone command (blur or crop the project ID if you prefer). --> | ||
[[File:Overleaf- | [[File:Overleaf-git-clone-url.png|thumb|center|500px|The project's Git clone URL.]] | ||
=== Step 3: Clone the paper inside your experiment repository === | === Step 3: Clone the paper inside your experiment repository === | ||
| Line 90: | Line 88: | ||
git commit -m "Ignore nested paper clone" | git commit -m "Ignore nested paper clone" | ||
git clone git | git clone https://git.overleaf.com/<project-id> paper | ||
</syntaxhighlight> | </syntaxhighlight> | ||
When Git asks for credentials, enter <code>git</code> as the username and your '''token''' as the password. To avoid typing the token every time, let Git remember it with your system's credential helper (e.g. <code>osxkeychain</code> on macOS, <code>manager</code> on Windows). On shared machines, avoid the plain-text <code>store</code> helper. | |||
Then check that the two repositories don't see each other: | Then check that the two repositories don't see each other: | ||
| Line 97: | Line 97: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
git status # should NOT list paper/ | git status # should NOT list paper/ | ||
git -C paper status # shows the | git -C paper status # shows the Overleaf project, on branch master | ||
</syntaxhighlight> | </syntaxhighlight> | ||
<!-- SCREENSHOT: Terminal showing both `git status` outputs (outer repo clean, inner repo on | <!-- SCREENSHOT: Terminal showing both `git status` outputs (outer repo clean, inner repo on master). --> | ||
[[File:Nested-clone-git-status.png|thumb|center|600px|The outer repository ignores <code>paper/</code>; the inner repository is independent.]] | [[File:Nested-clone-git-status.png|thumb|center|600px|The outer repository ignores <code>paper/</code>; the inner repository is independent.]] | ||
| Line 178: | Line 178: | ||
cd paper | cd paper | ||
git pull # pick up any text edits | git pull # pick up any text edits made on Overleaf first | ||
git add figures tables | git add figures tables | ||
git commit -m "Regenerate figures | git commit -m "Regenerate figures" | ||
git push | git push | ||
</syntaxhighlight> | </syntaxhighlight> | ||
That's it. Refresh the Overleaf editor and recompile, and the new figures are there. | |||
<!-- SCREENSHOT: Overleaf | <!-- SCREENSHOT: Overleaf file tree / editor right after a push, showing the updated figure (optionally with the History panel open). --> | ||
[[File:Overleaf- | [[File:Overleaf-after-git-push.png|thumb|center|600px|Pushed figures appear in Overleaf immediately.]] | ||
<div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;"> | <div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;"> | ||
'''Important: | '''Important: Always pull before you push'''<br> | ||
Overleaf | Your co-authors are editing the same repository through the Overleaf editor. If anything changed on Overleaf since your last pull, your push will be rejected until you <code>git pull</code> and merge. As long as your code only touches <code>figures/</code> and <code>tables/</code>, these merges are automatic. | ||
</div> | </div> | ||
=== Recording provenance === | |||
Since the two repositories aren't linked, have your script record which code produced the figures. Overleaf's Git integration does not reliably preserve your local commit messages, so write the information into a '''file''' in the paper instead: | |||
<syntaxhighlight lang="bash"> | |||
SHA=$(git rev-parse --short HEAD) | |||
git diff --quiet || SHA="$SHA-dirty" | |||
echo "Generated from my-experiments@$SHA on $(date -u +%Y-%m-%dT%H:%MZ)" > paper/figures/PROVENANCE.txt | |||
</syntaxhighlight> | |||
The <code>-dirty</code> suffix flags figures made from uncommitted code. Months later, opening <code>PROVENANCE.txt</code> (or its history in Overleaf) answers "where did these figures come from?" | |||
=== Automating it with a Makefile === | === Automating it with a Makefile === | ||
| Line 210: | Line 219: | ||
paper-sync: figures | paper-sync: figures | ||
git -C $(PAPER) pull --rebase | git -C $(PAPER) pull --rebase | ||
git -C $(PAPER) diff --quiet -- figures tables || \ | |||
echo "Generated from my-experiments@$(SHA) on $$(date -u +%Y-%m-%dT%H:%MZ)" > $(PAPER)/figures/PROVENANCE.txt | |||
git -C $(PAPER) add figures tables | git -C $(PAPER) add figures tables | ||
git -C $(PAPER) diff --cached --quiet || \ | git -C $(PAPER) diff --cached --quiet || \ | ||
git -C $(PAPER) commit -m "Regenerate figures from my-experiments@$(SHA)" | git -C $(PAPER) commit -m "Regenerate figures from my-experiments@$(SHA)" | ||
git -C $(PAPER) push | git -C $(PAPER) push | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Now <code>make paper-sync</code> regenerates everything, | Now <code>make paper-sync</code> regenerates everything, pulls in your co-authors' edits, updates the provenance file only when a figure or table actually changed, skips the commit entirely if nothing changed, and pushes straight to Overleaf. | ||
=== Avoiding conflicts === | === Avoiding conflicts === | ||
| Line 233: | Line 237: | ||
<div style="background-color: #f8f9fa; border-left: 4px solid #228B22; padding: 15px; margin-bottom: 10px;"> | <div style="background-color: #f8f9fa; border-left: 4px solid #228B22; padding: 15px; margin-bottom: 10px;"> | ||
'''<code>.tex</code> and <code>.bib</code> files belong to people.''' Write them | '''<code>.tex</code> and <code>.bib</code> files belong to people.''' Write them in the Overleaf editor (or locally, then push), and always pull before you push figures. | ||
</div> | </div> | ||
| Line 244: | Line 248: | ||
! style="background-color: #e9ecef; padding: 10px; text-align: left;" | Gitignored Clone | ! style="background-color: #e9ecef; padding: 10px; text-align: left;" | Gitignored Clone | ||
! style="background-color: #e9ecef; padding: 10px; text-align: left;" | Git Submodule | ! style="background-color: #e9ecef; padding: 10px; text-align: left;" | Git Submodule | ||
|- style="background-color: #f8f9fa;" | |- style="background-color: #f8f9fa;" | ||
| style="padding: 10px;" | '''Experiment repo records the paper version''' || style="padding: 10px;" | No || style="padding: 10px;" | Yes (a pointer | | style="padding: 10px;" | '''Experiment repo records the paper version''' || style="padding: 10px;" | No || style="padding: 10px;" | Yes (a pointer) | ||
|- | |- | ||
| style="padding: 10px;" | '''Extra commit in experiment repo per paper change''' || style="padding: 10px;" | No | | style="padding: 10px;" | '''Extra commit in experiment repo per paper change''' || style="padding: 10px;" | No || style="padding: 10px;" | Yes | ||
|- style="background-color: #f8f9fa;" | |- style="background-color: #f8f9fa;" | ||
| style="padding: 10px;" | ''' | | style="padding: 10px;" | '''Collaborators need Overleaf credentials to clone the code''' || style="padding: 10px;" | No || style="padding: 10px;" | Yes, for a full clone | ||
|- | |- | ||
| style="padding: 10px;" | '''Link between figure and code version''' || style="padding: 10px;" | Provenance file || style="padding: 10px;" | Automatic | |||
| style="padding: 10px;" | '''Link between figure and code version''' || style="padding: 10px;" | | |||
|} | |} | ||
A submodule gives you exact, automatic provenance, but at a price | A submodule gives you exact, automatic provenance, but at a price. Every change to the paper, including every typo fix a co-author makes on Overleaf, needs a matching "bump the submodule" commit in your experiment repository. On top of that, anyone cloning your code with submodules would need an Overleaf token. For a paper that is edited daily, this usually isn't worth it. The gitignored clone trades automatic provenance for a one-line provenance file, which is plenty in practice. | ||
== Variations == | == Variations == | ||
<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;"> | <div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;"> | ||
''' | '''Want a GitHub copy of the paper?''' Overleaf's GitHub Synchronization keeps the project mirrored in a GitHub repository, which you can clone into the gitignored directory in exactly the same way. You gain issues, pull-request reviews, and an off-site backup. The cost is a manual sync step, since Overleaf only exchanges changes with GitHub when someone clicks a button in the Overleaf menu. | ||
</div> | </div> | ||
<div style="background-color: #f8f9fa; border-left: 4px solid #FFD43B; padding: 15px; margin-bottom: 10px;"> | <div style="background-color: #f8f9fa; border-left: 4px solid #FFD43B; padding: 15px; margin-bottom: 10px;"> | ||
'''Generating figures on a cluster:''' Clone the | '''Generating figures on a cluster:''' Clone the Overleaf project on the cluster as well, set up your token with a credential helper, and run <code>make paper-sync</code> there. Alternatively, point <code>PAPER_DIR</code> at a scratch directory on the cluster, copy the outputs back to your laptop, and push from there, so your token never leaves your own machine. | ||
</div> | </div> | ||
| Line 275: | Line 276: | ||
|- style="vertical-align: top; background-color: #f8f9fa;" | |- style="vertical-align: top; background-color: #f8f9fa;" | ||
| style="padding: 15px;" | | | style="padding: 15px;" | | ||
* ''' | * '''Rejected pushes:''' Someone edited the paper on Overleaf since your last pull. Run <code>git pull</code>, then push again. Never force-push to Overleaf. | ||
* ''' | * '''Authentication failures:''' The username is literally <code>git</code>, and the password is your token, not your Overleaf password. If a token stops working, generate a new one and update your credential helper. | ||
* '''Bloating the | * '''Working on branches:''' Overleaf only knows about <code>master</code>. You can branch locally, but only commits merged into <code>master</code> can be pushed. | ||
* '''Bloating the project:''' PNGs and timestamped PDFs change on every run. Prefer vector PDFs with the timestamp removed, and only commit figures that actually changed. | |||
* '''Overleaf project limits:''' Overleaf limits project size and file counts. Keep raw outputs, checkpoints, and intermediate data in the experiment repository (or out of Git entirely); only put paper-ready figures and tables in <code>paper/</code>. | * '''Overleaf project limits:''' Overleaf limits project size and file counts. Keep raw outputs, checkpoints, and intermediate data in the experiment repository (or out of Git entirely); only put paper-ready figures and tables in <code>paper/</code>. | ||
* ''' | * '''Leaking tokens:''' Never paste the token into the clone URL in a shared script or commit it anywhere. Let a credential helper store it. | ||
|} | |} | ||
<div style="background-color: #e8f4f8; padding: 15px; text-align: center; margin-top: 20px; font-style: italic;"> | <div style="background-color: #e8f4f8; padding: 15px; text-align: center; margin-top: 20px; font-style: italic;"> | ||
'''Note:''' This is just one of several ways to connect your code and your paper. If | '''Note:''' This is just one of several ways to connect your code and your paper. If you want your paper on GitHub, try GitHub Synchronization instead; if you want airtight provenance, a submodule may be worth the extra commits. | ||
</div> | </div> | ||
Revision as of 23:24, 18 October 2026
If your experiment code lives in a Git repository and your paper lives on Overleaf, you have probably done the upload dance: rerun an experiment, hunt down the regenerated figure on disk, open Overleaf, delete the old file, upload the new one, and repeat for every figure and table that changed. It is slow, it is easy to leave a stale figure behind, and it leaves no record of which version of your code produced which figure in the paper.
Overleaf Premium's Git integration removes most of this busywork. Every Overleaf project can be cloned as an ordinary Git repository, with no GitHub account or extra repository required. This article describes one way to use it: cloning the Overleaf project inside your experiment repository, in a directory that the experiment repository ignores. Your analysis scripts then write figures and tables straight into the paper, and a single git push puts them on Overleaf.
How It Works
There are several ways to connect a code repository to an Overleaf project. In this one, the two repositories stay completely independent. The Overleaf project is cloned into a subdirectory (here called paper/), and that subdirectory is listed in the experiment repository's .gitignore. The experiment repository never records anything about the paper, and the paper never sees your experimental code. Your scripts simply write files into a folder that happens to be a different Git repository, one whose remote is Overleaf itself.
The final layout looks like this:
my-experiments/ ← experiment repo
├── .gitignore ← contains "paper/"
├── src/
├── scripts/
│ └── make_figures.py ← writes into paper/figures and paper/tables
├── Makefile
└── paper/ ← SEPARATE git repo, ignored by the outer repo
├── .git/ ← remote: https://git.overleaf.com/<project-id>
├── main.tex
├── refs.bib
├── figures/ ← written by code
│ ├── coverage.pdf
│ └── PROVENANCE.txt ← which code version made these
└── tables/ ← written by code
└── results.tex
Changes flow directly in both directions, with nothing in between:
scripts/make_figures.py ──writes──▶ paper/figures, paper/tables paper/ ──git push──▶ Overleaf Overleaf ──git pull──▶ paper/
| Strengths | Weaknesses |
|---|---|
|
|
| 🔗 Overleaf Documentation: Git Integration | |
Setting It Up
Step 1: Generate a Git authentication token
Overleaf doesn't accept your account password over Git. In Overleaf, open Account Settings, find the Git Integration section, and generate a token. Copy it somewhere safe right away, because Overleaf will only show it once.

Step 2: Find your project's clone URL
Open your paper in Overleaf, click Menu in the top left, and choose Git under the Sync section. The dialog shows a git clone command with your project's URL, which looks like https://git.overleaf.com/<project-id>. The project ID is the same string that appears in your browser's address bar when the project is open.

Step 3: Clone the paper inside your experiment repository
From the root of your experiment repository, tell Git to ignore the paper/ directory before you clone into it:
cd my-experiments
echo "paper/" >> .gitignore
git add .gitignore
git commit -m "Ignore nested paper clone"
git clone https://git.overleaf.com/<project-id> paperWhen Git asks for credentials, enter git as the username and your token as the password. To avoid typing the token every time, let Git remember it with your system's credential helper (e.g. osxkeychain on macOS, manager on Windows). On shared machines, avoid the plain-text store helper.
Then check that the two repositories don't see each other:
git status # should NOT list paper/
git -C paper status # shows the Overleaf project, on branch masterpaper/; the inner repository is independent.Step 4: Point your code at the paper directory
Rather than hard-coding paths in every script, define the paper location once. Using an environment variable with a sensible default means the code still runs for people who haven't cloned the paper (their figures just land in an ignored folder).
# src/paths.py
import os
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
PAPER_DIR = Path(os.environ.get("PAPER_DIR", REPO_ROOT / "paper"))
FIG_DIR = PAPER_DIR / "figures"
TAB_DIR = PAPER_DIR / "tables"
FIG_DIR.mkdir(parents=True, exist_ok=True)
TAB_DIR.mkdir(parents=True, exist_ok=True)For figures, save vector PDFs directly into the paper. Removing the creation timestamp means a figure that didn't actually change produces an identical file, so Git won't report a pointless change:
import matplotlib.pyplot as plt
from src.paths import FIG_DIR
fig, ax = plt.subplots(figsize=(3.25, 2.4)) # match your template's column width
...
fig.savefig(FIG_DIR / "coverage.pdf", bbox_inches="tight",
metadata={"CreationDate": None})For tables, write a LaTeX fragment rather than a full document:
from src.paths import TAB_DIR
tex = df.to_latex(index=False, escape=True, float_format="%.3f")
(TAB_DIR / "results.tex").write_text(tex)and pull both into the paper as usual:
\includegraphics[width=\linewidth]{figures/coverage}
...
\input{tables/results}For numbers quoted in the text, the same idea works. Have your code write LaTeX macros so the prose never goes stale:
(TAB_DIR / "numbers.tex").write_text(
f"\\newcommand{{\\coverageNinetyFive}}{{{cov95:.1f}\\%}}\n"
)% preamble
\input{tables/numbers}
% body
Our intervals achieve \coverageNinetyFive{} empirical coverage.Day-to-Day Workflow
Sending new results to the paper
After rerunning your experiments, commit and push from inside paper/:
python scripts/make_figures.py
cd paper
git pull # pick up any text edits made on Overleaf first
git add figures tables
git commit -m "Regenerate figures"
git pushThat's it. Refresh the Overleaf editor and recompile, and the new figures are there.
Recording provenance
Since the two repositories aren't linked, have your script record which code produced the figures. Overleaf's Git integration does not reliably preserve your local commit messages, so write the information into a file in the paper instead:
SHA=$(git rev-parse --short HEAD)
git diff --quiet || SHA="$SHA-dirty"
echo "Generated from my-experiments@$SHA on $(date -u +%Y-%m-%dT%H:%MZ)" > paper/figures/PROVENANCE.txtThe -dirty suffix flags figures made from uncommitted code. Months later, opening PROVENANCE.txt (or its history in Overleaf) answers "where did these figures come from?"
Automating it with a Makefile
Once the steps above feel familiar, wrap them in a single command:
PAPER := paper
SHA := $(shell git rev-parse --short HEAD)$(shell git diff --quiet || echo -dirty)
.PHONY: figures paper-sync
figures:
python scripts/make_figures.py
paper-sync: figures
git -C $(PAPER) pull --rebase
git -C $(PAPER) diff --quiet -- figures tables || \
echo "Generated from my-experiments@$(SHA) on $$(date -u +%Y-%m-%dT%H:%MZ)" > $(PAPER)/figures/PROVENANCE.txt
git -C $(PAPER) add figures tables
git -C $(PAPER) diff --cached --quiet || \
git -C $(PAPER) commit -m "Regenerate figures from my-experiments@$(SHA)"
git -C $(PAPER) pushNow make paper-sync regenerates everything, pulls in your co-authors' edits, updates the provenance file only when a figure or table actually changed, skips the commit entirely if nothing changed, and pushes straight to Overleaf.
Avoiding conflicts
Conflicts are rare if every file has exactly one "owner":
figures/ and tables/ belong to your code. Never edit them by hand on Overleaf; the next regeneration will silently overwrite your changes. Change the formatting in the code that generates them instead.
.tex and .bib files belong to people. Write them in the Overleaf editor (or locally, then push), and always pull before you push figures.
Why Not a Git Submodule?
Git submodules are the "official" way to put one repository inside another, so it is natural to ask why this workflow avoids them.
| Gitignored Clone | Git Submodule | |
|---|---|---|
| Experiment repo records the paper version | No | Yes (a pointer) |
| Extra commit in experiment repo per paper change | No | Yes |
| Collaborators need Overleaf credentials to clone the code | No | Yes, for a full clone |
| Link between figure and code version | Provenance file | Automatic |
A submodule gives you exact, automatic provenance, but at a price. Every change to the paper, including every typo fix a co-author makes on Overleaf, needs a matching "bump the submodule" commit in your experiment repository. On top of that, anyone cloning your code with submodules would need an Overleaf token. For a paper that is edited daily, this usually isn't worth it. The gitignored clone trades automatic provenance for a one-line provenance file, which is plenty in practice.
Variations
Want a GitHub copy of the paper? Overleaf's GitHub Synchronization keeps the project mirrored in a GitHub repository, which you can clone into the gitignored directory in exactly the same way. You gain issues, pull-request reviews, and an off-site backup. The cost is a manual sync step, since Overleaf only exchanges changes with GitHub when someone clicks a button in the Overleaf menu.
Generating figures on a cluster: Clone the Overleaf project on the cluster as well, set up your token with a credential helper, and run make paper-sync there. Alternatively, point PAPER_DIR at a scratch directory on the cluster, copy the outputs back to your laptop, and push from there, so your token never leaves your own machine.
Common Pitfalls
| Things to Watch Out For |
|---|
|
Note: This is just one of several ways to connect your code and your paper. If you want your paper on GitHub, try GitHub Synchronization instead; if you want airtight provenance, a submodule may be worth the extra commits.