Git Submodule Workflow¶

This document explains how to work with git submodules in the bazzite-ai repository.

Overview¶

The bazzite-ai repository uses git submodules to manage two separate codebases:

plugins (plugins/) - Claude Code skills and MCP server implementations
notebooks (notebooks/) - Jupyter notebooks for AI/ML experiments and documentation

Submodules are separate git repositories referenced at specific commits. When you clone bazzite-ai, the submodules are not automatically initialized unless you use `git clone --recurse-submodules`.

Architecture¶

Why Submodules?¶

Separation of Concerns:

Main repo: OS configuration, build system, documentation
plugins: Reusable skills and tools (can be used independently)
notebooks: Experiment notebooks (can be shared across projects)

Independent Development:

Each submodule has its own commit history
Can be versioned and released independently
Multiple projects can reference the same submodule

Clean Repository Structure:

Prevents mixing OS builds with experiment code
Makes CI/CD clearer (separate workflows per concern)
Easier to grant different access permissions

Repository URLs¶

Submodule	URL	Purpose
plugins	`git@github.com:atrawog/bazzite-ai-plugins.git`	Claude Code skills, MCP servers
notebooks	`git@github.com:atrawog/bazzite-ai-notebooks.git`	Jupyter notebooks for experiments

The Correct Workflow¶

**Critical Rule:** Always push submodule commits BEFORE pushing parent repo commits that reference them. Violating this causes CI/CD failures.

Step-by-Step: Making Changes¶

1. Make Changes in Submodule¶

# Navigate to submodule directory
cd notebooks/  # or plugins/

# Make your changes
vim finetuning/my-experiment.ipynb

# Commit in the submodule
git add finetuning/my-experiment.ipynb
git commit -m "Feat: Add new finetuning experiment"

2. CRITICAL: Push Submodule First¶

# Still in notebooks/ or plugins/
git push origin main

If you skip this step, the next step will fail with the pre-push hook, or worse, cause CI/CD failures if you bypass the hook.

3. Update Parent Repo Pointer¶

# Return to parent repo root
cd ..

# Git automatically detects submodule changes
git status
# Shows:
#   modified:   notebooks (new commits)

# Stage the submodule update
git add notebooks/

# Commit the parent repo update
git commit -m "Feat: Update notebooks submodule with finetuning experiment"

4. Push Parent Repo¶

# Pre-push hook validates submodule commits exist remotely
git push origin testing

Automation and Validation¶

Pre-Push Hook¶

The repository includes a pre-push hook that validates all submodule commits exist in their remotes before allowing you to push the parent repo.

Location: .git-hooks/pre-push-submodules.sh

Activation:

# Link the hook (done once during setup)
ln -sf ../../.git-hooks/pre-push-submodules.sh .git/hooks/pre-push
chmod +x .git-hooks/pre-push-submodules.sh

Behavior:

Runs automatically before every git push
Checks each submodule commit referenced by the parent repo
Blocks the push if any submodule commit is missing from its remote
Provides clear instructions on how to fix the issue

Example Output (when validation fails):

=== Validating submodule commits ===

Checking submodule: notebooks
  Referenced commit: 8191c487fe1afa9aa1e7115761ead6bf4edea79b
  Remote: git@github.com:atrawog/bazzite-ai-notebooks.git
  ❌ MISSING: Commit not found in remote

════════════════════════════════════════════════════════════════
❌ PRE-PUSH VALIDATION FAILED
════════════════════════════════════════════════════════════════

One or more submodule commits have not been pushed to their
remote repositories. This will cause CI/CD failures when GitHub
Actions tries to checkout the code.

Missing commits:
  • notebooks: 8191c487fe1afa9aa1e7115761ead6bf4edea79b

To fix this:

  1. cd notebooks
  2. git push origin HEAD:main
  3. cd ..
  4. git push  # Retry parent repo push

Bypassing the Hook:

**NEVER bypass the pre-push hook** unless you absolutely understand the consequences. Bypassing will break CI/CD.

If you must bypass (e.g., fixing a broken CI state):

git push --no-verify

CI/CD Validation¶

The repository includes a GitHub Actions workflow that provides an additional validation layer.

Workflow: .github/workflows/validate-submodules.yml

Behavior:

Runs on every push to main and testing branches
Runs on every pull request
Validates submodule commits exist in remotes
Blocks other workflows if validation fails
Shows clear error messages in GitHub UI

Benefits:

Catches issues even if developers bypass local hooks
Prevents cascading CI/CD failures
Provides visibility in pull requests

Common Operations¶

Cloning the Repository¶

Option 1: Clone with submodules (recommended)

git clone --recurse-submodules git@github.com:atrawog/bazzite-ai.git

Option 2: Initialize submodules after cloning

git clone git@github.com:atrawog/bazzite-ai.git
cd bazzite-ai
git submodule update --init --recursive

Updating Submodules¶

Pull latest changes in submodules:

# Update all submodules to their latest commits on tracked branches
git submodule update --remote --merge

# Or update a specific submodule
cd notebooks/
git pull origin main
cd ..

# Commit the submodule update in parent repo
git add notebooks/
git commit -m "Feat: Update notebooks submodule to latest"
git push

Checking Submodule Status¶

View current submodule commits:

git submodule status
# Output:
#   8191c487fe1afa9aa1e7115761ead6bf4edea79b notebooks (heads/main)
#   397dbbbcd538f31357a1977bcc9926fd970ba6be plugins (heads/main)

Check if submodule commits exist remotely:

# Run the pre-push hook manually
bash .git-hooks/pre-push-submodules.sh

Switching Branches¶

When switching branches in the parent repo, submodules may need to be updated:

# Switch branch in parent repo
git checkout main

# Update submodules to match the new branch
git submodule update --init --recursive

Making a Pull Request¶

Before creating a pull request:
1. Ensure all submodule commits are pushed to their remotes
2. Run `bash .git-hooks/pre-push-submodules.sh` to validate
3. Push the parent repo
4. CI will automatically validate submodules in the PR

Troubleshooting¶

Error: "fatal: remote error: upload-pack: not our ref ..."¶

Symptom:

fatal: remote error: upload-pack: not our ref 8191c487fe1afa9aa1e7115761ead6bf4edea79b
fatal: Fetched in submodule path 'notebooks', but it did not contain 8191c487...

Cause: The parent repo references a submodule commit that doesn't exist in the remote repository.

Solution:

# 1. Check which commits are unpushed
cd notebooks/  # or plugins/
git status
# Shows: "Your branch is ahead of 'origin/main' by N commits"

# 2. Push the missing commits
git push origin main

# 3. Return to parent and verify
cd ..
bash .git-hooks/pre-push-submodules.sh

# 4. Push parent repo
git push origin testing

Error: "Submodule path '...': checked out '...'"¶

Symptom: After pulling the parent repo, git shows a different commit in the submodule.

Cause: The parent repo's submodule pointer was updated, but your local submodule is still on an older commit.

Solution:

# Update submodules to match parent repo pointers
git submodule update --init --recursive

Error: "You have uncommitted changes in submodule ..."¶

Symptom: Git prevents operations because you have uncommitted changes in a submodule.

Solution:

# Option 1: Commit the changes
cd notebooks/  # or plugins/
git add .
git commit -m "Feat: Description"
git push origin main
cd ..
git add notebooks/
git commit -m "Feat: Update notebooks submodule"

# Option 2: Stash the changes
cd notebooks/
git stash
cd ..

CI Workflow Failing on Checkout¶

Symptom: GitHub Actions workflow fails during the checkout step with submodule errors.

Cause: Either:

Workflow missing submodules: recursive parameter
Referenced submodule commit not pushed to remote

Solution:

Fix 1: Add submodules parameter to workflow

- name: Checkout
  uses: actions/checkout@v4
  with:
    submodules: recursive  # Add this line

Fix 2: Push missing submodule commits

# See "Error: fatal: remote error: upload-pack" section above

Pre-Push Hook Not Running¶

Symptom: The pre-push hook doesn't run when you git push.

Cause: The hook is not linked or not executable.

Solution:

# Link the hook
ln -sf ../../.git-hooks/pre-push-submodules.sh .git/hooks/pre-push

# Make it executable
chmod +x .git-hooks/pre-push-submodules.sh

# Verify it works
bash .git-hooks/pre-push-submodules.sh

Submodule Directory is Empty¶

Symptom: After cloning, plugins/ or notebooks/ directories are empty.

Cause: Submodules were not initialized during clone.

Solution:

# Initialize and fetch all submodules
git submodule update --init --recursive

Best Practices¶

DO¶

✅ Always push submodule commits before parent repo commits

✅ Use the pre-push hook - It prevents 99% of issues

✅ Run git submodule status regularly to check synchronization

✅ Update submodules after pulling parent repo changes

git pull && git submodule update --init --recursive

✅ Test changes locally in both submodule and parent repo before pushing

✅ Use descriptive commit messages in both submodule and parent commits

DON'T¶

❌ Never push parent repo without pushing submodule commits first

❌ Never use git push --no-verify unless absolutely necessary

❌ Don't modify .gitmodules without understanding the consequences

❌ Don't commit submodule pointers unless you've actually changed the submodule

❌ Don't forget to initialize submodules after cloning or switching branches

Reference¶

Git Commands Quick Reference¶

# Clone with submodules
git clone --recurse-submodules <url>

# Initialize submodules after clone
git submodule update --init --recursive

# Check submodule status
git submodule status

# Update submodules to latest remote commits
git submodule update --remote --merge

# Pull latest parent repo and update submodules
git pull && git submodule update --init --recursive

# Push submodule changes
cd <submodule-path>
git push origin main
cd ..

# Validate submodule commits before pushing
bash .git-hooks/pre-push-submodules.sh

# Push parent repo
git push origin testing

Files and Locations¶

File/Directory	Purpose
`.gitmodules`	Submodule configuration (URLs, paths)
`.git/modules/`	Submodule git repositories
`.git-hooks/pre-push-submodules.sh`	Pre-push validation hook
`.git/hooks/pre-push`	Symlink to pre-push hook (when activated)
`.github/workflows/validate-submodules.yml`	CI/CD submodule validation

Additional Resources¶

Getting Help¶

If you encounter issues not covered in this guide:

Check CI/CD logs - GitHub Actions provides detailed error messages
Run the pre-push hook manually - bash .git-hooks/pre-push-submodules.sh
Ask in discussions - bazzite-ai discussions
Report a bug - bazzite-ai issues

When reporting submodule issues, always include:
- Output of `git submodule status`
- Output of `bash .git-hooks/pre-push-submodules.sh`
- Relevant CI/CD logs (if applicable)