General Introduction

2.1 The “Push/Pull” Model of Software Development

Git is a powerful, distributed source code management system. However, the way it’s used for gawk development purposely does not take advantage of all its features.

Instead, the model is rather simple, and in many ways much like more traditional distributed systems such as the Concurrent Versions System (CVS) or Subversion (SVN).

The central idea can be termed “push/pull.” You pull updates down from the central repository to your local copy, and if you have commit rights, you push your changes or updates up to the central repository.

Other developers work this way; pushing their changes up to the central repository and pulling your changes down into theirs.

Where Git does stand out is in its management of multiple branches of development. Git makes it very easy to set up a separate branch for use in fixing a bug or developing a feature. You can then easily keep that branch up to date with respect to the main development branch(es), and eventually merge the changes from your branch into the main branch.

Almost always Git does these merges for you without problem. When there is a problem (a merge conflict), usually it is very easy for you to resolve them and then complete the merge. We talk about this in more detail later (see Dealing With Merge Conflicts).

2.2 How Git Stores Branches and Their Copies

So how does Git work?¹

A repository consists of a collection of branches. Each branch represents the history of a collection of files and directories (a file tree). Each combined set of changes to this collection (files and directories added or deleted, and/or file contents changed) is termed a commit.

When you first create a local copy of a remote repository (“clone the repo”), Git copies all of the original repository’s branches to your local system. The original remote repository is referred to as being upstream, and your local repo is downstream from it. Git distinguishes branches from the upstream repo by prefixing their names with ‘origin/’. Let’s draw some pictures. Figure 2.1 represents the state of the repo on Savannah:

+======================+
|       Branches       |
+======================+
| master               |
+----------------------+
| gawk-4.1-stable      |
+----------------------+
| gawk-4.0-stable      |
+----------------------+
| feature/fix-comments |
+----------------------+
| ...                  |
+----------------------+

After you clone the repo, on your local system you will have a single branch named master that’s visible when you use ‘git branch’ to see your branches.

$ git clone http://git.savannah.gnu.org/r/gawk.git  Clone the repo
$ cd gawk                                           Change to local copy
$ git branch                                        See branch information
-| * master

The current branch is always indicated with a leading asterisk (‘*’).

Pictorially, the local repo looks like Figure 2.2 (you can ignore the ‘T’ column for the moment):

+===+======================++=============================+
| T |    Local Branches    ||      Remote Branches        |
+===+======================++=============================+
| X | master               || origin/master               |
+---+----------------------++-----------------------------+
|   |                      || origin/gawk-4.1-stable      |
+---+----------------------++-----------------------------+
|   |                      || origin/gawk-4.0-stable      |
+---+----------------------++-----------------------------+
|   |                      || origin/feature/fix-comments |
+---+----------------------++-----------------------------+
|   |                      || ...                         |
+---+----------------------++-----------------------------+

Note that what is simply gawk-4.1-stable in the upstream repo is now referred to as origin/gawk-4.1-stable. The ‘origin/’ branches are a snapshot of the state of the upstream repo. This is how Git allows you to see what changes you’ve made with respect to the upstream repo, without having to actually communicate with the upstream repo over the Internet. (When files are identical, Git is smart enough to not have two separate physical copies on your local disk.)

If you’re working on a simple bug fix or change, you can do so directly in your local master branch. You can then commit your changes, and if you have access rights, push them upstream to the Savannah repo. (However, there is a process to follow. Please read the rest of this Web page.)

2.3 Local Branches

Let’s talk about local branches in more detail. (The terminology used here is my own, not official Git jargon.) There are two kinds of local branches:

Tracking Branches ¶

Tracking branches track branches from the upstream repository. You first create a tracking branch simply by checking out a branch from the upstream. You use the branch name without the leading ‘origin/’ prefix. For example, ‘git checkout gawk-4.1-stable’.

You can then work on this branch, making commits to it as you wish. Once things are ready to move upstream, you simply use ‘git push’, and your changes will be pushed up to the main repo.²

You should never checkout a branch using the ‘origin/’ prefix. Things will get very confused. Always work on local tracking branches.

Purely Local Branches

A purely local branch exists only on your system. You may be developing some large new feature, or fixing a very difficult bug, or have a change for which paperwork has not yet been completed.

In such a case, you would keep your changes on a local branch, and periodically synchronize it with master (or whichever upstream branch you started from).

This may seem somewhat abstract so far. We demonstrate with commands and branches in Development Without Commit Access, later in this Web page.

Let’s say you have checked out a copy of gawk-4.1-stable and have created a purely local branch named better-random. Then our picture now looks like Figure 2.3, where the ‘T’ column indicates a tracking branch.

+===+======================++=============================+
| T |   Local Branches     ||      Remote Branches        |
+===+======================++=============================+
| X | master               || origin/master               |
+---+----------------------++-----------------------------+
| X | gawk-4.1-stable      || origin/gawk-4.1-stable      |
+---+----------------------++-----------------------------+
|   |                      || origin/gawk-4.0-stable      |
+---+----------------------++-----------------------------+
|   |                      || origin/feature/fix-comments |
+---+----------------------++-----------------------------+
|   |                      || ...                         |
+---+----------------------++-----------------------------+
|   | better-random        ||                             |
+---+----------------------++-----------------------------+

2.4 Branches Represent Development State

Branches represent development state. At any given time, when you checkout a particular branch (or create a new one), you have a copy of the gawk source tree that you should be able to build and test.

The following sections describe the different branches in the gawk repository and what they are for, as well as how to use your own branches.

• Branches in the Savannah Repository
• Branches in Your Local Repository
• A Closer Look at Branch Naming

$ git branch -a
-|   gawk-4.1-stable
-| * master
-|   remotes/origin/HEAD -> origin/master
-|   remotes/origin/dead-branches/async-events
-|   …
-|   remotes/origin/feature/api-mpfr
-|   remotes/origin/feature/array-iface
-|   remotes/origin/feature/fix-comments
-|   …

4.1 Cloning The Repo

Clone the Savannah repo using ‘git clone’. You should do so using using the HTTPS protocol; HTTPS is considered to be more secure than the native Git protocol and is preferred.⁴

To choose which method, you supply a URL for the repo when you clone it, as follows.

$ git clone https://git.savannah.gnu.org/r/gawk.git Clone the repo
-| ...
$ cd gawk                                           Start working

You only need to clone the repo once. From then on, you update its contents using other Git commands. For example, after coming back from your vacation in the Bahamas:

$ cd gawk               Move to the repo
$ make distclean        A good idea before updating
-| ...
$ git pull              Update it

To build, you should generally follow this recipe:

$ ./bootstrap.sh && ./configure && make -j && make check

NOTE: Unless you have installed all the tools described in GNU Tools, you must run ./bootstrap.sh every time you clone a repo, do a ‘git pull’ or checkout a different branch. (In the latter case, do ‘make distclean’ first.) Otherwise things will get messy very quickly. The bootstrap.sh script ensures that all of the file time stamps are up to date so that it’s not necessary to run the various configuration tools.

4.2 Switching Branches

So far, we’ve been working in the default master branch. Let’s check what’s happening in the gawk-4.1-stable branch:

$ make distclean                          Clean up
$ git checkout gawk-4.1-stable            Checkout a different branch
-| ...
$ git pull                                Get up to date
-| ...
$ ./bootstrap.sh && ./configure &&        Start working
> make -j && make check

4.3 Starting A New Branch

Let’s say you want to work on a new feature. For example, you might decide to add Python syntax support.⁵ You should create a new branch on which to work. First, switch back to master:

$ make distclean
$ git checkout master

Now, create a new branch. The easiest way to do that is with the -b option to ‘git checkout’:

$ git checkout -b feature/python
-| ...

You now do massive amounts of work in order to add Python syntax support. As you do each defined chunk of work, you update the ChangeLog file with your changes before committing them to the repo.

Let’s say you’ve added a new file python.c and updated several others. Use ‘git status’ to see what’s changed:

$ git status
-| ...

Before committing the current set of changes, you can use ‘git diff’ to view the changes. You may also use ‘git difftool’⁶ to run an external diff command, such as meld on GNU/Linux:

$ git diff                           Regular built-in tool for standard diffs
$ git difftool --tool=meld           GUI diff tool

When you’re happy with the changes, use ‘git add’ to tell Git which of the changed and/or new files you wish to have ready to be committed:

$ git add ...

Use ‘git status’ to see that your changes are scheduled for committing:

$ git status
-|

Now you can commit your changes to your branch:

$ git commit

Running ‘git commit’ causes Git to invoke an editor (typically from the $EDITOR environment variable) in which you can compose a commit message. Please supply a short message summarizing the commit. This message will be visible via ‘git log’.

4.4 Undoing A Change

Should you need to undo a change that you have not yet committed (so that you can start over), you can do so on per-file basis by simply checking out the file again:

$ git checkout awkgram.y     Undo changes to awkgram.y.  There is no output

To start over completely, use ‘git reset --hard’. Note that this will throw away all your changes, with no chance for recovery, so be sure you really want to do it.

4.5 Moving Changes Aside

Sometimes, you may be in the middle of a set of changes that are not yet completed, when you need to stop what you’re doing and work on something else. For example, you might be updating the documentation when a bug report comes in and you want to work on the bug. But you can’t just switch branches, since you haven’t finished your current changes.

The way to work around this problem is with ‘git stash’. This command saves your changes in a special place within Git from which they may be restored later. After executing ‘git stash’, your current branch is restored to its original, pristine state.

The workflow might go something like this:

$ git checkout my-local-branch                Checkout a work branch
...                                           Do some work
$ git stash                                   Save the work aside
$ git checkout gawk-4.1-stable                Work on a bug fix
...                                           Now we're done
$ git checkout my-local-branch                Go back to our local work
$ git stash pop                               Restore the earlier changes

The stash is maintained as a stack. Sets of changes are pushed onto the stack by ‘git stash’ and popped off of it with ‘git stash pop’. You may use ‘git stash list’ to see the list of saved changes.

4.6 Updating and Merging

As you work on your branch, you will occasionally want to bring it up to date with respect to master. This section discusses updating local branches and handling merge conflicts.

• Rebasing A Local Branch
• Dealing With Merge Conflicts

$ git checkout master                Checkout master
$ git pull                           Update it
$ git checkout feature/python        Move back to new, purely local branch
$ git rebase master                  ``Start over'' from current master

$ git checkout feature/python        Move back to new, purely local branch
$ git rebase master                  ``Start over'' from current master
-| First, rewinding head to replay your work on top of it...
-| Applying: Demo change.
-| Using index info to reconstruct a base tree...
-| M	main.c
-| Falling back to patching base and 3-way merge...
-| Auto-merging main.c
-| CONFLICT (content): Merge conflict in main.c
-| error: Failed to merge in the changes.
-| Patch failed at 0001 Demo change.
-| Use 'git am --show-current-patch' to see the failed patch
-|
-| Resolve all conflicts manually, mark them as resolved with
-| "git add/rm <conflicted_files>", then run "git rebase --continue".
-| You can instead skip this commit: run "git rebase --skip".
-| To abort and get back to the state before "git rebase", run "git rebase --abort".
$ gvim main.c                        Edit the file and fix the problem
$ git add main.c                     Tell Git everything is OK now …
$ git commit                         … and it's settled
$ git rebase --continue              Continue the rebase

4.7 Submitting Your Changes

So now your feature is complete. You’ve added test cases for it to the test suite⁷, you have ChangeLog entries that describe all the changes⁸, you have documented the new feature⁹, and everything works great. You’re ready to submit the changes for review, and with any luck, inclusion into gawk.

There are two ways to submit your changes for review.

Generate a single large patch

To do this, simply compare your branch to the branch off which it is based:

$ git checkout feature/python
$ git diff master > /tmp/python.diff

Mail the python.diff file to the appropriate mailing list along with a description of what you’ve changed and why.

The patch file will likely contain changes to generated files, such as awkgram.c or Makefile.in. If you are comfortable manually editing the patch file to remove those changes, do so. If not, then send the file as-is and the maintainer will handle it.

Generate a set of patches that in toto comprise your changes

To do this, use ‘git format-patch’:

$ git checkout feature/python
$ git format-patch

This creates a set of patch files, one per commit that isn’t on the original branch. Mail these patches, either separately, or as a set of attachments, to the appropriate mailing list along with a description of what you’ve changed and why.

Either way you choose to submit your changes, the gawk maintainer and development team will review your changes and provide feedback. If you have signed paperwork with the FSF for gawk and the maintainer approves your changes, he will apply the patch(es) and commit the changes.

Which list should you send mail to? If you are just starting to contribute, use bug-gawk@gnu.org. After making enough contributions, you may be invited to join the private gawk developers’ mailing list. If you do so, then submit your changes to that list.

If you make any substantial changes, you will need to assign copyright in those changes to the Free Software Foundation before the maintainer can commit those changes. See Assigning Copyrights to the FSF, for more information.

4.8 Removing Branches

Once the maintainer has integrated your changes, you can get rid of your local branch:

$ git checkout master                 Move to upstream branch
$ git pull                            Update
$ gvim ChangeLog ...                  Verify your changes are in
$ git branch -d feature/python        Remove your local branch

4.9 Points to Remember

There are some important points to remember:

• Always do a ‘make distclean’ before switching between branches. Things will get really confused if you don’t.
• For upstream branches, always work with tracking branches. Never use ‘git checkout origin/whatever’. Git will happily let you do something like that, but it’s just plain asking for trouble.
• Make sure your tracking branches are up-to-date before doing anything with them, particularly using them as the basis for a rebase or merge. This typically means a three-step process:

  $ git checkout master             Get to local copy
  $ git pull                        Bring it up to date
  $ git checkout feature/python     Go back to your branch
  

  You can then do the actual rebase:

  $ git rebase master               Now rebase your feature off of master
  

• Git always treats the currently checked-out branch as the object of operations. For example, when comparing files with the regular diff command, the usage is ‘diff oldfile newfile’. For ‘git diff’, the current branch takes the place of newfile, thus:

  $ git checkout feature/python
  $ git diff master                 Compare master to current branch
  

  or if merging:

  $ git checkout master             Checkout master
  $ git pull                        Update tracking branch
  $ git merge feature/python        Merge changes into master
  

5.1 Initial Setup

Congratulations! After becoming a quality contributor to gawk development, you’ve been invited to join the private development list and to accept having commit access to the repo.

The first thing to do is to create an account on Savannah, choosing a unique user name. To do so, go to the Savannah home page and click on the “New User” link. The setup will include uploading of your ssh key, as per the instructions on the Savannah web page.

After you’ve done all this, send email to the maintainer with your Savannah user name, and he will add you to the list of users who have commit access to the repo.

5.2 Cloning The Repo With An ssh URL

In order to be able to commit changes to the repo, you must clone it using an ‘ssh://’ URL. Cloning the repo with ssh is similar to cloning with HTTPS, but the URL is different:

$ git clone ssh://yourname@git.sv.gnu.org/srv/git/gawk.git
-| ...

Here, you should replace ‘yourname’ in the command with the user name you chose for use on Savannah.

5.3 Developing Patches

The first part of developing a patch is the same as for developers without commit access:

1. Develop the code and test it.
2. Update the ChangeLog.
3. If necessary, update the documentation: doc/gawktexi.in and/or doc/gawk.1.
4. Use ‘git diff > mychange.diff’ to create a patch file. (If you want, remove the diffs for generated files.)
5. Send it to the mailing list for discussion.
6. Iterate until the patch is ready to be committed.

However, now that you have commit access, you can commit the fix and push it up to the repo yourself! Let’s assume you’ve made a bug fix directly on master. Here’s how to commit your changes:

$ git diff            Review the patch one more time
$ git add …         Add any files for committing
$ git commit          Commit the files, with a commit message
$ git push            Push the files up to the repo.  Ta da!

The first three steps are the same described earlier (see Starting A New Branch). The ‘git push’ is what’s new, and it updates the repo on Savannah. Congratulations!

As a courtesy, you should send a note to the mailing list indicating that you have pushed your change.

5.4 Developing New Features

Developing a new feature can be easier once you have commit access to the repo. First, create a new branch to hold your feature:

$ git checkout master                     Start from master
$ git pull                                Be sure to be up to date
$ git checkout -b feature/python          Create and switch to a new branch

Now, you can develop as normal, adding new files if necessary (such as new tests), modifying code, updating the ChangeLog and documentation, and so on.

You can share changes with the mailing list as diffs, as usual. However, especially for a large feature, it would be better to push your branch up to Savannah. Then, everyone else can simply pull it down to their local systems and review your changes at their leisure.

To push your branch up initially:

$ git diff                                Review your changes
$ git add …                             Add any files for committing
$ git commit                              Commit the files with a commit message
$ git push -u origin feature/python       Push the branch up to the repo

When you use ‘push -u origin’, Git helpfully converts your purely local branch into a tracking branch. It becomes as if the branch had originated from the upstream repo and you checked it out locally.

You only need to do ‘git push -u origin’ once. As you continue to work on your branch, the workflow simplifies into this:

$ git diff                Review your changes
$ git add …             Add any files for committing
$ git commit              Commit the files
$ git push                Push your changes to the branch upstream

5.5 Developing Fixes

If you want to make a fix on master or on the current stable branch, you work the same way, by producing and discussing a diff on the mailing list. Once it’s approved, you can commit it yourself:

$ git checkout master     Move to master
$ git pull                Make sure we're up to date with the maintainer
$ gvim …                Make any fixes, compile, test
$ git diff                Review your changes
$ git add …             Add any files for committing
$ git commit              Commit the files with a commit message.

When you’re ready to push your changes:

$ git pull                Download latest version; Git will merge
$ gvim ...                Resolve any merge conflicts with git add and git commit
$ git push                Now you can push your changes upstream

See Dealing With Merge Conflicts for instructions on dealing with merge conflicts.

8.1 Coding Style

You should read the discussion about adding code in the gawk documentation. See the section Making Additions to gawk, in the online documentation for a discussion of the general procedure. In particular, pay attention to the coding style guidelines in the section Adding New Features, also in the online documentation.

8.2 Assigning Copyrights to the FSF

For any change of more than just a few lines, you will need to assign copyright in (that is, ownership of) those changes to the Free Software Foundation.

This is generally an easy thing to do. In particular, you can choose to use a version of the copyright assignment which assigns all your current and future changes to gawk to the FSF. This means that you only need to do the paperwork once, and from then on all your changes will automatically belong to the FSF. The maintainer recommends doing this.

The maintainer will help you with this process once you have a contribution that warrants it.

8.3 Software Tools You Will Need

This section discusses additional tools that you may need to install on your system in order to be in sync with what the gawk maintainer uses. It also discusses different C compiler options for use during code development, and how to compile gawk for debugging.

• GNU Tools
• Compilers

8.4 Compiling For Debugging

If you wish to compile for debugging, you should use GCC. After running configure but before running make, edit the Makefile and remove the -O2 flag from the definition of CFLAGS. Optionally, do the same for support/Makefile and/or extensions/Makefile. Then run make.

You can enable additional debugging code by creating a file named .developing in the gawk source code directory before running configure. Doing so enables additional conditionally-compiled debugging code within gawk, and adds additional warning and debugging options if compiling with GCC. It also disables optimization.