Re: [boost] Cannot add any file to GIT "libgit2 returned: Invalid data in index - invalid entry"

Subject: Re: [boost] Cannot add any file to GIT "libgit2 returned: Invalid data in index - invalid entry"
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-02-25 11:03:25

On 25 Feb 2015 at 14:09, Paul A. Bristow wrote:

> > > Cannot add any new file to GIT repo :-(
> > >
> > >  "libgit2 returned: Invalid data in index - invalid entry"
> > 
> > Yeah you've corrupted your index.
> 
> Gasp - that's not supposed to happen is it?

You'd be surprised at its frequency under certain use cases, even on 
Linux. 
> What have I done wrong?

Linus intended the index originally as a cache, so he didn't go out 
of his way to make it robust. I think that, with hindsight, was a 
mistake. Forthcoming TripleGit doesn't make that mistake 
incidentally. However the index is very much disposable, if 
inconvenient. Deleting it won't lose you much data, just what you 
have changed between now and the last commit. And it can always be 
regenerated from the last commit made to the backing object store.

> I'm only using Windows 8.1 64-bit and I have one copy of git (driven by a mixture of Tortoise GIT
> and command line).

Ah, but that install actually has three git instances! There is one 
git in msysgit, another in TortoiseGit, plus TortoiseGit keeps 
another copy of libgit inside itself.

If you install any mingw or cygwin, you get another git. If you 
install Visual Studio, yet another git comes into play.

Most tools just call 'git' and whichever on PATH or the current 
directory catches it first gets called. The fact it works at all 
without data loss is amazing.

> How can I tell if a git is good and what is corrupt?  My last commit didn't give any hints that all
> wasn't well.
> 
> Should I be doing any regular checking?

No, you just roll with the punches. As I mentioned, losing the index 
really isn't important. Your repo and everything you have committed 
is safe. You may just lose a bit of work since the last commit, 
that's all.

As I mentioned, occasionally deleting all traces of git and 
reinstalling on Windows is a very good idea. Syncs up the versions to 
a single version you see, and cleans up PATH so hopefully only one 
git is called.

> > Sometimes you can manually hand edit it back into life, but often it's easier just to zap it and
> recreate it.
> > Obviously doing that will reset your repo, including working directory, back to last good commit.
> > 
> > A safe alternative is to relocate the corrupted git checkout elsewhere, clone a new copy in its
> place, and
> > manually copy over the changes. No chance of losing anything important that way.
> 
> I have a backup and a zip of the files, but thousands of changes (to documentation).

None of these changes were committed? If they were, you have lost 
nothing. If you didn't, you'll need to manually copy over the changed 
files by hand (a bit of robocopy or rsync magic can do this for you 
into a new fresh git clone, have it copy if modified).

> > If the same corruption happens a second time, and if you are on Windows, I'd suggest wiping and
> > reinstalling your all your gits. A few months ago I had a persistent git index corruption problem
> on
> > Windows. git worked fine from mingw, but corrupted outside of mingw.
> > I did a total purge of all git, msysgit, tortoisegit and mingw from the system, and reinstalling
> everything
> > fresh. It fixed the problem, though it cost me a day of work. I think on Windows we get so many
> copies of
> > git that sometimes the versions get mismatched, and this corruption appears.
> 
> I'm not sure what you mean by 'copies of git' - do you so mean so many git repos, or ?
> 
> But thanks for the bad news anyway ;-)

It's not bad news, more a consequence of how git has been implemented 
on Windows. Due to the lack of a package manager, everything has to 
bundle a git. Sometimes one git doesn't like another git for some 
reason, and the index gets eaten.

Remember also that Windows has very different open handle semantics 
to POSIX. In particular, if one git tries to delete an index while 
another git is working with it, the deletion request returns an error 
on Windows. The error handling for that should be much better tested 
than it is, but in the end it is extremely unlikely for unlink() to 
ever fail on POSIX, so it isn't a debugging priority there. The 
Windows port shows the consequences of that.

Really what Windows needs is a ground up native reimplementation of 
git. All these problems go away then. Problem is who would fund such 
a thing?

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/