Fetching Organization Private Repos

[betobrandao]

As discussed in Issue #76 the current version of GitRob is not able to fetch private repositories of organizations.

This patch adds a new function that uses the GitHub API method Repositories.ListByOrg to retrieve a list of all the repositories from an organization including the private ones if the access token given has access to them.

After that, the CloneURL is set to the SSHURL attribute of the GitHub object , allowing the cloning of the repositories over ssh. In the future it's possible to check if the repository is public before setting the CloneURL, if public keeps using the HTTTPS URL, otherwise uses the SSHURL instead.
The selection of what function to use is based on the target type.

The command go fmt was ran on the core/github.go file, that's why the diff is showing so many modifications.

Added function GetRepositoriesFromOrganization for downloading of
private repo when target is an Organization using SSH URL

diff --git a/core/github.go b/core/github.go
index a194170..a32552b 100644
--- a/core/github.go
+++ b/core/github.go
@@ -91,6 +91,44 @@ func GetRepositoriesFromOwner(login *string, client *github.Client) ([]*GithubRe
        return allRepos, nil
 }

+func GetRepositoriesFromOrganization(login *string, client *github.Client) ([]*GithubRepository, error) {
+       var allRepos []*GithubRepository
+       loginVal := *login
+       ctx := context.Background()
+       opt := &github.RepositoryListByOrgOptions{
+               Type: "sources",
+       }
+
+       for {
+               repos, resp, err := client.Repositories.ListByOrg(ctx, loginVal, opt)
+               if err != nil {
+                       return allRepos, err
+               }
+               for _, repo := range repos {
+                       if !*repo.Fork {
+                               r := GithubRepository{
+                                       Owner:         repo.Owner.Login,
+                                       ID:            repo.ID,
+                                       Name:          repo.Name,
+                                       FullName:      repo.FullName,
+                                       CloneURL:      repo.SSHURL,
+                                       URL:           repo.HTMLURL,
+                                       DefaultBranch: repo.DefaultBranch,
+                                       Description:   repo.Description,
+                                       Homepage:      repo.Homepage,
+                               }
+                               allRepos = append(allRepos, &r)
+                       }
+               }
+               if resp.NextPage == 0 {
+                       break
+               }
+               opt.Page = resp.NextPage
+       }
+
+       return allRepos, nil
+}
+
 func GetOrganizationMembers(login *string, client *github.Client) ([]*GithubOwner, error) {
        var allMembers []*GithubOwner
        loginVal := *login
diff --git a/main.go b/main.go
--- a/main.go
+++ b/main.go
@@ -57,12 +57,18 @@ func GatherRepositories(sess *core.Session) {
   for i := 0; i < threadNum; i++ {
     go func() {
       for {
+        var repos []*core.GithubRepository
+       var err error
         target, ok := <-ch
         if !ok {
           wg.Done()
           return
         }
-        repos, err := core.GetRepositoriesFromOwner(target.Login, sess.GithubClient)
+       if *target.Type == "Organization" {
+               repos, err = core.GetRepositoriesFromOrganization(target.Login, sess.GithubClient)
+       } else {
+               repos, err = core.GetRepositoriesFromOwner(target.Login, sess.GithubClient)
+       }
         if err != nil {
           sess.Out.Error(" Failed to retrieve repositories from %s: %s\n", *target.Login, err)
         }

[njfox]

Hey @betobrandao, I'm trying to build your fork of Gitrob with this patch for an engagement but I'm getting the following error:

$ go get github.com/betobrandao/gitrob
# github.com/betobrandao/gitrob
/home/nick/go/src/github.com/betobrandao/gitrob/main.go:68:16: undefined: core.GetRepositoriesFromOrganization

And trying to build manually:

$ go build -o gitrob
# _/home/nick/Documents/projects/gitrob
./main.go:68:16: undefined: core.GetRepositoriesFromOrganization

Am I doing anything wrong? I tried to fix it but I don't know Go so I wasn't able to get it to compile.

[tmatias]

@njfox that's because the import path in the source doesn't match the directory structure of your clone (/home/nick/go/src/github.com/betobrandao/gitrob, from the error message).

https://golang.org/doc/code.html fwiw, is a concise overview of the basic mechanics for working with go code

[tmatias]

why did you change the format? Wasn't it gofmt'd?

[njfox]

Thanks @tmatias, that worked. The patch also seems to have worked for fetching private organization repositories.

[rayterrill]

Appreciate this work @betobrandao! Forked this repo as well and took much of your code, made a few more changes to get it working over HTTPS vs SSH using the same access token already provided. Thank you!

[hdyrawan]

@rayterrill @njfox
having issue with ssh failed, any ideas? git clone using ssh works well

Error cloning repository xx/xx: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain`