Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use http_archive with private Gitlab repo #16659

Closed
kilian-funk opened this issue Nov 4, 2022 · 10 comments
Closed

Use http_archive with private Gitlab repo #16659

kilian-funk opened this issue Nov 4, 2022 · 10 comments

Comments

@kilian-funk
Copy link

kilian-funk commented Nov 4, 2022

Description of the feature request:

I would like to use this in my WORKSPACE file:

http_archive(
    name = "my_repo",
    url = "https://gitlab.com/kilian-funk/test_project/-/archive/main/test_project-main.tar.gz",
    auth_patterns = {"gitlab.com": "PRIVATE-TOKEN: <password>"},
}

where test_project is a private repository.

What underlying problem are you trying to solve with this feature?

The tarball containing the repo can be downloaded on the command line with:

curl --header "PRIVATE-TOKEN: <token-goes-here>" <url>

Unfortunately the http_archive rule results in something equivalent to:

curl --header "Authentication: PRIVATE-TOKEN: <token-goes-here>" <url>

Therefore, http_archive is not compatible with gitlab.com for private repos, since it does not let the user customize the authentication header in the required way. http_archive is making too many assumptions on how an authentication header should look like.

Which operating system are you running Bazel on?

Linux, Ubuntu

What is the output of bazel info release?

release 5.3.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.


What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

---

Have you found anything relevant by searching the web?

no

Any other information, logs, or outputs that you want to share?

@Wyverald
Copy link
Member

Wyverald commented Nov 4, 2022

Sending a custom HTTP header is a highly uncommon use case. For GitLab specifically, you can use the standard Authentication header instead: https://docs.gitlab.com/ee/api/#personalprojectgroup-access-tokens

@Wyverald Wyverald closed this as not planned Won't fix, can't repro, duplicate, stale Nov 4, 2022
@thesayyn
Copy link
Contributor

@Wyverald I am hitting the same exact problem with the Accept header. docker hub registry expects specific media types to be sent via the HTTP request to use a newer registry feature.

Is there a way to do this under bazel? UrlRewriter etc didn't work.

@Wyverald
Copy link
Member

any links for context?

@thesayyn
Copy link
Contributor

https://github.com/opencontainers/distribution-spec/blob/main/spec.md#workflow-categories

Part that starts with The client SHOULD include an Accept header indicating which manifest content types it supports.

Which is currently can't be set; bazel automatically sets it to text/html, image/gif, image/jpeg, */*

@Wyverald
Copy link
Member

@thesayyn I'm much more sympathetic towards adding an "Accept" header than a completely random one called "PRIVATE-TOKEN". Could you file a separate issue about this? Thanks.

@thesayyn
Copy link
Contributor

@thesayyn I'm much more sympathetic towards adding an "Accept" header than a completely random one called "PRIVATE-TOKEN". Could you file a separate issue about this? Thanks.

filed #17829. I'll cc you

@sengelha
Copy link

@tjgq
Copy link
Contributor

tjgq commented Jun 23, 2023

FYI - you should be able to do this using a credential helper in Bazel 6.3.0 (soon to be released) or later.

@thesayyn
Copy link
Contributor

FYI - you should be able to do this using a credential helper in Bazel 6.3.0 (soon to be released) or later.

The problem with credential_helper is that I have to tell my ruleset users to either put a bash script into their workspace or download a binary from the internet for every CI/Dev machine just to add a header. it's not bazel-like

@CauhxMilloy
Copy link

CauhxMilloy commented Jan 11, 2024

I recently ran into this issue, but found out a solution. I'm responding here just to go ahead and document it (hopefully also helping anyone else who happens to stumble upon this in the future).

It is true that gitlab has historically used "PRIVATE-TOKEN: <access token>" as their header for their REST APIs / HTTP requests. However, as of 2019, the APIs now also accept the more common OAuth-style header, Authorization: Bearer <access token>. This can be seen here in their documentation, and it was added in this commit.

This can be shown working using curl -v --header "Authorization: Bearer $MY_ACCESS_TOKEN" "$MY_HTTP_REQUEST_URL".

This can be used with Bazel with:

http_archive(
    name = "my_external_repo",
    auth_patterns = {
        "gitlab.com": "Bearer <password>"
    },
    netrc = "/path/to/my/.netrc/if/not/wanting/to/use/the/one/in/my/home/dir/.netrc",
    sha256 = MY_EXT_REPO_SHA,
    url = MY_EXT_REPO_URL,
)

(one could also use the NETRC environment variable in lieu of using the netrc attribute.)

Note: the netrc file must be given in an absolute path. Relative paths are not supported; they resolve to the external/ directory in the cache. This is probably not where your .netrc file is -- nor anything else that you'd really want to reference (maybe should be addressed) 😇. The absolute path is required for both the attribute and the environment variable (they get piped into the same place).

Unfortunately, I believe none of this works with CI_JOB_TOKEN, only other access tokens.

Hope this helps!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants