Commit d916b11

Nick Faro committed on
Build incremental S3 Git publishing and static browser
commit d916b11b178d8be580b7d739038984c577cd6076
31 changed files +2886−0
Added.gitignore +4−0
@@ -0,0 +1,4 @@
1 .dumbforge/
2 dist/
3 coverage.out
4 npm/platforms/*/bin/
AddedLICENSE +21−0
@@ -0,0 +1,21 @@
1 MIT License
2
3 Copyright (c) 2026 Dumbforge contributors
4
5 Permission is hereby granted, free of charge, to any person obtaining a copy
6 of this software and associated documentation files (the "Software"), to deal
7 in the Software without restriction, including without limitation the rights
8 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 copies of the Software, and to permit persons to whom the Software is
10 furnished to do so, subject to the following conditions:
11
12 The above copyright notice and this permission notice shall be included in all
13 copies or substantial portions of the Software.
14
15 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 SOFTWARE.
AddedREADME.md +127−0
@@ -0,0 +1,127 @@
1 # dumbforge
2
3 `dumbforge` turns an S3-compatible bucket into a static Git host. Authors push
4 through a Git remote helper; everybody else clones with ordinary Git over dumb
5 HTTP. Each push also publishes a static repository browser at the same URL.
6
7 This is an early implementation. It has been exercised against Cloudflare R2,
8 including a full public clone and incremental multi-pack pushes.
9
10 ## Install
11
12 The planned primary distribution is npm:
13
14 ```sh
15 npm install --global dumbforge
16 ```
17
18 The npm package selects a native Go binary for the current OS and CPU from an
19 npm-hosted platform package. It does not run an installer that downloads a
20 GitHub release. Until the first npm release, build locally with Go 1.26 or
21 newer:
22
23 ```sh
24 go build -o dumbforge ./cmd/dumbforge
25 ln -s dumbforge git-remote-dumbforge
26 ```
27
28 Git itself must also be installed. The implementation deliberately asks Git to
29 create packfiles instead of reimplementing its storage format.
30
31 ## Configure an R2 remote
32
33 Create an R2 Object Read & Write token scoped to the target bucket and save it
34 in an AWS-compatible profile:
35
36 ```sh
37 aws configure --profile my-r2
38 ```
39
40 Use region `auto`, then add a remote from inside a local repository:
41
42 ```sh
43 dumbforge remote add origin \
44 --bucket my-bucket \
45 --prefix projects/example.git \
46 --endpoint https://ACCOUNT_ID.r2.cloudflarestorage.com \
47 --profile my-r2 \
48 --public-url https://pub-EXAMPLE.r2.dev
49 ```
50
51 This configures separate URLs:
52
53 - fetch: public HTTPS, usable by unmodified Git;
54 - push: `dumbforge://`, handled by `git-remote-dumbforge`.
55
56 Now push normally:
57
58 ```sh
59 git push --set-upstream origin main
60 ```
61
62 Anyone can clone without dumbforge:
63
64 ```sh
65 git clone https://pub-EXAMPLE.r2.dev/projects/example.git
66 ```
67
68 The repository browser is served from that exact URL. Git appends
69 `/info/refs`; browsers receive the HTML object stored at the repository prefix.
70
71 ## How a push works
72
73 1. Acquire an S3 conditional-write lease for the repository.
74 2. Read the currently advertised refs and pack list.
75 3. Validate fast-forward and concurrent-ref constraints.
76 4. Ask local Git for a non-thin pack containing objects reachable from the new
77 tips but not the known remote tips.
78 5. Upload the immutable `.pack` and `.idx`.
79 6. Publish `objects/info/packs`, individual refs, `HEAD`, and finally
80 `info/refs`.
81 7. Generate changed source/raw pages and affected ancestor tree pages.
82
83 Git data is never rebuilt on an ordinary push. Two small commits produce two
84 small packs. The mutable dumb-HTTP indexes are tiny and served with revalidation;
85 content-addressed packfiles are served as immutable objects.
86
87 ## Static browser
88
89 The generated browser includes:
90
91 - repository and directory trees;
92 - rendered GitHub-flavored Markdown README files;
93 - syntax highlighting and linked line numbers;
94 - raw file URLs and image previews;
95 - branch-aware paths;
96 - incremental regeneration based on `git diff --name-status`.
97
98 The UI is an original implementation using familiar source-hosting patterns;
99 it contains no copied GitHub HTML, CSS, logos, or assets.
100
101 ## Current constraints
102
103 - Shallow repositories are rejected because static dumb HTTP has no shallow
104 boundary negotiation.
105 - There is no pack compaction command yet. A future `dumbforge gc` will combine
106 accumulated incremental packs outside the normal push path.
107 - Git LFS is not implemented.
108 - Branch deletion removes Git refs, but full static-page cleanup for a deleted
109 branch is not implemented yet.
110 - The `r2.dev` hostname is intended for development; use an R2 custom domain for
111 production traffic and explicit cache rules.
112
113 ## Development
114
115 ```sh
116 go test ./...
117 go build ./cmd/dumbforge
118 ```
119
120 Build all npm platform packages with:
121
122 ```sh
123 ./scripts/build-npm-packages.sh 0.1.0
124 ```
125
126 The resulting tarballs are written to `dist/npm`. Publish the six platform
127 packages first and the `dumbforge` launcher package last.
Addedcmd/dumbforge/main.go +216−0
@@ -0,0 +1,216 @@
1 package main
2
3 import (
4 "context"
5 "flag"
6 "fmt"
7 "os"
8 "os/exec"
9 "path"
10 "path/filepath"
11 "strings"
12
13 "github.com/npfaro/dumbforge/internal/gitutil"
14 "github.com/npfaro/dumbforge/internal/helper"
15 "github.com/npfaro/dumbforge/internal/publish"
16 "github.com/npfaro/dumbforge/internal/remoteurl"
17 "github.com/npfaro/dumbforge/internal/s3store"
18 "github.com/npfaro/dumbforge/internal/site"
19 )
20
21 var version = "0.1.0-dev"
22
23 func main() {
24 ctx := context.Background()
25 base := strings.TrimSuffix(filepath.Base(os.Args[0]), ".exe")
26 if base == "git-remote-dumbforge" {
27 if len(os.Args) < 3 {
28 die("git remote helper requires a remote name and URL")
29 }
30 if err := helper.Run(ctx, os.Args[2], os.Stdin, os.Stdout, os.Stderr); err != nil {
31 die("%v", err)
32 }
33 return
34 }
35
36 if len(os.Args) < 2 {
37 usage()
38 os.Exit(2)
39 }
40 switch os.Args[1] {
41 case "version", "--version", "-v":
42 fmt.Printf("dumbforge %s\n", version)
43 case "remote":
44 if err := remoteCommand(os.Args[2:]); err != nil {
45 die("%v", err)
46 }
47 case "site":
48 if err := siteCommand(ctx, os.Args[2:]); err != nil {
49 die("%v", err)
50 }
51 case "__remote-helper":
52 if len(os.Args) < 4 {
53 die("internal remote helper requires a remote name and URL")
54 }
55 if err := helper.Run(ctx, os.Args[3], os.Stdin, os.Stdout, os.Stderr); err != nil {
56 die("%v", err)
57 }
58 default:
59 usage()
60 os.Exit(2)
61 }
62 }
63
64 func siteCommand(ctx context.Context, args []string) (err error) {
65 if len(args) == 0 || args[0] != "publish" {
66 return fmt.Errorf("usage: dumbforge site publish [--remote <name>] [--branch <name>]")
67 }
68 flags := flag.NewFlagSet("site publish", flag.ContinueOnError)
69 remoteName := flags.String("remote", "origin", "configured Git remote")
70 branch := flags.String("branch", "", "branch name; defaults to the current branch")
71 if err := flags.Parse(args[1:]); err != nil {
72 return err
73 }
74 if *branch == "" {
75 out, err := gitOutput("branch", "--show-current")
76 if err != nil {
77 return err
78 }
79 *branch = strings.TrimSpace(string(out))
80 }
81 if *branch == "" {
82 return fmt.Errorf("could not determine a branch; pass --branch")
83 }
84 pushURLBytes, err := gitOutput("config", "--get", "remote."+*remoteName+".pushurl")
85 if err != nil {
86 return fmt.Errorf("remote %s has no dumbforge push URL", *remoteName)
87 }
88 spec, err := remoteurl.Parse(strings.TrimSpace(string(pushURLBytes)))
89 if err != nil {
90 return err
91 }
92 store, err := s3store.New(ctx, spec)
93 if err != nil {
94 return err
95 }
96 lock, err := store.AcquireLock(ctx)
97 if err != nil {
98 return err
99 }
100 defer func() {
101 if releaseErr := lock.Release(context.WithoutCancel(ctx)); err == nil && releaseErr != nil {
102 err = releaseErr
103 }
104 }()
105 git := gitutil.Git{}
106 oid, err := git.RevParse(ctx, "refs/heads/"+*branch)
107 if err != nil {
108 oid, err = git.RevParse(ctx, "HEAD")
109 if err != nil {
110 return err
111 }
112 }
113 metadata, err := publish.LoadMetadata(ctx, store)
114 if err != nil {
115 return err
116 }
117 owner := spec.Bucket
118 prefixDir := path.Dir(spec.Prefix)
119 if prefixDir != "." && prefixDir != "/" {
120 owner = path.Base(prefixDir)
121 }
122 generator, err := site.New(git, spec.PublicRepoURL(), path.Base(spec.Prefix), owner)
123 if err != nil {
124 return err
125 }
126 generator.Progress = func(format string, args ...any) {
127 fmt.Printf("dumbforge: "+format+"\n", args...)
128 }
129 build, err := generator.BuildBranch(ctx, *branch, "", oid, metadata.Head == "refs/heads/"+*branch)
130 if err != nil {
131 return err
132 }
133 fmt.Printf("dumbforge: generated %d site object(s)\n", len(build.Pages))
134 if err := lock.Refresh(ctx); err != nil {
135 return err
136 }
137 return generator.Publish(ctx, store, build)
138 }
139
140 func remoteCommand(args []string) error {
141 if len(args) < 2 || args[0] != "add" {
142 return fmt.Errorf("usage: dumbforge remote add <name> [options]")
143 }
144 name := args[1]
145 flags := flag.NewFlagSet("remote add", flag.ContinueOnError)
146 bucket := flags.String("bucket", "", "S3 bucket")
147 prefix := flags.String("prefix", "", "repository object prefix")
148 endpoint := flags.String("endpoint", "", "S3-compatible API endpoint")
149 profile := flags.String("profile", "", "AWS shared-configuration profile")
150 publicURL := flags.String("public-url", "", "public bucket base URL")
151 if err := flags.Parse(args[2:]); err != nil {
152 return err
153 }
154 if *prefix == "" {
155 *prefix = name + ".git"
156 }
157 spec := remoteurl.Spec{
158 Bucket: *bucket,
159 Prefix: *prefix,
160 Endpoint: *endpoint,
161 Profile: *profile,
162 PublicURL: *publicURL,
163 }
164 pushURL, err := remoteurl.Build(spec)
165 if err != nil {
166 return err
167 }
168 fetchURL := spec.PublicRepoURL()
169 if fetchURL == "" {
170 return fmt.Errorf("public URL is required")
171 }
172 if err := runGit("remote", "add", name, fetchURL); err != nil {
173 return err
174 }
175 if err := runGit("config", "--add", "remote."+name+".pushurl", pushURL); err != nil {
176 return err
177 }
178 fmt.Printf("added %s\n fetch: %s\n push: dumbforge://%s/%s\n", name, fetchURL, spec.Bucket, spec.Prefix)
179 return nil
180 }
181
182 func runGit(args ...string) error {
183 cmd := exec.Command("git", args...)
184 cmd.Stdout = os.Stdout
185 cmd.Stderr = os.Stderr
186 if err := cmd.Run(); err != nil {
187 return fmt.Errorf("git %s failed: %w", strings.Join(args, " "), err)
188 }
189 return nil
190 }
191
192 func gitOutput(args ...string) ([]byte, error) {
193 cmd := exec.Command("git", args...)
194 var stderr strings.Builder
195 cmd.Stderr = &stderr
196 out, err := cmd.Output()
197 if err != nil {
198 return nil, fmt.Errorf("git %s: %s", strings.Join(args, " "), strings.TrimSpace(stderr.String()))
199 }
200 return out, nil
201 }
202
203 func usage() {
204 fmt.Fprintln(os.Stderr, `dumbforge stores Git repositories in S3-compatible object storage.
205
206 Usage:
207 dumbforge remote add <name> --bucket <bucket> --prefix <repo.git> \
208 --endpoint <s3-url> --profile <aws-profile> --public-url <https-url>
209 dumbforge site publish --remote <name> [--branch <name>]
210 dumbforge version`)
211 }
212
213 func die(format string, args ...any) {
214 fmt.Fprintf(os.Stderr, "dumbforge: "+format+"\n", args...)
215 os.Exit(1)
216 }
Addedgo.mod +27−0
@@ -0,0 +1,27 @@
1 module github.com/npfaro/dumbforge
2
3 go 1.26
4
5 require (
6 github.com/alecthomas/chroma/v2 v2.27.0 // indirect
7 github.com/aws/aws-sdk-go-v2 v1.43.0 // indirect
8 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 // indirect
9 github.com/aws/aws-sdk-go-v2/config v1.32.31 // indirect
10 github.com/aws/aws-sdk-go-v2/credentials v1.19.30 // indirect
11 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 // indirect
12 github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 // indirect
13 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 // indirect
14 github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 // indirect
15 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 // indirect
16 github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 // indirect
17 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 // indirect
18 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 // indirect
19 github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0 // indirect
20 github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 // indirect
21 github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 // indirect
22 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 // indirect
23 github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 // indirect
24 github.com/aws/smithy-go v1.27.4 // indirect
25 github.com/dlclark/regexp2/v2 v2.2.1 // indirect
26 github.com/yuin/goldmark v1.8.4 // indirect
27 )
Addedgo.sum +42−0
@@ -0,0 +1,42 @@
1 github.com/alecthomas/chroma/v2 v2.27.0 h1:FodwmyOBgJULFYmDqibcp9pvfDLWdtPRh9v/r5BXYZs=
2 github.com/alecthomas/chroma/v2 v2.27.0/go.mod h1:NjJ3ciIgrqBNeIkWZ4e46nseoLDslxU1LmfCoL+wcY8=
3 github.com/aws/aws-sdk-go-v2 v1.43.0 h1:fharf/WhbRAVZ1du0QL7roNFxZ6T/sWr+4Ni617bwSI=
4 github.com/aws/aws-sdk-go-v2 v1.43.0/go.mod h1:5pKeft2eJj+gElQ38Jqg4ibCqh+/AK33/0X3hip7IjM=
5 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 h1:3IZY0XAJquT3aHzbkHfPzy4ACPcEjVG0x87KOwtpqGY=
6 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14/go.mod h1:zwM6veDkhGgQFqkBy+uT28AAYpLu+uFMlPl+rCg/73E=
7 github.com/aws/aws-sdk-go-v2/config v1.32.31 h1:n4nY9O3QKoHIkL85EX+V8RcMFtOhlpTFhGArg915PXk=
8 github.com/aws/aws-sdk-go-v2/config v1.32.31/go.mod h1:PN0NYDCCoOpGGsZ2+elDUidmHfQBPyYzN2GCgl8HEBs=
9 github.com/aws/aws-sdk-go-v2/credentials v1.19.30 h1:TTCvvzFU6gXa4iJecNG/0F/B0oYTiazoRECr2XyLHrY=
10 github.com/aws/aws-sdk-go-v2/credentials v1.19.30/go.mod h1:jKxAp2AEncnliinzpgOSZDFv6+VjvWhjw/AtbfsWT9U=
11 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 h1:kfVL5wAunCJycL6MOQ6aNh6PlAYEymflcjuKmrWUA0o=
12 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31/go.mod h1:nWfRNDAppujCQgOUd43lKT4yeLv9z3nJ3bw1G3BgQKo=
13 github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 h1:Z8F3hfCY33IGpJjFAnv0wvtv1FIKj1GHmRDEYqy64tw=
14 github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31/go.mod h1:aVyUoytEyOViR6jhq6jula0xkc5NfBE2hgeF6BvOrao=
15 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 h1:hyOxUyXdh3AyjE93gBgsfziJag9ACwcs+ZpDBLzi8mw=
16 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31/go.mod h1:OERqI9k0draSLB8O8woxY3q25ZWTELRK4RRoLMuMZFo=
17 github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 h1:0MrUL35H/Y4kdFfItoR5jCgtDQ4Z/8LudAoIHRfA4hE=
18 github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32/go.mod h1:2tNZkuWz54arj8mHVf+8Y7cKkcD8Wr/fBpENgEXpjLc=
19 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 h1:mbRIur/BiHK6SKPjoBIXSE/hJ6g6JGRLuxQy1jGjlN4=
20 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13/go.mod h1:ITg9em2KbJx1s0y4aqRX5OYWG6HBZ5TVR//OdpEZ2CQ=
21 github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 h1:mdPwDQPqxlw9Sc62Nt15yjEcARaDbPXkjRYtXsUripo=
22 github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24/go.mod h1:ls5ytnwLTcQaUu32fMYXFI3MjpKuTwL840PAm9iqyEg=
23 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 h1:w2SIhW92DZPFrSL4ksVCr8IYff5OZwIcxg8+95tzvAI=
24 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31/go.mod h1:wAhpCQbkov+IcvjozJbd2xRCoZybUEHNkcFunssNACg=
25 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 h1:jWXtZdCnhXa9sGFixRaU2AxT4DIVse9HS4E2f+/KwV0=
26 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32/go.mod h1:9JS1UpfVvyD/ZPX8GsKb/Pq8scEM+7GP5fqh9SwH7po=
27 github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0 h1:7QZWVJZWzHivHWIa+5TELLaBBkbuoj0GPwQtMlJ0sqk=
28 github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0/go.mod h1:fcvq5L7dK+5cQFicEJwpI6e6Wn8NY2i6yT5wRLYVc7s=
29 github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 h1:OHH5iTQvVGmfHjX/5Q+vFuA/Rf2x6/95aJ/75QCQSm4=
30 github.com/aws/aws-sdk-go-v2/service/signin v1.5.0/go.mod h1:mCF3AK9PpL49oOrhniUXWAfhVBVQ/XbytoE5eccZUIs=
31 github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 h1:CaJyYhxBE0M/HJX/YvSaSmQlsI91VHB0lKU8LtLxL3A=
32 github.com/aws/aws-sdk-go-v2/service/sso v1.33.0/go.mod h1:+e6BMRMPjBQoCw/WovYR9GLy2IU0z4Q77smOB1DraSg=
33 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 h1:tC323YV77QdafeBr6LUhLDTsboyuyHLNRwAyCP44kGU=
34 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0/go.mod h1:SfLK1sgviHmbI+MozR9iDwDjL4cdCVZtahsjoR+z7wg=
35 github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 h1:Pd6PNlp4t8PTXxqzstICl52Wsy78vpjFZ7PRUj44mJc=
36 github.com/aws/aws-sdk-go-v2/service/sts v1.45.0/go.mod h1:rmQ0TnHzuLPmabgjPcsywhsSOmaBDgzR4zvDxSPsGdg=
37 github.com/aws/smithy-go v1.27.4 h1:JQcphmBN4f0q/sPqXqROIItRNV/hy10cgu7CsFy616M=
38 github.com/aws/smithy-go v1.27.4/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
39 github.com/dlclark/regexp2/v2 v2.2.1 h1:mf4KkFUj0gJuarK8P+LgiS+Lit7m9N1yAwEfPbee7R0=
40 github.com/dlclark/regexp2/v2 v2.2.1/go.mod h1:avUrQvPaLz2DrFNHJF0taWAFFX2C1GMSSoeiqFjcBmU=
41 github.com/yuin/goldmark v1.8.4 h1:oat/nd3U6NeQqFEL3xpEJq7d7c86NI+DbSNGAs4xnjA=
42 github.com/yuin/goldmark v1.8.4/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
Addedinternal/gitutil/git.go +232−0
@@ -0,0 +1,232 @@
1 package gitutil
2
3 import (
4 "bufio"
5 "bytes"
6 "context"
7 "encoding/binary"
8 "errors"
9 "fmt"
10 "os"
11 "os/exec"
12 "path/filepath"
13 "sort"
14 "strings"
15 )
16
17 type Git struct {
18 Dir string
19 }
20
21 func (g Git) command(ctx context.Context, args ...string) *exec.Cmd {
22 if g.Dir != "" {
23 args = append([]string{"-C", g.Dir}, args...)
24 }
25 return exec.CommandContext(ctx, "git", args...)
26 }
27
28 func (g Git) Run(ctx context.Context, args ...string) ([]byte, error) {
29 cmd := g.command(ctx, args...)
30 var stderr bytes.Buffer
31 cmd.Stderr = &stderr
32 out, err := cmd.Output()
33 if err != nil {
34 message := strings.TrimSpace(stderr.String())
35 if message == "" {
36 message = err.Error()
37 }
38 return nil, fmt.Errorf("git %s: %s", strings.Join(args, " "), message)
39 }
40 return out, nil
41 }
42
43 func (g Git) RevParse(ctx context.Context, rev string) (string, error) {
44 out, err := g.Run(ctx, "rev-parse", "--verify", rev)
45 if err != nil {
46 return "", err
47 }
48 return strings.TrimSpace(string(out)), nil
49 }
50
51 func (g Git) Peel(ctx context.Context, oid string) (string, error) {
52 out, err := g.Run(ctx, "rev-parse", "--verify", oid+"^{}")
53 if err != nil {
54 return "", err
55 }
56 return strings.TrimSpace(string(out)), nil
57 }
58
59 func (g Git) ObjectExists(ctx context.Context, oid string) bool {
60 cmd := g.command(ctx, "cat-file", "-e", oid+"^{object}")
61 return cmd.Run() == nil
62 }
63
64 func (g Git) IsShallow(ctx context.Context) (bool, error) {
65 out, err := g.Run(ctx, "rev-parse", "--is-shallow-repository")
66 if err != nil {
67 return false, err
68 }
69 return strings.TrimSpace(string(out)) == "true", nil
70 }
71
72 func (g Git) IsAncestor(ctx context.Context, older, newer string) (bool, error) {
73 cmd := g.command(ctx, "merge-base", "--is-ancestor", older, newer)
74 err := cmd.Run()
75 if err == nil {
76 return true, nil
77 }
78 var exitErr *exec.ExitError
79 if errors.As(err, &exitErr) && exitErr.ExitCode() == 1 {
80 return false, nil
81 }
82 return false, fmt.Errorf("check whether %s is an ancestor of %s: %w", older, newer, err)
83 }
84
85 func (g Git) CheckRef(ctx context.Context, ref string) error {
86 cmd := g.command(ctx, "check-ref-format", ref)
87 if err := cmd.Run(); err != nil {
88 return fmt.Errorf("invalid destination ref %q", ref)
89 }
90 return nil
91 }
92
93 type Pack struct {
94 Hash string
95 PackPath string
96 IndexPath string
97 Objects uint32
98 cleanup func()
99 }
100
101 func (p *Pack) Close() {
102 if p != nil && p.cleanup != nil {
103 p.cleanup()
104 }
105 }
106
107 func (g Git) PackObjects(ctx context.Context, wants, possibleHaves []string) (*Pack, error) {
108 wants = unique(wants)
109 if len(wants) == 0 {
110 return nil, nil
111 }
112 haves := make([]string, 0, len(possibleHaves))
113 for _, oid := range unique(possibleHaves) {
114 if g.ObjectExists(ctx, oid) {
115 haves = append(haves, oid)
116 }
117 }
118
119 tmpDir, err := os.MkdirTemp("", "dumbforge-pack-*")
120 if err != nil {
121 return nil, err
122 }
123 cleanup := func() { _ = os.RemoveAll(tmpDir) }
124 base := filepath.Join(tmpDir, "pack")
125 cmd := g.command(ctx, "pack-objects", "--revs", "--delta-base-offset", base)
126 var input strings.Builder
127 for _, oid := range wants {
128 fmt.Fprintln(&input, oid)
129 }
130 for _, oid := range haves {
131 fmt.Fprintf(&input, "^%s\n", oid)
132 }
133 cmd.Stdin = strings.NewReader(input.String())
134 var stderr bytes.Buffer
135 cmd.Stderr = &stderr
136 out, err := cmd.Output()
137 if err != nil {
138 cleanup()
139 return nil, fmt.Errorf("create incremental pack: %s", strings.TrimSpace(stderr.String()))
140 }
141 hash := strings.TrimSpace(string(out))
142 packPath := base + "-" + hash + ".pack"
143 indexPath := base + "-" + hash + ".idx"
144 count, err := packObjectCount(packPath)
145 if err != nil {
146 cleanup()
147 return nil, err
148 }
149 return &Pack{
150 Hash: hash,
151 PackPath: packPath,
152 IndexPath: indexPath,
153 Objects: count,
154 cleanup: cleanup,
155 }, nil
156 }
157
158 func packObjectCount(filename string) (uint32, error) {
159 f, err := os.Open(filename)
160 if err != nil {
161 return 0, err
162 }
163 defer f.Close()
164 header := make([]byte, 12)
165 if _, err := f.Read(header); err != nil {
166 return 0, fmt.Errorf("read pack header: %w", err)
167 }
168 if string(header[:4]) != "PACK" {
169 return 0, fmt.Errorf("invalid pack header")
170 }
171 return binary.BigEndian.Uint32(header[8:12]), nil
172 }
173
174 func unique(values []string) []string {
175 seen := make(map[string]struct{}, len(values))
176 result := make([]string, 0, len(values))
177 for _, value := range values {
178 if value == "" {
179 continue
180 }
181 if _, ok := seen[value]; ok {
182 continue
183 }
184 seen[value] = struct{}{}
185 result = append(result, value)
186 }
187 sort.Strings(result)
188 return result
189 }
190
191 type NameStatus struct {
192 Status string
193 Path string
194 OldPath string
195 }
196
197 func (g Git) DiffNameStatus(ctx context.Context, oldOID, newOID string) ([]NameStatus, error) {
198 out, err := g.Run(ctx, "diff", "--name-status", "-z", "--find-renames", oldOID, newOID)
199 if err != nil {
200 return nil, err
201 }
202 scanner := bufio.NewScanner(bytes.NewReader(out))
203 scanner.Split(splitNUL)
204 var result []NameStatus
205 for scanner.Scan() {
206 status := scanner.Text()
207 if !scanner.Scan() {
208 return nil, fmt.Errorf("malformed git diff output")
209 }
210 first := scanner.Text()
211 entry := NameStatus{Status: status, Path: first}
212 if strings.HasPrefix(status, "R") || strings.HasPrefix(status, "C") {
213 if !scanner.Scan() {
214 return nil, fmt.Errorf("malformed git rename output")
215 }
216 entry.OldPath = first
217 entry.Path = scanner.Text()
218 }
219 result = append(result, entry)
220 }
221 return result, scanner.Err()
222 }
223
224 func splitNUL(data []byte, atEOF bool) (advance int, token []byte, err error) {
225 if i := bytes.IndexByte(data, 0); i >= 0 {
226 return i + 1, data[:i], nil
227 }
228 if atEOF && len(data) > 0 {
229 return len(data), data, nil
230 }
231 return 0, nil, nil
232 }
Addedinternal/gitutil/git_test.go +74−0
@@ -0,0 +1,74 @@
1 package gitutil
2
3 import (
4 "context"
5 "os"
6 "os/exec"
7 "path/filepath"
8 "testing"
9 )
10
11 func TestPackObjectsIsIncremental(t *testing.T) {
12 dir := initTestRepository(t)
13 git := Git{Dir: dir}
14 ctx := context.Background()
15 first := commitFile(t, dir, "example.txt", "first\n", "first")
16 firstPack, err := git.PackObjects(ctx, []string{first}, nil)
17 if err != nil {
18 t.Fatal(err)
19 }
20 defer firstPack.Close()
21 if firstPack.Objects != 3 {
22 t.Fatalf("first pack has %d objects, want 3", firstPack.Objects)
23 }
24 if _, err := os.Stat(firstPack.IndexPath); err != nil {
25 t.Fatalf("pack index was not created: %v", err)
26 }
27
28 second := commitFile(t, dir, "example.txt", "second\n", "second")
29 secondPack, err := git.PackObjects(ctx, []string{second}, []string{first})
30 if err != nil {
31 t.Fatal(err)
32 }
33 defer secondPack.Close()
34 if secondPack.Objects != 3 {
35 t.Fatalf("incremental pack has %d objects, want 3", secondPack.Objects)
36 }
37 }
38
39 func initTestRepository(t *testing.T) string {
40 t.Helper()
41 dir := t.TempDir()
42 runTestGit(t, dir, "init", "-b", "main")
43 runTestGit(t, dir, "config", "user.name", "Dumbforge Test")
44 runTestGit(t, dir, "config", "user.email", "[email protected]")
45 return dir
46 }
47
48 func commitFile(t *testing.T, dir, name, contents, message string) string {
49 t.Helper()
50 filename := filepath.Join(dir, name)
51 if err := os.MkdirAll(filepath.Dir(filename), 0o755); err != nil {
52 t.Fatal(err)
53 }
54 if err := os.WriteFile(filename, []byte(contents), 0o644); err != nil {
55 t.Fatal(err)
56 }
57 runTestGit(t, dir, "add", name)
58 runTestGit(t, dir, "commit", "-m", message)
59 cmd := exec.Command("git", "-C", dir, "rev-parse", "HEAD")
60 out, err := cmd.Output()
61 if err != nil {
62 t.Fatal(err)
63 }
64 return string(out[:len(out)-1])
65 }
66
67 func runTestGit(t *testing.T, dir string, args ...string) {
68 t.Helper()
69 args = append([]string{"-C", dir}, args...)
70 cmd := exec.Command("git", args...)
71 if out, err := cmd.CombinedOutput(); err != nil {
72 t.Fatalf("git %v: %v\n%s", args, err, out)
73 }
74 }
Addedinternal/helper/helper.go +180−0
@@ -0,0 +1,180 @@
1 package helper
2
3 import (
4 "bufio"
5 "context"
6 "fmt"
7 "io"
8 "path"
9 "strings"
10
11 "github.com/npfaro/dumbforge/internal/gitutil"
12 "github.com/npfaro/dumbforge/internal/publish"
13 "github.com/npfaro/dumbforge/internal/remoteurl"
14 "github.com/npfaro/dumbforge/internal/s3store"
15 "github.com/npfaro/dumbforge/internal/site"
16 )
17
18 func Run(ctx context.Context, rawURL string, stdin io.Reader, stdout, stderr io.Writer) error {
19 spec, err := remoteurl.Parse(rawURL)
20 if err != nil {
21 return err
22 }
23 store, err := s3store.New(ctx, spec)
24 if err != nil {
25 return err
26 }
27 publisher := &publish.Publisher{
28 Store: store,
29 Git: gitutil.Git{},
30 Progress: func(format string, args ...any) {
31 fmt.Fprintf(stderr, "dumbforge: "+format+"\n", args...)
32 },
33 }
34 publisher.AfterPublish = func(ctx context.Context, result publish.Result) {
35 publishSite(ctx, publisher, result, stderr)
36 }
37
38 scanner := bufio.NewScanner(stdin)
39 writer := bufio.NewWriter(stdout)
40 var advertised map[string]string
41 for scanner.Scan() {
42 line := scanner.Text()
43 switch {
44 case line == "capabilities":
45 fmt.Fprintln(writer, "push")
46 fmt.Fprintln(writer, "option")
47 fmt.Fprintln(writer)
48 if err := writer.Flush(); err != nil {
49 return err
50 }
51 case line == "list" || line == "list for-push":
52 metadata, err := publisher.List(ctx)
53 if err != nil {
54 return err
55 }
56 advertised = make(map[string]string, len(metadata.Refs))
57 for name, ref := range metadata.Refs {
58 advertised[name] = ref.OID
59 }
60 if metadata.Head != "" {
61 fmt.Fprintf(writer, "@%s HEAD\n", metadata.Head)
62 }
63 for _, name := range sortedRefNames(metadata.Refs) {
64 fmt.Fprintf(writer, "%s %s\n", metadata.Refs[name].OID, name)
65 }
66 fmt.Fprintln(writer)
67 if err := writer.Flush(); err != nil {
68 return err
69 }
70 case strings.HasPrefix(line, "option "):
71 name := strings.Fields(line)
72 if len(name) >= 2 && (name[1] == "verbosity" || name[1] == "progress") {
73 fmt.Fprintln(writer, "ok")
74 } else {
75 fmt.Fprintln(writer, "unsupported")
76 }
77 if err := writer.Flush(); err != nil {
78 return err
79 }
80 case strings.HasPrefix(line, "push "):
81 lines := []string{strings.TrimPrefix(line, "push ")}
82 for scanner.Scan() {
83 next := scanner.Text()
84 if next == "" {
85 break
86 }
87 if !strings.HasPrefix(next, "push ") {
88 return fmt.Errorf("unexpected command in push batch %q", next)
89 }
90 lines = append(lines, strings.TrimPrefix(next, "push "))
91 }
92 updates, err := parsePushes(lines)
93 if err == nil {
94 _, err = publisher.Push(ctx, updates, advertised)
95 }
96 if err != nil {
97 message := strings.NewReplacer("\n", " ", "\r", " ").Replace(err.Error())
98 for _, update := range updates {
99 fmt.Fprintf(writer, "error %s %s\n", update.Dst, message)
100 }
101 if len(updates) == 0 {
102 return err
103 }
104 } else {
105 for _, update := range updates {
106 fmt.Fprintf(writer, "ok %s\n", update.Dst)
107 }
108 }
109 fmt.Fprintln(writer)
110 if err := writer.Flush(); err != nil {
111 return err
112 }
113 case line == "":
114 // Empty lines terminate command batches and need no response here.
115 default:
116 return fmt.Errorf("unsupported remote-helper command %q", line)
117 }
118 }
119 return scanner.Err()
120 }
121
122 func publishSite(ctx context.Context, publisher *publish.Publisher, result publish.Result, stderr io.Writer) {
123 spec := publisher.Store.Spec()
124 if spec.PublicRepoURL() == "" {
125 return
126 }
127 owner := spec.Bucket
128 prefixDir := path.Dir(spec.Prefix)
129 if prefixDir != "." && prefixDir != "/" {
130 owner = path.Base(prefixDir)
131 }
132 generator, err := site.New(publisher.Git, spec.PublicRepoURL(), path.Base(spec.Prefix), owner)
133 if err != nil {
134 fmt.Fprintf(stderr, "dumbforge: warning: generate site: %v\n", err)
135 return
136 }
137 generator.Progress = func(format string, args ...any) {
138 fmt.Fprintf(stderr, "dumbforge: "+format+"\n", args...)
139 }
140 for _, update := range result.Updates {
141 if !strings.HasPrefix(update.Dst, "refs/heads/") || update.NewOID == "" {
142 continue
143 }
144 branch := strings.TrimPrefix(update.Dst, "refs/heads/")
145 build, err := generator.BuildBranch(ctx, branch, update.OldOID, update.NewOID, update.Dst == result.Head)
146 if err == nil {
147 err = generator.Publish(ctx, publisher.Store, build)
148 }
149 if err != nil {
150 fmt.Fprintf(stderr, "dumbforge: warning: site publication failed: %v\n", err)
151 }
152 }
153 }
154
155 func parsePushes(specs []string) ([]publish.Update, error) {
156 updates := make([]publish.Update, 0, len(specs))
157 for _, spec := range specs {
158 force := strings.HasPrefix(spec, "+")
159 spec = strings.TrimPrefix(spec, "+")
160 parts := strings.SplitN(spec, ":", 2)
161 if len(parts) != 2 || parts[1] == "" {
162 return nil, fmt.Errorf("invalid push refspec %q", spec)
163 }
164 updates = append(updates, publish.Update{Src: parts[0], Dst: parts[1], Force: force})
165 }
166 return updates, nil
167 }
168
169 func sortedRefNames(refs map[string]publish.Ref) []string {
170 result := make([]string, 0, len(refs))
171 for name := range refs {
172 result = append(result, name)
173 }
174 for i := 1; i < len(result); i++ {
175 for j := i; j > 0 && result[j] < result[j-1]; j-- {
176 result[j], result[j-1] = result[j-1], result[j]
177 }
178 }
179 return result
180 }
Addedinternal/helper/helper_test.go +28−0
@@ -0,0 +1,28 @@
1 package helper
2
3 import "testing"
4
5 func TestParsePushes(t *testing.T) {
6 updates, err := parsePushes([]string{
7 "+refs/heads/local:refs/heads/main",
8 ":refs/heads/old",
9 })
10 if err != nil {
11 t.Fatal(err)
12 }
13 if len(updates) != 2 {
14 t.Fatalf("got %d updates", len(updates))
15 }
16 if !updates[0].Force || updates[0].Src != "refs/heads/local" || updates[0].Dst != "refs/heads/main" {
17 t.Fatalf("unexpected forced update %#v", updates[0])
18 }
19 if updates[1].Src != "" || updates[1].Dst != "refs/heads/old" {
20 t.Fatalf("unexpected deletion %#v", updates[1])
21 }
22 }
23
24 func TestParsePushesRejectsMissingDestination(t *testing.T) {
25 if _, err := parsePushes([]string{"refs/heads/main"}); err == nil {
26 t.Fatal("expected an error")
27 }
28 }
Addedinternal/publish/metadata.go +120−0
@@ -0,0 +1,120 @@
1 package publish
2
3 import (
4 "bufio"
5 "bytes"
6 "context"
7 "fmt"
8 "sort"
9 "strings"
10
11 "github.com/npfaro/dumbforge/internal/s3store"
12 )
13
14 type Ref struct {
15 OID string
16 Peeled string
17 }
18
19 type Metadata struct {
20 Head string
21 Refs map[string]Ref
22 Packs map[string]struct{}
23 }
24
25 func LoadMetadata(ctx context.Context, store *s3store.Store) (Metadata, error) {
26 metadata := Metadata{
27 Refs: map[string]Ref{},
28 Packs: map[string]struct{}{},
29 }
30 refsObject, err := store.Get(ctx, "info/refs")
31 if err != nil && !s3store.IsNotFound(err) {
32 return Metadata{}, fmt.Errorf("read remote refs: %w", err)
33 }
34 if err == nil {
35 if err := parseInfoRefs(refsObject.Data, metadata.Refs); err != nil {
36 return Metadata{}, err
37 }
38 }
39 headObject, err := store.Get(ctx, "HEAD")
40 if err != nil && !s3store.IsNotFound(err) {
41 return Metadata{}, fmt.Errorf("read remote HEAD: %w", err)
42 }
43 if err == nil {
44 line := strings.TrimSpace(string(headObject.Data))
45 if strings.HasPrefix(line, "ref: ") {
46 metadata.Head = strings.TrimSpace(strings.TrimPrefix(line, "ref: "))
47 }
48 }
49 packObject, err := store.Get(ctx, "objects/info/packs")
50 if err != nil && !s3store.IsNotFound(err) {
51 return Metadata{}, fmt.Errorf("read remote pack list: %w", err)
52 }
53 if err == nil {
54 parsePackList(packObject.Data, metadata.Packs)
55 }
56 return metadata, nil
57 }
58
59 func parseInfoRefs(data []byte, refs map[string]Ref) error {
60 scanner := bufio.NewScanner(bytes.NewReader(data))
61 for scanner.Scan() {
62 line := scanner.Text()
63 parts := strings.SplitN(line, "\t", 2)
64 if len(parts) != 2 {
65 return fmt.Errorf("malformed info/refs line %q", line)
66 }
67 oid, name := parts[0], parts[1]
68 if strings.HasSuffix(name, "^{}") {
69 base := strings.TrimSuffix(name, "^{}")
70 ref := refs[base]
71 ref.Peeled = oid
72 refs[base] = ref
73 continue
74 }
75 ref := refs[name]
76 ref.OID = oid
77 refs[name] = ref
78 }
79 return scanner.Err()
80 }
81
82 func parsePackList(data []byte, packs map[string]struct{}) {
83 scanner := bufio.NewScanner(bytes.NewReader(data))
84 for scanner.Scan() {
85 line := strings.TrimSpace(scanner.Text())
86 if strings.HasPrefix(line, "P ") {
87 packs[strings.TrimSpace(strings.TrimPrefix(line, "P "))] = struct{}{}
88 }
89 }
90 }
91
92 func renderInfoRefs(refs map[string]Ref) []byte {
93 names := make([]string, 0, len(refs))
94 for name := range refs {
95 names = append(names, name)
96 }
97 sort.Strings(names)
98 var result strings.Builder
99 for _, name := range names {
100 ref := refs[name]
101 fmt.Fprintf(&result, "%s\t%s\n", ref.OID, name)
102 if ref.Peeled != "" && ref.Peeled != ref.OID {
103 fmt.Fprintf(&result, "%s\t%s^{}\n", ref.Peeled, name)
104 }
105 }
106 return []byte(result.String())
107 }
108
109 func renderPackList(packs map[string]struct{}) []byte {
110 names := make([]string, 0, len(packs))
111 for name := range packs {
112 names = append(names, name)
113 }
114 sort.Strings(names)
115 var result strings.Builder
116 for _, name := range names {
117 fmt.Fprintf(&result, "P %s\n", name)
118 }
119 return []byte(result.String())
120 }
Addedinternal/publish/metadata_test.go +35−0
@@ -0,0 +1,35 @@
1 package publish
2
3 import (
4 "reflect"
5 "testing"
6 )
7
8 func TestInfoRefsRoundTrip(t *testing.T) {
9 want := map[string]Ref{
10 "refs/heads/main": {OID: "1111111111111111111111111111111111111111"},
11 "refs/tags/v1": {
12 OID: "2222222222222222222222222222222222222222",
13 Peeled: "3333333333333333333333333333333333333333",
14 },
15 }
16 got := map[string]Ref{}
17 if err := parseInfoRefs(renderInfoRefs(want), got); err != nil {
18 t.Fatal(err)
19 }
20 if !reflect.DeepEqual(got, want) {
21 t.Fatalf("got %#v, want %#v", got, want)
22 }
23 }
24
25 func TestPackListRoundTrip(t *testing.T) {
26 want := map[string]struct{}{
27 "pack-1111111111111111111111111111111111111111.pack": {},
28 "pack-2222222222222222222222222222222222222222.pack": {},
29 }
30 got := map[string]struct{}{}
31 parsePackList(renderPackList(want), got)
32 if !reflect.DeepEqual(got, want) {
33 t.Fatalf("got %#v, want %#v", got, want)
34 }
35 }
Addedinternal/publish/publish.go +253−0
@@ -0,0 +1,253 @@
1 package publish
2
3 import (
4 "context"
5 "fmt"
6 "path"
7 "sort"
8 "strings"
9
10 "github.com/npfaro/dumbforge/internal/gitutil"
11 "github.com/npfaro/dumbforge/internal/s3store"
12 )
13
14 const (
15 mutableCache = "no-cache, max-age=0, must-revalidate"
16 immutableCache = "public, max-age=31536000, immutable"
17 )
18
19 type Update struct {
20 Src string
21 Dst string
22 Force bool
23 }
24
25 type AppliedUpdate struct {
26 Dst string
27 OldOID string
28 NewOID string
29 }
30
31 type Result struct {
32 Updates []AppliedUpdate
33 PackName string
34 PackObjects uint32
35 Head string
36 }
37
38 type Publisher struct {
39 Store *s3store.Store
40 Git gitutil.Git
41 Progress func(format string, args ...any)
42 AfterPublish func(context.Context, Result)
43 }
44
45 func (p *Publisher) List(ctx context.Context) (Metadata, error) {
46 return LoadMetadata(ctx, p.Store)
47 }
48
49 // Push applies a batch as one publication. expected contains the refs advertised
50 // to Git before it sent the push commands and prevents a concurrent update from
51 // being silently overwritten.
52 func (p *Publisher) Push(ctx context.Context, updates []Update, expected map[string]string) (result Result, err error) {
53 if len(updates) == 0 {
54 return Result{}, nil
55 }
56 shallow, err := p.Git.IsShallow(ctx)
57 if err != nil {
58 return Result{}, err
59 }
60 if shallow {
61 return Result{}, fmt.Errorf("cannot publish from a shallow repository; fetch the complete history first")
62 }
63 lock, err := p.Store.AcquireLock(ctx)
64 if err != nil {
65 return Result{}, err
66 }
67 defer func() {
68 if releaseErr := lock.Release(context.WithoutCancel(ctx)); err == nil && releaseErr != nil {
69 err = releaseErr
70 }
71 }()
72
73 metadata, err := LoadMetadata(ctx, p.Store)
74 if err != nil {
75 return Result{}, err
76 }
77 if err := checkExpected(metadata, updates, expected); err != nil {
78 return Result{}, err
79 }
80
81 resolved, err := p.resolveAndValidate(ctx, metadata, updates)
82 if err != nil {
83 return Result{}, err
84 }
85 wants := make([]string, 0, len(resolved))
86 for _, update := range resolved {
87 if update.NewOID != "" {
88 wants = append(wants, update.NewOID)
89 }
90 }
91 haves := make([]string, 0, len(metadata.Refs))
92 for _, ref := range metadata.Refs {
93 haves = append(haves, ref.OID)
94 }
95 pack, err := p.Git.PackObjects(ctx, wants, haves)
96 if err != nil {
97 return Result{}, err
98 }
99 defer pack.Close()
100 if pack != nil && pack.Objects > 0 {
101 packName := "pack-" + pack.Hash + ".pack"
102 indexName := "pack-" + pack.Hash + ".idx"
103 p.progress("uploading %s (%d objects)", packName, pack.Objects)
104 if err := p.Store.PutFile(ctx, path.Join("objects/pack", packName), pack.PackPath, "application/x-git-packed-objects", immutableCache); err != nil {
105 return Result{}, err
106 }
107 if err := p.Store.PutFile(ctx, path.Join("objects/pack", indexName), pack.IndexPath, "application/x-git-packed-objects-toc", immutableCache); err != nil {
108 return Result{}, err
109 }
110 metadata.Packs[packName] = struct{}{}
111 result.PackName = packName
112 result.PackObjects = pack.Objects
113 }
114
115 for _, update := range resolved {
116 result.Updates = append(result.Updates, update)
117 if update.NewOID == "" {
118 delete(metadata.Refs, update.Dst)
119 continue
120 }
121 ref := Ref{OID: update.NewOID}
122 if strings.HasPrefix(update.Dst, "refs/tags/") {
123 peeled, peelErr := p.Git.Peel(ctx, update.NewOID)
124 if peelErr == nil && peeled != update.NewOID {
125 ref.Peeled = peeled
126 }
127 }
128 metadata.Refs[update.Dst] = ref
129 }
130 if metadata.Head == "" {
131 metadata.Head = chooseHead(resolved)
132 }
133 result.Head = metadata.Head
134 if err := lock.Refresh(ctx); err != nil {
135 return Result{}, err
136 }
137
138 // Publish packs first, then individual refs and HEAD, and info/refs last.
139 // A dumb-HTTP reader can therefore never discover a ref before its pack.
140 if err := p.Store.Put(ctx, "objects/info/packs", renderPackList(metadata.Packs), "text/plain; charset=utf-8", mutableCache); err != nil {
141 return Result{}, err
142 }
143 for _, update := range resolved {
144 if update.NewOID == "" {
145 if err := p.Store.Delete(ctx, update.Dst); err != nil {
146 return Result{}, err
147 }
148 continue
149 }
150 if err := p.Store.Put(ctx, update.Dst, []byte(update.NewOID+"\n"), "text/plain; charset=utf-8", mutableCache); err != nil {
151 return Result{}, err
152 }
153 }
154 if metadata.Head != "" {
155 if err := p.Store.Put(ctx, "HEAD", []byte("ref: "+metadata.Head+"\n"), "text/plain; charset=utf-8", mutableCache); err != nil {
156 return Result{}, err
157 }
158 }
159 if err := p.Store.Put(ctx, "info/refs", renderInfoRefs(metadata.Refs), "text/plain; charset=utf-8", mutableCache); err != nil {
160 return Result{}, err
161 }
162 p.progress("published %d ref update(s)", len(resolved))
163 if p.AfterPublish != nil {
164 p.AfterPublish(ctx, result)
165 }
166 return result, nil
167 }
168
169 func (p *Publisher) resolveAndValidate(ctx context.Context, metadata Metadata, updates []Update) ([]AppliedUpdate, error) {
170 seen := map[string]struct{}{}
171 resolved := make([]AppliedUpdate, 0, len(updates))
172 for _, update := range updates {
173 if err := p.Git.CheckRef(ctx, update.Dst); err != nil {
174 return nil, err
175 }
176 if _, ok := seen[update.Dst]; ok {
177 return nil, fmt.Errorf("destination ref %s appears more than once", update.Dst)
178 }
179 seen[update.Dst] = struct{}{}
180 oldOID := metadata.Refs[update.Dst].OID
181 if update.Src == "" {
182 if update.Dst == metadata.Head {
183 return nil, fmt.Errorf("refusing to delete the remote HEAD branch %s", update.Dst)
184 }
185 resolved = append(resolved, AppliedUpdate{Dst: update.Dst, OldOID: oldOID})
186 continue
187 }
188 newOID, err := p.Git.RevParse(ctx, update.Src)
189 if err != nil {
190 return nil, fmt.Errorf("resolve %s: %w", update.Src, err)
191 }
192 if oldOID != "" && oldOID != newOID && !update.Force {
193 if !strings.HasPrefix(update.Dst, "refs/heads/") {
194 return nil, fmt.Errorf("ref %s already exists; use force to replace it", update.Dst)
195 }
196 if !p.Git.ObjectExists(ctx, oldOID) {
197 return nil, fmt.Errorf("remote tip %s is missing locally; fetch before pushing", oldOID)
198 }
199 ancestor, err := p.Git.IsAncestor(ctx, oldOID, newOID)
200 if err != nil {
201 return nil, err
202 }
203 if !ancestor {
204 return nil, fmt.Errorf("non-fast-forward update to %s", update.Dst)
205 }
206 }
207 resolved = append(resolved, AppliedUpdate{Dst: update.Dst, OldOID: oldOID, NewOID: newOID})
208 }
209 return resolved, nil
210 }
211
212 func checkExpected(metadata Metadata, updates []Update, expected map[string]string) error {
213 if expected == nil {
214 return nil
215 }
216 for _, update := range updates {
217 current := metadata.Refs[update.Dst].OID
218 if advertised, ok := expected[update.Dst]; ok && advertised != current {
219 return fmt.Errorf("remote ref %s changed during push; retry", update.Dst)
220 }
221 if _, ok := expected[update.Dst]; !ok && current != "" {
222 return fmt.Errorf("remote ref %s was created during push; retry", update.Dst)
223 }
224 }
225 return nil
226 }
227
228 func chooseHead(updates []AppliedUpdate) string {
229 for _, preferred := range []string{"refs/heads/main", "refs/heads/master"} {
230 for _, update := range updates {
231 if update.Dst == preferred && update.NewOID != "" {
232 return preferred
233 }
234 }
235 }
236 names := make([]string, 0, len(updates))
237 for _, update := range updates {
238 if strings.HasPrefix(update.Dst, "refs/heads/") && update.NewOID != "" {
239 names = append(names, update.Dst)
240 }
241 }
242 sort.Strings(names)
243 if len(names) > 0 {
244 return names[0]
245 }
246 return ""
247 }
248
249 func (p *Publisher) progress(format string, args ...any) {
250 if p.Progress != nil {
251 p.Progress(format, args...)
252 }
253 }
Addedinternal/remoteurl/remoteurl.go +76−0
@@ -0,0 +1,76 @@
1 package remoteurl
2
3 import (
4 "fmt"
5 "net/url"
6 "path"
7 "strings"
8 )
9
10 // Spec describes the authenticated S3 location used for pushes and the
11 // optional public HTTP location used for clones.
12 type Spec struct {
13 Bucket string
14 Prefix string
15 Endpoint string
16 Profile string
17 PublicURL string
18 }
19
20 func Parse(raw string) (Spec, error) {
21 u, err := url.Parse(raw)
22 if err != nil {
23 return Spec{}, fmt.Errorf("parse remote URL: %w", err)
24 }
25 if u.Scheme != "dumbforge" {
26 return Spec{}, fmt.Errorf("unsupported remote scheme %q", u.Scheme)
27 }
28
29 spec := Spec{
30 Bucket: u.Hostname(),
31 Prefix: strings.Trim(u.Path, "/"),
32 Endpoint: strings.TrimRight(u.Query().Get("endpoint"), "/"),
33 PublicURL: strings.TrimRight(u.Query().Get("public"), "/"),
34 }
35 if u.User != nil {
36 spec.Profile = u.User.Username()
37 }
38 if spec.Bucket == "" {
39 return Spec{}, fmt.Errorf("remote URL is missing a bucket")
40 }
41 if spec.Prefix == "" {
42 return Spec{}, fmt.Errorf("remote URL is missing a repository prefix")
43 }
44 if spec.Endpoint == "" {
45 return Spec{}, fmt.Errorf("remote URL is missing the S3 endpoint query parameter")
46 }
47 return spec, nil
48 }
49
50 func Build(spec Spec) (string, error) {
51 if spec.Bucket == "" || spec.Prefix == "" || spec.Endpoint == "" {
52 return "", fmt.Errorf("bucket, prefix, and endpoint are required")
53 }
54 u := &url.URL{
55 Scheme: "dumbforge",
56 Host: spec.Bucket,
57 Path: "/" + strings.Trim(spec.Prefix, "/"),
58 }
59 if spec.Profile != "" {
60 u.User = url.User(spec.Profile)
61 }
62 q := url.Values{}
63 q.Set("endpoint", strings.TrimRight(spec.Endpoint, "/"))
64 if spec.PublicURL != "" {
65 q.Set("public", strings.TrimRight(spec.PublicURL, "/"))
66 }
67 u.RawQuery = q.Encode()
68 return u.String(), nil
69 }
70
71 func (s Spec) PublicRepoURL() string {
72 if s.PublicURL == "" {
73 return ""
74 }
75 return strings.TrimRight(s.PublicURL, "/") + "/" + path.Clean(strings.Trim(s.Prefix, "/"))
76 }
Addedinternal/remoteurl/remoteurl_test.go +27−0
@@ -0,0 +1,27 @@
1 package remoteurl
2
3 import "testing"
4
5 func TestRoundTrip(t *testing.T) {
6 want := Spec{
7 Bucket: "bucket",
8 Prefix: "owner/repo.git",
9 Endpoint: "https://example.invalid",
10 Profile: "r2",
11 PublicURL: "https://public.invalid",
12 }
13 raw, err := Build(want)
14 if err != nil {
15 t.Fatal(err)
16 }
17 got, err := Parse(raw)
18 if err != nil {
19 t.Fatal(err)
20 }
21 if got != want {
22 t.Fatalf("got %#v, want %#v", got, want)
23 }
24 if got.PublicRepoURL() != "https://public.invalid/owner/repo.git" {
25 t.Fatalf("unexpected public URL %q", got.PublicRepoURL())
26 }
27 }
Addedinternal/s3store/store.go +242−0
@@ -0,0 +1,242 @@
1 package s3store
2
3 import (
4 "bytes"
5 "context"
6 "crypto/rand"
7 "encoding/hex"
8 "errors"
9 "fmt"
10 "io"
11 "os"
12 "path"
13 "strings"
14 "time"
15
16 "github.com/aws/aws-sdk-go-v2/aws"
17 "github.com/aws/aws-sdk-go-v2/config"
18 "github.com/aws/aws-sdk-go-v2/service/s3"
19 "github.com/aws/smithy-go"
20 "github.com/npfaro/dumbforge/internal/remoteurl"
21 )
22
23 type Object struct {
24 Data []byte
25 ETag string
26 }
27
28 type Store struct {
29 client *s3.Client
30 spec remoteurl.Spec
31 }
32
33 func New(ctx context.Context, spec remoteurl.Spec) (*Store, error) {
34 opts := []func(*config.LoadOptions) error{config.WithRegion("auto")}
35 if spec.Profile != "" {
36 opts = append(opts, config.WithSharedConfigProfile(spec.Profile))
37 }
38 cfg, err := config.LoadDefaultConfig(ctx, opts...)
39 if err != nil {
40 return nil, fmt.Errorf("load AWS configuration: %w", err)
41 }
42 client := s3.NewFromConfig(cfg, func(o *s3.Options) {
43 o.BaseEndpoint = aws.String(spec.Endpoint)
44 o.UsePathStyle = true
45 })
46 return &Store{client: client, spec: spec}, nil
47 }
48
49 func (s *Store) Spec() remoteurl.Spec { return s.spec }
50
51 func (s *Store) key(name string) string {
52 return path.Join(strings.Trim(s.spec.Prefix, "/"), strings.TrimLeft(name, "/"))
53 }
54
55 // RootKey returns the object key used for the repository's HTML landing page.
56 // It intentionally has no trailing slash; Git can still append /info/refs to
57 // the same URL when cloning.
58 func (s *Store) RootKey() string { return strings.Trim(s.spec.Prefix, "/") }
59
60 func (s *Store) Get(ctx context.Context, name string) (Object, error) {
61 out, err := s.client.GetObject(ctx, &s3.GetObjectInput{
62 Bucket: aws.String(s.spec.Bucket),
63 Key: aws.String(s.key(name)),
64 })
65 if err != nil {
66 return Object{}, err
67 }
68 defer out.Body.Close()
69 data, err := io.ReadAll(out.Body)
70 if err != nil {
71 return Object{}, fmt.Errorf("read s3://%s/%s: %w", s.spec.Bucket, s.key(name), err)
72 }
73 return Object{Data: data, ETag: aws.ToString(out.ETag)}, nil
74 }
75
76 func (s *Store) Put(ctx context.Context, name string, data []byte, contentType, cacheControl string) error {
77 input := &s3.PutObjectInput{
78 Bucket: aws.String(s.spec.Bucket),
79 Key: aws.String(s.key(name)),
80 Body: bytes.NewReader(data),
81 ContentType: aws.String(contentType),
82 }
83 if cacheControl != "" {
84 input.CacheControl = aws.String(cacheControl)
85 }
86 _, err := s.client.PutObject(ctx, input)
87 if err != nil {
88 return fmt.Errorf("put s3://%s/%s: %w", s.spec.Bucket, s.key(name), err)
89 }
90 return nil
91 }
92
93 func (s *Store) PutRoot(ctx context.Context, data []byte, contentType, cacheControl string) error {
94 input := &s3.PutObjectInput{
95 Bucket: aws.String(s.spec.Bucket),
96 Key: aws.String(s.RootKey()),
97 Body: bytes.NewReader(data),
98 ContentType: aws.String(contentType),
99 }
100 if cacheControl != "" {
101 input.CacheControl = aws.String(cacheControl)
102 }
103 _, err := s.client.PutObject(ctx, input)
104 if err != nil {
105 return fmt.Errorf("put s3://%s/%s: %w", s.spec.Bucket, s.RootKey(), err)
106 }
107 return nil
108 }
109
110 func (s *Store) PutFile(ctx context.Context, name, filename, contentType, cacheControl string) error {
111 f, err := os.Open(filename)
112 if err != nil {
113 return err
114 }
115 defer f.Close()
116 input := &s3.PutObjectInput{
117 Bucket: aws.String(s.spec.Bucket),
118 Key: aws.String(s.key(name)),
119 Body: f,
120 ContentType: aws.String(contentType),
121 }
122 if cacheControl != "" {
123 input.CacheControl = aws.String(cacheControl)
124 }
125 _, err = s.client.PutObject(ctx, input)
126 if err != nil {
127 return fmt.Errorf("put s3://%s/%s: %w", s.spec.Bucket, s.key(name), err)
128 }
129 return nil
130 }
131
132 func (s *Store) Delete(ctx context.Context, name string) error {
133 _, err := s.client.DeleteObject(ctx, &s3.DeleteObjectInput{
134 Bucket: aws.String(s.spec.Bucket),
135 Key: aws.String(s.key(name)),
136 })
137 if err != nil {
138 return fmt.Errorf("delete s3://%s/%s: %w", s.spec.Bucket, s.key(name), err)
139 }
140 return nil
141 }
142
143 type Lock struct {
144 store *Store
145 name string
146 token string
147 etag string
148 }
149
150 func (s *Store) AcquireLock(ctx context.Context) (*Lock, error) {
151 random := make([]byte, 16)
152 if _, err := rand.Read(random); err != nil {
153 return nil, err
154 }
155 token := hex.EncodeToString(random)
156 body := []byte(fmt.Sprintf("%s\n%s\n", token, time.Now().UTC().Format(time.RFC3339Nano)))
157 name := ".dumbforge/lock"
158 input := &s3.PutObjectInput{
159 Bucket: aws.String(s.spec.Bucket),
160 Key: aws.String(s.key(name)),
161 Body: bytes.NewReader(body),
162 ContentType: aws.String("text/plain; charset=utf-8"),
163 IfNoneMatch: aws.String("*"),
164 }
165 out, err := s.client.PutObject(ctx, input)
166 if err != nil {
167 if IsPreconditionFailed(err) {
168 existing, getErr := s.Get(ctx, name)
169 if getErr != nil {
170 return nil, fmt.Errorf("repository is locked by another push")
171 }
172 lines := strings.Split(string(existing.Data), "\n")
173 if len(lines) < 2 {
174 return nil, fmt.Errorf("repository is locked by another push")
175 }
176 lockedAt, timeErr := time.Parse(time.RFC3339Nano, lines[1])
177 if timeErr != nil || time.Since(lockedAt) < time.Hour {
178 return nil, fmt.Errorf("repository is locked by another push")
179 }
180 input.IfNoneMatch = nil
181 input.IfMatch = aws.String(existing.ETag)
182 out, err = s.client.PutObject(ctx, input)
183 if err != nil {
184 return nil, fmt.Errorf("replace stale repository lock: %w", err)
185 }
186 } else {
187 return nil, fmt.Errorf("acquire repository lock: %w", err)
188 }
189 }
190 return &Lock{store: s, name: name, token: token, etag: aws.ToString(out.ETag)}, nil
191 }
192
193 // Refresh proves that this process still owns the lease before it publishes
194 // mutable repository metadata.
195 func (l *Lock) Refresh(ctx context.Context) error {
196 body := []byte(fmt.Sprintf("%s\n%s\n", l.token, time.Now().UTC().Format(time.RFC3339Nano)))
197 out, err := l.store.client.PutObject(ctx, &s3.PutObjectInput{
198 Bucket: aws.String(l.store.spec.Bucket),
199 Key: aws.String(l.store.key(l.name)),
200 Body: bytes.NewReader(body),
201 ContentType: aws.String("text/plain; charset=utf-8"),
202 IfMatch: aws.String(l.etag),
203 })
204 if err != nil {
205 return fmt.Errorf("refresh repository lock: %w", err)
206 }
207 l.etag = aws.ToString(out.ETag)
208 return nil
209 }
210
211 func (l *Lock) Release(ctx context.Context) error {
212 _, err := l.store.client.DeleteObject(ctx, &s3.DeleteObjectInput{
213 Bucket: aws.String(l.store.spec.Bucket),
214 Key: aws.String(l.store.key(l.name)),
215 IfMatch: aws.String(l.etag),
216 })
217 if err != nil {
218 return fmt.Errorf("release repository lock: %w", err)
219 }
220 return nil
221 }
222
223 func IsNotFound(err error) bool {
224 var apiErr smithy.APIError
225 if !errors.As(err, &apiErr) {
226 return false
227 }
228 switch apiErr.ErrorCode() {
229 case "NoSuchKey", "NotFound", "404":
230 return true
231 default:
232 return false
233 }
234 }
235
236 func IsPreconditionFailed(err error) bool {
237 var apiErr smithy.APIError
238 if !errors.As(err, &apiErr) {
239 return false
240 }
241 return apiErr.ErrorCode() == "PreconditionFailed" || apiErr.ErrorCode() == "412"
242 }
Addedinternal/site/site.go +672−0
@@ -0,0 +1,672 @@
1 package site
2
3 import (
4 "bytes"
5 "context"
6 "fmt"
7 "html/template"
8 "mime"
9 "net/url"
10 "path"
11 "sort"
12 "strconv"
13 "strings"
14 "sync"
15 "unicode/utf8"
16
17 "github.com/alecthomas/chroma/v2"
18 "github.com/alecthomas/chroma/v2/formatters/html"
19 "github.com/alecthomas/chroma/v2/lexers"
20 "github.com/alecthomas/chroma/v2/styles"
21 "github.com/npfaro/dumbforge/internal/gitutil"
22 "github.com/npfaro/dumbforge/internal/s3store"
23 "github.com/yuin/goldmark"
24 "github.com/yuin/goldmark/extension"
25 "github.com/yuin/goldmark/parser"
26 )
27
28 const (
29 siteCache = "no-cache, max-age=0, must-revalidate"
30 rawCache = "no-cache, max-age=0, must-revalidate"
31 )
32
33 type Page struct {
34 Route string
35 Body []byte
36 ContentType string
37 Cache string
38 Root bool
39 }
40
41 type Build struct {
42 Pages []Page
43 Deletes []string
44 }
45
46 type Generator struct {
47 Git gitutil.Git
48 BaseURL string
49 RepoName string
50 Owner string
51 CloneURL string
52 Progress func(format string, args ...any)
53 templates *template.Template
54 }
55
56 type Commit struct {
57 OID string
58 ShortOID string
59 Author string
60 Date string
61 Message string
62 Initials string
63 }
64
65 type TreeEntry struct {
66 Mode string
67 Type string
68 OID string
69 Size int64
70 Name string
71 URL string
72 }
73
74 type Crumb struct {
75 Name string
76 URL string
77 }
78
79 type pageData struct {
80 Kind string
81 Title string
82 BaseURL string
83 RepoName string
84 Owner string
85 CloneURL string
86 Branch string
87 Path string
88 ParentURL string
89 Breadcrumbs []Crumb
90 Entries []TreeEntry
91 Commit Commit
92 CommitCount int
93 ReadmeName string
94 Readme template.HTML
95 Code template.HTML
96 RawURL string
97 Language string
98 FileSize int64
99 IsImage bool
100 }
101
102 func New(git gitutil.Git, baseURL, repoName, owner string) (*Generator, error) {
103 if repoName == "" {
104 repoName = "repository"
105 }
106 tmpl, err := template.New("page").Funcs(template.FuncMap{
107 "humanSize": humanSize,
108 }).Parse(pageTemplate)
109 if err != nil {
110 return nil, err
111 }
112 return &Generator{
113 Git: git,
114 BaseURL: strings.TrimRight(baseURL, "/"),
115 RepoName: strings.TrimSuffix(repoName, ".git"),
116 Owner: owner,
117 CloneURL: strings.TrimRight(baseURL, "/"),
118 templates: tmpl,
119 }, nil
120 }
121
122 // BuildBranch generates a full branch browser when oldOID is empty. For later
123 // pushes it only regenerates changed blobs and the directory pages that contain
124 // them. The repository landing page is refreshed when isHead is true.
125 func (g *Generator) BuildBranch(ctx context.Context, branch, oldOID, newOID string, isHead bool) (Build, error) {
126 commit, err := g.commit(ctx, newOID)
127 if err != nil {
128 return Build{}, err
129 }
130 commitCount, err := g.commitCount(ctx, newOID)
131 if err != nil {
132 return Build{}, err
133 }
134 result := Build{}
135 result.Pages = append(result.Pages, Page{
136 Route: ".dumbforge/assets/site.css",
137 Body: []byte(siteCSS + g.chromaCSS()),
138 ContentType: "text/css; charset=utf-8",
139 Cache: siteCache,
140 })
141
142 if oldOID == "" {
143 entries, err := g.listTreeRecursive(ctx, newOID)
144 if err != nil {
145 return Build{}, err
146 }
147 dirs := map[string]struct{}{"": {}}
148 for _, entry := range entries {
149 if entry.Type == "tree" {
150 dirs[entry.Name] = struct{}{}
151 continue
152 }
153 pages, err := g.blobPages(ctx, branch, newOID, entry, commit, commitCount)
154 if err != nil {
155 return Build{}, err
156 }
157 result.Pages = append(result.Pages, pages...)
158 }
159 for _, dir := range sortedKeys(dirs) {
160 if dir == "" && isHead {
161 continue
162 }
163 page, err := g.treePage(ctx, branch, newOID, dir, commit, commitCount, false)
164 if err != nil {
165 return Build{}, err
166 }
167 result.Pages = append(result.Pages, page)
168 }
169 } else {
170 changes, err := g.Git.DiffNameStatus(ctx, oldOID, newOID)
171 if err != nil {
172 g.progress("could not diff %s..%s; rebuilding branch", short(oldOID), short(newOID))
173 return g.BuildBranch(ctx, branch, "", newOID, isHead)
174 }
175 dirs := map[string]struct{}{"": {}}
176 for _, change := range changes {
177 if change.OldPath != "" {
178 result.Deletes = append(result.Deletes,
179 path.Join("blob", branch, change.OldPath),
180 path.Join("raw", branch, change.OldPath),
181 )
182 addAncestorDirs(dirs, change.OldPath)
183 }
184 if strings.HasPrefix(change.Status, "D") {
185 result.Deletes = append(result.Deletes,
186 path.Join("blob", branch, change.Path),
187 path.Join("raw", branch, change.Path),
188 )
189 addAncestorDirs(dirs, change.Path)
190 continue
191 }
192 entry, err := g.entry(ctx, newOID, change.Path)
193 if err != nil {
194 return Build{}, err
195 }
196 pages, err := g.blobPages(ctx, branch, newOID, entry, commit, commitCount)
197 if err != nil {
198 return Build{}, err
199 }
200 result.Pages = append(result.Pages, pages...)
201 addAncestorDirs(dirs, change.Path)
202 }
203 for _, dir := range sortedKeys(dirs) {
204 if dir == "" && isHead {
205 continue
206 }
207 exists, err := g.treeExists(ctx, newOID, dir)
208 if err != nil {
209 return Build{}, err
210 }
211 if !exists {
212 result.Deletes = append(result.Deletes, path.Join("tree", branch, dir))
213 continue
214 }
215 page, err := g.treePage(ctx, branch, newOID, dir, commit, commitCount, false)
216 if err != nil {
217 return Build{}, err
218 }
219 result.Pages = append(result.Pages, page)
220 }
221 }
222
223 if isHead {
224 landing, err := g.treePage(ctx, branch, newOID, "", commit, commitCount, true)
225 if err != nil {
226 return Build{}, err
227 }
228 result.Pages = append(result.Pages, landing)
229 }
230 result.Deletes = uniqueStrings(result.Deletes)
231 return result, nil
232 }
233
234 func (g *Generator) Publish(ctx context.Context, store *s3store.Store, build Build) error {
235 for _, route := range build.Deletes {
236 if err := store.Delete(ctx, route); err != nil {
237 return err
238 }
239 }
240
241 ctx, cancel := context.WithCancel(ctx)
242 defer cancel()
243 jobs := make(chan Page)
244 errCh := make(chan error, 1)
245 var once sync.Once
246 var workers sync.WaitGroup
247 for range 8 {
248 workers.Add(1)
249 go func() {
250 defer workers.Done()
251 for page := range jobs {
252 var err error
253 if page.Root {
254 err = store.PutRoot(ctx, page.Body, page.ContentType, page.Cache)
255 } else {
256 err = store.Put(ctx, page.Route, page.Body, page.ContentType, page.Cache)
257 }
258 if err != nil {
259 once.Do(func() {
260 errCh <- err
261 cancel()
262 })
263 return
264 }
265 }
266 }()
267 }
268 sendPages:
269 for _, page := range build.Pages {
270 select {
271 case jobs <- page:
272 case <-ctx.Done():
273 break sendPages
274 }
275 }
276 close(jobs)
277 workers.Wait()
278 select {
279 case err := <-errCh:
280 return err
281 default:
282 }
283 g.progress("published %d site object(s), removed %d", len(build.Pages), len(build.Deletes))
284 return nil
285 }
286
287 func (g *Generator) treePage(ctx context.Context, branch, oid, dir string, commit Commit, commitCount int, root bool) (Page, error) {
288 entries, err := g.listTree(ctx, oid, dir)
289 if err != nil {
290 return Page{}, err
291 }
292 for i := range entries {
293 fullPath := path.Join(dir, entries[i].Name)
294 if entries[i].Type == "tree" {
295 entries[i].URL = routeURL(g.BaseURL, path.Join("tree", branch, fullPath))
296 } else {
297 entries[i].URL = routeURL(g.BaseURL, path.Join("blob", branch, fullPath))
298 }
299 }
300 data := pageData{
301 Kind: "tree",
302 Title: g.RepoName,
303 BaseURL: g.BaseURL,
304 RepoName: g.RepoName,
305 Owner: g.Owner,
306 CloneURL: g.CloneURL,
307 Branch: branch,
308 Path: dir,
309 Breadcrumbs: g.breadcrumbs(branch, dir),
310 Entries: entries,
311 Commit: commit,
312 CommitCount: commitCount,
313 }
314 if dir != "" {
315 data.Title = path.Base(dir) + " · " + g.RepoName
316 data.ParentURL = routeURL(g.BaseURL, path.Join("tree", branch, path.Dir(dir)))
317 }
318 if root {
319 data.Kind = "repo"
320 readmeName, readme, err := g.readme(ctx, oid, entries)
321 if err != nil {
322 return Page{}, err
323 }
324 data.ReadmeName = readmeName
325 data.Readme = readme
326 }
327 body, err := g.render(data)
328 if err != nil {
329 return Page{}, err
330 }
331 if root {
332 return Page{Body: body, ContentType: "text/html; charset=utf-8", Cache: siteCache, Root: true}, nil
333 }
334 return Page{Route: path.Join("tree", branch, dir), Body: body, ContentType: "text/html; charset=utf-8", Cache: siteCache}, nil
335 }
336
337 func (g *Generator) blobPages(ctx context.Context, branch, oid string, entry TreeEntry, commit Commit, commitCount int) ([]Page, error) {
338 var content []byte
339 var err error
340 if entry.Type == "commit" {
341 content = []byte("Submodule commit " + entry.OID + "\n")
342 } else {
343 content, err = g.Git.Run(ctx, "show", oid+":"+entry.Name)
344 if err != nil {
345 return nil, err
346 }
347 }
348 rawRoute := path.Join("raw", branch, entry.Name)
349 contentType := mime.TypeByExtension(path.Ext(entry.Name))
350 if contentType == "" {
351 contentType = "application/octet-stream"
352 }
353 data := pageData{
354 Kind: "blob",
355 Title: path.Base(entry.Name) + " · " + g.RepoName,
356 BaseURL: g.BaseURL,
357 RepoName: g.RepoName,
358 Owner: g.Owner,
359 CloneURL: g.CloneURL,
360 Branch: branch,
361 Path: entry.Name,
362 Breadcrumbs: g.breadcrumbs(branch, entry.Name),
363 Commit: commit,
364 CommitCount: commitCount,
365 RawURL: routeURL(g.BaseURL, rawRoute),
366 FileSize: int64(len(content)),
367 IsImage: strings.HasPrefix(contentType, "image/"),
368 }
369 if data.IsImage {
370 data.Language = "Image"
371 } else if bytes.IndexByte(content, 0) >= 0 || !utf8.Valid(content) {
372 data.Language = "Binary"
373 } else if len(content) > 2*1024*1024 {
374 data.Language = "Large text file"
375 } else {
376 code, language, err := highlight(entry.Name, content)
377 if err != nil {
378 return nil, err
379 }
380 data.Code = code
381 data.Language = language
382 }
383 body, err := g.render(data)
384 if err != nil {
385 return nil, err
386 }
387 return []Page{
388 {Route: path.Join("blob", branch, entry.Name), Body: body, ContentType: "text/html; charset=utf-8", Cache: siteCache},
389 {Route: rawRoute, Body: content, ContentType: contentType, Cache: rawCache},
390 }, nil
391 }
392
393 func (g *Generator) readme(ctx context.Context, oid string, entries []TreeEntry) (string, template.HTML, error) {
394 for _, entry := range entries {
395 if entry.Type != "blob" || !strings.HasPrefix(strings.ToLower(entry.Name), "readme") {
396 continue
397 }
398 content, err := g.Git.Run(ctx, "show", oid+":"+entry.Name)
399 if err != nil {
400 return "", "", err
401 }
402 var rendered bytes.Buffer
403 markdown := goldmark.New(
404 goldmark.WithExtensions(extension.GFM),
405 goldmark.WithParserOptions(parser.WithAutoHeadingID()),
406 )
407 if err := markdown.Convert(content, &rendered); err != nil {
408 return "", "", err
409 }
410 return entry.Name, template.HTML(rendered.String()), nil
411 }
412 return "", "", nil
413 }
414
415 func highlight(filename string, content []byte) (template.HTML, string, error) {
416 lexer := lexers.Match(filename)
417 if lexer == nil {
418 lexer = lexers.Analyse(string(content))
419 }
420 if lexer == nil {
421 lexer = lexers.Fallback
422 }
423 lexer = chroma.Coalesce(lexer)
424 iterator, err := lexer.Tokenise(nil, string(content))
425 if err != nil {
426 return "", "", err
427 }
428 formatter := html.New(
429 html.WithClasses(true),
430 html.WithLineNumbers(true),
431 html.LineNumbersInTable(true),
432 html.WithLinkableLineNumbers(true, "L"),
433 html.TabWidth(4),
434 )
435 var rendered bytes.Buffer
436 if err := formatter.Format(&rendered, styles.Get("github"), iterator); err != nil {
437 return "", "", err
438 }
439 return template.HTML(rendered.String()), lexer.Config().Name, nil
440 }
441
442 func (g *Generator) chromaCSS() string {
443 formatter := html.New(html.WithClasses(true))
444 var css bytes.Buffer
445 _ = formatter.WriteCSS(&css, styles.Get("github"))
446 return "\n" + css.String()
447 }
448
449 func (g *Generator) render(data pageData) ([]byte, error) {
450 var result bytes.Buffer
451 if err := g.templates.Execute(&result, data); err != nil {
452 return nil, err
453 }
454 return result.Bytes(), nil
455 }
456
457 func (g *Generator) commit(ctx context.Context, oid string) (Commit, error) {
458 out, err := g.Git.Run(ctx, "show", "-s", "--format=%H%x00%h%x00%an%x00%cs%x00%s", oid)
459 if err != nil {
460 return Commit{}, err
461 }
462 parts := strings.Split(strings.TrimSpace(string(out)), "\x00")
463 if len(parts) != 5 {
464 return Commit{}, fmt.Errorf("unexpected git show output")
465 }
466 return Commit{
467 OID: parts[0],
468 ShortOID: parts[1],
469 Author: parts[2],
470 Date: parts[3],
471 Message: parts[4],
472 Initials: initials(parts[2]),
473 }, nil
474 }
475
476 func (g *Generator) commitCount(ctx context.Context, oid string) (int, error) {
477 out, err := g.Git.Run(ctx, "rev-list", "--count", oid)
478 if err != nil {
479 return 0, err
480 }
481 return strconv.Atoi(strings.TrimSpace(string(out)))
482 }
483
484 func (g *Generator) listTree(ctx context.Context, oid, dir string) ([]TreeEntry, error) {
485 treeish := oid
486 if dir != "" {
487 treeish += ":" + dir
488 }
489 out, err := g.Git.Run(ctx, "ls-tree", "-z", "-l", treeish)
490 if err != nil {
491 return nil, err
492 }
493 entries, err := parseTree(out)
494 if err != nil {
495 return nil, err
496 }
497 sortTree(entries)
498 return entries, nil
499 }
500
501 func (g *Generator) listTreeRecursive(ctx context.Context, oid string) ([]TreeEntry, error) {
502 out, err := g.Git.Run(ctx, "ls-tree", "-r", "-t", "-z", "-l", oid)
503 if err != nil {
504 return nil, err
505 }
506 return parseTree(out)
507 }
508
509 func (g *Generator) entry(ctx context.Context, oid, name string) (TreeEntry, error) {
510 out, err := g.Git.Run(ctx, "ls-tree", "-z", "-l", oid, "--", name)
511 if err != nil {
512 return TreeEntry{}, err
513 }
514 entries, err := parseTree(out)
515 if err != nil {
516 return TreeEntry{}, err
517 }
518 if len(entries) != 1 {
519 return TreeEntry{}, fmt.Errorf("path %s is not a file at %s", name, short(oid))
520 }
521 return entries[0], nil
522 }
523
524 func (g *Generator) treeExists(ctx context.Context, oid, dir string) (bool, error) {
525 if dir == "" || dir == "." {
526 return true, nil
527 }
528 out, err := g.Git.Run(ctx, "cat-file", "-t", oid+":"+dir)
529 if err != nil {
530 return false, nil
531 }
532 return strings.TrimSpace(string(out)) == "tree", nil
533 }
534
535 func parseTree(data []byte) ([]TreeEntry, error) {
536 var result []TreeEntry
537 for _, record := range bytes.Split(data, []byte{0}) {
538 if len(record) == 0 {
539 continue
540 }
541 parts := bytes.SplitN(record, []byte{'\t'}, 2)
542 if len(parts) != 2 {
543 return nil, fmt.Errorf("malformed ls-tree record")
544 }
545 fields := strings.Fields(string(parts[0]))
546 if len(fields) != 4 {
547 return nil, fmt.Errorf("malformed ls-tree header %q", parts[0])
548 }
549 size := int64(-1)
550 if fields[3] != "-" {
551 var err error
552 size, err = strconv.ParseInt(fields[3], 10, 64)
553 if err != nil {
554 return nil, err
555 }
556 }
557 result = append(result, TreeEntry{Mode: fields[0], Type: fields[1], OID: fields[2], Size: size, Name: string(parts[1])})
558 }
559 return result, nil
560 }
561
562 func sortTree(entries []TreeEntry) {
563 sort.Slice(entries, func(i, j int) bool {
564 if (entries[i].Type == "tree") != (entries[j].Type == "tree") {
565 return entries[i].Type == "tree"
566 }
567 return strings.ToLower(entries[i].Name) < strings.ToLower(entries[j].Name)
568 })
569 }
570
571 func (g *Generator) breadcrumbs(branch, filePath string) []Crumb {
572 crumbs := []Crumb{{Name: g.RepoName, URL: g.BaseURL}}
573 parts := strings.Split(strings.Trim(filePath, "/"), "/")
574 for i, part := range parts {
575 if part == "" {
576 continue
577 }
578 prefix := path.Join(parts[:i+1]...)
579 routeKind := "tree"
580 if i == len(parts)-1 && path.Ext(filePath) != "" {
581 routeKind = "blob"
582 }
583 crumbs = append(crumbs, Crumb{Name: part, URL: routeURL(g.BaseURL, path.Join(routeKind, branch, prefix))})
584 }
585 return crumbs
586 }
587
588 func routeURL(base, route string) string {
589 parts := strings.Split(strings.Trim(route, "/"), "/")
590 for i := range parts {
591 parts[i] = url.PathEscape(parts[i])
592 }
593 if len(parts) == 1 && parts[0] == "" {
594 return strings.TrimRight(base, "/")
595 }
596 return strings.TrimRight(base, "/") + "/" + strings.Join(parts, "/")
597 }
598
599 func addAncestorDirs(dirs map[string]struct{}, filePath string) {
600 dir := path.Dir(filePath)
601 for dir != "." && dir != "/" && dir != "" {
602 dirs[dir] = struct{}{}
603 dir = path.Dir(dir)
604 }
605 dirs[""] = struct{}{}
606 }
607
608 func sortedKeys(values map[string]struct{}) []string {
609 result := make([]string, 0, len(values))
610 for value := range values {
611 result = append(result, value)
612 }
613 sort.Strings(result)
614 return result
615 }
616
617 func uniqueStrings(values []string) []string {
618 seen := map[string]struct{}{}
619 result := make([]string, 0, len(values))
620 for _, value := range values {
621 if _, ok := seen[value]; ok {
622 continue
623 }
624 seen[value] = struct{}{}
625 result = append(result, value)
626 }
627 sort.Strings(result)
628 return result
629 }
630
631 func initials(name string) string {
632 fields := strings.Fields(name)
633 if len(fields) == 0 {
634 return "?"
635 }
636 result := string([]rune(fields[0])[0])
637 if len(fields) > 1 {
638 result += string([]rune(fields[len(fields)-1])[0])
639 }
640 return strings.ToUpper(result)
641 }
642
643 func humanSize(size int64) string {
644 if size < 0 {
645 return ""
646 }
647 if size < 1024 {
648 return fmt.Sprintf("%d B", size)
649 }
650 units := []string{"KB", "MB", "GB"}
651 value := float64(size)
652 for _, unit := range units {
653 value /= 1024
654 if value < 1024 {
655 return fmt.Sprintf("%.1f %s", value, unit)
656 }
657 }
658 return fmt.Sprintf("%.1f TB", value/1024)
659 }
660
661 func short(oid string) string {
662 if len(oid) > 7 {
663 return oid[:7]
664 }
665 return oid
666 }
667
668 func (g *Generator) progress(format string, args ...any) {
669 if g.Progress != nil {
670 g.Progress(format, args...)
671 }
672 }
Addedinternal/site/site_test.go +89−0
@@ -0,0 +1,89 @@
1 package site
2
3 import (
4 "context"
5 "os"
6 "os/exec"
7 "path/filepath"
8 "strings"
9 "testing"
10
11 "github.com/npfaro/dumbforge/internal/gitutil"
12 )
13
14 func TestFullAndIncrementalBuild(t *testing.T) {
15 dir := t.TempDir()
16 runGit(t, dir, "init", "-b", "main")
17 runGit(t, dir, "config", "user.name", "Site Test")
18 runGit(t, dir, "config", "user.email", "[email protected]")
19 write(t, dir, "README.md", "# Hello bucket\n")
20 write(t, dir, "src/main.go", "package main\n\nfunc main() {}\n")
21 runGit(t, dir, "add", ".")
22 runGit(t, dir, "commit", "-m", "initial")
23 first := outputGit(t, dir, "rev-parse", "HEAD")
24
25 generator, err := New(gitutil.Git{Dir: dir}, "https://example.invalid/repo.git", "repo.git", "owner")
26 if err != nil {
27 t.Fatal(err)
28 }
29 full, err := generator.BuildBranch(context.Background(), "main", "", first, true)
30 if err != nil {
31 t.Fatal(err)
32 }
33 assertPage(t, full, "", "Hello bucket")
34 assertPage(t, full, "blob/main/src/main.go", "chroma")
35 assertPage(t, full, "tree/main/src", "main.go")
36
37 write(t, dir, "src/main.go", "package main\n\nfunc main() { println(\"changed\") }\n")
38 runGit(t, dir, "add", "src/main.go")
39 runGit(t, dir, "commit", "-m", "change one file")
40 second := outputGit(t, dir, "rev-parse", "HEAD")
41 incremental, err := generator.BuildBranch(context.Background(), "main", first, second, true)
42 if err != nil {
43 t.Fatal(err)
44 }
45 if len(incremental.Pages) != 5 {
46 t.Fatalf("incremental build generated %d pages, want 5", len(incremental.Pages))
47 }
48 assertPage(t, incremental, "blob/main/src/main.go", "changed")
49 assertPage(t, incremental, "tree/main/src", "change one file")
50 }
51
52 func assertPage(t *testing.T, build Build, route, contains string) {
53 t.Helper()
54 for _, page := range build.Pages {
55 if page.Route == route && page.Root == (route == "") {
56 if !strings.Contains(string(page.Body), contains) {
57 t.Fatalf("page %q does not contain %q", route, contains)
58 }
59 return
60 }
61 }
62 t.Fatalf("page %q was not generated", route)
63 }
64
65 func write(t *testing.T, dir, name, contents string) {
66 t.Helper()
67 filename := filepath.Join(dir, name)
68 if err := os.MkdirAll(filepath.Dir(filename), 0o755); err != nil {
69 t.Fatal(err)
70 }
71 if err := os.WriteFile(filename, []byte(contents), 0o644); err != nil {
72 t.Fatal(err)
73 }
74 }
75
76 func runGit(t *testing.T, dir string, args ...string) {
77 t.Helper()
78 _ = outputGit(t, dir, args...)
79 }
80
81 func outputGit(t *testing.T, dir string, args ...string) string {
82 t.Helper()
83 args = append([]string{"-C", dir}, args...)
84 out, err := exec.Command("git", args...).CombinedOutput()
85 if err != nil {
86 t.Fatalf("git %v: %v\n%s", args, err, out)
87 }
88 return strings.TrimSpace(string(out))
89 }
Addedinternal/site/style.go +123−0
@@ -0,0 +1,123 @@
1 package site
2
3 const siteCSS = `:root {
4 --bg: #ffffff;
5 --canvas: #f6f8fa;
6 --border: #d0d7de;
7 --border-muted: #d8dee4;
8 --text: #1f2328;
9 --muted: #656d76;
10 --blue: #0969da;
11 --green: #1f883d;
12 --nav: #25292e;
13 --shadow: 0 1px 0 rgba(31,35,40,.04);
14 font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
15 color: var(--text);
16 background: var(--bg);
17 }
18 * { box-sizing: border-box; }
19 body { margin: 0; min-width: 320px; background: var(--bg); font-size: 14px; line-height: 1.5; }
20 a { color: var(--blue); text-decoration: none; }
21 a:hover { text-decoration: underline; }
22 button { font: inherit; }
23 .topbar { height: 64px; padding: 0 28px; display: flex; align-items: center; gap: 18px; background: var(--nav); color: white; }
24 .brand { display: inline-flex; align-items: center; gap: 9px; color: white; font-size: 16px; font-weight: 650; letter-spacing: -.2px; }
25 .brand:hover { text-decoration: none; }
26 .brand-mark { display: grid; place-items: center; width: 32px; height: 32px; border: 1px solid #6e7681; border-radius: 50%; font-family: ui-monospace, monospace; font-weight: 800; }
27 .top-search { width: 300px; height: 34px; border: 1px solid #57606a; border-radius: 6px; display: flex; align-items: center; padding: 0 12px; color: #b6bec8; font-size: 13px; }
28 .top-link { margin-left: auto; color: white; font-weight: 600; }
29 .repo-head { background: var(--canvas); border-bottom: 1px solid var(--border); padding: 22px 32px 0; }
30 .repo-title { max-width: 1280px; margin: 0 auto; display: flex; align-items: center; gap: 7px; font-size: 20px; }
31 .repo-icon { color: var(--muted); font-size: 20px; margin-right: 2px; }
32 .repo-name { font-weight: 650; }
33 .slash { color: var(--muted); }
34 .visibility { color: var(--muted); border: 1px solid var(--border); border-radius: 999px; padding: 1px 8px; font-size: 12px; font-weight: 600; margin-left: 5px; }
35 .tabs { max-width: 1280px; margin: 20px auto 0; display: flex; gap: 8px; }
36 .tab { position: relative; display: flex; align-items: center; gap: 7px; padding: 9px 12px 12px; color: var(--text); font-weight: 500; }
37 .tab:hover { text-decoration: none; background: rgba(175,184,193,.16); border-radius: 6px 6px 0 0; }
38 .tab.active { font-weight: 650; }
39 .tab.active:after { content: ""; height: 2px; background: #fd8c73; position: absolute; left: 8px; right: 8px; bottom: -1px; }
40 .content-shell { max-width: 1280px; margin: 24px auto 64px; padding: 0 24px; }
41 .repo-grid { display: grid; grid-template-columns: minmax(0, 1fr) 270px; gap: 28px; }
42 .primary { min-width: 0; }
43 .full-width { width: 100%; }
44 .toolbar { min-height: 38px; margin-bottom: 14px; display: flex; align-items: center; gap: 10px; }
45 .branch-button, .clone-button, .blob-actions a, .blob-actions button { min-height: 32px; border: 1px solid rgba(31,35,40,.15); border-radius: 6px; background: #f6f8fa; box-shadow: var(--shadow); color: var(--text); display: inline-flex; align-items: center; gap: 7px; padding: 5px 12px; font-weight: 600; font-size: 13px; }
46 .branch-icon { font-size: 17px; line-height: 1; }
47 .chevron { color: var(--muted); margin-left: 4px; }
48 .clone-button { margin-left: auto; background: var(--green); border-color: rgba(31,35,40,.15); color: white; cursor: pointer; }
49 .toolbar-path { border-bottom: 1px solid var(--border-muted); padding-bottom: 14px; }
50 .breadcrumbs { min-width: 0; display: flex; align-items: center; gap: 6px; font-size: 16px; overflow: hidden; }
51 .breadcrumbs a { color: var(--text); white-space: nowrap; }
52 .breadcrumbs a:last-child { font-weight: 650; }
53 .breadcrumbs span { color: var(--muted); }
54 .tree-card, .readme-card, .blob-card { border: 1px solid var(--border); border-radius: 6px; overflow: hidden; background: var(--bg); }
55 .commit-row { min-height: 54px; display: grid; grid-template-columns: 32px auto minmax(80px, 1fr) auto auto; align-items: center; gap: 9px; padding: 9px 16px; background: #f6f8fa; border-bottom: 1px solid var(--border); }
56 .avatar { width: 28px; height: 28px; display: grid; place-items: center; border-radius: 50%; color: white; background: linear-gradient(135deg,#8250df,#0969da); font-size: 10px; font-weight: 700; }
57 .commit-message { color: var(--text); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
58 .commit-hash, .commit-row time { color: var(--muted); font-family: ui-monospace, SFMono-Regular, Consolas, monospace; font-size: 12px; }
59 .commit-row time { font-family: inherit; white-space: nowrap; }
60 .tree-rows { width: 100%; }
61 .tree-row { min-height: 42px; display: grid; grid-template-columns: 24px minmax(180px, 2fr) minmax(120px, 3fr) 75px; align-items: center; gap: 8px; padding: 6px 16px; color: var(--text); border-top: 1px solid var(--border-muted); }
62 .tree-row:first-child { border-top: 0; }
63 .tree-row:hover { background: #f6f8fa; text-decoration: none; }
64 .file-icon { color: var(--muted); font-size: 16px; }
65 .file-icon.folder { color: #54aeff; }
66 .file-name { color: var(--blue); font-weight: 500; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
67 .row-message { color: var(--muted); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
68 .file-size { color: var(--muted); text-align: right; font-size: 12px; white-space: nowrap; }
69 .parent-row { grid-template-columns: 24px 1fr 1fr 75px; }
70 .readme-card { margin-top: 18px; }
71 .card-heading { height: 46px; display: flex; align-items: center; gap: 9px; padding: 0 16px; border-bottom: 1px solid var(--border); font-weight: 650; }
72 .markdown-body { padding: 30px 32px; font-size: 16px; line-height: 1.6; overflow-wrap: anywhere; }
73 .markdown-body > :first-child { margin-top: 0 !important; }
74 .markdown-body > :last-child { margin-bottom: 0 !important; }
75 .markdown-body h1, .markdown-body h2 { border-bottom: 1px solid var(--border-muted); padding-bottom: .3em; }
76 .markdown-body h1 { font-size: 2em; }
77 .markdown-body h2 { font-size: 1.5em; }
78 .markdown-body h3 { font-size: 1.25em; }
79 .markdown-body h1, .markdown-body h2, .markdown-body h3 { margin: 24px 0 16px; line-height: 1.25; }
80 .markdown-body img { max-width: 100%; }
81 .markdown-body code { padding: .2em .4em; border-radius: 6px; background: rgba(175,184,193,.2); font: 85% ui-monospace, SFMono-Regular, Consolas, monospace; }
82 .markdown-body pre { padding: 16px; overflow: auto; border-radius: 6px; background: var(--canvas); }
83 .markdown-body pre code { padding: 0; background: transparent; }
84 .markdown-body blockquote { margin-left: 0; padding-left: 1em; color: var(--muted); border-left: .25em solid var(--border); }
85 .sidebar { padding-top: 48px; }
86 .sidebar h2 { font-size: 16px; margin: 0 0 10px; }
87 .sidebar h3 { font-size: 13px; margin: 0 0 8px; }
88 .sidebar p { color: var(--muted); margin: 0; }
89 .sidebar-rule { height: 1px; background: var(--border-muted); margin: 20px 0; }
90 .clone-url { display: block; max-width: 100%; padding: 8px 10px; border-radius: 6px; background: var(--canvas); overflow: hidden; text-overflow: ellipsis; font-size: 11px; white-space: nowrap; }
91 .stat { display: flex; gap: 6px; padding: 3px 0; }
92 .stat span { color: var(--muted); }
93 .blob-card { border-radius: 6px; }
94 .blob-heading { min-height: 54px; padding: 10px 14px; display: flex; align-items: center; gap: 12px; background: var(--canvas); border-bottom: 1px solid var(--border); }
95 .blob-heading > div:first-child { min-width: 0; display: flex; flex-direction: column; }
96 .blob-heading strong { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
97 .blob-heading span { color: var(--muted); font-size: 12px; }
98 .blob-actions { margin-left: auto; display: flex; gap: 6px; }
99 .blob-actions a, .blob-actions button { min-height: 28px; padding: 3px 10px; cursor: pointer; }
100 .code-view { overflow: auto; font: 12px/1.5 ui-monospace, SFMono-Regular, Consolas, "Liberation Mono", monospace; }
101 .code-view .chroma { margin: 0; padding: 0; background: white; }
102 .code-view .chroma table { width: 100%; border-spacing: 0; }
103 .code-view .chroma td:first-child { width: 1%; min-width: 50px; padding: 8px 10px; color: #6e7781; text-align: right; vertical-align: top; border-right: 1px solid var(--border-muted); user-select: none; }
104 .code-view .chroma td:last-child { padding: 8px 16px; }
105 .code-view pre { margin: 0; }
106 .image-preview { padding: 32px; display: grid; place-items: center; min-height: 260px; background: repeating-conic-gradient(#f6f8fa 0 25%, #fff 0 50%) 50%/20px 20px; }
107 .image-preview img { max-width: 100%; max-height: 70vh; }
108 .binary-notice { padding: 70px 20px; display: flex; flex-direction: column; align-items: center; color: var(--muted); }
109 .binary-notice > span { font-size: 44px; }
110 .binary-notice strong { color: var(--text); font-size: 16px; }
111 .binary-notice p { margin: 5px 0 14px; }
112 footer { min-height: 90px; max-width: 1232px; margin: 0 auto; border-top: 1px solid var(--border-muted); color: var(--muted); display: flex; justify-content: center; gap: 28px; align-items: center; font-size: 12px; }
113 @media (max-width: 800px) {
114 .topbar { padding: 0 16px; }.top-search,.top-link { display: none; }
115 .repo-head { padding-left: 16px; padding-right: 16px; }
116 .content-shell { padding: 0 12px; margin-top: 16px; }
117 .repo-grid { grid-template-columns: 1fr; }.sidebar { padding-top: 0; }
118 .commit-row { grid-template-columns: 32px auto 1fr; }.commit-hash,.commit-row time { display: none; }
119 .tree-row { grid-template-columns: 22px minmax(100px,1fr) 70px; }.row-message { display: none; }
120 .markdown-body { padding: 22px 18px; }
121 .toolbar-path { align-items: flex-start; flex-direction: column; }
122 footer { flex-direction: column; gap: 4px; }
123 }`
Addedinternal/site/template.go +112−0
@@ -0,0 +1,112 @@
1 package site
2
3 const pageTemplate = `<!doctype html>
4 <html lang="en">
5 <head>
6 <meta charset="utf-8">
7 <meta name="viewport" content="width=device-width, initial-scale=1">
8 <meta name="color-scheme" content="light">
9 <title>{{.Title}}</title>
10 <link rel="stylesheet" href="{{.BaseURL}}/.dumbforge/assets/site.css">
11 </head>
12 <body>
13 <header class="topbar">
14 <a class="brand" href="{{.BaseURL}}" aria-label="dumbforge home">
15 <span class="brand-mark">D</span><span>dumbforge</span>
16 </a>
17 <div class="top-search">Search this repository</div>
18 <a class="top-link" href="https://git-scm.com/docs/git-clone">Docs</a>
19 </header>
20
21 <main>
22 <section class="repo-head">
23 <div class="repo-title">
24 <span class="repo-icon">⌘</span>
25 <a href="{{.BaseURL}}">{{.Owner}}</a><span class="slash">/</span><a class="repo-name" href="{{.BaseURL}}">{{.RepoName}}</a>
26 <span class="visibility">Public</span>
27 </div>
28 <nav class="tabs">
29 <a class="tab active" href="{{.BaseURL}}"><span>‹›</span> Code</a>
30 <a class="tab" href="{{.BaseURL}}"><span>◷</span> {{.CommitCount}} commits</a>
31 </nav>
32 </section>
33
34 <div class="content-shell">
35 {{if eq .Kind "repo"}}
36 <div class="repo-grid">
37 <section class="primary">
38 <div class="toolbar">
39 <span class="branch-button"><span class="branch-icon">⑂</span>{{.Branch}}<span class="chevron">⌄</span></span>
40 <button class="clone-button" data-copy="{{.CloneURL}}"><span>▣</span> Clone</button>
41 </div>
42 {{template "tree" .}}
43 {{if .Readme}}
44 <article class="readme-card">
45 <div class="card-heading"><span>☰</span>{{.ReadmeName}}</div>
46 <div class="markdown-body">{{.Readme}}</div>
47 </article>
48 {{end}}
49 </section>
50 <aside class="sidebar">
51 <h2>About</h2>
52 <p>A Git repository hosted directly from an object-storage bucket.</p>
53 <div class="sidebar-rule"></div>
54 <h3>Clone over HTTPS</h3>
55 <code class="clone-url">{{.CloneURL}}</code>
56 <div class="sidebar-rule"></div>
57 <div class="stat"><strong>{{.CommitCount}}</strong><span>commits</span></div>
58 <div class="stat"><strong>{{len .Entries}}</strong><span>top-level items</span></div>
59 </aside>
60 </div>
61 {{else if eq .Kind "tree"}}
62 <section class="primary full-width">
63 <div class="toolbar toolbar-path">
64 <span class="branch-button"><span class="branch-icon">⑂</span>{{.Branch}}<span class="chevron">⌄</span></span>
65 <div class="breadcrumbs">{{range $i, $crumb := .Breadcrumbs}}{{if $i}}<span>/</span>{{end}}<a href="{{$crumb.URL}}">{{$crumb.Name}}</a>{{end}}</div>
66 </div>
67 {{template "tree" .}}
68 </section>
69 {{else}}
70 <section class="primary full-width">
71 <div class="toolbar toolbar-path">
72 <span class="branch-button"><span class="branch-icon">⑂</span>{{.Branch}}<span class="chevron">⌄</span></span>
73 <div class="breadcrumbs">{{range $i, $crumb := .Breadcrumbs}}{{if $i}}<span>/</span>{{end}}<a href="{{$crumb.URL}}">{{$crumb.Name}}</a>{{end}}</div>
74 </div>
75 <div class="blob-card">
76 <div class="blob-heading">
77 <div><strong>{{.Path}}</strong><span>{{humanSize .FileSize}} · {{.Language}}</span></div>
78 <div class="blob-actions"><a href="{{.RawURL}}">Raw</a><button data-copy="{{.RawURL}}">Copy link</button></div>
79 </div>
80 {{if .IsImage}}<div class="image-preview"><img src="{{.RawURL}}" alt="{{.Path}}"></div>
81 {{else if .Code}}<div class="code-view">{{.Code}}</div>
82 {{else}}<div class="binary-notice"><span>◫</span><strong>Preview unavailable</strong><p>This file is binary or too large to render safely.</p><a href="{{.RawURL}}">Download raw file</a></div>{{end}}
83 </div>
84 </section>
85 {{end}}
86 </div>
87 </main>
88 <footer><span>Forged from a static bucket.</span><span>Git data and pages share one URL.</span></footer>
89 <script>document.querySelectorAll('[data-copy]').forEach(function(el){el.addEventListener('click',function(){navigator.clipboard.writeText(el.getAttribute('data-copy'));var old=el.textContent;el.textContent='Copied';setTimeout(function(){el.textContent=old},1200)})})</script>
90 </body>
91 </html>
92
93 {{define "tree"}}
94 <div class="tree-card">
95 <div class="commit-row">
96 <span class="avatar">{{.Commit.Initials}}</span>
97 <strong>{{.Commit.Author}}</strong>
98 <span class="commit-message">{{.Commit.Message}}</span>
99 <a class="commit-hash" href="#">{{.Commit.ShortOID}}</a>
100 <time>{{.Commit.Date}}</time>
101 </div>
102 <div class="tree-rows">
103 {{if .ParentURL}}<a class="tree-row parent-row" href="{{.ParentURL}}"><span class="file-icon folder">↰</span><span class="file-name">..</span><span></span><span></span></a>{{end}}
104 {{range .Entries}}<a class="tree-row" href="{{.URL}}">
105 {{if eq .Type "tree"}}<span class="file-icon folder">▰</span>{{else}}<span class="file-icon">▤</span>{{end}}
106 <span class="file-name">{{.Name}}</span>
107 <span class="row-message">{{$.Commit.Message}}</span>
108 <span class="file-size">{{humanSize .Size}}</span>
109 </a>{{end}}
110 </div>
111 </div>
112 {{end}}`
Addednpm/launcher/dumbforge.cjs +3−0
@@ -0,0 +1,3 @@
1 #!/usr/bin/env node
2
3 require("./run.cjs")(false);
Addednpm/launcher/git-remote-dumbforge.cjs +3−0
@@ -0,0 +1,3 @@
1 #!/usr/bin/env node
2
3 require("./run.cjs")(true);
Addednpm/launcher/run.cjs +48−0
@@ -0,0 +1,48 @@
1 "use strict";
2
3 const path = require("node:path");
4 const { spawnSync } = require("node:child_process");
5
6 const packages = {
7 "darwin-arm64": "@dumbforge/darwin-arm64",
8 "darwin-x64": "@dumbforge/darwin-x64",
9 "linux-arm64": "@dumbforge/linux-arm64",
10 "linux-x64": "@dumbforge/linux-x64",
11 "win32-arm64": "@dumbforge/win32-arm64",
12 "win32-x64": "@dumbforge/win32-x64"
13 };
14
15 module.exports = function run(remoteHelper) {
16 const target = `${process.platform}-${process.arch}`;
17 const packageName = packages[target];
18 if (!packageName) {
19 console.error(`dumbforge: unsupported platform ${target}`);
20 process.exit(1);
21 }
22
23 let packageRoot;
24 try {
25 packageRoot = path.dirname(require.resolve(`${packageName}/package.json`));
26 } catch (error) {
27 console.error(
28 `dumbforge: the native package ${packageName} is missing. ` +
29 "Reinstall without --no-optional."
30 );
31 process.exit(1);
32 }
33
34 const executable = path.join(
35 packageRoot,
36 "bin",
37 process.platform === "win32" ? "dumbforge.exe" : "dumbforge"
38 );
39 const args = remoteHelper
40 ? ["__remote-helper", ...process.argv.slice(2)]
41 : process.argv.slice(2);
42 const result = spawnSync(executable, args, { stdio: "inherit" });
43 if (result.error) {
44 console.error(`dumbforge: ${result.error.message}`);
45 process.exit(1);
46 }
47 process.exit(result.status === null ? 1 : result.status);
48 };
Addednpm/platforms/darwin-arm64/package.json +10−0
@@ -0,0 +1,10 @@
1 {
2 "name": "@dumbforge/darwin-arm64",
3 "version": "0.1.0",
4 "description": "dumbforge native binary for macOS ARM64",
5 "license": "MIT",
6 "os": ["darwin"],
7 "cpu": ["arm64"],
8 "files": ["bin"],
9 "publishConfig": {"access": "public"}
10 }
Addednpm/platforms/darwin-x64/package.json +10−0
@@ -0,0 +1,10 @@
1 {
2 "name": "@dumbforge/darwin-x64",
3 "version": "0.1.0",
4 "description": "dumbforge native binary for macOS x64",
5 "license": "MIT",
6 "os": ["darwin"],
7 "cpu": ["x64"],
8 "files": ["bin"],
9 "publishConfig": {"access": "public"}
10 }
Addednpm/platforms/linux-arm64/package.json +10−0
@@ -0,0 +1,10 @@
1 {
2 "name": "@dumbforge/linux-arm64",
3 "version": "0.1.0",
4 "description": "dumbforge native binary for Linux ARM64",
5 "license": "MIT",
6 "os": ["linux"],
7 "cpu": ["arm64"],
8 "files": ["bin"],
9 "publishConfig": {"access": "public"}
10 }
Addednpm/platforms/linux-x64/package.json +10−0
@@ -0,0 +1,10 @@
1 {
2 "name": "@dumbforge/linux-x64",
3 "version": "0.1.0",
4 "description": "dumbforge native binary for Linux x64",
5 "license": "MIT",
6 "os": ["linux"],
7 "cpu": ["x64"],
8 "files": ["bin"],
9 "publishConfig": {"access": "public"}
10 }
Addednpm/platforms/win32-arm64/package.json +10−0
@@ -0,0 +1,10 @@
1 {
2 "name": "@dumbforge/win32-arm64",
3 "version": "0.1.0",
4 "description": "dumbforge native binary for Windows ARM64",
5 "license": "MIT",
6 "os": ["win32"],
7 "cpu": ["arm64"],
8 "files": ["bin"],
9 "publishConfig": {"access": "public"}
10 }
Addednpm/platforms/win32-x64/package.json +10−0
@@ -0,0 +1,10 @@
1 {
2 "name": "@dumbforge/win32-x64",
3 "version": "0.1.0",
4 "description": "dumbforge native binary for Windows x64",
5 "license": "MIT",
6 "os": ["win32"],
7 "cpu": ["x64"],
8 "files": ["bin"],
9 "publishConfig": {"access": "public"}
10 }
Addedpackage.json +37−0
@@ -0,0 +1,37 @@
1 {
2 "name": "dumbforge",
3 "version": "0.1.0",
4 "description": "Push Git repositories to S3-compatible buckets and clone them over static dumb HTTP",
5 "license": "MIT",
6 "repository": {
7 "type": "git",
8 "url": "git+https://github.com/npfaro/dumbforge.git"
9 },
10 "keywords": [
11 "git",
12 "s3",
13 "cloudflare-r2",
14 "object-storage",
15 "git-remote-helper"
16 ],
17 "files": [
18 "npm/launcher",
19 "README.md",
20 "LICENSE"
21 ],
22 "bin": {
23 "dumbforge": "npm/launcher/dumbforge.cjs",
24 "git-remote-dumbforge": "npm/launcher/git-remote-dumbforge.cjs"
25 },
26 "optionalDependencies": {
27 "@dumbforge/darwin-arm64": "0.1.0",
28 "@dumbforge/darwin-x64": "0.1.0",
29 "@dumbforge/linux-arm64": "0.1.0",
30 "@dumbforge/linux-x64": "0.1.0",
31 "@dumbforge/win32-arm64": "0.1.0",
32 "@dumbforge/win32-x64": "0.1.0"
33 },
34 "engines": {
35 "node": ">=18"
36 }
37 }
Addedscripts/build-npm-packages.sh +35−0
@@ -0,0 +1,35 @@
1 #!/usr/bin/env bash
2 set -euo pipefail
3
4 version="${1:-0.1.0}"
5 project_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
6 output_dir="$project_root/dist/npm"
7 mkdir -p "$output_dir"
8
9 build_target() {
10 local goos="$1"
11 local goarch="$2"
12 local npm_platform="$3"
13 local suffix="$4"
14 local package_dir="$project_root/npm/platforms/$npm_platform"
15 mkdir -p "$package_dir/bin"
16 CGO_ENABLED=0 GOOS="$goos" GOARCH="$goarch" go build \
17 -trimpath \
18 -ldflags "-s -w -X main.version=$version" \
19 -o "$package_dir/bin/dumbforge$suffix" \
20 "$project_root/cmd/dumbforge"
21 }
22
23 build_target darwin arm64 darwin-arm64 ""
24 build_target darwin amd64 darwin-x64 ""
25 build_target linux arm64 linux-arm64 ""
26 build_target linux amd64 linux-x64 ""
27 build_target windows arm64 win32-arm64 ".exe"
28 build_target windows amd64 win32-x64 ".exe"
29
30 for package_dir in "$project_root"/npm/platforms/*; do
31 npm pack "$package_dir" --pack-destination "$output_dir"
32 done
33 npm pack "$project_root" --pack-destination "$output_dir"
34
35 echo "npm packages written to $output_dir"