Skip to content

Commit 855cdf0

Browse files
authored
Offer protoc's language targets alongside protobuf-net's (#3)
The old ASP.NET site generated C++, Java, Objective-C, PHP, Python and Ruby by shelling out to protoc; the static rewrite dropped them, because protoc is a native executable and there is no maintained WebAssembly build of the compiler. This brings them back through protoc.protobuf-net.dev, a Cloudflare Worker in front of a container that runs the real thing - plus Kotlin and Google's own C# generator, which is a genuinely different answer to the same question as the C# sitting above it in the picker. The language picker now has two groups, and the values are namespaced (protoc:csharp) because protoc has a csharp generator too. Choosing a protoc target generates once - choosing it is the decision to use it - and after that editing marks the output stale rather than posting every keystroke to somebody else's server. The header badge and the footer say which of the two is in play; "runs locally" was true of everything here until now and should not quietly stop being true. The service answers with the same GenerateResponse shape the WebAssembly path returns, so both paths share applyResponse: one code path for output, tabs, diagnostics and the editor gutter. Diagnostics against an imported file are shown but not projected onto the editor, since their line numbers point into a file that is not on screen. Imports need no plumbing here: the service carries the same .proto corpus that protobuf-net.Reflection embeds, so google/** and protobuf-net/** resolve the same way on both paths. Output highlighting reuses the legacy modes already present for C# - clike covers C++, Java, Kotlin and Objective-C - plus python and ruby, which is +14 kB raw and +5 kB gzipped on the bundle. PHP gets no tokenizer: legacy-modes has no PHP mode and lang-php would drag in the HTML, CSS and JavaScript parsers to colour output that is read once and copied.
1 parent 7570519 commit 855cdf0

7 files changed

Lines changed: 383 additions & 40 deletions

File tree

‎README.md‎

Lines changed: 45 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,15 @@
11
# protobuf-net.dev
22

33
The site behind **[protobuf-net.dev](https://protobuf-net.dev)**: generate C# and
4-
VB.NET from `.proto` schemas, and pull apart raw protobuf payloads without a schema.
4+
VB.NET from `.proto` schemas, reach `protoc` for the languages Google's compiler provides, and
5+
pull apart raw protobuf payloads without a schema.
56

6-
Everything runs in the browser. There is no server, no API and no upload — schemas and payloads
7-
never leave the machine they were pasted on. The site is a folder of static files.
7+
The site is a folder of static files. C#, VB.NET and payload decoding run in the browser through
8+
WebAssembly, so a schema or payload pasted into either of those never leaves the machine it was
9+
pasted on. The `protoc` targets are the exception and cannot be otherwise — `protoc` is a native
10+
executable — so they are compiled by
11+
[protoc.protobuf-net.dev](https://github.com/protobuf-net/protoc.protobuf-net.dev), and only when
12+
the user picks one of those languages and presses Generate.
813

914
This replaces the older ASP.NET-hosted site that lived in the
1015
[protobuf-net](https://github.com/protobuf-net/protobuf-net) repo under `src/protogen.site`.
@@ -40,6 +45,11 @@ npm test # vitest, unit tests for the payload parsing
4045
npm run build # wasm + typecheck + tests + bundle -> web/dist
4146
```
4247

48+
The `protoc` targets point at the live service by default. To develop against a local one, set
49+
`VITE_PROTOC_ENDPOINT` — `VITE_PROTOC_ENDPOINT=http://localhost:8787 npm run dev` alongside
50+
`wrangler dev` in the
51+
[service repo](https://github.com/protobuf-net/protoc.protobuf-net.dev).
52+
4353
`npm run wasm` alone re-publishes the .NET side into `web/public/_framework`. That folder is
4454
generated and git-ignored; it is copied verbatim rather than bundled, because the .NET boot process
4555
resolves its own content-hashed filenames.
@@ -152,20 +162,42 @@ are live.
152162

153163
## Scope
154164

155-
**C# and VB.NET only.** The old site also offered C++, Java, JavaScript, Objective-C, PHP, Python
156-
and Ruby by shelling out to `protoc` on the server. `protoc` is a native executable and cannot run
157-
client-side, so those targets are gone rather than quietly broken.
158-
159-
Compiling `protoc` itself to WebAssembly was investigated and parked. There is no maintained build:
160-
`kwonoj/protobuf-wasm` and `mjz20/protobuf_wasm` patch the protobuf *runtime* for Emscripten and
161-
explicitly do not build the compiler, and the one recent attempt at the compiler proper
165+
**C# and VB.NET locally; everything else through `protoc`.** protobuf-net generates the first two,
166+
in the browser. C++, Java, Kotlin, Objective-C, PHP, Python, Ruby and Google's own C# come from
167+
`protoc`, which is a native executable and cannot run client-side, so they are compiled by
168+
[protoc.protobuf-net.dev](https://github.com/protobuf-net/protoc.protobuf-net.dev) — a Cloudflare
169+
Worker in front of a container. That repository documents the service; what matters here is the
170+
contract:
171+
172+
- `web/src/protoc.ts` holds the target list, the endpoint and the response reader. Adding a target
173+
protoc already supports means one entry there and one in the service's own table.
174+
- The service answers with the same `GenerateResponse` shape the WebAssembly path returns, so both
175+
render through `applyResponse` in `web/src/schema.ts` and there is one code path for output,
176+
diagnostics and the editor gutter.
177+
- Nothing is sent while typing. Choosing a `protoc` target generates once, because choosing it is
178+
the decision to use it; after that the output is marked stale on edit and re-sent only when the
179+
user presses Generate.
180+
- The service carries the same `.proto` corpus that `protobuf-net.Reflection` embeds, so imports of
181+
`google/**` and `protobuf-net/**` resolve identically on both paths. Nothing about imports is
182+
sent from here.
183+
184+
Compiling `protoc` itself to WebAssembly — which would have avoided the server altogether — was
185+
investigated and parked. There is no maintained build: `kwonoj/protobuf-wasm` and
186+
`mjz20/protobuf_wasm` patch the protobuf *runtime* for Emscripten and explicitly do not build the
187+
compiler, and the one recent attempt at the compiler proper
162188
([protobuf#20819](https://github.com/protocolbuffers/protobuf/issues/20819)) produces a `protoc.js`
163189
that builds but fails code generation on path resolution. The real work is MEMFS plumbing for
164190
`--proto_path` and generated outputs, unmaintained upstream, on top of a C++ binary that links
165-
every language generator and would likely dwarf the .NET runtime.
191+
every language generator and would likely dwarf the .NET runtime. If that ever lands, the service
192+
becomes unnecessary rather than merely unfortunate.
193+
194+
JavaScript and TypeScript are on neither list: `protoc` dropped its JavaScript generator in v21.
195+
`protobufjs` and `@bufbuild/protobuf` generate both in pure JS, so that one belongs back in the
196+
browser if it is ever worth having.
166197

167-
If a target language is ever worth restoring, JavaScript/TypeScript is the cheap one: `protobufjs`
168-
and `@bufbuild/protobuf` generate it in pure JS with no `protoc` involved.
198+
PHP output is not syntax-highlighted. `@codemirror/legacy-modes` has no PHP mode, and
199+
`@codemirror/lang-php` brings the HTML, CSS and JavaScript parsers with it — too much weight for
200+
colouring output that is read once and copied.
169201

170202
See [`docs/trimming.md`](docs/trimming.md) for why the build suppresses `IL2104`.
171203

‎web/index.html‎

Lines changed: 27 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
<title>protobuf-net — protobuf schema codegen and payload analysis</title>
77
<meta
88
name="description"
9-
content="Generate C# and VB.NET from .proto schemas, and pull apart raw protobuf payloads without a schema. Runs entirely in your browser."
9+
content="Generate C# and VB.NET from .proto schemas in your browser, reach protoc for C++, Java, Python and the rest, and pull apart raw protobuf payloads without a schema."
1010
/>
1111
<link rel="icon" href="/protobuf-net-mark.svg" type="image/svg+xml" />
1212
</head>
@@ -21,7 +21,8 @@
2121
<a href="#decode" data-view="decode">Decode payload</a>
2222
</nav>
2323
<div class="header-right">
24-
<span class="privacy" title="Nothing you paste leaves your browser">runs locally</span>
24+
<!-- says where the current target runs; schema.ts flips it when a protoc target is picked -->
25+
<span class="privacy" id="privacy-badge" title="Nothing you paste leaves your browser">runs locally</span>
2526
<a href="https://docs.protobuf-net.dev/" rel="noopener">Docs</a>
2627
<a href="https://github.com/protobuf-net/protobuf-net" rel="noopener">GitHub</a>
2728
</div>
@@ -34,29 +35,39 @@
3435
<label>
3536
Language
3637
<select name="language" id="opt-language">
37-
<option value="csharp">C#</option>
38-
<option value="vb">VB.NET</option>
38+
<optgroup label="protobuf-net — in your browser">
39+
<option value="csharp">C#</option>
40+
<option value="vb">VB.NET</option>
41+
</optgroup>
42+
<!-- filled from PROTOC_LANGUAGES, so the picker and the service cannot disagree -->
43+
<optgroup label="protoc — on protoc.protobuf-net.dev" id="opt-language-protoc"></optgroup>
3944
</select>
4045
</label>
41-
<label id="opt-langver-wrap">
46+
<button type="button" class="ghost remote-only" id="run-protoc">Generate</button>
47+
<span class="note remote-only">
48+
compiled by <code>protoc</code> on
49+
<a href="https://protoc.protobuf-net.dev" rel="noopener">protoc.protobuf-net.dev</a> —
50+
your schema is sent there when you press Generate
51+
</span>
52+
<label id="opt-langver-wrap" class="net-only">
4253
Version
4354
<select name="languageVersion" id="opt-langver"></select>
4455
</label>
45-
<label>
56+
<label class="net-only">
4657
Naming
4758
<select name="namingConvention" id="opt-naming">
4859
<option value="auto">Auto (.NET conventions)</option>
4960
<option value="noplural">Auto, no pluralising</option>
5061
<option value="original">Original (as written)</option>
5162
</select>
5263
</label>
53-
<label class="check"><input type="checkbox" name="services" checked /> services</label>
54-
<label class="check"><input type="checkbox" name="oneOfEnum" /> <code>oneof</code> as enum</label>
55-
<label class="check"><input type="checkbox" name="listSet" /> <code>repeated</code> with setters</label>
56-
<label class="check"><input type="checkbox" name="repeatedAsList" /> <code>repeated</code> as list</label>
57-
<label class="check"><input type="checkbox" name="nullableValueType" /> nullable value types</label>
58-
<label class="check"><input type="checkbox" name="disableNullWrappers" /> no null wrappers</label>
59-
<label class="check"><input type="checkbox" name="disableCompatLevel" /> no <code>CompatibilityLevel</code></label>
64+
<label class="check net-only"><input type="checkbox" name="services" checked /> services</label>
65+
<label class="check net-only"><input type="checkbox" name="oneOfEnum" /> <code>oneof</code> as enum</label>
66+
<label class="check net-only"><input type="checkbox" name="listSet" /> <code>repeated</code> with setters</label>
67+
<label class="check net-only"><input type="checkbox" name="repeatedAsList" /> <code>repeated</code> as list</label>
68+
<label class="check net-only"><input type="checkbox" name="nullableValueType" /> nullable value types</label>
69+
<label class="check net-only"><input type="checkbox" name="disableNullWrappers" /> no null wrappers</label>
70+
<label class="check net-only"><input type="checkbox" name="disableCompatLevel" /> no <code>CompatibilityLevel</code></label>
6071
</form>
6172

6273
<div class="split">
@@ -125,7 +136,9 @@
125136
<footer class="app-footer">
126137
<span id="engine-version">loading engine…</span>
127138
<span>
128-
Everything runs in your browser via WebAssembly — no payload or schema is uploaded.
139+
C#, VB.NET and payload decoding run in your browser via WebAssembly and are never uploaded.
140+
protoc's own languages are compiled on
141+
<a href="https://protoc.protobuf-net.dev" rel="noopener">protoc.protobuf-net.dev</a>.
129142
</span>
130143
</footer>
131144

‎web/src/editor.ts‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,27 @@ import { StreamLanguage, syntaxHighlighting, bracketMatching } from '@codemirror
66
import { highlightStyle } from './highlight';
77
import { lintGutter, setDiagnostics, type Diagnostic } from '@codemirror/lint';
88
import { protobuf } from './protobufMode';
9-
import { csharp } from '@codemirror/legacy-modes/mode/clike';
9+
import { csharp, cpp, java, kotlin, objectiveC } from '@codemirror/legacy-modes/mode/clike';
1010
import { vb } from '@codemirror/legacy-modes/mode/vb';
11+
import { python } from '@codemirror/legacy-modes/mode/python';
12+
import { ruby } from '@codemirror/legacy-modes/mode/ruby';
1113
import type { SchemaError } from './types';
1214

15+
// protobuf-net generates the first two; the rest arrive from protoc and are read-only output like
16+
// any other, so they need a tokenizer and nothing else.
1317
const languages = {
1418
protobuf: StreamLanguage.define(protobuf),
1519
csharp: StreamLanguage.define(csharp),
1620
vb: StreamLanguage.define(vb),
21+
cpp: StreamLanguage.define(cpp),
22+
java: StreamLanguage.define(java),
23+
kotlin: StreamLanguage.define(kotlin),
24+
objectiveC: StreamLanguage.define(objectiveC),
25+
// deliberately unhighlighted: @codemirror/legacy-modes has no PHP mode, and @codemirror/lang-php
26+
// brings the HTML, CSS and JavaScript parsers with it to colour output nobody edits here
27+
php: [] as Extension,
28+
python: StreamLanguage.define(python),
29+
ruby: StreamLanguage.define(ruby),
1730
} as const;
1831

1932
export type LanguageName = keyof typeof languages;

‎web/src/protoc.test.ts‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
import { describe, expect, it } from 'vitest';
2+
import { PROTOC_LANGUAGES, protocLanguage, protocValue, readResponse } from './protoc';
3+
4+
describe('language values', () => {
5+
it('round-trips every target', () => {
6+
for (const language of PROTOC_LANGUAGES) {
7+
expect(protocLanguage(protocValue(language))).toBe(language);
8+
}
9+
});
10+
11+
it('does not claim the local generators', () => {
12+
// protoc has a csharp generator too, which is exactly why the values are namespaced
13+
expect(protocLanguage('csharp')).toBeUndefined();
14+
expect(protocLanguage('vb')).toBeUndefined();
15+
});
16+
17+
it('ignores a target it does not offer', () => {
18+
expect(protocLanguage('protoc:fortran')).toBeUndefined();
19+
});
20+
});
21+
22+
describe('readResponse', () => {
23+
it('reads a successful generation', () => {
24+
const response = readResponse(
25+
200,
26+
JSON.stringify({ files: [{ name: 'my_pb2.py', text: 'x = 1' }], errors: [] }),
27+
);
28+
expect(response.files).toEqual([{ name: 'my_pb2.py', text: 'x = 1' }]);
29+
expect(response.exception).toBeUndefined();
30+
});
31+
32+
it('reads diagnostics, which are an answer rather than a failure', () => {
33+
const response = readResponse(
34+
200,
35+
JSON.stringify({
36+
files: [],
37+
errors: [{ isError: true, lineNumber: 2, columnNumber: 3, message: 'Expected ";".' }],
38+
}),
39+
);
40+
expect(response.errors).toHaveLength(1);
41+
expect(response.exception).toBeUndefined();
42+
});
43+
44+
it('keeps an exception the service reports', () => {
45+
const response = readResponse(429, JSON.stringify({ files: [], errors: [], exception: 'too many requests' }));
46+
expect(response.exception).toBe('too many requests');
47+
});
48+
49+
it('survives a reply that is not the service at all', () => {
50+
const response = readResponse(502, '<html>Bad gateway</html>');
51+
expect(response.files).toEqual([]);
52+
expect(response.exception).toContain('502');
53+
expect(response.exception).toContain('Bad gateway');
54+
});
55+
56+
it('survives an empty body', () => {
57+
expect(readResponse(504, '').exception).toContain('504');
58+
});
59+
});

‎web/src/protoc.ts‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
import type { GenerateResponse } from './types';
2+
import type { LanguageName } from './editor';
3+
4+
/**
5+
* The language targets that come from Google's compiler rather than from protobuf-net.
6+
*
7+
* protoc is a native executable, so unlike everything else on this site these cannot run in the
8+
* browser: the schema is posted to protoc.protobuf-net.dev, which runs protoc in a container and
9+
* hands back what it wrote. That service carries the same .proto corpus protobuf-net.Reflection
10+
* embeds, so an import that resolves locally resolves there too.
11+
*
12+
* See https://github.com/protobuf-net/protoc.protobuf-net.dev.
13+
*/
14+
const ENDPOINT: string = import.meta.env['VITE_PROTOC_ENDPOINT'] ?? 'https://protoc.protobuf-net.dev';
15+
16+
/** The single virtual file the service compiles; it names the diagnostics that come back, too. */
17+
export const SCHEMA_FILE = 'my.proto';
18+
19+
/** Namespaces the picker's values: protoc's `csharp` is not protobuf-net's `csharp`. */
20+
const PREFIX = 'protoc:';
21+
22+
export interface ProtocLanguage {
23+
/** what the service calls it */
24+
id: string;
25+
/** what the picker shows */
26+
label: string;
27+
/** how the output pane highlights it */
28+
editor: LanguageName;
29+
}
30+
31+
export const PROTOC_LANGUAGES: readonly ProtocLanguage[] = [
32+
{ id: 'cpp', label: 'C++', editor: 'cpp' },
33+
// Google's C# generator, which emits Google.Protobuf types — a genuinely different answer to
34+
// the same question as the C# above it, and worth being able to compare against
35+
{ id: 'csharp', label: 'C# (Google.Protobuf)', editor: 'csharp' },
36+
{ id: 'java', label: 'Java', editor: 'java' },
37+
{ id: 'kotlin', label: 'Kotlin', editor: 'kotlin' },
38+
{ id: 'objc', label: 'Objective-C', editor: 'objectiveC' },
39+
{ id: 'php', label: 'PHP', editor: 'php' },
40+
{ id: 'python', label: 'Python', editor: 'python' },
41+
{ id: 'ruby', label: 'Ruby', editor: 'ruby' },
42+
];
43+
44+
/** The picker value for a target, e.g. `protoc:cpp`. */
45+
export function protocValue(language: ProtocLanguage): string {
46+
return PREFIX + language.id;
47+
}
48+
49+
/** The target a picker value names, or undefined when it is one of the local generators. */
50+
export function protocLanguage(value: string): ProtocLanguage | undefined {
51+
if (!value.startsWith(PREFIX)) return undefined;
52+
const id = value.slice(PREFIX.length);
53+
return PROTOC_LANGUAGES.find((language) => language.id === id);
54+
}
55+
56+
/**
57+
* Reads the service's answer. It replies with a GenerateResponse whether the schema compiled or
58+
* not — a schema that does not compile is an answer, not a failure — but anything between here
59+
* and there (a cold start that times out, a proxy, an outage) will reply with something else
60+
* entirely, so the body is not assumed to be JSON.
61+
*/
62+
export function readResponse(status: number, body: string): GenerateResponse {
63+
let parsed: Partial<GenerateResponse> | undefined;
64+
try {
65+
parsed = JSON.parse(body) as Partial<GenerateResponse>;
66+
} catch {
67+
parsed = undefined;
68+
}
69+
70+
if (!parsed || !Array.isArray(parsed.files) || !Array.isArray(parsed.errors)) {
71+
const detail = body.trim().slice(0, 200) || '(no detail)';
72+
return { files: [], errors: [], exception: `protoc.protobuf-net.dev replied ${status}: ${detail}` };
73+
}
74+
return { files: parsed.files, errors: parsed.errors, ...(parsed.exception ? { exception: parsed.exception } : {}) };
75+
}
76+
77+
export async function generateWithProtoc(
78+
language: ProtocLanguage,
79+
schema: string,
80+
signal: AbortSignal,
81+
): Promise<GenerateResponse> {
82+
const response = await fetch(`${ENDPOINT}/generate`, {
83+
method: 'POST',
84+
headers: { 'content-type': 'application/json' },
85+
body: JSON.stringify({
86+
language: language.id,
87+
files: { [SCHEMA_FILE]: schema },
88+
entry: [SCHEMA_FILE],
89+
}),
90+
signal,
91+
});
92+
return readResponse(response.status, await response.text());
93+
}

0 commit comments

Comments
 (0)