Skip to content
Yuvraj 🧢
Github - yindiaGithub - tqindiaContact

Proto at Scale: Buf, Remote Plugins, and ConnectRPC

— proto, grpc, buf, connectrpc, bsr — 8 min read

I have maintained Protocol Buffers at Flyte and consumed them across services at Union. The schema was never the hard part. The hard part was the machinery around it: generating code for five languages, versioning it, catching breaking changes, distributing it, and consuming it across repositories without copy-paste. This post walks through the three pieces that fixed it for us, with working code: Buf for the schema, remote plugins for code generation, and ConnectRPC for serving. If gRPC itself feels heavy, the seven-minute talk at the end is the simplest explanation of what an RPC actually is.

The problem: proto does not scale as a pile of scripts

A schema-to-code pipeline has a fan-out problem. One .proto file must become idiomatic code in every language a consumer uses, and each target has its own plugin, options, and packaging. Flyte's generation script is the honest version of what most teams start with:

# roughly what a hand-rolled pipeline looks like
protoc -I protos \
  --go_out=gen/go --go_opt=paths=source_relative \
  --python_out=gen/py \
  --js_out=import_style=commonjs,binary:gen/js \
  --java_out=gen/java \
  protos/**/*.proto
generate_protos.sh
                                │
        ┌───────────┬───────────┼───────────┬───────────┐
        ▼           ▼           ▼           ▼           ▼
   protoc-gen-go  py plugins  js plugin   java plugin  c plugin
        │           │           │           │           │
     Go pkg      PyPI pkg    npm pkg     (neglected)   C code

Three failure modes show up fast, and all three cost real engineering time:

  1. Plugin drift. protoc and each protoc-gen-* live on developer machines and CI images. Two people generate slightly different code because they have different plugin versions. Nothing pins them.
  2. No breaking-change gate. Someone renames a field or reuses a field number, the code still compiles for the producer, and a downstream consumer breaks at runtime. The schema has no CI check that understands wire compatibility.
  3. Manual distribution. Every language target is a separate package to publish. The low-attention target gets dropped. At Flyte the Java package was exactly that, so the Flytekit-Java team at Spotify ended up vendoring the protos by hand.

At Union we consumed flyteidl through a git submodule plus bash. A submodule pins a commit, someone forgets to bump it, and now the generated code and the upstream schema silently disagree. The tooling, not gRPC, was the problem.

Part 1: Buf makes the schema a versioned artifact

Buf treats a set of .proto files as a module with a config, a linter, a breaking-change detector, and a registry. The module is declared in buf.yaml:

version: v2
modules:
  - path: protos
    name: buf.build/flyteorg/flyteidl
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE
deps:
  - buf.build/googleapis/googleapis

Two commands turn this into CI gates, and they are the reason to adopt Buf even if you change nothing else.

buf lint enforces a consistent style: package naming, file layout, RPC request/response naming. buf breaking compares your current schema against a baseline (the last published version, or main) and fails on any wire-incompatible change.

buf lint
buf breaking --against '.git#branch=main'

If someone changes a field's type, the build stops with something like:

protos/order/v1/order.proto:12:3: Field "1" with name "id" on message
"GetOrderRequest" changed type from "string" to "int64".

That single check moves a class of production incidents left, into the pull request, where a broken change is a code review comment instead of a pager alert.

The Buf Schema Registry (BSR) is a package registry for the schema itself, the way npm is for JavaScript and PyPI is for Python. A producer runs buf push once. buf.lock pins the exact versions of your dependencies, so builds are reproducible.

buf push   # publish a new version to the BSR

We published flyteidl to the BSR and depended on it from the Union protos. That deleted the submodule and the bash in one move. The shift is the whole point: distribute the schema, and let each consumer generate what it needs.

Producer                BSR                          Consumers
   ────────                ───                          ─────────
   buf push ─────────────▶ buf.build/flyteorg/flyteidl ─▶ Go service:  buf generate → Go
                           (versioned; lint + breaking     Python job:  buf generate → Python
                            checked; deps pinned)           Rust tool:   buf generate → Rust

Part 2: remote plugins make code generation reproducible

To understand why this matters, it helps to know how a Protobuf plugin actually works. protoc (or Buf) parses your schema into a CodeGeneratorRequest and pipes it to a plugin binary named protoc-gen-<name> over stdin. The plugin writes a CodeGeneratorResponse, a set of files, back over stdout. That is the entire contract. go, java, grpc, and connect-go are all just binaries that speak it.

buf generate
        │   CodeGeneratorRequest (parsed schema) ──▶ stdin
        ▼
   protoc-gen-go / protoc-gen-connect-go / ...
        │   CodeGeneratorResponse (generated files) ──▶ stdout
        ▼
   files written under ./gen

The problem with local plugins is that every developer and CI image needs the exact same plugin binaries at the exact same versions, or the output drifts. Buf's remote plugins remove that entirely. You reference a plugin hosted on the BSR, pinned to a version, and Buf sends the input to the BSR's plugin executor, runs the plugin there, and writes the output to your disk. No protoc, no protoc-gen-* installs, and the version lives in the config, not on someone's laptop.

# buf.gen.yaml
version: v2
managed:
  enabled: true
plugins:
  - remote: buf.build/protocolbuffers/go:v1.36.6
    out: gen
    opt: paths=source_relative
  - remote: buf.build/grpc/go:v1.5.1
    out: gen
    opt: paths=source_relative

Two things to note. Every remote: reference is pinned (:v1.36.6), so two machines produce byte-identical code. And managed: enabled: true turns on managed mode, which moves language-specific options like Go package paths out of the .proto files and into this file. The producer's schema stays clean and language-agnostic; each consumer chooses its own namespaces.

Generate straight from the registry, imports included:

buf generate buf.build/flyteorg/flyteidl --include-imports

That input is a BSR module by name. There is no submodule to sync and no plugin chain to keep aligned across a team.

Part 3: ConnectRPC removes the gRPC-versus-REST tax

Schema distribution was one problem. The second was a protocol argument every team seems to have: engineers want to write gRPC, but many consumers, browsers above all, want plain HTTP and JSON.

The usual answer is grpc-gateway. You annotate each RPC with an HTTP mapping, generate a reverse-proxy that translates JSON over HTTP to gRPC, and run that proxy in front of your service. It works, but it is a second server, a second code-generation path, and a set of annotations that drift out of sync with your RPCs.

grpc-gateway:                          ConnectRPC:


   browser ──JSON/HTTP──▶ gateway proxy    browser ─────┐
                              │             grpc client ─┼─▶ one handler
                           gRPC│                          │   gRPC + gRPC-Web + Connect
                              ▼             grpcurl ──────┘   on a single HTTP port
                         gRPC server

ConnectRPC collapses that into one server. A Connect handler speaks three protocols from a single implementation: gRPC, gRPC-Web, and Connect's own protocol, which is ordinary HTTP you can hit with curl. Here is the full path for one service.

The schema:

syntax = "proto3";
package order.v1;
option go_package = "example/gen/order/v1;orderv1";


message GetOrderRequest { string id = 1; }
message GetOrderResponse {
  string id = 1;
  string status = 2;
}


service OrderService {
  rpc GetOrder(GetOrderRequest) returns (GetOrderResponse);
}

Add the connect-go plugin to buf.gen.yaml:

version: v2
plugins:
  - remote: buf.build/protocolbuffers/go:v1.36.6
    out: gen
    opt: paths=source_relative
  - remote: buf.build/connectrpc/go:v1.18.1
    out: gen
    opt: paths=source_relative

buf generate produces gen/order/v1/orderv1connect/order.connect.go with a server interface and an HTTP handler. Implement the interface:

package main


import (
	"context"
	"net/http"


	"connectrpc.com/connect"
	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"


	orderv1 "example/gen/order/v1"
	"example/gen/order/v1/orderv1connect"
)


type OrderServer struct{}


func (s *OrderServer) GetOrder(
	ctx context.Context,
	req *connect.Request[orderv1.GetOrderRequest],
) (*connect.Response[orderv1.GetOrderResponse], error) {
	return connect.NewResponse(&orderv1.GetOrderResponse{
		Id:     req.Msg.Id,
		Status: "SHIPPED",
	}), nil
}


func main() {
	mux := http.NewServeMux()
	path, handler := orderv1connect.NewOrderServiceHandler(&OrderServer{})
	mux.Handle(path, handler)


	// h2c serves HTTP/2 without TLS, so gRPC clients work over cleartext locally.
	_ = http.ListenAndServe("localhost:8080", h2c.NewHandler(mux, &http2.Server{}))
}

That is the whole server: no gateway, no second binary. Now the payoff. The same running process answers a browser-style JSON call over HTTP:

curl \
  --header "Content-Type: application/json" \
  --data '{"id": "A-1001"}' \
  http://localhost:8080/order.v1.OrderService/GetOrder
# {"id":"A-1001","status":"SHIPPED"}

and a gRPC call, on the same port:

grpcurl -plaintext -d '{"id": "A-1001"}' \
  localhost:8080 order.v1.OrderService/GetOrder

Because Connect handlers also speak the gRPC and gRPC-Web protocols, they interoperate with existing gRPC clients, Envoy, and grpcurl, and browsers can call them directly over gRPC-Web without a proxy. Streaming works too. You define the service in proto once, distribute it through the BSR, generate per consumer with pinned remote plugins, and serve it over a protocol both your browsers and your backends can call.

How the pieces fit

schema (.proto)
        │  buf push
        ▼
   BSR module ──────────────┐
        │ buf generate       │ (remote plugins, pinned, reproducible)
        ▼                     ▼
   Go + Python + Rust stubs   connect-go stubs
                                   │
                                   ▼
                        one Connect handler on one port
                          │            │            │
                        curl         grpcurl      browser (gRPC-Web)

If RPC still feels like magic, this talk is the clearest short explanation of what a gRPC server really is under the covers, which is what convinced me the complexity lived in the libraries, not the protocol. It is included purely as a reference.

Takeaways

  • Publish the schema, not the generated code. Producers buf push to the BSR; consumers buf generate what they need.
  • Make buf lint and buf breaking CI gates. Catching a wire-incompatible change in review is far cheaper than in production.
  • Use pinned remote plugins so code generation is reproducible across a team, and managed mode so the schema carries no language-specific options.
  • Use ConnectRPC to serve gRPC, gRPC-Web, and plain HTTP JSON from one handler, and delete the gateway proxy.