From multiple proto files, generate only one file.pb.go

[Emixam23]

Hey folks !

I am creating my first service in go with protobuf and I was wondering if I could generate only one task.pg.go from my 3 different protofiles?

They are in the same package, referencing the same output package.

- Task
- Date
- Content

Task imports the 2 others at the beginning of the file.

import "date.proto";
import "content.proto";

I then generate my task.pb.go from protoc task.proto --go_out=plugins=grpc:.. It works, however, the pb file won't contain the Date messages and the Content messages.

In fact, the output will contains the rpc functions but not the messages from the other files.

service Tasks {
     rpc AddTask (Content) returns (Content) {}
     rpc CompleteTask (Date) returns (Date) {}
}

The generated protofile task.pb.go contains the definitions, so the methods are found by my main.go file, but the messages cannot be find.

Thanks for any help

PS: feel free to ask more details if you need to

Max

[dsnet]

Hi, thanks for the report. This is a duplicate of #39. It's something we may want to address in the near future (no guarantees), but there are currently other protobuf items we're working on. So feel free to subscribe to that issue.

[Emixam23]

Thanks for the fast reply :)

Sorry I didn't notice it when I was searching in the issues..

However, it seems that people are looking for merging different packages when I am trying to just merge different proto files of the same package into only one output.pg.gofile

Am I wrong?

Max

[dsnet]

Sorry. I misread your post. You actually want the contents of all three .proto files to be in the same generated .pb.go file. That is a different issue from #39. I apologize.

[dsnet]

One of the assumptions made by protoc is that you can run it sequentially on each .proto file individually or a set of .proto files and the output be the same. I don't see how you can maintain that property by having multiple .proto files feed into the same .pb.go file.

The best you can do today is to do import public "date.proto", which will forward all declarations in data.proto inside the currently generated file as well. However:

• import public was never designed for this purpose. It's use case is for migration, where you move a .proto file to a new location, but need declarations to continue to exist in the former location.
• If the target .proto file is generating its output to the same Go package, then you will run into problems with conflicting Go declarations.

[Emixam23]

Ok thank you first for reopening :)

It's just about readability, tbh the 3 proto files could be just one, I just prefere to separate my class (or model) from my methods/functions

I will give it a try asap and come back to you

Thanks

Max

[dsnet]

I tested this out and it should satisfy what you want:

$ tree
.
├── bar
│   └── bar.proto
├── foo
│   └── foo.proto
└── test.proto

File "test.proto" has:

syntax = "proto2";

import public "foo/foo.proto";
import public "bar/bar.proto";

message MyMessage {
	optional FooMessage foo = 1;
	optional BarMessage bar = 2;
}

File "foo.proto" has:

syntax = "proto2";
option go_package = "github.com/user/project/foo";
message FooMessage { optional string f = 1; }

File "bar.proto" has:

syntax = "proto2";
option go_package = "github.com/user/project/bar";
message BarMessage { optional string f = 1; }

When compiling with protoc --go_out=. test.proto, you get a file:

package test

import bar "github.com/user/project/bar"
import foo "github.com/user/project/foo"

// FooMessage from public import foo/foo.proto
type FooMessage = foo.FooMessage

// BarMessage from public import bar/bar.proto
type BarMessage = bar.BarMessage

type MyMessage struct {
	Foo                  *foo.FooMessage `protobuf:"bytes,1,opt,name=foo" json:"foo,omitempty"`
	Bar                  *bar.BarMessage `protobuf:"bytes,2,opt,name=bar" json:"bar,omitempty"`
        ...
}

As you can see, this generates test.pb.go with FooMessage and BarMessage as declarations in that local file, but are aliased to declarations in a different file.

The caveats of this:

• It relies on type aliases, which are only available in Go1.9 and above.
• It's an abuse of the original use case for import public.

Since this kinda accomplishes what you want, I'm going to close this issue as a mixture of "working as intended" and "workaround available". I don't think we would ever provide first-class support for the feature since that would probably interact poorly with #39.

[Emixam23]

Hey !

Ok but import public doesn't work, so why are you using it?

protoc --proto_path proto/ proto/api_web_service.proto --go_out=plugins=grpc:proto/

libprotoc 3.5.1

...
 
// This is a compile-time assertion to ensure that this generated file
// is compatible with the proto package it is being compiled against.
// A compilation error at this line likely means your copy of the
// proto package needs to be updated.
const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package

// Ignoring public import of Area from area.proto

// Ignoring public import of Place from place.proto

...

[neild]

"Ignoring public import" means that you're doing an import public of a .proto file that's in the same Go package as the current one, so there's no need to add forwarding references to the imported package.

[dsnet]

If your import paths are sufficiently simple, you may be hitting #646.

I think it will work if you put area.proto and place.proto in sub-directories.

[Emixam23]

@neild But neither "import public" or "import" works if I don't compile as well the other protofile..

The thing is, I have 3 protofile in the example and I want only on generated file, either if I import or not, only the main file is generated, but not the 2 others, I have to add them to the CLI which generates me 3 files instead of just one

@dsnet They are part of the same package actually :/

[neild]

Each input .proto file generates a different output .pb.go. I don't believe this is ever likely to change. It is entirely normal for a Go package to be composed of multiple files.

[Emixam23]

That's why it's weird to me... It means that, if I have an API in Go, and my client in Android, I have to pass all of my proto file (the exchanged one between mobile and API) to my client...

If I have 25 classes, I need to duplicate there 25classes file in Android and Go? Then generate the pg.go and the pb.java