Been digging into this lately and figured I'd share, since it cleared up something I was fuzzy on: how BigQuery actually discovers Iceberg tables, and what's genuinely new here vs. what's just been repackaged.
The old ways of doing this:
- Pointing BigQuery straight at a metadata file. Something like
gs://bucket/orders/metadata/00003.metadata.json. It works, but it's fragile. Iceberg writes new metadata files instead of editing the old ones, so if any other engine commits a new snapshot, your pointer is instantly stale. You're stuck manually bumping it to 00004.metadata.json every time.
- Catalog based access, like using AWS Glue as an external catalog. This actually fixes the staleness problem since the catalog tracks current state for you. But every engine still needed its own custom integration to talk to that catalog.
What's actually new: Google now has a managed Lakehouse Runtime Catalog (BigLake) that exposes a real Apache Iceberg REST Catalog endpoint, using the same open REST Catalog spec other engines already speak. So instead of building a one off integration per engine, anything that already understands Iceberg REST can just talk to BigLake directly. The data itself never moves, it stays in Cloud Storage. BigLake is really just a metadata and governance layer with IAM sitting on top of it.
There's also a version detail worth knowing: Iceberg 1.10 shipped a native BigQueryMetastoreCatalog and BigQueryMetastoreClient, plus a GoogleAuthManager that handles Google credential auth inside Iceberg's REST auth framework. It's easy to conflate these two things, but they're not the same. One's a catalog implementation built for BigQuery Metastore specifically, the other is the standardized REST interface. Google's steering people toward the REST endpoint for anything new where cross engine interoperability matters.
The post I read also gets into the practical setup side: creating a BigLake catalog (single bucket vs multi bucket), the IAM roles Google recommends (BigLake Admin, Editor, Viewer, plus Storage Object User), and shows an open source tool called OLake Go (full disclosure, I work on this) writing Iceberg tables directly into that catalog so BigQuery can see them right away, queryable through the usual four part PROJECT.CATALOG.NAMESPACE.TABLE identifier with no manual metadata pointer updates. There's a decent troubleshooting section too. Apparently if your connection test passes but the actual sync fails, it's almost always a missing storage level IAM role rather than bad credentials.
Full post here if you want the details: https://olake.io/blog/biglake-iceberg-rest-catalog-olake-go-setup/
Curious if anyone here has already moved existing Iceberg on BigQuery setups over to the REST catalog, or if you're all still doing the manual metadata file dance like I was.
