Airflow upgrade on GCP Composer

manager Fri, 04/09/2021 - 15:32

I get the following error when trying to upgrade Composer versions on GCP

UPDATE operation on this environment failed 1 hour ago with the following error message:
Failed to update image version. The Cloud Build image build failed: Build failed; check build logs for details

The logs for this are in the Cloud builder but they have no useful information. Turns out I had to remove the PYPI packages before the upgrade.  For more information see:

This guide in Composer Docs https://cloud.google.com/composer/docs/how-to/managing/updating that gave me some clue about what was going on.

Then I succeeded after doing this:

  1. ensured that these conditions from the link above were met
  2. became Admin, I was editor (that should have permission), but I became Admin to eliminate permissions doubts
  3. removed all PYPI packages from the environment