Skip to main content

Airbyte 1.4.0

Happy new year! Airbyte version 1.4.0 was released on January 17, 2025. We’re excited to share new improvements and changes to the Airbyte platform.

🚀 Platform Changes​

Platform changes improve Airbyte for everyone with a self-managed instance.

Configure the schema refresh rate​

At the start of a sync, Airbyte occasionally re-runs schema discovery to detect changes in your source, such as new fields or a change in column type. But running schema discovery has a performance cost, and may not always be necessary before every sync.

You may now use a new toggle, DISCOVER_REFRESH_WINDOW_MINUTES, to set the minimum number of minutes Airbyte will wait to refresh the schema for your sources. By setting a larger number, you run automatic schema detection less frequently, which can result in gains in sync performance. The default configuration in Airbyte Open Source is 1440, which refreshes schemas every 24 hours. The lowest interval you can set is 1, which refreshes schemas before every sync.

values.yaml
worker:
  env_vars:
    DISCOVER_REFRESH_WINDOW_MINUTES: 1440 # Airbyte automatically refreshes schemas no more than once per day (1440 minutes).

Set this to 0 to disable automatic schema refreshes. All schema refreshes will need to be done manually via the Airbyte UI or API.

values.yaml
worker:
  env_vars:
    DISCOVER_REFRESH_WINDOW_MINUTES: 0 # Airbyte does not automatically detect schema changes.

Connectors support custom image registries​

Connectors can now use custom image registries rather than Airbyte’s public Docker registry. If you configure Airbyte to use a custom image registry, it now automatically uses that registry for connector images as well as platform images. Previously, only platform images supported this. In this example, we set Airbyte’s values.yaml file to pull all images from GitHub.

values.yaml
global:
  image:
    registry: ghcr.io/NAMESPACE

You must ensure copies of platform and connector images are available in your custom image registry. Learn how to set up custom image registries.

Custom Docker connectors in your workspace that specify an image using a fully qualified domain name (for example, example.com/airbyte/your-custom-source) ignore your configured custom image registry and pull images from the domain specified by that connector.

Simplified OAuth flow for connectors​

We've simplified how Self-Managed Airbyte instances use OAuth to connect to data sources. You no longer need to implement your own web service to handle OAuth flows, removing the need for an entire piece of web infrastructure so you can start moving data faster and more easily.

Version 1.3 and earlier​

In the past, you implemented your own web service to handle the OAuth flow, passing your client ID and client secret, setting a redirect URL, handling the redirect response, and obtaining an access token and refresh token. You then input that access token and refresh token into Airbyte while setting up a connection.

Version 1.4 and later​

Airbyte handles the parts of the OAuth flow that once required a separate web server. Existing connections set up using the old flow continue to work. However, when you set up new connections, you use this simplified flow.

  1. Create your OAuth application for a given data source.
  2. Input your Client ID and Client Secret in Airbyte while setting up a connection.
  3. Click the authenticate button.

Airgapped instances support this new flow. The redirect is handled by your web browser and not the Airbyte server.

Learn more about OAuth in Airbyte.

🐛 Bug fixes​

  • Reduced resource consumption: Instances of Airbyte running multiple connections at a time now consume fewer resources, thanks to optimizations to the airbyte-worker pod.

  • Canceled syncs do not rerun until the next scheduled sync: If you previously cancelled an in-progress sync, and the next sync as configured in the connection frequency was behind schedule, Airbyte would immediately start a new sync. In practice, many users had to choose to ‘Cancel Sync’ twice in a row to stop moving data. Airbyte now automatically waits until the next scheduled sync to move data.

  • Reduce rate limit errors from the airbyte-cron service: We fixed an issue reported by the community that caused excessive rate limit errors on the airbyte-cron pod when users scheduled connections using the Airbyte CRON capability.

  • Autorecovery for hanging connections: Airbyte now packages a service to detect connections blocked by the unlikely event that a sync becomes stuck, and remains in a perpetual ‘in-progress’ state without moving data. This heartbeat service will detect syncs that are hanging, and automatically create a new job attempt.

Was this page helpful?