Blog

Enhanced Job Management for modern data pipelines with Fyrefuse

Written by Fabrizio Rocco | Jun 26, 2024 9:20:35 AM

At Fyrefuse, we were happy to meet our community’s requests to streamline job management features in order to harness agility in authoring new jobs and their reusability in multiple pipelines. Now, we’re pleased to announce that the updated Fyrefuse Job Manager is now available.

Deployment of new jobs for data pipelines has never been an easy task. Traditional use cases adopt a combination of a set of most common pre-built connectors as well as custom jobs built ad-hoc by a vendor for the specific final customer. In addition, each data source could require its own connector developed in almost any programming language. Either way, such an approach is often inflexible and creates weighty engineering dependencies.

At Fyrefuse, we were happy to meet our community’s requests to streamline job management features in order to harness agility in authoring new jobs and their reusability in multiple pipelines. Now, we’re pleased to announce that the updated Fyrefuse Job Manager is now available.

The enhanced job management features are here to deliver several benefits over the existing traditional vendor-dependent job creation and management process:

  • To dynamically deploy jobs in Fyrefuse
  •  To allow developers’ teams to handle the jobs’ release in an fast and intuitive way
  • To introduce a connection between a job and its own repository
  • To grant developers the possibility to deploy any job written in any programming language

Until today, to deploy a new job users were constrained to declare it manually using SDKs inside the core, or wait for future releases of native jobs in the roadmap. With the new Job Manager, users are able to access an interface presenting a list of existing jobs and create and configure new jobs directly from the Fyrefuse portal.

Moreover, users can introduce their own jobs written in numerous programming languages, from C++ to Python and many more. Fyrefuse manager will handle it as a native job to ensure maximum compatibility.

Job Management in Fyrefuse

In the Fyrefuse platform, the Job Management section is accessible from the left sidebar( Pipelines , Jobs ):

Each job has its own properties: main information such as title, description, language, and more technical details like repositories’ credentials or build version.

Finally, each job has its own category to differentiate among the stakeholder’s needs.

Since each job is associated with a specific repository, Git’s username and password are not necessary since they have already been linked to the specified repository automatically. Whenever a job is updated, Fyrefuse provides the possibility to the user to choose the type of release such as:

  • New release (i.e. 1.0.0 to 2.0.0)
  • Major release (i.e. 1.0.0 to 1.1.0)
  • Minor release (i.e. 1.0.0 to 1.0.1)

Depending on the needs of your organization, each release type represents a different impact on your existing users.

Job management facilitates the deployment of a new job from development production as never happened before. Your company will be able to integrate existing business processes and exploit them with Fyrefuse in a new cutting-edge way.