Data Engineer

Data Engineer

Data Engineers design and implement the systems that allow data to be accessible and useful.

Data Platform
Job Family
AU$110k
Salary
Average salary in Australia
19%
Job Growth
The number of positions relative to last year
46
Open Roles
Job openings on Alooba Jobs

Data Engineers are responsible for building and maintaining the architecture used for data storage and processing. They develop, construct, test, and maintain data management systems, ensuring that they meet organizational requirements. Data Engineers work closely with data scientists and analysts to provide them with usable data and are essential for data-driven decision making.

What are the responsibilities & duties of a Data Engineer

  • Collaborate with Technology Teams, Global Analytical Teams, and Data Scientists across programs
  • Improve database and reporting tools performance
  • Create dashboards and visualization using Tableau
  • Develop UI, tools, and applications to digitize processes
  • Lead business growth and enhance product experiences by gaining experience in large-scale data processing systems (batch and streaming)
  • Design and develop business intelligence solutions for complex business problems
  • Work with stakeholders for business analytics tool development and data models
  • Improve the quality of data by adding sources, coding rules, and producing metrics as requirements evolve
  • Cooperate with other organizations on data governance, KPIs, and reporting tools
  • Direct ETL development demonstrating understand key concepts of ETL/ELT
  • Apply multi-dimensional and tabular design patterns
  • Work within the Software Development Life Cycle (SDLC) across multiple environments

What are the required skills & experiences of a Data Engineer?

  • Expertise in SQL and experience with database technologies (e.g., MySQL, PostgreSQL, Microsoft SQL Server)
  • Proficiency in big data technologies (e.g., Hadoop, Spark)
  • Experience with data pipeline and workflow management tools (e.g., Airflow, Luigi)
  • Knowledge of scripting languages (e.g., Python, Java)
  • Understanding of ETL (Extract, Transform, Load) processes
  • Familiarity with cloud services (e.g., AWS, Azure, GCP)
  • Strong problem-solving and analytical skills
  • Ability to work in a team and communicate effectively
  • Experience with version control tools (e.g., Git)
  • Understanding of data warehousing concepts
  • Knowledge of data modeling techniques
  • Familiarity with machine learning algorithms and data science principles

Core Data Engineer Required Skills

.NET.NETAARRRAARRRAccessibilityAccessibilityAdaptabilityAdaptabilityAdvanced AnalyticsAdvanced AnalyticsAgileAgileAirtableAirtableAlgorithmsAlgorithmsAlteryx DesignerAlteryx DesignerAmazon AthenaAmazon AthenaAmazon DynamoDBAmazon DynamoDBAmazon GlueAmazon GlueAmazon KinesisAmazon KinesisAmazon Web ServicesAmazon Web ServicesAnalytical MindsetAnalytical MindsetAnalytical ReasoningAnalytical ReasoningAnalytics DatabasesAnalytics DatabasesAnalytics EngineeringAnalytics EngineeringAnalytics ManagementAnalytics ManagementAnalytics ProgrammingAnalytics ProgrammingAnalytics Project ManagementAnalytics Project ManagementAnomaly DetectionAnomaly DetectionAnsibleAnsibleApache AirflowApache AirflowApache BeamApache BeamApache CassandraApache CassandraApache FlinkApache FlinkApache FlumeApache FlumeApache HadoopApache HadoopApache HBaseApache HBaseApache HiveApache HiveApache IcebergApache IcebergApache ImpalaApache ImpalaApache NiFiApache NiFiApache SparkApache SparkApache SqoopApache SqoopArea ChartsArea ChartsArraysArraysAssertivenessAssertivenessAssociation RulesAssociation RulesAtomicityAtomicityAutomated Data Quality ChecksAutomated Data Quality ChecksAutomationAutomationAvailability HeuristicAvailability HeuristicAWS LambdaAWS LambdaAzureAzureAzure Data FactoryAzure Data FactoryAzure Data LakeAzure Data LakeAzure DatabricksAzure DatabricksBalancing TreesBalancing TreesBar ChartsBar ChartsBashBashBayesian AnalysisBayesian AnalysisBehavioral AnalyticsBehavioral AnalyticsBERTBERTBiasBiasBig DataBig DataBig Data MiningBig Data MiningBinary SearchBinary SearchBinary TreesBinary TreesBinomial DistributionBinomial DistributionBonferroni CorrectionBonferroni CorrectionBoxplotsBoxplotsBusiness AcumenBusiness AcumenBusiness AnalyticsBusiness AnalyticsBusiness InsightsBusiness InsightsBusiness IntelligenceBusiness IntelligenceBusiness Intelligence ArchitectureBusiness Intelligence ArchitectureBusiness Intelligence DevelopmentBusiness Intelligence DevelopmentCCC++C++CachingCachingCardinalityCardinalityCausal InferenceCausal InferenceCausationCausationCause & EffectCause & EffectCentral Limit TheoremCentral Limit TheoremChart InterpretationChart InterpretationChatGPTChatGPTChi-Squared DistributionChi-Squared DistributionClassesClassesClassificationClassificationClassification MetricsClassification MetricsClassification ModelsClassification ModelsClickHouseClickHouseClojureClojureCloud AnalyticsCloud AnalyticsCloud ComposerCloud ComposerCloud ComputingCloud ComputingCloud Data EngineeringCloud Data EngineeringCloudera Data PlatformCloudera Data PlatformClusteringClusteringCode ReviewsCode ReviewsCollaborationCollaborationCollectionsCollectionsCollectorsCollectorsCollinearityCollinearityColumn ChartsColumn ChartsColumnar DatabasesColumnar DatabasesCommittingCommittingComparatorsComparatorsComplexityComplexityComputer ScienceComputer ScienceConcurrencyConcurrencyConcurrency ControlConcurrency ControlConcurrency ControlsConcurrency ControlsConditional ProbabilityConditional ProbabilityConflict ManagementConflict ManagementConfluentConfluentConfusion MatricesConfusion MatricesContent Management SystemsContent Management SystemsContinuous LearningContinuous LearningContinuous VariablesContinuous VariablesControl StructuresControl StructuresConvolutionConvolutionCorrelationCorrelationcsv filescsv filesCustomer Data PlatformsCustomer Data PlatformsD3.jsD3.jsDagsterDagsterDashboardingDashboardingDaskDaskDataDataData AcquisitionData AcquisitionData AdvocacyData AdvocacyData AnonymizationData AnonymizationData ArchitectureData ArchitectureData BlendingData BlendingData CatalogingData CatalogingData EngineeringData EngineeringData Engineering InfrastructureData Engineering InfrastructureData EthicsData EthicsData ExplorationData ExplorationData FabricData FabricData FederationData FederationData FormatsData FormatsData GovernanceData GovernanceData InfrastructureData InfrastructureData IntegrationData IntegrationData InterpretationData InterpretationData LakeData LakeData LakehouseData LakehouseData LeakageData LeakageData LineageData LineageData LiteracyData LiteracyData ManagementData ManagementData ManipulationData ManipulationData MartData MartData MaskingData MaskingData MeshData MeshData MiningData MiningData ModellingData ModellingData MonitoringData MonitoringData OrchestrationData OrchestrationData Pipeline OrchestrationData Pipeline OrchestrationData PipelinesData PipelinesData PrivacyData PrivacyData ProcessingData ProcessingData Quality AssuranceData Quality AssuranceData ScienceData ScienceData ScrapingData ScrapingData SecurityData SecurityData ShardingData ShardingData SplittingData SplittingData StewardshipData StewardshipData Storage FrameworkData Storage FrameworkData StoresData StoresData StrategyData StrategyData StreamingData StreamingData StructuresData StructuresData SynchronisationData SynchronisationData TransferData TransferData TransformationsData TransformationsData TypesData TypesData VaultData VaultData VirtualizationData VirtualizationData VisualizationData VisualizationData WarehousingData WarehousingData WranglingData WranglingData-Driven Decision MakingData-Driven Decision MakingData-Driven InsightsData-Driven InsightsDatabase & Storage SystemsDatabase & Storage SystemsDatabase DesignDatabase DesignDatabase ManagementDatabase ManagementDatabase Management ToolDatabase Management ToolDatabase ModelingDatabase ModelingDatabase Performance OptimisationDatabase Performance OptimisationDatabase Scaling StrategiesDatabase Scaling StrategiesDatabricksDatabricksDatadogDatadogDataflowDataflowDataFramesDataFramesDataOpsDataOpsDAXDAXdbtdbtDebuggingDebuggingDecision TreesDecision TreesDell BoomiDell BoomiDenodoDenodoDependency GraphsDependency GraphsDesign PatternsDesign PatternsDifference in DifferencesDifference in DifferencesDimension TablesDimension TablesDimensional ModellingDimensional ModellingDistance MatricesDistance MatricesDistance MetricsDistance MetricsDistributed ComputingDistributed ComputingDistributed Data ProcessingDistributed Data ProcessingDistributed Event StoreDistributed Event StoreDistributed SQL Query EngineDistributed SQL Query EngineDistributionsDistributionsDomoDomodplyrdplyrDynamic ProgrammingDynamic ProgrammingEconometric ModelingEconometric ModelingElasticsearchElasticsearchEncapsulationEncapsulationEncryptionEncryptionEnglishEnglishEntity Relationship DiagramsEntity Relationship DiagramsError of DecompositionError of DecompositionETL/ELT ProcessesETL/ELT ProcessesEvaluation MetricsEvaluation MetricsEvent AnalyticsEvent AnalyticsEvent Data AnalysisEvent Data AnalysisEvent Driven ArchitectureEvent Driven ArchitectureEvent StreamingEvent StreamingExploratory Data AnalysisExploratory Data AnalysisFact TablesFact TablesFeature DependenciesFeature DependenciesFeature StoresFeature StoresFew-Shot PromptingFew-Shot PromptingFFTFFTFinancial ModelingFinancial ModelingFivetranFivetranForeach LoopsForeach LoopsForeign KeysForeign KeysFormulasFormulasFrequency GraphsFrequency GraphsFunctional ProgrammingFunctional ProgrammingFunctional RequirementsFunctional RequirementsFunctionsFunctionsGDPRGDPRGgplot2Ggplot2GitHubGitHubGLMGLMGoGoGoogle BigQueryGoogle BigQueryGoogle Cloud PlatformGoogle Cloud PlatformGoogle SheetsGoogle SheetsGradientsGradientsGrafanaGrafanaGraphQLGraphQLGraphsGraphsGrowth AnalyticsGrowth AnalyticsHadoop Distributed File SystemHadoop Distributed File SystemHashed DataHashed DataHaskellHaskellHeat MapsHeat MapsHeteroscedasticityHeteroscedasticityHistogramsHistogramsHomoscedasticityHomoscedasticityHTTP MethodsHTTP MethodsHypothesis TestingHypothesis TestingIBM DataStageIBM DataStageImputationImputationIncremental LoadingIncremental LoadingIndexingIndexingIndexing StrategiesIndexing StrategiesInductive ReasoningInductive ReasoningIndustriousnessIndustriousnessInformaticaInformaticaInformation RetrievalInformation RetrievalInformation SecurityInformation SecurityInfrastructure as CodeInfrastructure as CodeInteractive Query ServiceInteractive Query ServiceInternet SecurityInternet SecurityInterpersonal SkillsInterpersonal SkillsIteratorsIteratorsJavaJavaJSONJSONJuliaJuliaJupyter NotebookJupyter NotebookK-MeansK-MeansKanbanKanbanKerasKerasKeysKeysKNIMEKNIMEKnowledge GraphsKnowledge GraphsLanguage ModelingLanguage ModelingLFSLFSLine ChartsLine ChartsLinear ExtrapolationLinear ExtrapolationLinear Model AnalysisLinear Model AnalysisLinear ModellingLinear ModellingLinked ListsLinked ListsLinuxLinuxListsListsLocksLocksLog CollectionLog CollectionLog ManagementLog ManagementLogistic RegressionsLogistic RegressionsLookerLookerLooker StudioLooker StudioLoopsLoopsLSILSILuaLuaManaging UpManaging UpMapReduceMapReduceMariaDBMariaDBMarkdownMarkdownMarkov ChainsMarkov ChainsMathematicsMathematicsMATLABMATLABMatricesMatricesMeasures of Central TendencyMeasures of Central TendencyMeasures of DispersionMeasures of DispersionMergingMergingMetaBaseMetaBaseMetadata ManagementMetadata ManagementMetricsMetricsMicrosoft ExcelMicrosoft ExcelMissing Value TreatmentMissing Value TreatmentMixpanelMixpanelMode AnalyticsMode AnalyticsModel BiasModel BiasModel MonitoringModel MonitoringMongoDBMongoDBMoving AveragesMoving AveragesMulti-factor AuthenticationMulti-factor AuthenticationMulti-threadingMulti-threadingMulticollinearityMulticollinearityMVCMVCMySQLMySQLNaive BayesNaive BayesNatural Language ProcessingNatural Language ProcessingNested LoopsNested LoopsNeural NetworksNeural NetworksNo Code DatabaseNo Code DatabaseNon-Functional RequirementsNon-Functional RequirementsNormalizationNormalizationNoSQL DatabasesNoSQL DatabasesNumerical ReasoningNumerical ReasoningNumPyNumPyOAuth2OAuth2Object-Oriented ProgrammingObject-Oriented ProgrammingObjective-CObjective-COLAPOLAPOLTPOLTPOne-Hot EncodingOne-Hot EncodingOpen-Closed PrincipleOpen-Closed PrincipleOperating SystemsOperating SystemsOperation AnalyticsOperation AnalyticsOptimizationOptimizationOracle Business Intelligence Enterprise Edition PlusOracle Business Intelligence Enterprise Edition PlusOrganisational AnalyticsOrganisational AnalyticsOutlier RemovalOutlier RemovalOutlier TreatmentOutlier TreatmentOutliersOutliersP-ValueP-ValuePandasPandasParallel Computing FrameworkParallel Computing FrameworkPartitioned TablesPartitioned TablesPartitioningPartitioningPassword HandlingPassword HandlingPendoPendoPercentagesPercentagesPersonal SkillsPersonal SkillsPivot TablesPivot TablesPlotlyPlotlyPower BIPower BIPowerQueryPowerQueryPowerShellPowerShellPre-processingPre-processingPrescriptive AnalyticsPrescriptive AnalyticsPresentationsPresentationsPrestoPrestoPrimary KeysPrimary KeysPrincipal Component AnalysisPrincipal Component AnalysisProbabilityProbabilityProbability DensityProbability DensityProbability DistributionsProbability DistributionsProblem SolvingProblem SolvingProduct AnalyticsProduct AnalyticsProgrammingProgrammingProject ManagementProject ManagementPrompt EngineeringPrompt EngineeringPub/SubPub/SubPythonPythonQlikQlikQuantitative ResearchQuantitative ResearchQuantum Machine LearningQuantum Machine LearningQuboleQuboleQuery Execution PlansQuery Execution PlansQuery OptimisationQuery OptimisationQuickSightQuickSightR LanguageR LanguageR^2R^2Radar ChartsRadar ChartsRecommendation SystemsRecommendation SystemsRecursionRecursionRedshiftRedshiftRegression ModelsRegression ModelsRegressionsRegressionsRegular ExpressionsRegular ExpressionsRelational DatabasesRelational DatabasesReportingReportingRequirements GatheringRequirements GatheringRequirements TranslationRequirements TranslationReverting ChangesReverting ChangesRidge RegressionRidge RegressionRisk AnalysisRisk AnalysisRobustnessRobustnessROCROCRudderStackRudderStackS3S3Salesforce Customer 360Salesforce Customer 360SamplingSamplingSAP Data ServicesSAP Data ServicesSAP HANASAP HANASASSASScatter ChartsScatter ChartsSeabornSeabornSearch EnginesSearch EnginesSearching ArraysSearching ArraysSearching TreesSearching TreesSeasonality AnalysisSeasonality AnalysisSegmentSegmentSegmentationSegmentationServerless Architectures in DataServerless Architectures in DataServerless ComputingServerless ComputingSingle Responsibility PrincipleSingle Responsibility PrincipleSisenseSisenseSisense for Cloud Data TeamsSisense for Cloud Data TeamsSnapLogicSnapLogicSnowflake Data CloudSnowflake Data CloudSOAPSOAPSoftware EngineeringSoftware EngineeringSolarWindsSolarWindsSolution DesignSolution DesignSortingSortingSplunkSplunkSpreadsheetsSpreadsheetsSQLSQLSQL DevelopmentSQL DevelopmentSQL ServerSQL ServerSQLiteSQLiteSSASSSASSSISSSISStandard DeviationStandard DeviationStandardizationStandardizationStatisticsStatisticsStitch DataStitch DataStored ProceduresStored ProceduresStrategic InsightsStrategic InsightsStrategic ThinkingStrategic ThinkingStrategies for Missing DataStrategies for Missing DataString ManipulationString ManipulationStringsStringsStructured DataStructured DataSummary StatsSummary StatsSupermetricsSupermetricsSyntaxSyntaxT-ScoresT-ScoresT-TestsT-TestsTablesTablesTalend Data FabricTalend Data FabricTask ManagementTask ManagementTask SchedulingTask SchedulingTeradataTeradataTerraformTerraformText PreprocessingText PreprocessingThrottlingThrottlingtidyrtidyrtidyversetidyverseTime ComplexityTime ComplexityTime Series AnalysisTime Series AnalysisTinybirdTinybirdTopic ModelingTopic ModelingTransactionsTransactionsTransport Layer SecurityTransport Layer SecurityTrend AnalysisTrend AnalysisTrinoTrinoTuplesTuplesType 1 ErrorType 1 ErrorTypes of DataTypes of DataTypes of ErrorsTypes of ErrorsUnixUnixUnstructured DataUnstructured DataUnsupervised AlgorithmsUnsupervised AlgorithmsUnsupervised LearningUnsupervised LearningUser Experience ResearchUser Experience ResearchUserflowUserflowVarianceVarianceVerbal CommunicationVerbal CommunicationVerbal ReasoningVerbal ReasoningViewsViewsVirusesVirusesVLOOKUPVLOOKUPVolatilityVolatilityWaterfallWaterfallWaterfall ChartsWaterfall ChartsWeb CrawlingWeb CrawlingWeighted AveragesWeighted AveragesWikiWikiWindowsWindowsWindows Task SchedulerWindows Task SchedulerWorkflow AutomationWorkflow AutomationWorkflow ManagementWorkflow ManagementWormsWorms