Recursos Data Engineer

---------------------- ARTICULOS SPARK SCALA ---------------------- https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/

https://medium.com/@Sushil_Kumar/setting-up-spark-with-scala-development-environment-using-intellij-idea-b22644f73ef1

Instalar Java 8 en Ubuntu 

- https://computingforgeeks.com/how-to-install-java-8-on-ubuntu/ 

Instalar intellij-idea en Ubuntu 20.04 LTS 

- https://youtu.be/zCkozcDBJ-A 

Instalar plugin Scala y crear Proyecto en IntelliJ 

https://docs.scala-lang.org/getting-started/intellij-track/getting-started-with-scala-in-intellij.html

Mi primera aplicacion Apache Spark en Scala con Maven en IntelliJ 

- https://www.diegocalvo.es/mi-primera-aplicacion-apache-spark-en-scala-con-maven-en-intelligent-idea/ 

Create Scala Project with Maven in IntelliJ IDEA

https://javahelps.com/create-scala-project-with-maven-in-intellij-idea?amp=1

Spark Scala con Maven en IntelliJ 

- https://josedeveloper.com/2020/03/23/spark-scala-con-maven-en-intellij/ 

Configurar Scala SDK Modificar fichero pom.xml con version de scala y dependencias spark - https://github.com/martinprobson/Spark-Scala-Maven-Example/blob/master/pom.xml 

Scala con Maven 

- https://docs.scala-lang.org/tutorials/scala-with-maven.html#the-scala-maven-plugin 

Scala on IntelliJ using Maven

https://mxnet.apache.org/versions/1.7/api/scala/docs/tutorials/mxnet_scala_on_intellij

Crear App Spark usando el IDE de IntelliJ en Ubuntu 

- https://youtu.be/AP5JvdgiBME 

Usar Gradle para crear una aplicación simple en Spark Scala 

- https://medium.com/@yanggao1119/using-gradle-to-create-a-simple-scala-spark-application-d5f1740e741b 

Crear de manera rapida un proyecto scala usando gradle en intellij 

- http://www.legendu.net/misc/blog/scala-with-gradle-in-intellij/ 

Iniciarse en Gradle con IntelliJ 

- https://www.jetbrains.com/help/idea/getting-started-with-gradle.html 

Generar un fichero JAR de una aplicación Scala con SBT 

- https://alvinalexander.com/scala/sbt-how-build-single-executable-jar-file-assembly/ 

 Como configurar y estructurar una aplicación Spark en Scala 

- https://godatadriven.com/blog/how-to-setup-and-structure-a-spark-application-in-scala/

How to create maven project for Spark and Scala in Scala IDE - https://medium.com/@manojkumardhakad/how-to-create-maven-project-for-spark-and-scala-in-scala-ide-1a97ac003883

Create your first scala project with Maven - https://docs.scala-lang.org/tutorials/scala-with-maven.html

Create scala project with maven in IntelliJ Idea - https://www.javahelps.com/2021/03/create-scala-with-maven-project-in.html

Setting up a Spark development environment with Scala

https://www.cloudera.com/tutorials/setting-up-a-spark-development-environment-with-scala.html

Instalar Apache Spark y Scala

- https://sundog-spark.s3.amazonaws.com/spark-install.pdf

Setup Development Environment - Scala and Spark

https://kaizen.itversity.com/courses/aws-emr-and-spark-2-using-scala/lessons/setup-development-environment-for-scala-and-spark/topic/setup-development-environment-scala-and-spark/

Creating a Spark Standalone Cluster with Docker and Docker Compose

https://dev.to/mvillarrealb/creating-a-spark-standalone-cluster-with-docker-and-docker-compose-2021-update-6l4

Learning Scala on MacBook with MacOS 12 (Monterey)

https://janrock.medium.com/learning-scala-and-akka-on-new-macbook-with-macos-12-monterey-f516193f1b5e

Setup homebrew on macOS Monterey

https://tech-cookbook.com/2021/10/25/how-to-setup-homebrew-brew-install-on-macos-12-monterey/

Install Java on Mac with brew 

https://adamtheautomator.com/install-java-on-mac/

Setup Scala Spark project in IntelliJ using SBT build tool

https://medium.com/@Sushil_Kumar/setting-up-spark-with-scala-development-environment-using-intellij-idea-b22644f73ef1

 

Kernel de Scala para Jupyter

- https://almond.sh/docs/intro

- https://github.com/almond-sh/examples

Instalar Spark en Ubuntu

- https://phoenixnap.com/kb/install-spark-on-ubuntu

Spark setup with Scala and Run in IntelliJ

https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/?expand_article=1

——— LEER CONFIGURACIONES ——-

Configuration Library for JVM Languages - https://github.com/lightbend/config

Reading configurations In Scala - https://medium.com/@ramkarnani24/reading-configurations-in-scala-f987f839f54d

TypeSafe Config Features and Example Usage - https://www.stubbornjava.com/posts/typesafe-config-features-and-example-usage

Hoy to use the Lightbend Config library In a Scala of Java application - https://alvinalexander.com/scala/how-use-lightbend-config-library-scala-java-application/

Configuring Spark Applications with Typesafe Config - https://florentfo.rest/2019/01/07/configuring-spark-applications-with-typesafe-config.html

 ----- IDEs --—

Working with Scala on IntelliJ 

- https://www.jetbrains.com/help/idea/discover-intellij-idea-for-scala.html 

Compile and build applications with IntelliJ 

- https://www.jetbrains.com/help/idea/compiling-applications.html 

Write and edit source code 

- https://www.jetbrains.com/help/idea/working-with-source-code.html

————- JAVA —————————————————————-

Java full course for beginners (using IntelliJ) - https://youtu.be/Qgl81fPcLc8

 ----------------------------- PROGRAMA DE FORMACION DEVOPS -----------------------------

 Git, GitHub and Gitlab - Curso de Atlassian en Coursera: Version Control with Git CI/CD Pipeline with JEnkins - Cursos Udemy: - Jenkins, from zero to hero. Become a DevOps Jenkins Master - GitLab CI: Pipelines, CI/CD and DevOps for Beginners Configuration Management with Chef, Puppet and Ansible - Ansible for the absolute beginner - Hands-on - DevOps - Curso Udemy: Dive into Ansible - From begginer to expert - Todos los cursos en kodekloud.com Docker Certified Associate Certification - Todos los cursos en kodekloud.com - Instalar Docker en Ubuntu: https://docs.docker.com/engine/install/ubuntu/ - Curso Docker y Devops: de novato a experto: https://www.udemy.com/share/102RT4BUQdeF5UQX4=/ - Desplegar una aplicación NodeJS con Docker: https://www.luigicolella.it/blog/how-to-create-nodejs-app-with-docker-1 - Dockerizar una App Scala: https://medium.com/@ievstrygul/dockerizing-scala-app-3fdf08cffda4 - dockerizar una aplicación Django - https://platzi.com/blog/django-docker/ - Como crear imagenes docker para Django - https://www.section.io/engineering-education/django-docker/ - Spark cluster with docker and docker-compose: https://github.com/fgarcialopez605/spark-essentials/tree/master/spark-cluster Certified Kubernetes Administrator - Todos los cursos en kodekloud.com Devops on AWS - Curso Udemy: AWS Certified DevOps Engineer (Requiere al menos el AWS Certified Developer) 

Apache Spark Cluster on Docker:

- https://www.kdnuggets.com/2020/07/apache-spark-cluster-docker.html

Spark clúster with docker

https://github.com/mvillarrealb/docker-spark-cluster

Apache Spark Standalone Cluster on Docker

https://github.com/cluster-apps-on-docker/spark-standalone-cluster-on-docker#tech-stack

 ---------------------------- PROGRAMA CERTIFICACION CLOUD ----------------------------

 https://acloudguru.com/

AWS Cloud Practitioner Certification 

- https://www.whizlabs.com/aws-certified-cloud-practitioner/

AWS Certified Cloud Practitioner

https://www.udemy.com/share/103a09BUQdeF5UQX4=/

Official Google Cloud Certified Professional Data Engineer Study Guide

https://www.amazon.es/Official-Google-Certified-Professional-Engineer/dp/1119618436/ref=sr_1_1?__mk_es_ES=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=Official+Google+Cloud+Certified+Professional+Data+Engineer+Study+Guide&qid=1620994245&sr=8-1

AWS Certified Solutions Architect

https://www.udemy.com/course/aws-certified-solutions-architect-associate-saa-c02/

AWS Certified Solutions Architect Study Guide

https://www.amazon.com/Certified-Solutions-Architect-Study-Guide-dp-111950421X/dp/111950421X?tag=javamysqlanta-20

Prepare for AWS Solution Architect Associate Certification

https://medium.com/javarevisited/how-to-prepare-for-aws-solution-architect-associate-certification-saa-c01-saa-c02-exam-in-2021-a6e7e7e771fc

GCP Professional Data Engineer Dmitri Lerko - https://deploy.live/blog/google-cloud-certified-professional-data-engineer/

Como superar el examen de data Engineer de Google cloud - https://dbibyhavas.io/es/blog/data-engineer-de-google-cloud/

Desarrollo Python en Google Cloud 

- https://cloud.google.com/python/docs/setup

- https://cloud.google.com/python

Desarrollo Java en Google Cloud

- https://cloud.google.com/java/docs/setup

- https://cloud.google.com/java

—————————- PROGRAMA CERTIFICACIÓN SPARK ——————————-

Como aprobar la certificación Databricks Spark 3.0

- http://josedeveloper.com/2021/04/30/como-aprobar-la-certificacion-databricks-spark-3-0/

 ------------------- MAQUINAS VIRTUALES ------------------- 

 Install Ubuntu 20.04 LTS on VirtualBox in Windows 10 - https://youtu.be/x5MhydijWmc - https://gadgetliv.com/install-ubuntu-on-virtualbox-in-windows/ 

How to install Ubuntu in VirtualBox on Windows 10

- https://youtu.be/IOwlnpWPuj0

Como empezar a trabajar con un contenedor Docker de Cloudera QuickStart 

https://medium.com/@SnazzyHam/how-to-get-up-and-running-with-clouderas-quickstart-docker-container-732c04ed0280

Proyecto Vagrant para arrancar una máquina virtual con Hadoop/Spark

https://github.com/martinprobson/vagrant-hadoop-hive-spark

 -------------------- DESARROLLO PYTHON --------------------

 Jupyter Notebooks, JupyterLab, Zeppelin, Anaconda, Spyder 

 

 Iniciar un proyecto de ciencia de datos con Anaconda en Python - http://spaceanalytics.blogspot.com/2020/11/iniciar-un-proyecto-de-ciencia-de-datos.html 

Empezar a trabajar con Apache Spark y Python

- https://www.sundog-education.com/spark-python/

 

 Introducción práctica al analisis de datos en python 

- https://towardsdatascience.com/a-beginners-guide-to-data-analysis-in-python-188706df5447 

 

Instalar un entorno de desarrollo Python3 en Ubuntu 20.04 

- https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-programming-environment-on-an-ubuntu-20-04-server 

 

Creación de proyectos Python en IntelliJ 

- https://www.jetbrains.com/help/idea/plugin-overview.html 

Introducción a MongoDB, PySpark y Jupyter Notebook

- https://www.mongodb.com/blog/post/getting-started-with-mongodb-pyspark-and-jupyter-notebook

Setup PySpark environment on Ubuntu

- https://medium.com/@GalarnykMichael/install-spark-on-ubuntu-pyspark-231c45677de0

Setup JupyterLab Environment on Ubuntu

https://www.digitalocean.com/community/tutorials/how-to-set-up-a-jupyterlab-environment-on-ubuntu-18-04

Install PySpark on Ubuntu

https://t.co/UdzmhAaTKi

https://www.roseindia.net/bigdata/pyspark/install-pyspark-on-ubuntu.shtml

Install PySpark locally and use it with Jupyter Notebook

- https://youtu.be/MLXOy-OhWRY

PySpark crash course in 50 minutes

- https://youtu.be/6rgoqfzX_YI

PySpark Tutorial for Beginners

https://sparkbyexamples.com/pyspark-tutorial/

Instalar Anaconda Python en Ubuntu 20.04

https://www.digitalocean.com/community/tutorials/how-to-install-the-anaconda-python-distribution-on-ubuntu-20-04-es

Documentación oficial de Anaconda

- https://docs.anaconda.com/anaconda/

JupyterLab in a virtual environment 

- https://dev.to/joeneville_/jupyterlab-in-a-virtual-environment-gf5

Use Virtual Environments Inside Jupyter Notebooks & JupyterLab

- https://www.zainrizvi.io/blog/jupyter-notebooks-best-practices-use-virtual-environments/

Create Django App on Ubuntu Server

- https://linuxhint.com/create_django_app_ubuntu/

Documentación oficial de Conda

https://conda.io/projects/conda/en/latest/user-guide/getting-started.html

Como instalar Python 3 y entornos virtuales con miniconda en Ubuntu

https://thucnc.medium.com/how-to-install-python-3-and-virtual-environments-with-miniconda-in-ubuntu-18-04-c9e000a39994

Gestionar múltiples versiones de Python en Ubuntu

https://hackersandslackers.com/multiple-versions-python-ubuntu/

 Otros ----- http://www.blogdelujo.com/2010/01/ver-incrustar-otra-web-en-blog.html

—————————DESARROLLO KAFKA ——————-

Desarrollo de aplicaciones Kafka con Scala - 

https://www.confluent.io/blog/kafka-scala-tutorial-for-beginners/

Kafka Streams DSL en Scala - https://medium.com/@danieljameskay/kafka-streams-dsl-for-scala-the-basics-11d603295f5c

———— KUDU / IMPALA —————

Montar una infraestructura Docker que usa Impala para operar con Kudu

https://github.com/apache/kudu/tree/master/examples/quickstart/impala

Oficial Kudu - https://kudu.apache.org/

Oficial Impala - https://impala.apache.org/index.html

Tutoríales prácticos de Impala - https://impala.apache.org/docs/build/html/topics/impala_tutorial.html

Building real time BI systems with Kafka, Spark and Kudu - https://databricks.com/session/building-real-time-bi-systems-with-kafka-spark-and-kudu

Apache Kudu and Spark SQL for fast Analytics on fast data - https://databricks.com/session/apache-kudu-and-spark-sql-for-fast-analytics-on-fast-data

CERTIFICACIÓN GCP DATA ENGINEER

https://www.cloudskillsboost.google/paths/16

https://youtu.be/w2_ukhoLGEM

Usar Google Dataproc con PySpark y Jupyter Notebook

https://morioh.com/p/2c1e7be91f53

Run PySpark job on Dataproc

https://acloudguru.com/hands-on-labs/running-a-pyspark-job-on-cloud-dataproc-using-google-cloud-storage

GENERAL DATA ENGINEER

https://awesomedataengineering.com/

https://medium.com/coriers/data-engineering-roadmap-for-2021-eac7898f0641

Setting up a Dev Environment for Data Engineers

- https://discuss.itversity.com/t/data-engineers-setting-up-the-development-environment/19879

Data Architecture Articles

https://medium.com/tag/data-architecture

How to become a Data Architect

https://techguide.org/careers/data-architect/

Top Data Architect and Data Engineer Certifications

https://www.techtarget.com/whatis/feature/8-top-data-architect-and-data-engineer-certifications