Prework

Sign up for Databricks Community Edition

Databricks is the company behind Apache Spark. The community edition offers you a cost-free environment (15GB RAM and 2 Cores) with the latest Apache Spark & much more.

• To sign up for Databricks Community Edition, follow the instructions here: https://docs.databricks.com/en/getting-started/community-edition.html
  • We recommend you use UMN email for the sign up.
  • On the page, "How do you plan on using Databricks?", choose the Community Edition.
  • At the final page, be sure to choose the "Get started with Community Edition" (the grey button instead of the red button) -- see a picture below.
  • As suggested, after signing in, you may try the Get started: Query and visualize data from a notebook to familiarize yourself with Databricks.

• After that, you can bookmark this page community.cloud.databricks.com for future log on.

Linux Shell Commands

We will spend some of our course time on Linux shell commands, which are often used in big data, cloud computing, and MLOps. It would be helpful for you to warm up to the concepts and commands with the following video and short web course.

Here are some materials that warm you up to the concepts:

• Learning the Shell (web tutorial): just focus on Learning the Shell chapter
• Introduction to Linux Operating System (A gentle overview): only need to watch the segment of 18:05-52:03 (or the following topics)
  • 18:05 Introduction to Linux operating system and comparison with windows
  • 24:32 Terminal vs. File Manager
  • 27:20 Command Line Interfaces on Ubuntu Operating system
  • 49:19 Brief of Linux commands

Some Useful Concepts in Computing

Big data is about completing tasks in parallel, using multiple computers. Naturally, it needs some background on how computing is done on a single computer. These are a series of short videos explaining various concepts and aspects of computing.

• https://www.youtube.com/watch?v=p3q5zWCw8J4 (how do computer memory work)
• https://www.youtube.com/watch?v=H_M--weEzpA (Memory versus Storage)
• https://www.youtube.com/watch?v=IIvbEn54ZAY (CPUs and Cores)
• https://www.youtube.com/watch?v=H4l42nbYmrU (video explaining ASCII files)
• https://www.youtube.com/watch?v=v7IpCq5YL68 (comparing plain text and binary files)
• https://www.youtube.com/watch?v=BKgRaHMUul0 (explaining binary files and demonstrating difference in file types)