Appendix B — Supported file types
Below is a list of supported file types and how they work out of the box.
You can always use parse hooks to add additional file types for your use cases, or override gusty’s default parsers.
All Airflow task ids are the Task Definition Files’ file names.
.yml
Behavior
Declare an operator
and pass in any operator parameters using YAML.
Example
operator: airflow.operators.bash.BashOperator
bash_command: echo hello world
.py
Behavior
For starters, you can just write Python code and by default gusty will execute your file using a PythonOperator
.
To expand, you can declare a python_callable
in the Frontmatter and define the function in the body.
While default behavior for .py
files specifies PythonOperator
as the operator
, as with any Task Definition File, you can specify any operator
.
Example
A Task Definition File, hello_world.py
, with no Frontmatter:
print("hello world")
A task_definition file, hello_world.py
, with Frontmatter:
# ---
# python_callable: main
# ---
def main():
print("hello world")
The callable name is up to you, but it must match the function name in the Body.
.sql
Behavior
Declare an operator
in a YAML header, then write SQL in the main .sql file. The SQL automatically gets sent to the operator.
Example
---
operator: airflow.providers.sqlite.operators.sqlite.SqliteOperator
---
SELECT 'hello world'
.ipynb
Behavior
Put a YAML block at the top of your notebook and specify an operator
that renders your Jupyter Notebook.
Example
See the gusty demo Jupyter Notebook Example and sample JupyterOperator.
.Rmd
Behavior
Use the YAML block at the top of your notebook and specify an operator
that renders your R Markdown Document.
Example
See the gusty demo Rmd Example and sample RmdOperator.