Conceptually is this sort of a database-agnostic SQL view? Seems like these could be stacked to simplify (or complicate?) an entire ETL pipeline.
I haven’t decided where I land on this. In some ways, stacking SQL views looks like it simplifies a bunch of ETL jobs, but I also fear a few things:
* It either breaks catastrophically when there’s a change in the source data
* Fails silently and just passes the incorrect data along
* More challenging to debug than an ETL pipeline where we have a clear point of error, can see the input and output of each stage, etc
* Source control of SQL views seems less great than code. Often when we have too many views, you can’t update one without dropping all of the dependencies and recreating them all
But I also wonder if I feel this way because I know programming better than SQL
No you are right, views have an use case but they get abused as ersatz orchestration, creating a nested structure that multipkies the amount of needed compute.
edit: now I read your question more carefully. I think the s3 data is meant to be managed by other orchestration. This is a quick easy way to share a data source with an analyst, PM or end consumer.
I do not expect poster is advocating this as any intermediate stage in a data pipeline.
I haven’t decided where I land on this. In some ways, stacking SQL views looks like it simplifies a bunch of ETL jobs, but I also fear a few things:
* It either breaks catastrophically when there’s a change in the source data
* Fails silently and just passes the incorrect data along
* More challenging to debug than an ETL pipeline where we have a clear point of error, can see the input and output of each stage, etc
* Source control of SQL views seems less great than code. Often when we have too many views, you can’t update one without dropping all of the dependencies and recreating them all
But I also wonder if I feel this way because I know programming better than SQL