Full disclosure: I have not dabbled with data science, nor I'm an experienced programmer.
But many of the CS pioneers dealt with the issue of reading input, processing it, and giving a meaningful output. Recently I've stumbled upon Jackson's Principles of Program Design and it has really helped me in writing a parser. IMHO general understanding of structuring a program goes a long way than arbitrary design patterns. I now am a propenent of modular programming. For me it resembles functional programming and TDD. Basically it states your program should be made up of numerous self-contained modules that can be independently tested.
For data science workflow it could be an import package, further divided in modules such as general helpers and implementations for different file types; a computation package etc.
Its always hard for me to decide how those modules are going to end up communicating though. Should there be a lingua franca to all your modules? Or do they all have specialized APIs?
Programming to interfaces help in this matter. e.g. if I have a data interface, I could use the same type on any format I liked. If it is the same interface, methods are the same too.
For encapsulation and message passing, coroutines work wonders. You could use a generator in Python, or a goroutine in Golang. And then you could treat different modules of your program, as if they were a standalone independent part.
But many of the CS pioneers dealt with the issue of reading input, processing it, and giving a meaningful output. Recently I've stumbled upon Jackson's Principles of Program Design and it has really helped me in writing a parser. IMHO general understanding of structuring a program goes a long way than arbitrary design patterns. I now am a propenent of modular programming. For me it resembles functional programming and TDD. Basically it states your program should be made up of numerous self-contained modules that can be independently tested.
For data science workflow it could be an import package, further divided in modules such as general helpers and implementations for different file types; a computation package etc.