I'm a Senior Machine Learning Engineer with >11 years of experience. I've specialized in NLP/NLU, most recently with fine-tuning SLMs for embedded deployment in a Rust application. I'd like to continue to apply machine learning techniques and learn new things, whether that's in the domain of SLMs + agentic systems or in a new domain.
I think a lot of these may have improved since your last experience with Keras. It's pretty easy to override the training loop and/or make custom loss. The below is for overriding training / test step altogether, custom loss is easier by making a new loss function/class.
> - Keras's training loop assumes you can fit all the data in memory and that the data is fully preprocessed, which in the world of LLMs and big data is infeasible.
The Tensorflow backend has the excellent tf.data.Dataset API, which allows for out of core data and processing in a streaming way.
That's a fair implementation of custom loss. Hugging Face's Trainer with transformers suggests a similar implementation, although their's has less boilerplate.
> To return to the point about image augmentations being hard to add: It's so easy to explain what your training code should do "Just distort the hue a bit" and there seem to be operations explicitly for that: https://www.tensorflow.org/api_docs/python/tf/image/adjust_h.... but when you go to train with them, you'll discover that backpropagation isn't implemented, i.e. they break in training code.
Why not do the data augmentation during preprocessing (so that the transformations don't have to be done by differentiable transforms)? I.e., map over a tf.Dataset with the transformation (and append to the original dataset).
This is cool - might be worth training a simple discriminator model to identify your utterances, and then you can use the plug-and-play language model (PPLM - https://github.com/huggingface/transformers/blob/master/exam...) to generate utterances modeling a specific speaker without special tokens. Could also take less time to fine-tune.
Additionally, the Nim code was not compiled with many optimizations turned on! (I.e., without -d:release).
$ nim c -o:base64_test_nim -d:danger --cc:gcc --verbosity:0 base64_test.nim
$ nim c -o:json_test_nim -d:danger --cc:gcc --verbosity:0 json_test.nim
IIRC the -d:danger flag is necessary for some optimizations (like disabling bounds checking) but -d:release is necessary for most optimizations to be enabled.
Edit: It appears I'm incorrect, -d:danger does imply -d:release in newer Nim versions.
It does imply release in latest Nim versions, `-d:release` still has some checks enabled, and `-d:danger` is full-on release mode with all possible checks disabled.
I'd recommend Elements of Statistical Learning or ISLR instead, if you want to start with a theory-heavy introduction. Most of what you need for DS you'd I think better learn through projects or on-the-job.
Also, as others have mentioned, some of the most important skills for DS are data munging, data "presentation", and soft skills like managing expectations / relationships / etc.
I would not recommend this book if you want to get into DS with the idea that, "I'll read this and then I'll know everything I need to." It's too dense and academically-focused, and it would probably be discouraging if you try to read this all without getting your feet wet.
Remote: Yes
Willing to relocate: No
Technologies: PyTorch, Tensorflow 2 / Keras, Rust, Go, Python. Model training, deployment, LLMs/SLMs. Docker/OCI, Helm and K8s (some). PostgreSQL, SQLite. GCP (Vertex AI).
Résumé/CV: https://drive.google.com/file/d/1sZ8JKJKXM5O8vfVYt172BqMIO2a...
Email: sloanes.k@gmail.com
I'm a Senior Machine Learning Engineer with >11 years of experience. I've specialized in NLP/NLU, most recently with fine-tuning SLMs for embedded deployment in a Rust application. I'd like to continue to apply machine learning techniques and learn new things, whether that's in the domain of SLMs + agentic systems or in a new domain.