Amazon Language Data Scientist II AWS Q Business 

United States, California 

566852203

In this role, you will leverage data-centric AI principles to assess the impact of data on model performance and the broader machine learning pipeline. You will apply Generative AI techniques to evaluate how well our data represents human language and conduct experiments to measure downstream interactions.Key job responsibilities
* oversee end-to-end evaluation data pipeline and propose evaluation metrics and methods
* incorporate your knowledge of linguistic fundamentals, NLU, NLP to the data pipeline
* process and analyze diverse media formats including audio recordings, video, images and text
* perform statistical analysis of the data
* write intuitive data generation & annotation guidelines
* write advanced and nuanced prompts to optimize LLM outputs
* write python scripts for data wrangling
* automate repetitive workflows and improve existing processes
* perform background research and vet available public datasets on topics such as long text retrieval, text generation, summarization, question-answering, and reasoning
* leverage and integrate AWS services to optimize data collection workflows
* collaborate with scientists, engineers, and product managers in defining data quality metrics and guidelines.
* lead dive deep sessions with data annotators

- 2+ years of data scientist experience
- 3+ years of data querying languages (e.g. SQL), scripting languages (e.g. Python) or statistical/mathematical software (e.g. R, SAS, Matlab, etc.) experience
- 3+ years of machine learning/statistical modeling data analysis tools and techniques, and parameters that affect their performance experience
- Excellent knowledge of semantics, pragmatics, conversation analysis, and/or discourse analysis
- Experience designing and executing data collection projects, including guidelines, label set and annotation workflow development
- Experience developing and evaluating data annotation and data quality metrics
- Experience designing and executing psychology/linguistic/cognitive science surveys or experiments with human participants