Data Scientist, Machine Learning Evaluation

Our Company Changing the world through digital experiences is what Adobe's all about. We give everyone-from emerging artists to global brand...

Our Company Changing the world through digital experiences is what Adobe's all about. We give everyone-from emerging artists to global brands-everything they need to design and deliver exceptional digital experiences! We're passionate about empowering people to create beautiful and powerful images, videos, and apps, and transform how companies interact with customers across every screen. We're on a mission to hire the very best and are committed to creating exceptional employee experiences where everyone is respected and has access to equal opportunity. We realize that new ideas can come from everywhere in the organization, and we know the next big idea could be yours!The OpportunityMachine Learning is a critical part of Adobe's Cloud offering. Adobe Clouds enable customers to build and manage digital content, such as assets, composites, 3D, documents etc., and digital experience and transformations.Creative Cloud is focused on visual imagination and creation, and includes such well-known products as Photoshop, Illustrator, Lightroom, and Adobe Stock. Document Cloud is centered around the creation and management of textual documents, including Acrobat and Adobe Sign. Experience Cloud is the set of offerings for the large-scale enterprise, covering marketing, advertising, and analytics.Adobe Sensei powers machine learning and artificial intelligence across all of these Adobe Cloud product lines. This platform enables thousands of applied researchers, millions of users, and billions of content pieces. The objective is to make machine learning offerings a world-class, leading-edge, differentiating product in Adobe Cloud ecosystem. We match the pace, innovation and excitement of a startup, backed by the resources and infrastructure of Adobe!Data is critical piece of developing machine learning solutions. ML data can include the datasets that train algorithms, but also large-scale evaluation datasets which give insights into algorithm behavior that helps to refine and evolve ML products.Training data - Adobe's training datasets often contain digital art, photos, videos, text, or audio files, along with some type of additional knowledge data (user behaviors, annotations, manipulations) to enable meaningful machine learning. How do we build these datasets to maximize performance and accuracy for the customer? How do we ensure that our models are robust and unbiased?Evaluation - How do we assess the quality of our algorithms and models? What is the ground truth we use to judge them? Where do they succeed and how do they fail? These problems are even more subtle in the complex, creative domains of Adobe products - for example, how should we judge the quality of similarity search, where a user finds new images with an image as the query?We're looking for data scientists to define, drive, and coordinate training data construction and model evaluation for the Applied Machine Learning team at Adobe. This will involve thinking hard about product quality, the role of machine learning in those products, and the capabilities of large-scale human-judgment tasks to support scalable solutions that generate the fuel for success of our AI/ML algorithms.Responsibilities: Work closely with product, research, and engineering leaders to understand ML applications and define success.Interact with APIs to effectively sample raw data to optimize for ML training and evaluation.Design tasks and guidelines for human judgment jobs to annotate images, videos, text, audio; and also to measure the performance of machine-learned and other advanced algorithms.Help design A/B tests for our cloud offerings, and use human judgment vs. end-user A/B tests as appropriateCreate a winning culture built on collaboration and shared accomplishments- have fun along the way! Qualifications: BS or MS in a technical field (Computer Science, Physics, Math, Statistics) OR in a human-centered field such as Psychology, Linguistics, Human-Computer Interaction.Exposure to applied machine learning in an enterprise setting.Proven experience in validation and evaluation of machine learned models.Strong quantitative and analytical orientation. Experience in evaluation metrics, Python or R, JSON parsing, custom HTML for visualization.Strong qualitative skillset. Experience in taxonomies, survey design.Understanding of issues around ground truth and training/evaluation data for ML, including human judgment and A/B testingHands-on experience using third-party judgment services such as Figure 8, Appen, or Amazon Mechanical Turk.At Adobe, you will be immersed in an exceptional work environment that is recognized throughout the world on Best Companies lists . You will also be surrounded by colleagues who are committed to helping each other grow through our unique Check-In approach where ongoing feedback flows freely.If you're looking to make an impact, Adobe's the place for you. Discover what our employees are saying about their career experiences on the Adobe Life blog and explore the meaningful benefits we offer.Adobe is an equal opportunity employer. We welcome and encourage diversity in the workplace regardless of gender, race or color, ethnicity or national origin, age, disability, religion, sexual orientation, gender identity or expression, or veteran status.